Re: Some preliminaries and a claim (about EA and Bay Area Rationality)
Sorry, but I don't think it's good practice to publicly make a big, bold, vague claim that apparently contradicts a lot of expected readers' beliefs, and then ask that no one respond to it in any way unless they've spent hours reading hundreds of pages of other material. Personally, it seems to me like tightly coupled feedback loops are very useful, often necessary, but sometimes unfortunately not available (there are some ideas that are very hard or expensive to test). But eg. OpenAI clearly has such a feedback loop already; they've written a bunch of software that visibly either works or doesn't work, has worked surprisingly well, and has sometimes gone on to be quite famous. People have credibly told me that they worry that OpenAI focuses too much on issues where they get rapid feedback (eg. how skilled GPT-3 is or how much money it makes) over issues where they don't (safety is inherently harder to test since most of the badness comes from unlikely or non-obvious edge cases); one could debate that but it seems implausible that they simply lack any feedback loop and are lost in the woods. One could say that an organization like CSET, supported by OpenPhil, lacks a strong feedback loop, since ideas in foreign and technology policy usually aren't implemented and then take years to show results when they are implemented. Because of this, the entire field of American foreign policy has a long and visible track record of stupid mistakes (Iraq, Libya, Syria, China trade policy.... ); if you have no eyes you will often get lost. But it's not clear what the alternative is; someone has to do foreign policy, unless one thinks one should replace all the diplomats with GPT-3 and make all decisions randomly.