The pool is pre-selected: on Leverage and trusted-advisor evidence
Leverage Research, founded in 2011 by Geoff Anders, was a Bay Area organization that ran a closed-door “psychology research” program intended (in its founders’ telling) to map the human mind well enough to coordinate large-scale world improvement. It was unusually secretive for a nonprofit, publishing little, and many in the rationalist and effective altruist communities were skeptical. Nevin Freeman, a startup founder and CFAR alumnus, had argued in a thread that Leverage’s legitimacy could be vouched for by the prestigious investors, advisors, and employees who had seen its work in person and come away convinced. The reply below pushes back on the “trusted insiders saw it and believe” form of evidence in general, with a startup case study.
The trusted-insiders argument doesn’t actually differentiate Leverage. Theranos (and essentially all similar startups, or at least all who manage to get funding) has a bunch of investors, employees, and outside advisors with prestigious backgrounds who saw the technology in private and will swear that it works. This is true even for technology which is pretty clearly impossible — eg. Andrea Rossi’s cold fusion device was “tested” by Hanno Essen, a prominent Swedish physicist, who published a paper describing the test in some detail. The reason for that, of course, is selection bias; people are busy, so those who are approached by Theranos (or Rossi or whoever) and don’t find the claims credible just shrug and go on their way. The pool of investors, employees, advisors, and so on is strongly pre-selected, so those people having a positive opinion means very little, even when each person in the pool is very capable. Eg. two and a half years ago, noted VC Mark Suster, an investor in uBeam, wrote this when challenged on uBeam’s physical plausibility: “Here’s what you need to know for now: It does work. I have witnessed it working. So anybody telling you the physics is impossible is simply wrong. We hired outside experts. We kept our skepticism and like many who initially doubted we were convinced. We checked patents. We checked regulatory rules. We checked efficiency calculations. We checked safety. We checked charge times. (…) If anything I’d like to fund 5 more teams and projects this ambitious. And if one of them succeeds it would have been worth doing. But my chips are in on uBeam and I’m not afraid to put my reputation on the line the same way entrepreneurs must each day. Some people have publicly or privately asked me to define exactly what uBeam IS doing and why this backbencher is wrong. We will do that. Of course. When we ship product.” ( Mark Suster on uBeam ) Of course, shipping the product didn’t happen and there’s every indication it will never happen. Since then, high-ranking uBeam employees have quit and publicly confirmed (in full technical detail) that, yes, the claims were always physically implausible just as electrical engineers had been saying all along ( Lies and Startup PR ).
uBeam is also a good illustration of the problems with the “talk to us in person” model. uBeam offered tours of their lab, but only to people they selected and only under NDA. Suppose you’re touring the lab, and you are smart and scientifically literate, but not an expert in ultrasonics (since there are relatively few of those). You get to meet a lot of genuinely cool, smart, experienced, and friendly people, you get to play with awesome machines, you get to see a (carefully constructed) demo, you get to ask people questions about how they got interested in power transmission while fleeing Soviet Estonia. An hour into the three-hour tour, you notice that range might be a problem, since ultrasound is absorbed in air; the uBeam staff just say that they’ll make the transmitters more efficient. When the tour ends two hours later, you’ve of course forgotten all about it, and never look up that it’s impossible to make transmitters that efficient. You also never talk to the seven other people who took the tour, the first of which noted the problems with safety, the second of which noted the problems with line-of-sight, the third of which asked about cost efficiency, the fourth of which asked about interference with animal hearing, and so on and so forth. And so there goes thirty million dollars.