“Try it and see” is not enough evidence in psychology
Continuing the exchange with Nevin Freeman about Leverage Research, the Bay Area organization running a closed-door psychology research program. Nevin had defended Leverage’s methods on the grounds that the people they work with sit down, do the techniques, get the described results, and end up convinced. The note below explains why that kind of personal-experience evidence has a long history of misleading even well-intentioned researchers in psychology.
As someone who has studied science a fair amount, sentences like this set off major alarm bells: “The evidence that ends up convincing people that things are working is seeing it work directly. We tell them what we can do, they sit down and do it with us, and they get the results we described.”
The history of psychology is replete with ideas for interventions — from the most well-intentioned and nobly altruistic to the nastiest of scams, from the most prestigious academics to the rankest amateurs, from the most elaborate and formal theories to the sketchiest of guesswork — where people were convinced by this kind of evidence, took this as positive feedback that they were on the right track, made a wrong turn down a dark alley, and ultimately got nowhere. And that’s not even just imagination, or overconfidence bias, or the placebo effect. Many times, at least, the results they saw were very real. It’s just that the reason why they were real had nothing to do with what the experimenter thought the reason was. Quoth Eliezer: “Yes, patients who see psychotherapists have been known to get better faster than patients who simply do nothing. But there is no statistically discernible difference between the many schools of psychotherapy. There is no discernible gain from years of expertise. And there’s also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It’s just talking to anyone that helps you get better, apparently.” ( Schools Proliferating Without Evidence ) So if Bob Smith invents a new psychotherapy idea, tries it on his friends, and measures the results, he really will see genuinely positive results. It’s just that his new idea, whatever it is, will (almost certainly) have nothing to do with them.
As just one especially famous example, neurolinguistic programming has a ton of fans, some of whom have even promoted it on Less Wrong: “I think this site could find A LOT of benefit in delving into NLP. I mean, the whole field is basically a quest to find the machine-code of the human psyche. The version of NLP that is represented on sites like SkepDic seems like a poor representation of the amazing stuff I am always reading about, which is a shame as it turns people off from reading more about it. (…) Oh, and this concept of asking better questions is something that Anthony Robbins is always talking about as one of the most important factors in self improvement. He says something to the effect of ‘the quality of your life is determined by the quality of the questions you ask yourself’. And Robbins’ background is in, you guessed it, NLP :) TLDR: LessWrong needs to investigate NLP. I am a somewhat smart and non-kooky cookie and I find NLP interesting and worthwhile AND I am a big fan of LW. This might indicate that others could find value in the field as well.” ( LessWrong comment ) I have no reason to doubt that this commenter is smart, honest, and gets a lot of personal benefit from studying NLP. But when you run controlled studies, what you see is: “In the early 1980s, NLP was advertised as an important advance in psychotherapy and counseling, and attracted some interest in counseling research and clinical psychology. However, as controlled trials failed to show any benefit from NLP and its advocates made increasingly dubious claims, scientific interest in NLP faded. Numerous literature reviews and meta-analyses have failed to show evidence for NLP’s assumptions or effectiveness as a therapeutic method. While some NLP practitioners have argued that the lack of empirical support is due to insufficient research testing NLP, the consensus scientific opinion is that NLP is pseudoscience and that attempts to dismiss the research findings based on these arguments ‘[constitute]s an admission that NLP does not have an evidence base and that NLP practitioners are seeking a post-hoc credibility.’ Surveys in the academic community have shown NLP to be widely discredited among scientists. Among the reasons for considering NLP a pseudoscience are that evidence in favor of it is limited to anecdotes and personal testimony, that it is not informed by scientific understanding of neuroscience and linguistics, and that the name ‘neuro-linguistic programming’ uses jargon words to impress readers and obfuscate ideas, whereas NLP itself does not relate any phenomena to neural structures and has nothing in common with linguistics or programming. In fact, in education, NLP has been used as a key example of pseudoscience.” ( Wikipedia: NLP scientific criticism ) The historical record is extremely strong evidence that self-reports of benefits from better therapy techniques (etc.) simply don’t contain enough information to make much real progress in psychology. It’s not even a question of intelligence, or funding, or good intentions; none of those will help you if the information you’re looking for isn’t in the place you’re looking for it. It’s like trying to navigate a spaceship to Pluto with a pair of handheld binoculars; the tool is just not suitable for the task.