On CFAR’s longitudinal study
The Center for Applied Rationality (CFAR) is a Berkeley-based nonprofit that runs paid workshops teaching cognitive techniques drawn from the LessWrong rationalist community. In late 2015 it published a one-year longitudinal study of workshop participants, reporting modest positive effects across several life-outcome metrics. This response, written to Oliver Habryka, walks through the study’s methodological gaps with the standards of charity evaluation used by GiveWell as a comparison point.
On the one hand, it’s great that CFAR has released some of their data, and several positive effects were found. On the other, the writeup is pretty clearly biased towards pushing their point of view, much more than I’d expect from eg. GiveWell. It might not be feasible to mitigate every problem at a reasonable cost, but I think possible problems should at least be mentioned, as part of having good epistemic hygiene. Going over specific issues: They discuss several possible methodological flaws, but they notably don’t really discuss a very well-known and well-studied cognitive bias called choice-supportive bias . If you choose to attend a CFAR workshop (or pretty much anything else), you’ll retroactively decide that the advantages of your choice were larger, and the disadvantages smaller, than you would have otherwise. Hence, someone who chooses to attend a CFAR workshop is likely to (possibly unconsciously) give responses that would tend to justify that decision.
There is no real discussion of a possible placebo effect (ie. someone tells you that doing X will make your life better, you do X, and then your life becomes better because you believed it would work), except to dismiss the option of a placebo-controlled trial out-of-hand. Placebo effects are known to be very large in psychology; for antidepressants, one study found that “the effect size of placebo is 0.92 and the effect size of antidepressants is 1.24, which means antidepressants have a 0.32 SD benefit over placebo.” 0.92 is huge ; it’s more than twice as large as the largest effect CFAR reported on any of their metrics. Whatever the practical constraints are, it’s easy enough to construct and discuss a hypothetical placebo group; eg. find a group of people who believe in psychic powers, recruit them for workshops where cold-readers try to “psychically” improve their lives, and then do a similar survey. It’s true that as stated, this plan is likely impractical. But the way CFAR is structured actually allows for some neat tricks here, because CFAR’s classes are not static (they change from month to month). One could, before a workshop, ask the instructors how effective they thought the techniques in this particular workshop would be compared to past workshops, on a scale from 1–10. Then you could run a correlation between this score and the various outcomes measured.
(Also, quoting Eliezer in Schools Proliferating Without Evidence : “Yes, patients who see psychotherapists have been known to get better faster than patients who simply do nothing. But there is no statistically discernible difference between the many schools of psychotherapy. There is no discernible gain from years of expertise. And there’s also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It’s just talking to anyone that helps you get better, apparently.”)
There isn’t really any discussion of possible causal confounders. They note that, on average, people in their 20s don’t change much in terms of things like personality metrics and overall life happiness. However, there are many possible values of X for which X causes both major life benefits and attending a CFAR workshop, or where X causes both Y (which in turn causes major life benefits) and attending a CFAR workshop. These include things like graduating from college, learning how to program, moving to the Bay Area, moving to the US from <poor country>, becoming interested in existential risk, becoming interested in AI, becoming interested in effective altruism, hanging out with MIRI researchers, and reading HPMOR. Of course, all of these might also be caused by attending a CFAR workshop, but it’s pretty unclear how often the causality runs in which direction given the data we have.
They discuss an alternative form of study (peer assessment) which hopes to avoid some of the possible biases. However, while this is indeed better in some respect, it’s likely that (for logistical reasons) the people doing the peer assessments would also be fans of CFAR/have attended a CFAR workshop themselves. Hence, things like choice-supportive bias would still likely apply to them.
The data publication only includes adjusted affect sizes, and not things like averages or standard deviations. Thus, it’s quite hard to tell how large the effects actually are. Making up numbers, Scott says that “the guy who invented effect size suggested that 0.2 be called ‘small’, 0.5 be called ‘medium’, and 0.8 be called ‘large’.” By this largely arbitrary standard, all of the effects that CFAR identified are “small” (or nonexistent). However, a small effect size could still be quite important if the standard deviation were very large — if, for example, many of the participants were students and this caused a wide range of incomes, a “small” income effect would still make workshops profitable on net (though CFAR didn’t actually find any effect on income).