Community metrics
Hi! Since the New Years' Eve party, I've been thinking about the problem of how to measure community strength and progress in community-building. I think this question is interesting, important, and unsolved - Eliezer wrote a post about measuring "rationality" way back in the day (http://lesswrong.com/lw/2s/3_levels_of_rationality_verification/), which just looked at individuals rather than groups, but even that never really went anywhere AFAIK.
I believe Tara mentioned that CEA was currently tracking web traffic and newsletter signups, but this seems like a terrible metric, for a number of reasons. For a for-profit business, which is what YC was built around helping, the end goal is simply to get lots of people to pay you - why doesn't matter that much, as long as the credit card numbers are valid. But in this case, the end goal is not to get newsletter signups, that's just a means to the end. Optimizing heavily for the means is likely to fall victim to Goodhart's law (https://en.wikipedia.org/wiki/Goodhart's_law). In particular, the easiest way to boost those metrics is to sign up large numbers of random people who (at best) do nothing, or (at worst) divert resources from important projects to pointless or counter-productive ones.
Environmentalism seems like a useful case study here... a year or two ago, I had this conversation (paraphrased slightly) with a very smart and dedicated scientist I know working on the climate change problem:
Scientist (who I was already friends with, being quoted in a newspaper): Antarctica is melting! The West Antarctic ice sheet could collapse at any moment! Me: Wait, what? Antarctica doesn't work like that. That sounds completely impossible. Me (looks up paper): This paper uses the RCP 8.5 scenario, and looks at the result of 8 C of warming around the year 2300. 8 degrees C! Well of course the ice will melt if you're warming by eight bloody degrees C, by then you'll have much worse problems though.... Me (email to scientist): Ummm, I looked up this paper, and you know it's talking about an extreme scenario three centuries out, right? Scientist: Yeah, I know. But I couldn't say that in public. Me: Why not? Scientist: Well, we've decided that since everyone will understate things by a factor of four, we have to overstate them by a factor of four, to make it balance out.
Aside from the obvious ethics/trust problems with this, it makes it impossible to solve problems by backward chaining, because the movement is built around large numbers of people who have strong opinions, but whose knowledge is shallow or flat-out wrong. For example, there are endless political fights about legislation to "reduce emissions" in a generic sense, but this doesn't solve the underlying problem that almost everyone in the US is de facto legally required to live in a large house and drive long distances every day. Rooftop solar installation doesn't solve the base load generation problem, resulting in the now-famous duck curve (https://assets.bwbx.io/images/users/iqjWHBFdfxIU/ilVXr6BF7iAU/v4/-1x-1.png). Various types of recycling are either impractically inefficient, or actively make things worse (eg. growing lots of trees, using the wood or paper once, and then burying it in landfills is equivalent to carbon sequestration). Corn ethanol... well, I won't bother to continue beating a dead horse. :)
That all raises the question, of course, of other what metrics would be better. Unfortunately, I don't think there's any easy way to solve this - there is, as of yet, no real "natural" way of collecting good data about large numbers of people, many of them strangers, scattered all over the world (though I do dream of solving this someday). I think the two main options are survey results and web databases, and both are a significant amount of work. For a survey, of course, someone has to do the work to design, write, market, administer, and summarize the survey data. Web databases are easier, since everything is already "there", but to get solid data you need a site where a significant fraction of the community spends a large chunk of their time (eg. Facebook, Hacker News, Overcoming Bias, SL4, Usenet groups), and maintaining that is also labor-intensive. (I believe HN now has two full-time professional moderators plus a full-time developer, and it's still not enough.) Otherwise, you can't track things like what do people say, how many people read it, how engaged are people, what words do people use, what's the ratio of active people to lurkers, how deeply nested are conversations, and so on, which are orders of magnitude more informative than just looking at hit counts.
What types of data should be collected specifically is complicated, with many different criteria to optimize for, but I'll take a crack at it anyway because it seems under-explored. The first thing I'd look at