Thoughts on community flow-through effects
Many people in effective altruism talk about “flow-through effects” — places where you achieve one goal, and then side effects of the things you do also help achieve other goals. For example, when you give a dollar to a homeless person, you might help them rise out of homelessness and get a job, which generates tax dollars, which makes more money available to scientific grants, which can be used for mitigating existential risk. This type of “general flow-through” relies on a very large intermediate pool — “the economy”, “the government”, “the global conversation”, and so on — and so the effects are normally microscopically small .
However, there is another type of flow-through effect, which I’ll call “community flow-through”. In community flow-through, some particular group (“the community”) tries to achieve a goal. While doing so, that group acquires assets — money, knowledge, institutional capital, and so on — which can help it to achieve other goals. Unlike general flow-through, because “the community” is usually small, the effects can be fairly important. In fact, sometimes it’s more effective to achieve a goal with community flow-through, instead of tackling it directly.
Anti-aging research is a good example. Suppose you’re a software billionaire, and you want to solve the aging problem. This will probably require a large, world-class biology research team. But setting up and running biology research teams — especially ones that accomplish weird, ambitious goals — is really hard. If you just naively hired a bunch of biologists off the street, they’d probably fall flat on their faces (as happened with eg. Google+).
So what you can do is start off by assembling a small team, and tackling a less ambitious goal, like producing a better anti-cancer drug. In the course of doing that, you (and the managers you hire) will learn how to run a lab, how to identify skilled biologists, how to assess research directions in biology, how to deal with government regulators, how to work well with each other, how to establish trust that nobody is a scam artist, and so on. As you accomplish more goals, you can grow the team and scale up your ambitions, until eventually you have an organization that can tackle the aging problem.
As a historical example, suppose someone was teleported back to 1975. Coming from the future, they’d know that web browsers will eventually have lots of security holes. What’s the best strategy to prevent this failure? The wrong strategy would be to try and build secure web browsers directly; people did try that, but Project Xanadu never went anywhere. There was no demand for web browsers in 1975. But a good strategy, as it turned out, was to write a BASIC interpreter in 8080 assembly language. On the object level, the BASIC interpreter is mostly unrelated — there’s no shared code or algorithms or even design ideas, really — but it’s a) commercially useful in 1975, and b) upstream of web browser technology, when it eventually got built.
Danielle Fong describes the general principle : “This is the analogy our investors at Khosla Ventures have adopted: An ambitious startup can be like an attempt to climb Mount Everest. First the resources have to be marshaled, and a great way to do that is to first establish a ‘base camp.’ For a business, a base camp can be an operating business that produces revenue, and potentially profits. But, most important, that base camp produces and collects the expertise and capabilities necessary to attempt the greater climb to Everest.
In our case, we’re using advanced-pressure-vessel production, sold into the natural-gas business, to establish the profits and production capability necessary to crack the global energy-storage problem. Tanks are our base camp, and the global energy-storage problem is our Everest.
Often startups are seemingly faced with the question, ‘Do I go for the huge win, or do I go for base camp?’ But this is the wrong way to think about it. Arriving at base camp is not the goal; it’s setting you up for the big win.”
Even if some resources should be used for directly tackling end goals, it might still be good to use some for community flow-through, because resources often aren’t fungible. In particular, research teams usually require very specific qualifications; anyone without those qualifications would do much less good as a direct researcher, giving flow-through effects a comparative advantage.
For existential risk, there are many, many projects which might deliver significant community flow-through. Instead of evaluating them ad-hoc, therefore, it might be good to start from the top and work backwards. What type of community is best suited to tackle existential risk? Which kinds of flow-through are most likely to help create that community? And finally, of the possible flow-through routes, which achieve the most gain per unit of effort invested?
In a blog post, MIRI head Nate Soares breaks down the research needed to solve x-risk into four categories: safety engineering, target selection, alignment, and strategy . In addition, there are two other major tasks not mentioned by Nate: using the research to build real AI systems, and getting enough money to finance everything. What kind of community could best achieve these goals? At a guess, such a community would: Be larger than Dunbar’s Number (~150). Each of the six problems is so large, and the skills to solve them so diverse, that we’d realistically need a substantial team to tackle them; it can’t be so small that “everyone knows everyone else”.
Be smaller than “everyone”. Humans are designed to work in small groups, and there are real coordination costs to growing, costs that rise super-linearly with group size. When big companies grow to tens or hundreds of thousands of people, they get bogged down and become less effective at research, which is the single biggest goal. Since it’s hard to kick people out of a group en masse, caution should therefore be taken to not grow “too fast”.
Have a high proportion of very smart people, with strong quantitative skills. The three largest tasks involved are (broadly) research, engineering, and finance, and top performers in these fields tend to be intelligent and good at math. This needn’t be everyone, of course, and there are significant advantages to skillset diversity, but in general people shouldn’t be recruited at random.
Make efforts to acquire and accumulate all types of capital. The most obvious type is financial capital (money). But there is also human capital (skills and knowledge); social capital (connections to important people); institutional capital (groups who know each other, have shared knowledge and trust, and work well together); reputational capital (impressive past achievements); and probably more. Some may be more useful than others, especially on the margin, but in general all are necessary.
Be partially transparent. Being completely opaque (unknown to the world) is almost certainly impractical, given the number of people and the amount of time needed. It might also be undesirable for other reasons, like recruiting. However, since many of the plans involved in x-risk must be “weird”, it’s also good to avoid full transparency; we shouldn’t let journalists simply wander into any meeting they want. Almost all successful organizations, including governments, corporations, nonprofits, and social groups like Y Combinator, have a policy of partial transparency; the group’s existence is public, but what is shown to the press is heavily regulated.
Award social status to create good incentives. Since most people are very motivated by status, at least among some group, this is critical to very difficult goals. In some cases, good incentives are “naturally” aligned with status; status will by default, for example, go to people who get rich to fund research. But in other cases they aren’t. Eg., George R.R. Martin has (by default) extremely high status from his fantasy fiction, but more fantasy fiction isn’t something we should incentivize (although we could incentivize related things, like other kinds of good writing). When something is secret to other group members, like anonymous donations, it can never have any effect on status.
Have high internal trust, to make communication and collaboration easier. This requires that people be kicked out (or ideally not allowed to enter) if they egregiously break social norms, through repeated harassment, large-scale and continuous deception, vandalism, violence, and so on. Similarly, people shouldn’t be added to the group purely because they’re enthusiastic or friendly. Bad people do damage directly, but do much more damage indirectly, since the possibility of a bad person destroys opportunities for positive-sum trade.
Have strong internal communication. Some details may be sensitive and closely-held, but in general, groups shouldn’t be kept in the dark (internally) about what other groups are doing. For this reason, Google has an internal search engine at the address “who”; typing “who/Bob Smith” (as a URL) brings up a page about Bob, with standardized information about where Bob is and what he’s up to.
Make strong efforts to keep people’s System 1s well informed. This isn’t a standard goal, so it might be slightly confusing, but it seems very important. In the Mary’s Room paradox, a neuroscientist named Mary knows everything about the biology of color vision, but lives in a room where everything is black and white. Does she learn anything when the door is opened, and she sees colors for the first time? The answer is yes, and it’s because certain parts of the brain can only accept data in certain formats, like an Apple computer that only has one kind of port. The neurons that do color vision can only respond to signals from the retina, not information about color delivered with words and abstractions. Similarly, Kahneman’s System 1 works on the basis of “what you see is all there is”; anything not immediately in front of you isn’t considered. If you’re in a room with some nuclear launch codes, your System 2 probably knows they’re important, but your System 1 might think that they’re what your eyes see, a few random scraps of paper; a very dangerous mistake! System 1 must be kept up-to-date with information it understands; most importantly, direct, in-person experiences whenever possible. (For example, a team working on global poverty should fly to poor countries, and see how things actually are there.)
Be comfortable with, and award status to executing, ambitious ideas at all timescales (months, years, decades, and centuries). Per Y Combinator, “One of the best ways to help a society generally is to create events and institutions that bring ambitious people together. It’s like pulling the control rods out of a reactor: the energy they emit encourages other ambitious people, instead of being absorbed by the normal people they’re usually surrounded with.” For timeframes, any ambitious large goal (eg. Tesla Motors) must be backed up by smaller ambitious goals (eg., getting a car prototype built in only six months), or it will never happen.
Be comfortable with changing or discarding ideas as new information comes in; while determination is very important, there must be some new data which, if observed, would cause people to give up. High status people can get away with wrong ideas for longer, so they should be especially careful.
Doing a full analysis of how all possible strategies would affect each of these different factors would take a while, so I’ll just go through some of the past alternatives people have suggested. I’ll also describe (what seems to me like) the main rationale for each strategy. One important note is that, when you select a goal for community flow-through, you bias the community towards being the kind of person interested in that goal. Picking AMF as a goal tends to attract professional fundraisers, and people who think fundraising is important and high-status. Picking global warming as a goal tends to attract engineers, public policy wonks and media advocates. Picking anti-aging as a goal tends to attract bio geeks and human-enhancement theorists. These groups all have different demographics and skill distributions. I’d never want eg. pick-up artistry as a goal; not because sex and good relationships are bad, but because the people that goal tends to attract are often creepy and/or manipulative and/or evil. Conversely, if we want to recruit many mathematicians, that’s extra reason for adopting math-heavy fields like cryptography and robotics as goals.
Rationality, in the sense of better science. The goal is to have accurate beliefs about things that are confusing or hard to test, and this would flow through to the community by making people better researchers and strategists.
Rationality, in the sense of positive psychology and self-help. The goal is to improve people’s normal lives, and the flow-through is from increased community capital and possible flow-through towards better science.
Effective altruism. The goal is to help the poor or help other mundane causes, and the flow-through comes from building capital and recruiting via people switching cause areas.
Starting narrow AI companies. The goal is to create commercially successful narrow AI, and flow-through comes from raising money, recruiting researchers, and building expertise in technology that’s eventually useful for more powerful AI.
General earning-to-give. The goal is simply to make money, and flow-through comes from additional availability of capital, although there are also some effects from skill-building, increased prestige and so on.
Human enhancement and embryo selection; the goal is to build better humans, and the flow-through is when they help us tackle the AI problem down the road.