The meta problem and the object-level problem
A reply to AI alignment researcher Paul Christiano, who had circulated a Google doc on “cause prioritization” — the effective-altruist project of figuring out, before donating any money, which broad cause area (global poverty, animal welfare, AI safety, etc.) gives the most good per dollar. Christiano had argued that working on the meta-level question of how to prioritize is itself among the highest-priority things one can do. Alyssa pushes back: certain meta problems are best attacked indirectly, by working on a well-chosen object-level problem instead, and she points to Geoff Anders’s research organization Leverage as a case where she thinks the ratio of meta to object-level work has gone wrong. Rather than argue the point head-on, she gathers three quotations that she thinks gesture at the cluster of intuitions behind the position.
The meta problem here is, almost by definition, the most important problem to solve; but it seems likely that some types of meta problems are best addressed by tackling certain, specific kinds of object-level problems instead of the meta-problem directly. (Eg. I think Leverage would be much more effective if they shifted time budget to reasonably well-chosen object-level problems.) I’m pretty uncertain here, so I’ll just quote some people who seem like they have this type of perspective on things, and hopefully I can point at the relevant cluster in conceptspace: “If it takes years to articulate great questions, what do you do now, at sixteen? Work toward finding one. Great questions don’t appear suddenly. They gradually congeal in your head. And what makes them congeal is experience. So the way to find great questions is not to search for them — not to wander about thinking, what great discovery shall I make? You can’t answer that; if you could, you’d have made it.
The way to get a big idea to appear in your head is not to hunt for big ideas, but to put in a lot of time on work that interests you, and in the process keep your mind open enough that a big idea can take roost. Einstein, Ford, and Beckenbauer all used this recipe. They all knew their work like a piano player knows the keys. So when something seemed amiss to them, they had the confidence to notice it.” — Paul Graham, What You’ll Wish You’d Known “But let me say why age seems to have the effect it does. In the first place if you do some good work you will find yourself on all kinds of committees and unable to do any more work. You may find yourself as I saw Brattain when he got a Nobel Prize. The day the prize was announced we all assembled in Arnold Auditorium; all three winners got up and made speeches. The third one, Brattain, practically with tears in his eyes, said, ‘I know about this Nobel-Prize effect and I am not going to let it affect me; I am going to remain good old Walter Brattain.’ Well I said to myself, ‘That is nice.’ But in a few weeks I saw it was affecting him. Now he could only work on great problems.
When you are famous it is hard to work on small problems. This is what did Shannon in. After information theory, what do you do for an encore? The great scientists often make this error. They fail to continue to plant the little acorns from which the mighty oak trees grow. They try to get the big thing right off. And that isn’t the way things go. So that is another reason why you find that when you get early recognition it seems to sterilize you. In fact I will give you my favorite quotation of many years. The Institute for Advanced Study in Princeton, in my opinion, has ruined more good scientists than any institution has created, judged by what they did before they came and judged by what they did after. Not that they weren’t good afterwards, but they were superb before they got there and were only good afterwards.” — Richard Hamming, You and Your Research “In Archimedes’s time, slavery was thought right and proper; in our time, it is held an abomination. If, today, you need to argue that slavery is bad, you can invent all sorts of moral arguments which lead to that conclusion — all sorts of justifications leap readily to mind. If you could talk to Archimedes of Syracuse directly, you might even be able to persuade him to your viewpoint (or not). But the really odd thing is that, at some point in time, someone must have turned against slavery — gone from pro-slavery to anti-slavery — even though they didn’t start out wanting to persuade themselves against slavery. By the time someone gets to the point of wanting to construct persuasive anti-slavery arguments, they must have already turned against slavery. If you know your desired moral destination, you are already there. Thus, that particular cognitive strategy — searching for ways to persuade people against slavery — can’t explain how we got here from there, how Western culture went from pro-slavery to anti-slavery. (…)
Can you suggest that Archimedes pay attention to facts, and authorities, and think about which one should ought to take precedence — by way of leading him down a garden path to the scientific method? But humanity did not invent the scientific method by setting out to invent the scientific method — by looking for a garden path that would lead to the scientific method. If you know your desired destination, you are already there. And no matter how you try to prevent your garden path from looking like a garden path, the laws of time travel know the difference.” — Eliezer Yudkowsky, Archimedes’s Chronophone