Series on Instrumental Rationality
A draft series of posts written for the LessWrong rationalist community, then in its early days as a Bay Area online forum about Bayesian reasoning, cognitive bias, and existential risk from artificial intelligence. The community had been heavily focused on what it called “epistemic rationality” — the art of having accurate beliefs — under the slogan that rationality is “the art of winning.” This series argues that what they actually had was the art of not believing false things, and sets out, post by post, a more practical alternative. Eliezer Yudkowsky, founder of the Singularity Institute (later MIRI), is the LessWrong figure referenced throughout.
It’s What You Do This is the first post in a series on what is sometimes called “instrumental rationality” (though I personally dislike that term). I am writing this series because I can find no good summaries of it on Less Wrong (or elsewhere), and I feel that this material is important and should be covered. I am, of course, wholly inadequate for the task, but I’m going ahead and doing it anyway, and after some as-yet-undetermined number of posts, I’ll explain why this makes perfect sense from an instrumental rationality perspective.
The goal of what is sometimes called “instrumental rationality” is to push the world into a certain state. The world is in place A, you want it to be in place B, and it doesn’t look like it’s going to get to place B anytime soon, so you have decided to get out and push. The first and most important principle is: by analogy to the Bottom Line principle, the only things that count in instrumental rationality are the actions that you take . A given set of actions moves the state of the world from A to B, and is scored based on whether B is a better or worse place than A. If, after you have finished, B is a worse place than A, or sufficiently similar such that no one can tell the difference, you have failed as an instrumental rationalist — period.
The most striking difference between this and the normal conception of rationalism is that thoughts don’t count, and I suspect that this is a major paradigm shift for a lot of people. If you save the lives of a thousand people, it matters not a whit if you did it in the name of atheism, Christianity, or Scientology. I, for one, am sure that the thousand people won’t care (if they are at all sane). One who saves a thousand while shouting about the hidden conspiracy of Xenu is a hundred times as virtuous as one who saves ten while quoting Feynman and Dawkins. If you have evaluated all the charities known to you, and by some crazy chance, the most effective one is run by believing Catholics, donate to it anyway, because you will be doing more good than if you did not.
One might say, perhaps, that by assigning high status to and helping irrational people who reliably do lots of good (either because of, or despite, their irrationality), we would be preventing rationality from spreading. This argument is roughly similar in form to the astrologers who point out that, through gravity, the motions of the planets do exert some influence over human bodies. While the effect is certainly there, it is tiny; the effect that any one of us could have in the next year on the religiousness or rationality of human civilization as a whole is so absolutely miniscule as to be not worth worrying about. With the planet in danger from existential risks that no one else seems to be paying much attention to, we have more important things that should concern us. The battle between Dawkins and the creationists is fun to watch, sure, but we should not make it our primary goal to fight on Dawkins’ side.
Good thoughts, of course, are important (and in some cases essential) for good actions. But I strongly suspect that we, since we are much better thinkers than average, tend to overvalue the importance of good thought. In the worlds of math, computer science, philosophy, and theoretical physics, where a large number of us come from, one can achieve greatness purely by thinking. This is the exception and not the rule — there are a huge number of great thinkers who achieve very few of their goals. Consider the case of the average postdoc in the sciences at a U.S. university, an overwhelmingly better thinker than the US average. He (postdocs in the sciences are largely male) probably makes less than the U.S. median income, might work 60 to 70 hours a week, has no job security, little respect or social status, probably no kids, and most likely works on problems that, in the scheme of things, are utterly trivial. He probably (with some luck) gets to work on problems he considers interesting at least some of the time, but, with the exception of fields like particle physics that require prohibitively expensive equipment, there are a large number of interesting problems that can be tackled in one’s garage (this, in fact, is how most science was done up until the late 19th century). His thoughts may be world-class, but his actions, in many cases, are nowhere near as effective. And in the long run, this hurts, not only him, but science itself — wouldn’t science be so much more reputable and so much better funded if the best practitioners were all billionaires and CEOs?
Rationality has been sometimes described on Less Wrong as “the art of winning”. However, it is only reasonable that, before we call something the “art of winning”, we should check to make sure whether it is actually the art of winning; putting on a sign saying “COLD” does not make a refrigerator chilly. Any fair, reasonable evaluator of the material currently on Less Wrong would be forced to conclude that it is not the art of winning; a much more accurate name for it is the art of not believing false things. Where, for instance, are the articles explaining how to become friends with someone, talk with them, and persuade them to your point of view, without getting into an argument with them or seeming hostile? I will not write about this myself, for it is not my specialty, but it is obvious that it is terribly useful in virtually every situation, and yet Less Wrong is essentially mum on the subject.
Now, this does not mean that the material currently on Less Wrong is stupid or useless — quite the contrary, it is tremendously valuable for a huge number of things. But this doesn’t mean that we should go into a happy death spiral around it, and assume that it is the solution to literally everything. An art of not believing false things can be useful and admirable in its own right, without also being an art of persuading people or investing money or avoiding procrastination or doing the dishes. We should be happy to gaze upon the adorableness of kittens, just because they are adorable; they need not also fly, teleport, or cure smallpox. A major part of choosing actions that are effective is knowing the limitations of any one technique, a point I’ll return to later.
A clever argument, or a wonderful new belief, even if it is actually correct, will not help you — will not aid you in achieving your goals — if it doesn’t change the things that you do. Even if you believe in it. Even if you make it your life’s motto. Even if you write twenty books about it, and edit them until they are the highest quality nonfiction in the world, and leave them sitting around in a desk drawer instead of publishing them. Do good, or do bad, or do nothing — that is the ultimate test.
The Marketing Arms Race In today’s world, how do we determine which actions are useful, and which actions are ineffective? It wasn’t always this way, but nowadays, one of the most important underlying concepts is that of the marketing arms race, which we as a society continue to spiral deeper and deeper into.
Consider a town where there are two soda stores, Vendor A and Vendor B. Both A and B sell soda from roadside stands. Initially, Vendors A and B are identical — they have the same locations, same prices, the same quality, and they each have 50% of the market. Now, suppose that Vendor A wants to make more money. What can he do? He has four options: He can expand the market for soda, raise the price of soda, lower his own cost of production, or steal market share from Vendor B.
The first one would be very difficult; raising the city’s population would be hard, and the people of the city already drink so much soda that it’s rotting their teeth. The second, of course, requires that he develop a better quality product, or else people would have no reason to buy his soda, instead of the cheaper Soda B. The third requires that he find a way to cut costs without lowering quality, or, again, people will just switch to Soda B. The second and third could be, and are, done, but they are still quite difficult. To improve soda quality or lower cost would require thinking about the soda production process, and most people dislike thinking.
What about the fourth option, stealing market share from Vendor B? Well, it still requires work, but it’s a lot more straightforward than the other three. All you have to do is persuade people who drink B to switch to A, and the human brain is built to persuade other people to your point of view. So, Vendor A goes out on the town every night, and puts up big posters telling people to switch to Soda A. The campaign is a spectacular success — next week, his market share rises to 80%.
Now, what does Vendor B do? Naturally, he notices Vendor A’s success, and the next week, he goes around and puts up his own, identical posters. Since both A and B now have posters, people are once again indifferent between the two, and next week A’s market share drops back to 50%.
Notice how, after both rounds of marketing, society is strictly worse off: no one has benefited. The citizens of the city are still drinking the same soda, at the same price, from the same stores as before. Both A and B still have the same revenue, and the same cost of production. The only difference is that A and B have wasted a large amount of money and time putting up their posters; A and B are now both worse off than they were at the beginning.
By now, I hope people have noticed that this sequence of events is isomorphic to the Prisoner’s Dilemma: both parties would benefit if they agreed to not put up posters, but each individual is strictly better off if they do put up posters. There is, however, one important difference. In the Prisoner’s Dilemma (or, more generally, the Iterated Prisoner’s Dilemma), there are only two options: cooperate or defect. The damage that can be done by defecting is significant, but it doesn’t include the possibility of arbitrarily large negative payoffs. With marketing, on the other hand, you can continue developing more and more effective marketing strategies, which your competitors can then copy, requiring you to develop even more effective strategies, and so on ad infinitum. It’s as if the Prisoner’s Dilemma, in addition to having “cooperate” and “defect”, also had “super-defect”, and “super-super-defect”, and “super-super-super-defect”…
We, as a civilization, are already a long, long way into this slide. It’s been going on for well over eighty years now (it was the main basis for Hitler’s rise to power). So, if you ever want to get people’s attention — for any reason at all — don’t expect it to be easy. The flip side of this is that, with a few very unusual exceptions, if you don’t try to get people to notice you, they won’t. This means that there are all sorts of incredible, wonderful things lying around undiscovered, because the people who know about them either aren’t good at marketing, or don’t have the resources for it. We already know about immortality (the Immortality Institute) and saving lives for under $1,000 (GiveWell) and changing the destiny of all mankind (the Singularity Institute). What else is out there? I wouldn’t be surprised at all if some scientist somewhere has already cured cancer, and people just didn’t notice.
Eliezer Yudkowsky once said, “You have a rather idealized view of academia if you think that they descend on every new idea in existence to approve or disapprove it. It takes a tremendous amount of work to get academia to notice something at all — you have to publish article after article, write commentaries on other people’s work from within your reference frame so they notice you, go to conferences and promote your idea, et cetera.” This principle applies to, not just academia, but the world in general. For the most part, it takes a tremendous amount of work to get anyone to notice anything.
The Web has not made marketing irrelevant. What the Web has done, if anything, is to make marketing even more important than it was already. Consider the magazine market in 1990. What did it take to successfully run a magazine? Marketing skill, of course, but you also needed things like capital, distribution, a printing shop, a graphics design team, paid professional writers… Nowadays, you don’t need any of these. But people still have a limited amount of time — one person can only read so many magazines. Some factor must let people choose between them. What moves in to pick up the slack? Largely, marketing.
The way the Web fools people is a little subtle, since many Web companies don’t have a large marketing department or big advertising budget. Facebook doesn’t run advertisements. It doesn’t have a huge PR department, with thousands of people who talk to reporters and try and get Facebook in the news. It doesn’t call people up and try to sell them anything. How can they be doing more marketing than, say, the local piano company, which runs ads in the newspaper all the time?
The answer is that a lot of Web companies have come up with an ingenious plan: hire your users to do the marketing for you, in exchange for their use of your website. Suppose it’s 2006, and Joe Schmo, a random member of the public, signs up on Facebook. What does he need in order to use Facebook? Well, he needs Facebook’s software and databases. But he also needs his friends to sign up — if none of his friends are on Facebook, the service is useless. So, of course, he asks all of his friends to sign up on Facebook. In effect, he’s paying for Facebook with time instead of money — instead of paying for the service, he is required to spend time marketing the service in order to use it effectively.
How is this different from people just telling their friends about something they like? In the pre-marketing days, if I built a better mousetrap, and you bought it and it worked well, you might tell your friend about it (this still does happen sometimes, e.g. with Apple computers). And for that to happen, I don’t need to have a marketing strategy — I can just concentrate on making good mousetraps. The incentives are directly aligned; I am motivated to do those things, and only those things, which actually help my customers. With Facebook, on the other hand, the user benefits directly from evangelizing the service. If I buy an Apple laptop, and then recommend it to my friend, there’s nothing in it for me. But with Facebook and MySpace and Twitter and FarmVille, each time you refer one of your friends, it directly makes the service more useful for you. So all these companies aren’t really a return to the old days, where good ideas and products spread by word of mouth; they’re a sign of the new days, where only things painstakingly designed to spread by word of mouth spread by word of mouth.
How The World Counts What’s five, minus two, plus seven, minus four? The obvious answer is “six”. But sometimes (though not always!), the obvious answer is not the correct one. For this post, the answer happens to be twelve. For that is, in large part, how the world actually does arithmetic.
As a first example, consider the beginning of Arnold Schwarzenegger’s Wikipedia page: Arnold Alois Schwarzenegger (born July 30, 1947) is an Austrian American bodybuilder, actor, model, businessman, and politician, who is currently serving as the 38th Governor of California.
Schwarzenegger began weight-training at fifteen. He was awarded the title of Mr. Universe at age 22 and went on to win the Mr. Olympia contest a total of seven times. Schwarzenegger has remained a prominent face in the bodybuilding sport long after his retirement, and has written several books and numerous articles on the sport.
Schwarzenegger gained worldwide fame as a Hollywood action film icon, noted for his lead role in such films as Conan the Barbarian and The Terminator . He was nicknamed the “Austrian Oak” and the “Styrian Oak” in his bodybuilding days, “Arnold Strong” and “Arnie” during his acting career, and more recently the “Governator” (a portmanteau of “Governor” and “Terminator”).
As a Republican, he was first elected on October 7, 2003, in a special recall election to replace then-Governor Gray Davis. Schwarzenegger was sworn in on November 17, 2003, to serve the remainder of Davis’s term. Schwarzenegger was then re-elected on November 7, 2006, in California’s 2006 gubernatorial election, to serve a full term as governor, defeating Democrat Phil Angelides, who was California State Treasurer at the time. Schwarzenegger was sworn in for his first elected term on January 5, 2007.
Schwarzenegger is married to journalist Maria Shriver. The two have four children.
The beginning of the article, all five paragraphs, is entirely about Arnold’s successes. None of his failures are even mentioned. I don’t know all that much about Arnold, but he must have had failures on the way to success — everyone does. So why doesn’t the article talk about them?
When evaluating the bottom line of one’s actions, one of the most useful tools is to go look at other people who we know have accomplished a lot, and go see what they did. And when one looks at their biographies, one discovers something interesting. For the most part, the only things that count when trying to accomplish something, in politics or science or business or art, are your successes — the things you did well. Your failures, the things you did badly, are very nearly irrelevant . People count the hits and forget the misses. You can see this pattern when you Google the biography of nearly any famous person — it’ll mostly be about the things they achieved, and not the things they failed to achieve.
However, for the most part, people don’t act like the hits will be counted and the misses will be forgotten. I don’t act like it myself, a lot of the time. Why? Well, in the ancestral environment, if you lived in a group of under a hundred people, those hundred people would sure remember every time you screwed up. A bad enough screwup might negatively affect their opinion of you for years and years. Under those circumstances, the obvious thing to do would be to develop an instinctive fear of failing in front of other people — and to not bother checking if those people were actually paying attention, since thousands of years ago, they fairly reliably were.
Nowadays, one of the biggest advantages in living a world after the start of the marketing arms race is that people don’t spontaneously notice anything , including failure; something has to be marketed before anyone notices. And some things are much easier to market than others. To be marketable, an idea should be simple, believable, and unexpected : people mostly aren’t interested in hearing about the same things over and over. If some random person fails at something, it isn’t unexpected. Random people fail at random things all the time: everyone’s life, no matter who they are, is full of failures and mistakes. Eliezer Yudkowsky once said, “If I tried to make a better car from bronze squares, and failed completely, ending up with a heap of scrap metal, there would be nothing surprising about that. More experienced engineers would just shake their heads wisely and say, ‘That’s why we use iron triangles, kiddo.’”
Most people, including most Less Wrongers (and me), aren’t already famous enough to be paid attention to by default . As far as I know, out of the top 10 karma scorers on Less Wrong today, only Eliezer Yudkowsky is. And if you aren’t already famous, if you try to (say) start a web business for six months and fail, few people will notice, and even fewer will remember for any length of time. So, if you try a bunch of projects in succession, the net sum is likely to be positive even if most of the projects are failures. It also follows immediately that, all else being equal, one should try to do more projects rather than less, because the expected utility of an additional project (even one likely to fail!) is still likely to be positive.
Needless to say, this is terrible advice when dealing with potentially existential-risk-causing technologies. If I build a hundred superintelligent AIs, the world will not keep the ones that are Friendly and forget about the ones that aren’t. The world will, instead, quite possibly be destroyed if the very first AI I build is unFriendly. And, because existential risks are the most important thing in the world, I suspect they have a higher-than-average availability in these circles.
Luckily, very few of our activities involve directly building technologies that might increase the probability of existential risk. So, for most ordinary goals, like getting funding, finding a girlfriend/boyfriend, or getting hired for a great programming job, counting the hits and forgetting the misses is still an incredibly useful heuristic, even if preventing existential risk is a major long-term goal of ours.
Do Something Instead Of Nothing We all have algorithms for choosing between actions. How well do the algorithms work? Well, for the most part, not very well at all. Every human sometimes makes terrible decisions. However, do these algorithms work better than chance ? Are humans any better at distinguishing between good and bad actions than a random number generator?
I’d say that the answer is fairly clearly yes. Most of the decisions we make every day are simple — doing laundry, driving to work, filling out forms, watering the plants, and so on. And you don’t need very good algorithms to make better-than-entropy decisions when the decisions are simple. If that’s the case, then, on average, we should try to do something instead of doing nothing, for even if our algorithms are poor, they still do better than the null case. (This has become even more true over the past century.) However, it seems to me like a lot of people intuitively believe the opposite — that in most cases, we should opt to do nothing, because taking action would do more harm than good. Why is that?
It is a very basic principle that deliberately trying to do something is more likely to accomplish it than not deliberately trying. Finding a random $20 in the street is nice, but if you deliberately optimize your life for obtaining as much money as possible, you can go a lot farther than that, even without much skill. Conversely, it may seem terrible when your computer randomly breaks, but if you try to do as much damage to your life as possible, you can make things several orders of magnitude worse.
This means that, if you do a mental search for “big disasters”, the most available ones will involve direct, deliberate action by some person, whether yourself or someone else. I suspect that this gives deliberate action an unfairly bad name — for, in counting all of the big problems caused by action (for instance, WWII), it ignores all of the little problems caused by inaction, and those look collectively like a much bigger deal.
On the small scale, we are all familiar with this situation. If you take the garbage out, the garbage can becomes empty; if you don’t, the room starts to stink. If you do the dishes, you have clean dishes to eat off of; if not, you wind up eating off of dirty dishes or a table. If you go and buy food, you have lots of food to eat; if not, you go hungry, or pay a lot for food at a restaurant. And so on. We all know that doing a lot of things, on a small scale, leads to a nice life; while not doing anything, on a small scale, leads to living in poverty and squalor.
I think that, collectively, we are all living in a world that’s isomorphic to a pigsty house, where no one ever does the dishes and no one ever takes out the trash. We’re doing better now than we have in the past. But the world of today is nothing, compared to the shining, sparkling utopia that could exist if people simply did more things, using the decision algorithms that they already know how to execute. So many things never get done by anyone. Whose job is it to notice new cults, so they can be prevented from growing exponentially? Whose job is it to prevent sectors of industry from becoming oligopolies? (The government takes action against monopolies sometimes, but almost never against oligopolies.) Whose job is it to prevent indefinite growth in “education” spending at twice the rate of GDP? These are all vitally important things. And we all realize that they’re important things! I know of not a single person, Less Wrong reader or not, who is a fan of Scientology or the Moonies, who likes dealing with big company bureaucracies and “customer service”, or who enjoys paying hefty school property taxes. Yet, in all three cases, with the exception of a tiny handful of part-time volunteers, nothing ever gets done.
It is, of course, also important to choose effective actions, in addition to simply choosing to act. Preventing existential risk is a more important goal by far than any of these. However, the first and most important step is to just do something instead of nothing . If, at the end of the day, what you actually wind up doing is nothing instead of X, it doesn’t matter that there are ten other things you could do which would be a hundred times more effective than X. Those don’t show up on the bottom line. Your final score is still zero, instead of at least being positive.
Even if we consciously agree that doing X could be useful and is unlikely to result in serious harm, I strongly suspect that there are still barriers to acting: unconscious ones. Most of the machinery of our brain operates below the level of deliberate reasoning. While typing this sentence, I remembered to breathe oxygen, match the words I was thinking to the motions of my hands (so as to make them appear on the screen), remain sitting upright, blink every so often, look around to make sure my stuff hadn’t gotten lost, and move around a bit so my legs don’t go numb, all without paying any deliberate attention at all. Yet, each of these things are quite difficult to do, in the sense that it would be a lot of work to build a robot that can do them. Just as we frequently breathe without noticing, we all refrain from doing useful things, out of unjustified fear, without noticing.
One possible solution to this is to lower your general level of inhibition , by practicing doing things that you feel inhibited about. Studies have shown that there is a significant positive correlation between teenage alcohol use, and income later in life. Why would that be? Drinking alcohol doesn’t make you smarter. Nor does it make you work harder, or become more skilled, or gain additional knowledge. I think the reason is that alcohol is disinhibitory . People do things while drunk that they wouldn’t do otherwise, and even though a lot of them are stupid and destructive, some of them are useful (like meeting new people). And the world counts the good things and forgets about the bad things, like how everyone forgot about George W. Bush being branded with a coat hanger by his fraternity.
As an experiment, I’d like to suggest that people try this: Fill a good-sized glass with tap water. Take everything valuable out of your pockets, get a towel, stand in a suitable place, and dump the glass of water over your head. Yes, I actually think you should go and do this. Yes, I really did it myself a few minutes ago. Yes, actually stop reading and go do it.
Right now . It won’t take very long. I’ll upvote anyone who does.
<PAUSE> Objectively, there’s no point to this at all. But it might help as practice for actually going out and doing something, even something that feels a little awkward, after one has decided to do so. Having taken action of some sort once, I think, people might be more inclined to go out and do it again. And that would be quite valuable for a lot of things.
Carpe Diem How can one, on a day-to-day level, reliably do something instead of nothing? There’s (supposedly) a Chinese saying about how opportunity is always knocking, but only some people are able to hear it. I think this is a bad metaphor (although not completely inaccurate), because knocking implies something loud and forceful; a knock is something that makes you stop what you’re doing, and pay attention to the interruption. This is, in fact, the complete opposite of the truth. Finding real opportunity, in my (albeit limited) experience, is more like fishing. You sit there, and wait, and watch things happen; then, from out of the distance, something approaches. But it won’t demand your attention or interrupt what you’re doing, and it’s probably not going to bite on its own; you have to make an effort to notice it, and then take action yourself to lure it in.
There are a large number of goals that are useful for just about anything. These include things like money, power, friends, skill, toughness, experience, and so forth. These goals are sufficiently broad that there are always little opportunities, floating by on the wind, that can help fulfill them. An email from a stranger, asking for help on a project. A poster about participating in an economics study. A chance to volunteer somewhere. An ad for a job in your field.
My advice is: if these things look like they might be useful, do them! Carpe diem — seize the day, seize the moment, seize the opportunity. Don’t let the opportunity disappear back into the ether. Grab it, hold on to it, and only let it back if you’ve investigated it and you know it’s a dud (which is different from the intuitive guesstimate that “it probably won’t amount to anything”).
One of the main objections to this, I suspect, is that seizing the day demands resources — investments of time, money, energy. This, of course, is true. But ask the question — are these resources being used for anything? Are you doing something instead of nothing with the resources you have? (“Something” certainly includes things like vacations, if a vacation would genuinely be useful.) If not, why not seize the day, and at least do something with your spare three hours? Why let opportunity go to waste?
A corollary to this is: if you are busy seizing opportunity on a regular basis, don’t feel too bad about any single, missed chance, or any one thing that you made a mistake on. More chances will come along; they always do. The phrase “once in a lifetime opportunity”, in my experience, is hogwash. And the opportunities that come later will probably be comparable in importance to the one you lost earlier. Exceptional things, by definition, are rare; most of the time, the magnitude of the missed opportunity will be about average.
Level One and Level Two Actions Suppose that you go onto Mechanical Turk, open an account, and spend a hundred hours transcribing people’s speeches. At current market rates, you’d get paid around $100 for your labor. By taking this action, you have made yourself $100 wealthier. This is an example of what I’d like to call a Level 1 or object-level action: something that directly moves the external world from a less desirable state into a more desirable state.
On the other hand, suppose you take a typing class, which teaches you to type twice as fast. On the object level, this doesn’t move the world into a better state — nothing about the world has changed, other than you. However, the typing class can still be very useful, because for every Level One project you tackle later that involves typing, you’ll be able to do it more efficiently — you’ll get a higher return on your time. This is what I’d call a Level Two or meta-level action, because it doesn’t make things better directly, but makes things better indirectly, through Level 1 actions. You can also get Level 3 (meta-meta-level) actions, Level 4 (meta-meta-meta-level actions), and so on and so forth.
The most important difference between Level 1 and Level 2 actions is that Level 1 actions tend to be additive , while Level 2 actions tend to be multiplicative . If you do ten hours of work at McDonald’s, you’ll get paid ten times as much as if you did one hour; the benefits of the hours add together. On the other hand, if you take ten independent typing classes, each one of which improves your ability by 20%, you’ll be (1.20)^10 = 6.2 times better at typing at the end than at the beginning: the benefits of the classes multiply.
One of the most obvious results is that spending time on Level 2 actions can have a much greater net return than spending time on Level 1 actions. If your labor is worth $20 an hour, and you can’t change that number, then the amount of money you can earn in a year has a fairly hard upper bound — no matter how you slice it, there are only 168 hours in a week. If you spend that year trying to increase the value of your time, on the other hand, the upper bound on your performance is both a lot higher (because you can then make more money every year for the next three decades), and a lot more fuzzy. It’s a lot more fuzzy because, while everyone has the same number of hours in a week, how effective you are at Level 2 actions depends a great deal on your intelligence, what methods you use, and lots of other stuff. Most adults in the US spend too little time on Level 2 actions.
It is, of course, possible to have the opposite problem, and I suspect that quite a few people in this general community do. People sometimes fall into the trap of noticing that Level 2 is (when applied properly) far more powerful than Level 1, and then reacting by giving blind praise to the meta level at the expense of the object level. One cultural example is the ancient Greeks, who, though they were good thinkers for their day, didn’t invent science — because science involved actually going out and looking at the world, and that was manual labor and manual labor was for slaves. The ultimate extreme of this is Aristotle, who got philosophy off to a rather bad start by starting with the assumption that the most noble knowledge would be the most useless.
The key problem there is that, because Level 2 actions are multiplicative and not additive, you need at least some Level 1 actions to multiply by. It doesn’t matter how high the value of one’s labor is, if one never actually goes out and does labor. A very large number, multiplied by zero, is still zero. If one just does Level 2 actions, without any Level 1 actions, it is a failure to do something instead of nothing. Taking only meta-level actions accomplishes less than the ten-year-old who just mowed the neighbor’s lawn for a dollar.
On a societal level, one can run into this problem even more easily, because having a large society allows one to build up more meta-levels. Consider people’s day-to-day labor as Level 1 actions — the stuff that directly improves the world. Engineering technological devices that help improve people’s productivity is then a Level 2 action. Doing science that helps with engineering is then a Level 3 action (meta-meta), and doing math that helps with science is a Level 4 action (meta-meta-meta).
In order for working on Level 4 to actually be effective, there have to be three steps chaining back to Level 1: from math to science, from science to engineering, and from engineering to productive work. If any one of these steps fails, then working on Level 4 won’t accomplish anything. And even if other people are chaining backward from Level 4 to Level 1, if you’re working on Level 4, the other people doing the backward chaining will steal a large percentage of your surplus. If, at every step, 90% of the surplus is taken by others, then by the time you get to Level 4, only 0.1% of the surplus you create will be returned to you. (US GDP is $14 trillion, so if 90% is lost at every stage, it implies that $1.4 trillion goes to engineering, $140 billion goes to science and $14 billion goes to math, which sounds about right.)
Going meta can be very powerful, for the reasons outlined above — namely, that each action taken on Level N + 1 makes it easier to do lots of things on Level N. However, in order for it to be useful within a specific project , that project has to incorporate all the steps from the meta level back down to the object level, and this becomes much more difficult with each meta level added. The Manhattan Project managed to pull it off with two meta-levels — science to engineering and engineering to real-world consequences — but the Manhattan Project had dozens of absolutely world-class people, and hundreds of very good people. Attempting too many meta levels without having the intelligence and skill to support such a project is roughly as effective as filing an IPO for a part-time beer pong table business.
What specific examples of this are there? Consider the researchers at Xerox PARC, who built much of the technology used in modern computers, only to get copied by Apple and Microsoft. Or Philo Farnsworth, who invented television, and then spent years slugging it out with RCA over who had the right to profit from his invention. Or Nikola Tesla, whose alternating current was superior to Edison’s direct current, but who died broke for a lack of ability to compete with Edison’s business acumen.
Eventually , if something is genuinely worth doing, someone somewhere will reap the rewards — but that someone need not be you. Whether you should do it anyway is a subject I’ll leave for later posts.
Combining Levels There is a very interesting way, which I highly recommend, of getting both the benefits of Level 1 and Level 2 actions. One does this by going out and doing things that one hasn’t done before.
Ordinarily, going to the grocery store is a Level 1 action. But what if you’ve never been to a grocery store before? Then, going to the store is actually both a Level 1 and a Level 2 action. By going to the grocery store, you acquire food. And you also learn lots of useful things about how grocery stores in general work, which will help you on all of your subsequent trips to the store.
The downside of this policy is that, considered purely as a Level 1 action, it might be less worthwhile to do something new than something you’re already familiar with. If you’ve never ridden a bus before, it might be faster to walk than to take the risk of getting lost. But, thanks to the world’s general policy of counting the hits and forgetting the misses, this tends not to be a very big deal. If you do it badly, it’s no biggie, you can just try again later when you’re more skilled. This is the reason why, as I explained in my first post, I’m writing this sequence on “instrumental rationality” even though I’m wholly inadequate for the task. It’s both a Level 1 and Level 2 action, and no one will much notice or care if I wind up falling flat on my face.
In the ancestral environment, the range of skills one could acquire was fairly limited. Hence, we humans evolved to employ a two-part strategy: try new things during your childhood, and then when you mature (note that “mature” means reaching age 14 or so), forget about trying new things, and concentrate only on Level 1 actions. Now that the range of possible skills is so much larger, this is terribly suboptimal — but humans have this thing about continuing to do stuff, like eating chocolate, that has long since lost its utility.
The world is an extremely complicated place, and as a general rule, no matter how much you read and learn about something, there’s always some sort of surprise when you actually go and do it; something that the authors of the stuff you read didn’t notice, or forgot to write down. In computer programming, we have the general principle of humility with regard to bugs: even if you can’t think of anything that you did wrong when writing some software, you had better go and test it before releasing it anyway, because the odds are pretty darn good that you made a mistake somewhere . The analogous principle is, never assume that you can do something (even if it seems simple) unless you’ve actually done it before, because there will quite likely be some sort of hidden surprise.
The counterpart to this, of course, is that if you do go out and do things that you haven’t done before, you’ll be able to pass over mostly-invisible barriers that other people will smack into. Suppose you try to construct a ten-step plan, where each of the ten steps is something seemingly simple, but something that you haven’t ever done before. Will it work? Probably not. Even if the probability of success on each step is 90%, the probability of the whole plan working is only 0.9^10 = 0.35. You can go over each and every one of the steps, analyze them, figure out that they’re very likely to succeed individually — and still fail, because of the unlikelihood of being able to execute all of them in the proper sequence.
On the other hand, if you have lots of experience doing something, you can understand it well enough to actually make the probability of failure arbitrarily small, not just smallish seeming. How narrow or wide your probability distribution is for X is a function of how much information you have about X. And doing X offers you the possibility of gaining arbitrarily complete information about X, not just the sort of information that one can communicate effectively in words. I think the evidence is fairly conclusive that someone who hasn’t experienced, for example, love, war, or torture can’t really have complete information about it, because there are parts of the brain which are only wired to receive information directly from the external environment. (Of course, the reverse is also true, which is why people who don’t read a lot can be very capable in their domains of expertise, but still terrible at abstract thought.)
Making the probability of failure arbitrarily small, then, allows you to construct long chains of sequential events which will work reliably. An average human can take a step forward ten thousand times in a row without falling down once, because they (and by “they” I mean the mostly subconscious systems in their brain that handle movement) understand it so well. And, once a particular chain of events has become reliable enough, you can then use it as a building block to construct new, higher-level chains of events. This is how society can accomplish extremely complicated things, like taking a company public, with any reliability at all — by having people build larger chains out of smaller building blocks that they already understand, and can execute many times without slipping up.
Not by One Skill Alone All big actions, from taking a company public, to launching the Space Shuttle, to saving mankind from existential risk, are made of innumerable smaller actions, glued together by layer upon layer of causal chains, which join them to each other and to the final goal. To do anything big well, this means that a large percentage of the smaller events must be done well, and there can’t be lots of smaller events which are done really badly (or the chains will break down).
One of the most important things one can notice about these long chains of small tasks, is that the individual tasks tend to be dissimilar . Consider how many things you have to succeed at if you want to get promoted in a big company. You have to do your work well, of course. But you also have to be able to communicate clearly. You also have to be charismatic, so your boss will like you. You also have to arrive at work on time, which involves things like driving and consistently getting out of bed in the morning. You also have to have enough fashion sense to not stick out much in your company’s dress environment. You also have to know how to avoid getting into feuds, so there won’t be anyone who really hates you, and who will block your promotion…
Because academia is so much more specialized than the rest of the world, one tends to excel in academia by being really, really, really good at one, very specific thing. This tends not to be true for the 99% of the world that is not academia — you can’t get what you want by just becoming better and better and better at the thing you’re already good at. You have to diversify, because becoming really good at any one thing will have diminishing marginal returns.
Consider these three case studies. Jim Simons, the CEO and owner of Renaissance Technologies, has a personal net worth of $6 billion, and controls tens of billions more. The American military is fairly obviously the best military in the world, and routinely smacks down its opponents with 100:1 casualty ratios. And Linus Torvalds, the creator of the Linux kernel, is one of the best programmers in the world. All of them, if they wanted to, could plausibly take over the world in reasonably short order.
Yet, notice how it’s completely implausible that any of them could take over the world just by becoming better at what they’re already good at. It’s very obvious that Jim Simons isn’t going to take over, just by making another ten billion. It’s very obvious that the American military isn’t going to take over the world if they achieve 1,000:1 casualty ratios, instead of just 100:1. It’s very obvious that Linus Torvalds isn’t going to take over the world, just by becoming a better programmer. These are things that none of us would ever anticipate happening.
What skills are the people in these examples missing? I don’t know exactly, of course, but I suspect that the biggest one is doing something instead of nothing; they aren’t trying to take over the world, even if it would actually help them achieve their goals. (Taking over the world is one of those things that turns out to be incredibly useful for a wide variety of objectives.)
Now, why would this be the case; why might specialization lose? The conventional wisdom is that specialization is one of the primary causes of economic growth. As people specialize, they can become better at one specific thing than a generalist ever could. So a team of ten people, where each person specializes in one stage of a production process, should be able to outcompete a team where every member goes through the entire production process individually.
However, specialization also has costs, which act as a drag on performance, preventing people who are too highly specialized from achieving a lot. The most general is simply the communication problem. As the size of the team increases, the parallelization difficulties also go up, until they start to outweigh the benefits of increasing specialized skill. If the chain of small events composing the big project is simple, like on an assembly line, this doesn’t happen for a very long time, because most people only have to interact with the people immediately next to them in the chain. However, if a process is highly complex and nonlinear (like launching a major website, or starting an airline, say), then each specialist has to be “plugged in” to many different points in the process, which means that the number of communication lines needed grows more like n^1.5 than like n.
The second problem is the averaging effect. Most people are, by definition, average, and so it’s a lot easier to assemble a team of average people than it is to assemble a team of very good people. And so while an organization of ten highly competent specialists may wind up beating an organization of ten highly competent generalists, this doesn’t happen very often, because teams of ten highly competent specialists are rare. What tends to end up happening is that a team of a single, competent generalist and nine average generalists beats a team of one competent specialist and nine average specialists, because the competent generalist can use his team much more effectively than the competent specialist (who, being a specialist, lacks overall project management ability).
The third problem is that the surplus of a specialist may get stolen by a generalist. A specialist, who is very good at making widgets, might produce a million dollars of value in a year. However, by virtue of being a specialist, he probably hasn’t gotten very good at the skill of “getting the largest percentage of any value I create”. Hence, someone else, skilled at stealing other people’s surpluses, might just come along and take 90% of it.
Two Kinds of Lying (This begins somewhat of a digression, but I believe it needs to be said.)
On Less Wrong, we have already covered what we called “the ethics of lying”, and the tradeoffs between the possible benefits of lying to people, and the added costs of doing so. This is all well and good; I have little of importance to add. However, I believe that this kind of discussion is not the whole story. There is a whole, different category of things, also called “lies”, to which the last discussion about the costs and benefits of lying doesn’t really apply. It is a great shame that we call them both by the same name, because how one should handle them is quite different.
The sort of lies we discussed earlier are what one might call “long-term lies”. These lies tend to be big, and to be about important things. For instance, if I’m the CFO of a public company, and I lie to the CEO about how much the company made last quarter, it’s a long-term lie, because I have to remember it later on in case he asks again, or in case someone else goes over the books. If I’m the head of a project, and I promise that the project will accomplish goals that I know are technically infeasible, it’s a long-term lie, because I’ll have to remember to make excuses when the results don’t show up. There is a tendency for long-term lies to seriously hurt people, by causing them to take suboptimal actions, and from this, we generally conclude that such lies are immoral, or at least morally questionable. If I lie to my girlfriend/boyfriend about having cheated on them, even if the lie causes a net increase in happiness, a lot of people would still condemn it as wrong. Long-term lies also have the practical cost of having to keep track of the lie. Every long-term lie you tell has to be remembered and kept track of, lest you get caught in a contradiction. This phenomenon is fairly well-known.
There is, however, another kind of lies altogether, which I’ll call “short-term lies”. (I will not use the term “white lies”, since this is usually applied to long-term lies whose morality is being questioned.) Short-term lies are ones where, thanks to the marketing arms race, whoever you tell the lie to will soon forget about it. If you lie about catching the 9:00 PM bus instead of the 9:10 PM bus, no one is going to remember, or care. You don’t have to remember this kind of lie to avoid contradicting it, because if you do contradict it, no one will notice, or you can write off the contradiction as a memory lapse (yours or theirs).
If you are very concerned about your own level of epistemic rationality, do be warned that using short-term lies may tend to degrade it. With long-term lies, because they’re important and you have to remember them, you probably won’t be tempted to confuse the truth with the lie. With short-term lies, on the other hand, your brain may start to subtly blend together the lie with the truth, until you remember the lie as being true. If you use short-term lies in a systematic way in one particular domain (for instance, to make yourself look more impressive or more skilled), you may wind up biasing your view of the world in that domain. So there are indeed significant trade-offs to using short-term lies, even in purely practical terms.
How to Actually Run a Conspiracy There has been much talk on Less Wrong about conspiracies. However, I strongly suspect that many people on Less Wrong have no idea how to actually run a real conspiracy. This is a great shame, as many things we would like to do, such as reducing existential risk, would be aided by the presence of a friendly conspiracy or two. Hence, I will now set out to correct this deficiency.
The key, defining characteristic of a real conspiracy is: working together on projects where keeping information secret actually matters . In the Scientific Conspiracy, if the latest paper about the properties of cyclopentanol happens to get leaked, it’s no big deal. The purpose of secrecy isn’t to actually keep the information secret (the world gets along just fine now, with scientific information accessible to everyone), it’s to take advantage of human psychological motivations like information scarcity. But if you’re working on a project to build technology which could flatten a city with the push of a button, it’s really serious if some blueprints leak to the public; it could cause an immense amount of damage.
The goal of a real conspiracy, then, is not to seem secretive, but to remain completely hidden . Outsiders, when they consider you and your group, shouldn’t be thinking “oh, it’s a conspiracy, they’re not allowed to tell me stuff”; they shouldn’t be thinking anything at all . You and your group should be as ignorable as that little speck of dust on your screen that you probably didn’t notice until you read these words. The best case scenario is when, not only does no one think of your group as a conspiracy, but no one would even consider the possibility , because it would seem so preposterously absurd. If there were an article on CNN tomorrow about how the Russians were scheming to take control of the American water supply, most people would probably believe it. But if there were an article on CNN tomorrow about how the Hungarian clowns were scheming to take over the American ping-pong ball factories, everyone would think that someone had hacked CNN’s website. You could probably run a story like that in different news outlets, every week for a decade, and people still wouldn’t really believe it.
The most important ally, in the quest for ignorability, is the marketing death spiral that has engulfed Western civilization. Thanks to the increased power of marketing, being ignored is the default state; it takes special effort to get noticed. Your goal should be to remain in this default state. If people are trying to pry in and discover your secrets, it doesn’t much matter what systems you’ve built to keep them out, because at that point, you’ve already lost. How can you remain ignored? It’s difficult and complicated, but here are some ideas: Don’t go around talking about conspiracies or secrets, of any sort at all. I am, of course, violating the crap out of this rule by writing this blog post, but heck, someone has to write it. Don’t mention the name of your conspiracy, what it’s about, what its goals are, or anything about it.
If you do mention conspiracies, or if someone asks you about your secrets or about your involvement in conspiracies, deflect them by talking about something trivial. Talk about the surprise birthday party you’re planning. Talk about how you and your co-worker dislike the guy three doors down. Talk about who you think might be hooking up with who. Act like it’s a big deal, and say that you’ll kill the guy if he tells anyone. If you do this enough, people who know you won’t even bother to ask about it if a hint of real conspiracy comes up. They’ll already have dismissed it as unimportant, just on priors.
If something related to your conspiracy comes up in conversation, don’t say much. Don’t act nervous or clam up, just be quiet and listen to everyone else talk. If you do say anything, make it short, bland, generic, inoffensive. Other people (unless they’re also involved in the conspiracy!) probably won’t know much about the subject, and will expect you not to know much either. For instance, if you’re running a conspiracy to engineer a beneficial superbacterium, and your friend at a bar mentions the genetic engineering debate in Congress, don’t launch into a discussion of genetics, or correct him if he makes mistakes, or try to educate him about proper terminology. The average person doesn’t have the foggiest clue what reverse transcriptase is or what the 3’ and 5’ ends of a DNA strand are, and they’ll think it’s odd if you do. (Unless you really do have a day job as a biology professor, but most DIYBio people don’t.)
If you’re anonymously doing things for the conspiracy, don’t mention obvious identifying info, but also don’t mention stuff like the name of the town you’re in, the names of people you know, groups you’re involved with, or unusual hobbies you have. If you do mention such things, also mix in false information that sounds similar-ish (so as not to raise alarm bells or create story incongruities). If you’re in Mountain View, say that you’re in Redwood City; if you work for Boeing, say that you work for Northrop Grumman; if you just went on vacation in the Bahamas, say that you went to Jamaica. Constructing a believable identity completely from scratch is a lot harder than just starting with an existing story, and changing details.
Buy stuff for the conspiracy using your own credit/debit cards. If you don’t have one, you can buy Visa/Mastercard gift cards with cash that can be used online. If you’re engineering a bacterium, and you order a few thousand dollars worth of DIYBio kits, it’s going to look awfully strange to anyone who sees your credit card statement. Use your own card, not your wife’s/husband’s card, your father’s/mother’s card, your corporate card, or your friends’ card (unless your friend is in on the conspiracy).
Try to avoid creating long, extended chains of secrets and meta-secrets and meta-meta-secrets with other conspirators (either people in your own conspiracy, or someone else’s). You really shouldn’t have to keep track of what person A knows about what person B knows about what person C knows about person D. This is a lot of what conspiracy games like Illuminati consist of, because it can be fun. But it’s also a tremendous waste of time and energy, and prone to failure if a single link breaks. You should never be in a position to say I Know You Know I Know. If someone else is conspiracy-minded enough to successfully keep track of what things they should keep secret from person A and what things they should keep secret from person B about person A and what things they should let person B know about person A while also keeping secret the fact that they know what person B knows about person A (most people aren’t), they’re probably trustworthy anyway.
If you have an anonymous website or blog, cloak your domain name! This is very easy to do. When you buy a domain name (either from your web hosting company or from a third party), there will almost always be a little checkbox which says “cloak this domain name” or “keep my information private”, or something to that effect. Sometimes it’s free, sometimes it costs a dollar or four, but it’s worth it. If you don’t check that box, anyone will be able to find your real name and address by doing a simple WHOIS search. If you use a subdomain, like conspiracyblog.myfriendswebsite.com, be sure that the main domain is cloaked, so it can’t be traced to your friend or organization and then to you.
And finally, never, ever, ever act like any supposed “conspirators” you see in the movies, on TV, or in other mass media. For instance, spies in real life, like the Russian spies who were found in the Lifeboat Foundation in July, look and act absolutely nothing like James Bond. Real spies look like ordinary people, dress in ordinary clothing, live in ordinary houses, own ordinary stuff, and work in ordinary jobs; that’s why they’re inconspicuous. But stuff in the media has to be easily marketable, and anything that’s inconspicuous is, ipso facto, boring, and therefore difficult to market. Hence, when the mass media tries to depict inconspicuousness, they always do an absolutely terrible job.
Science and Communism At this point, I’m going to stop the main sequence of posts for a bit and enter a digression, to consider a very interesting question: Suppose you write a good computer program, and you have the option of keeping it to yourself and your friends, or giving it away. Which one do you choose?
Well, suppose you’re the supreme dictator of an Eastern European country. You look out the window at your country, and you see that some people are very wealthy and live in huge mansions, while other people are poor and starve in the street. What do you do? Money has decreasing marginal utility, so one obvious solution is to take all the rich people’s money and give it to all the poor people. That will make society as a whole much better off — won’t it?
This logic works fine, if you assume that wealth is a zero-sum game. Of course, everyone here knows the problem with this reasoning: By taking all the rich people’s money, you mess with their incentives, and this causes less total wealth to be created. The poor get a larger slice, but it’s a larger slice of a much smaller pie. (Wealth was largely a zero-sum game for most of history, which is largely why a lot of people still have trouble with this idea.)
Much the same logic applies to giving away computer programs or research findings, although the problems are more subtle. When discussing communist economics, there is a very obvious, first-order incentive effect: if you get paid the same for being productive and not being productive, there’s no reason why you should be productive, and so most people don’t do any work. With computer programming (and science), the effects are harder to notice, because they’re second-order. Science does a good job of solving the first-order incentive problem by using a peer reputation system — if you’re lazy, people won’t think well of you, and science is small enough and isolated enough for this to be a strong motivator (ditto for computer hacking).
However, there is still a huge second-order problem, because this peer reputation system is only effective within science. If scientists give away all of their output, regardless of what the rest of the world does, then everyone else has no reason to care what scientists think. And a lot of people don’t want to go into fields where no one outside their field cares what they think, so they get MBAs or go into corporate law, instead of becoming scientists. Giving your output away can work on the first level (people within science are still motivated to be productive), but largely fails on the second level (people outside of science have no incentives to enter the field in the first place).
There is a strong community norm in the science and free software movements in favor of giving something away, instead of keeping it for yourself, because it’s seen as being the noble and virtuous thing to do. Individually, giving away a piece of research or a computer program does benefit society. However, over the long term, the community norm harms society, by changing people’s incentives in a way that keeps the best and most productive from entering science. A good majority of the grad students, and therefore of the scientific labor, in the United States exists purely because of a quirk in the American immigration system that makes it easier for grad students to get visas than experienced workers.
“‘A scientist!’ There was genuine indignation on his face, and his voice had grown stronger and sharper. ‘You could be the best of all my students! The greatest fighting wizard to come out of Hogwarts in five decades! I cannot picture you wasting your days in a white lab coat doing pointless things to rats!’” says Professor Quirrell in Harry Potter and the Methods of Rationality . Professor Quirrell is right, not with regards to Harry’s plans for creating a science of magic (which would be immensely valuable in the Methods-verse), but with regards to the utility of science as a career if one has a fairly typical degree of self-interest.