Thoughts on standardized testing
Various people have written about the proposals to stop using the SAT and ACT for American college admissions. Of course, there are some pretty concerning trends happening in the US right now, but I think some of the responses to this particular issue have been overblown. More importantly, they might also be a distraction from good questions that a person or small group could actually try to answer. Diving in: The classic reason for using large-scale standardized testing is the case of someone like Ramanujan : he was poor, socially scorned, foreign, and had no wealth or resume, but was extremely talented in a particular field. An impartial test, whether it’s the SAT or an older system like the Chinese imperial examinations , should be able to identify people like that, regardless of their wealth or background. To quote Thomas Ptacek: “Imagine your candidate is a ‘natural’ who has never done the work, but has a preternatural aptitude for it. Could you hire that person, or would you miss them because they have the wrong kind of Github profile?” ( Thomas Ptacek, “The Hiring Post” ) So when considering how important a particular test is, I think it’s good to ask, “if Ramanujan existed today, how useful would this test be to him?”. In the case of the SAT and ACT, it seems like the real answer is “not very useful at all”, for a number of reasons: A lot of colleges use these tests mainly as a first-pass filter; that is, they screen out those who score badly, but don’t give much extra credit to someone who does very well. Once someone has reached the top decile of scores, the benefit to doing better may be quite small, or even negative ( Steve Sachs on gaming the system ) — colleges don’t want to admit people with lots of other opportunities, who will then turn down their acceptance.
The ceiling of these tests is not very high; that is, they can’t reliably distinguish 99th percentile scores from 99.99th percentile scores. To quote the Harvard website: “The large majority of the 40,000+ applicants to Harvard College are academically qualified, requiring the College to consider more than grades and test scores. In a recent admissions cycle (in which there are fewer than 2,000 available slots): more than 8,000 domestic applicants had perfect GPAs; over 3,400 applicants had perfect SAT math scores; and over 2,700 applicants had perfect SAT verbal scores.” ( Harvard admissions key points ) Hence, a modern-day Ramanujan, with the best possible math scores but no other credentials, would probably not get in. That link notably discusses other, ah, interesting things… to quote Paul Graham on college admissions: “If getting into college were merely a matter of having the quality of one’s mind measured by admissions officers the way scientists measure the mass of an object, we could tell teenage kids ‘learn a lot’ and leave it at that. You can tell how bad college admissions are, as a test, from how unlike high school that sounds. In practice, the freakishly specific nature of the stuff ambitious kids have to do in high school is directly proportionate to the hackability of college admissions. The classes you don’t care about that are mostly memorization, the random ‘extracurricular activities’ you have to participate in to show you’re ‘well-rounded,’ the standardized tests as artificial as chess, the ‘essay’ you have to write that’s presumably meant to hit some very specific target, but you’re not told what.
As well as being bad in what it does to kids, this test is also bad in the sense of being very hackable. So hackable that whole industries have grown up to hack it. This is the explicit purpose of test-prep companies and admissions counsellors, but it’s also a significant part of the function of private schools.
Why is this particular test so hackable? I think because of what it’s measuring. Although the popular story is that the way to get into a good college is to be really smart, admissions officers at elite colleges neither are, nor claim to be, looking only for that. What are they looking for? They’re looking for people who are not simply smart, but admirable in some more general sense. And how is this more general admirableness measured? The admissions officers feel it. In other words, they accept who they like.” ( Paul Graham, “The Lesson to Unlearn” ) Although elite American schools claim to be need-blind, in practice, going through the application and financial aid process is so onerous that most people who have less privileged backgrounds are effectively screened out by it anyway. In my case, the whole mess required both me and my parents to submit hundreds of pages of documentation every year, both on academics and on every aspect of our financial and personal lives. Needless to say, this doesn’t seem practical for most Indian villagers, or even many Americans in poor neighborhoods.
If a modern-day genius managed to navigate this anyway, how much is really being offered to them? Certainly, many fields like investment banking, management consulting, and corporate law are obsessed with educational credentials and rankings. However, my impression is that the reputation of these fields (especially among older generations) far exceeds their actual ability to make life nicer, either for you or for other people. People do these jobs for status, but, without carefully managing one’s life, that simply leads to an unending arms race. People do these jobs for money, but give little thought to what, exactly, they would want to spend it on. People do these jobs for power, but often have no clear idea of what they would want to change, or how. They mostly don’t do them to help others, and (unsurprisingly) others are mostly not helped. And, of course, there are many serious disadvantages. To quote Zvi Mowshowitz: “‘Success,’ in context, does not mean happiness. It does not make you healthy. It does not improve your reproductive fitness. It does not reflect or spread the values that you (one would hope) had when you stared down that road.
It gives you money. But in terms of actual meaningful personal consumption, you can’t really do much with it beyond status competitions. If you had plans to do something good with the money, by the time the day arrives, it is highly unlikely you’ll do it. You have changed yourself to succeed on your journey.
Even after you ‘succeed’ you probably keep putting tons of hours into the job in ways no amount of money can compensate for, once you already had basically enough.
What was the point? What are you even doing?
Note that failure is indeed much worse than success. You still paid all the sunk costs, including everything you are. You’ve invested a ton in and become very invested in local status hierarchies, and in the quest to climb them, which you have failed. Being ‘under the thumb’ of others who succeeded where you failed is deeply unpleasant — and is the most likely outcome, since the math says most who try will fail.” ( Zvi Mowshowitz, “What is Success in an Immoral Maze?”
) So, in the end, my best guess is that this will not really change very much. However, all of this still ignores what (IMO) is a much more interesting question. Over the last few months, the American government’s various failures have been the number one story almost every day, in every media outlet from the New York Times to 4chan. It gets rather repetitive. So rather than simply asking “will the US do a bad job?”, a much more fun, motivating and productive angle is to pick any government function — stopping the pandemic, defending against China, funding research, settling business disputes, validating drug safety, etc. — assume the US will do a bad job, and then ask “how might I, or people I know, realistically help do this task better?”. After thinking for an hour or two, there are many fascinating ideas, some of which I hope to write up later.
In the case of testing, the key question is “how can a person, at large scale, robustly and automatically pick out a very small fraction of the world’s population in a way that has important predictive power, ie., correlates strongly with some ability, trait, or skill that people might care about, years or decades later?”. It’s not at all an obvious question. In math, there are some well-known examples like the IMO and the Putnam contest, but the track record for almost everything else is progressively less impressive. Some quick research on the SAT reveals clear room for improvement. The SAT is nearly a century old. The major technology it uses, automated scoring via optical mark recognition (“Scantron”), was developed in the 1920s and 30s by a guy almost nobody has heard of named Benjamin Wood . Surely, with nearly a century of additional research findings, with the vast amounts of data one could collect with modern tools, and with technologies like the computer, the Internet, and machine learning, one should expect to be able to do better, and to have more interpretable results than one number for language skill and another for math. With such a tool, one would be able to put together a much better class or lab or team (or even music studio?) than what an American university does now.
But, even if better tests were developed, what if (say) the government still only hired Harvard graduates? My guess is that the best way to avoid this outcome is to produce and popularize better, competing tests. The historical record shows that the Ivies didn’t start using tests in the first place out of the goodness of their hearts; they would have been happy to keep admitting students based on race, religion, family background, and height (yes really). They started, in large part, because other universities and organizations started competing with them, and it was embarrassing to lose (see Jerome Karabel, The Chosen ). With effort, maybe they can wind up with egg on their faces again.