IMO scores as a high-ceiling test of intellectual ability
A draft paper, sent to Carl Shulman in April 2010, exploring whether performance at the International Mathematical Olympiad predicts later academic productivity. The motivation is a long-running argument in psychometrics: standard IQ tests are normed on small samples and so measure poorly above the 99th percentile, which leaves popular writers like Malcolm Gladwell free to claim that very high intelligence doesn’t correlate with achievement. The IMO, a contest in which entry from a country like the US already requires roughly one-in-a-million mathematical ability, offers an unusually high-ceiling alternative. Alyssa scrapes the IMO’s public results, cross-references contestants on Google Scholar, and finds that scores predict citation counts even within an already extraordinarily selected pool.
It is a well-known result in psychometrics that a large number of human cognitive abilities all correlate positively with a single central parameter, commonly called the “general intelligence factor” or “Spearman’s g” (Carroll 1993). Researchers have found that general intelligence is one of the most important predictors of things such as job performance (Schmidt 1998) and annual income (Murray 1998), while negatively correlating with undesirable characteristics such as violent crime (McDaniel 2006).
The most common means of measuring general intelligence is through an IQ test, such as the Wechsler Adult Intelligence Scale (WAIS). However, these tests are standardized on relatively small numbers of people; the WAIS-IV was standardized on a sample of 2,200 individuals (Glass 2009). Hence, there is a strong possibility of inaccurate measurement at both the low and high end. A sample size of 2,200 individuals implies that there will only be, on average, six people more than 3 SD from the mean (IQ > 145 or IQ < 55, given SD = 15) — far from a statistically adequate data source. Therefore, even if the nominal ceiling of the test is higher than 145, we should expect such tests to have difficulty measuring the general intelligence of the top-scoring 0.1% of the population.
This prediction appears to be borne out in practice. Famously, Marilyn vos Savant and Christopher Langan have both been identified as “the smartest person in America,” on the basis of very high IQ scores (McFarlan 1988, Sager 1999). However, as of 2010, neither has accomplished very much by traditional academic measures of intellectual productivity; neither has, so far as we are aware, published a peer-reviewed paper indexed by Google Scholar. Meanwhile, the famous Nobel Laureate physicist Richard Feynman was measured as having an IQ of only 125 (Gleick 1992). More generally, Terman’s Genetic Studies of Genius, a long-term longitudinal study involving 1,528 individuals tested for extremely high IQs in childhood, showed that, although the children in question had better life outcomes than average, few became very wealthy or successful (Terman 1926).
This data has been interpreted by many, including Malcolm Gladwell, to mean that high IQ — above 130 or thereabouts — does not correlate significantly with wealth, fame, professional success, or other positive life outcomes (Gladwell 2008). Although this is not necessarily inaccurate, especially in fields far removed from the hard sciences (such as politics and entertainment), it would be very difficult, given only the data from IQ tests, to distinguish this from the alternative hypothesis that very high g is important but difficult to measure.
Indeed, a significant amount of work has been done that supports the latter hypothesis, such as Lubinski’s studies of children who scored exceptionally well on the SAT at age 12 or 13, a test with a very high ceiling (Lubinski 2001). (Frey 2004 found an SAT/g correlation of 0.82.) These studies show that such children “pursued doctoral degrees at rates over 50 times base-rate expectations, with several participants having created noteworthy literary, scientific, or technical products by their early 20s.” One participant, Terence Tao — one of only two people known to have scored 700 or more on the SAT-M at age 8 (Gross 1986) — went on to win a Fields Medal and a MacArthur Foundation “genius grant.”
To help further flesh out the relationship between very high g and very high achievement later in life, we looked at data from the International Mathematical Olympiad. The IMO is considered one of the premier contests for high-school-age students of mathematics. Only six students from one country are allowed to compete in any given year; hence, in a large country such as the US, China, or Russia, where there are millions of high school students eligible to compete, merely entering the contest may require a demonstration of mathematical ability at the 1:1,000,000 level. The IMO is, therefore, a test with an extremely high ceiling. Although no studies have been done on its correlation with g, the test is similar in character to tests that have been studied: it is timed, requires the solution of standardized, pre-determined problems (is not open-ended), allows no use of books or other references, allows no collaboration, and is scored relatively objectively, with individual solution steps being either clearly right or clearly wrong.
The IMO website provides data on the scores of individual participants in the Olympiad, including name (first and last), country of residence, the score on each individual question, and medal won (if any). We performed analysis on this data for the IMOs between the years 1981 and 1995; before 1981 the data becomes increasingly spotty, while after 1995 even very capable performers might be insufficiently advanced in their careers for their abilities to be fairly judged (the average contestant in 1996 would only be 31 at the time the data was collected). We used Google Scholar to search for papers authored by people who were listed as participating in the IMO, and then counted both total number of papers and total number of citations for each individual. The total number of individuals identified across all years was 3,871.
There are difficulties with this method. Because the IMO data only gives name and country of residence at the time of the exam, there is no reliable way of determining that the persons who authored the papers on Google Scholar are the same persons who participated in the IMO ten or twenty years previously. Hence, there are a significant number of false positives, particularly for unusually common names that might be shared by many different authors.
The distribution of the resulting data is also extremely far from normal. For example, in the data from participants in the 1991 IMO, the mean number of citations among those at the 90th percentile or above was 2,885, even though the overall median was 1, and the overall mean was 345. To get an accurate picture of the data, we believe that it is necessary to use a number of different statistical measures.
One such measure is the percentage of contestants who are recorded as having exactly zero citations. This measure should be somewhat resistant to false positives; while a large number of citations might merely be indicative of having a common name, having zero citations is more definitive evidence of a lack of academic publication. The baseline rate among the 3,871 participants was 47.1%. Among the 299 gold medalists, 98 (32.7%) are recorded as having no citations at all. However, of the 299 bottom-scoring participants, 181 (60.5%) — nearly twice as many — have zero citations. The over-performance of the first group and the under-performance of the second are statistically significant at p < 10 -6 and p < 10 -5 , respectively.
Another measure is the logarithm of the total number of papers (assigning a zero value in the case of zero papers). Using a logarithmic measure helps to damp out the effects of the long tail, whereby a single person who co-authors a hundred papers can swamp any effect created by five who write two and five more who write eight. It also damps false positives, by removing the overwhelming bias created by people with exceptionally common names, who are sometimes recorded as having written more than a thousand papers. Among the total group of participants, the average for log(papers written) was 1.99, corresponding to 7.32 papers. Among the 299 gold medalists, this number was 2.75, while among the 299 lowest-scoring participants it was 1.73 — corresponding to 15.64 and 5.64 papers, respectively. We calculate an overall correlation of r = 0.11 between log(papers written) and IMO percentile rank, with p < 10 -6 .
A similar measure is the logarithm of the total number of citations (again assigning a zero value in the case of zero citations). For the overall group, the average was 2.52, corresponding to 12.43 citations. For the gold medalists, it was 3.62 (37.34 citations); among the lowest-scoring group, 2.01 (7.46 citations). We calculate an overall correlation of r = 0.15 between log(citations) and IMO percentile rank, with p < 10 -6 .
In conclusion, the data clearly shows that performance on the International Mathematical Olympiad is a significant predictor of future academic output. This holds even though the vast majority of IMO participants have already been through multiple rounds of very difficult mathematical exams. Hence, the data do not support the hypothesis of a “maxing out” effect on such tests, whereby they are supposed to predict performance only up to a certain percentile and not significantly thereafter.
References Carroll, John B.
Human Cognitive Abilities: A Survey of Factor-Analytic Studies . Cambridge University Press, 1993.
Frey, M. C., & Detterman, D. K. (2004). “Scholastic Assessment or g? The relationship between the Scholastic Assessment Test and general cognitive ability.”
Psychological Science 15(6): 373–8.
Gladwell, Malcolm.
Outliers: The Story of Success . Little, Brown, 2008.
Glass, L. A., Ryan, J. J., Charter, R. A., & Bartels, J. M. (2009). “Discrepancy Score Reliabilities in the WISC-IV Standardization Sample.”
Journal of Psychoeducational Assessment 27: 138–144.
Gleick, James.
Genius: The Life and Science of Richard Feynman . Pantheon, 1992.
Gross, M. (1986). “Radical acceleration in Australia: Terence Tao.”
Gifted Child Today 9: 2–11.
International Mathematical Olympiad Individual Results . IMO Advisory Board.
Lubinski, D., Webb, R. M., Morelock, M. J., & Benbow, C. P. (2001).
Journal of Applied Psychology 86(4): 718–729.
McDaniel, M. A. (2006). “Estimating state IQ: Measurement challenges and preliminary correlates.”
Intelligence 34(6).
McFarlan, Donald.
1989 Guinness Book of World Records . Bantam, 1988.
Murray, Charles A.
Income Inequality and IQ . AEI, 1998.
Sager, Mike. “The Smartest Man in America.”
Esquire , November 1999.
Schmidt, F. L., & Hunter, J. E. (1998). “The validity and utility of selection methods in personnel psychology.”
Psychological Bulletin 124: 262.
Terman, L. M. (1926).
Mental and Physical Traits of a Thousand Gifted Children . Genetic Studies of Genius, Vol. 1 (2nd ed.). Stanford University Press.