Measures of AGI progress
Hi everyone. Thinking about AGI timelines and so on, it seems like a major part of the problem with making estimates is that we still don't (as far as I know) have even a qualitative benchmark of AI power that goes up to/beyond the human level. We can track the progress of specific benchmarks (http://intelligence.org/files/AlgorithmicProgress.pdf), both for software and hardware, but it seems very hard to say that these can give us good AGI time estimates, or even to be sure that they represent progress in AGI at all. Eg. for something like a fast SAT solver, it seems plausible to me (not knowing the details of the algorithms) that the knowledge needed to do better is all domain-specific, and need not show any advancement towards general AI.
My current best guess for a semi-quantitative measure of AI software progress is the size or complexity of the programs that an AI can generate. A human programmer, sitting at their desk, can write several hundred lines of high-level-language code a day. However, the vast majority of humans that are not programmers also 'write code', in the sense of learning to manipulate machines or other physical objects. A human programmer, wanting to make a robot perform various tasks, would have to write a program to apply appropriate voltages to the motors; likewise, it seems like you can think of learning to perform various tasks as 'writing code', in a 'language' where each instruction is a muscle manipulation. http://www.udel.edu/educ/gottfredson/reprints/1997whygmatters.pdf is a good summary of training tasks, the intra-species variation in how quickly humans can complete them, and how this correlates well with intelligence tests; squinting and over-generalizing some, the intra-species variation seems to be roughly one order of magnitude. I haven't examined the data in detail, but my guess is that there is a fairly smooth continuum of 'lines of code written' over many orders of magnitude from top humans, through other mammals, down to worms and other simple organisms which just have genetically pre-programmed responses to certain stimuli (like, say, a non-electronic car engine might).