AI results on toy problems
There’s high uncertainty here, but one reason I’ve been more skeptical of very aggressive AI timelines is that I don’t think we’ve learned a lesson from the last two AI winters: apparently strong performance on “toy problems” often doesn’t mean much. For example, one of the big AI successes of the 1960s was called SHRDLU. It had a simple physics model of a “blocks world”, and could have an English-language dialogue with the user about seeing and manipulating the blocks: Person: Stack up two pyramids.
Computer: I CAN’T.
Person: Will you please stack up both of the red blocks and either a green cube or a pyramid?
Computer: OK.
Person: Which cube is sitting on the table?
Computer: THE LARGE GREEN ONE WHICH SUPPORTS THE RED PYRAMID.
Person: Is there a large block behind a pyramid?
Computer: YES, THREE OF THEM: A LARGE RED ONE, A LARGE GREEN CUBE, AND THE BLUE ONE.
Person: Put a small one onto the green cube which supports a pyramid.
Computer: OK.
Person: Put the littlest pyramid on top of it.
Computer: OK.
Person: Does the shortest thing the tallest pyramid’s support supports support anything green?
Computer: YES, THE GREEN PYRAMID.
Person: What color is it?
Computer: BY “IT”, I ASSUME YOU MEAN THE SHORTEST THING THE TALLEST PYRAMID’S SUPPORT SUPPORTS.
Computer: RED ( Wikipedia: SHRDLU ) It seemed natural to people that this “simple conversation” was a solved problem, so it would quickly be extended to “complex conversation” or “general conversation”. Of course, that didn’t happen. Fifty years later, we have computers that are a million times as powerful, but nothing that can have even a simple conversation reliably: Alyssa: What do you do?
Cleverbot: I am a student. You?
Alyssa: Software engineer.
Cleverbot: Ohh.
Alyssa: Where do you live?
Cleverbot: In the US.
Alyssa: Where in the US?
Cleverbot: California.
Alyssa: Where in California?
Cleverbot: Bay area.
Alyssa: Oh me too.
Cleverbot: Do you know where the house of Pain is?
(me just now, on cleverbot.com ) I have no “proof” or hard data backing this, other than my intuition, but my sense is that many modern AI papers fall into one of two buckets: The paper solves a “toy problem”, which itself has no or negligible value, but it implies (usually without arguing very strongly) that it can be readily extended to solve an important problem.
The paper solves a problem with a new technique, and gets the same or worse results than existing techniques, but implies that this new technique can ultimately achieve greater performance down the road.
Of course, there’s nothing wrong with toy problems, they’re often essential intermediate steps. But my intuition (not being an academic myself) is that the follow-up — the extension from toy problem to real problem, or from exploring a new method to superior performance — most often never happens, in large part because the incentives don’t favor pushing prototypes through to production. (I still remember, eg., a 2009 SIGGRAPH paper I dug up, with a new algorithm for frame interpolation that handily beat all competitors; the code was never released and everyone’s still using the older crap, eight years later.) It reminds me of McDermott’s old paper, Artificial Intelligence Meets Natural Stupidity (1981): “Long after he has his Ph.D. or his tenure, inquiring students will be put off by the document he has left behind. He seems to have solved everything already, so the report says, yet there is no tangible evidence of it besides the report itself. No one really wants to take up the problem again, even though the original research is essentially a partial success or even a failure! If a student decides sidetracking is a good idea, and wants to study it, people will assume he is ‘merely implementing’ an already fully designed program. (No Ph.D. for that!) He would be willing or even eager to start from a smoothly running Version II and write Version III, incorporating a new theoretical idea like Syntactic Network Data Bases, but there is no Version II. Even a Version I would help, but it isn’t really working very well and its author has no desire for it to be publicized.” — McDermott (1981)