On the Interestingness of Text
Notes sent to Scott Roy, a Google engineer Alyssa had spoken with the previous week while interviewing for a position on Ray Kurzweil’s conversational-AI team. Roy had mentioned the project’s need for an “interestingness” function — a way of measuring whether a piece of text was worth surfacing or saying back to a user. Alyssa wrote up some half-formed ideas about how such a function might also be useful for general-purpose search.
For a standard Google search, one might be trying to find one page in particular, and the search ranking is then ideally like a probabilistic model — you want result #1 to be most likely to be “the target”, #2 to be second most likely, and so on. Pages might be very similar, because if pages are embedded in smooth high-dimensional “text space” and you’re searching for one particular optimum, the next-most-likely points will be tightly clustered around the first point (and the ease of copy-pasting + CPM ad money makes a lot of neighborhoods pretty dense).
But if I’m searching for information on e.g. China’s Gang of Four, I’m imagining you could treat that like a conversational model. Once you find one page, you want to see other pages of maximum relative “interestingness” about the same topic, but with different facts, and ideally facts which are of maximum surprise (they update my internal model a lot) and maximum importance (those parts of my model have steep utility gradients). In the Gang of Four example, you might have 9/10 results be the same basic info re-phrased in different ways, and then one “great” page with interesting new content about the exact timeline of October 1976.
So one thing I’m imagining you could do is, run a pre-computation for each page with reasonably high PageRank, where you go through the Google archives and find the hundred pages that are maximally “interesting” with respect to this page. You then put a link saying e.g. “See more like this” next to the external link, and then you display the “interesting” results, which then link to their own “most interesting” results, and so on. That would save you from, when you find one page that’s “unusually good”, having to try a bunch of permutations of the query to stumble on similar pages.
That would have… curious effects, I’m guessing, if it got to be popular. With current social media, if you have a graph of users which are statistically homogeneous and the graph is reasonably big, then most stories are either “viral” or “not viral”, and so you mostly get the same few “viral” stories that everyone sees, and then CNN reports on it because everyone is talking about it. But what would be the algorithmically “most interesting” cluster around cnn.com? A naive answer would be msnbc.com, reuters.com, etc., but those mostly have boringly similar content. A simple NLP algorithm might look at words like “plane crash” which are of high importance, and then find sites which talk about “plane crash” but include new words like “hijacking”. A more complex algorithm which parses sentence structure could then do things like take “a plane crashed into the Pacific Ocean today” and select “a plane crashed into the Atlantic Ocean today”, since the difference between the sentence meaning is “interesting” even though the words are very close. That would incentivize news content which is both relevant and actually new (my friend Louie set up his site with exactly this goal a few months ago, if you’re interested), instead of the “old model” of lots of local distributors copying a wire service.
Or, for e.g. educational materials, if something is commonly taught in high school, it’s annoying to dig up info about it because websites catering to students will flood results with the same information rephrased slightly differently. (E.g. quotes from A Tale of Two Cities usually bring up Cliff Notes as the top results.) I’m a noob here, but maybe you could detect and exclude very similar things by looking at word count — weighted by inverse of word frequency, to exclude things like “the”, and you’d also need a “matrix of synonym-ness” or something like that, maybe built from MTurk data — and then comparing the distributions. (Though this would work less well for generating a conversational response, since there you have strict rules about “making sense”, e.g. “the plane crash was hijacked” is nonsense but very similar to “the plane was hijacked”. Although I wonder how far you could get by chopping up Wikipedia etc. into individual sentences and selecting the “most interesting”, since then at least you know it’s not a nonsense reply…)
And then, if you had a large group of facts in Knowledge Graph, it might be easier to synthesize text into “interesting” articles than it would be to process the articles into facts going the other direction. A failure of parsing can be unrecoverable — “the yellow is sun” — while it seems naively like synthesized text can be “robot-like” but still understandable with less effort (“The Sun is yellow. The Sun is a star. The Sun has a temperature of 5,800 Kelvin…”)
Don’t know if any of this is practical to implement, but I’ve found it fun to think about.