Stay on Top of Emerging Technology Trends Get updates impacting your industry from our GigaOm Research Community

The Paul G. Allen Foundation announced on Wednesday that it has awarded $5.7 million in grants to five projects that aim to teach machines to understand what they see and read, including anything from a photograph to a chart, a diagram to an entire textbook.

The grants, which are part of the foundation’s Allen Distinguished Investigator program, were awarded to Devi Parikh of Virginia Tech; Maneesh Agrawala of the University of California and Jeffrey Heer of the University of Washington; Sebastian Riedel of University College London; Ali Farhadi and Hannaneh Hajishirzi of the University of Washington; and Luke Zettlemoyer of the University of Washington. Their projects span multiple techniques in the fields of machine reading and reasoning, including teaching machine to understand diagrams, data visualizations, photographs and textbooks.

The grants comprise a small portion of the $79.1 million Allen said he has now invested in artificial intelligence research. Among those expenditures is the Allen Institute of Artificial Intelligence, which launched in 2013 and has a core mission quite similar in scope to the types of projects the new round of grants will fund. In some cases, it’s also working on some very complementary research, including its flagship, Project Aristo (which Etzioni discusses in the presentation embedded below).

In a blog post discussing the grants, Oren Etzioni, executive director of the Allen Institute, explained why that institute and the Allen Foundation are so interested in machines that can reason and even apply common sense:

[blockquote person=”” attribution=””]While AI is increasingly part of our everyday lives – in our phones, in our cars, in our advertising – it has the potential to do so much more. Computers process incomprehensibly large amounts of data, yet they lack innate human abilities such as common sense reasoning, and the ability to learn and apply knowledge acquired from text. A child “knows” that bears are scary and can hurt them, and react by running away. College students study increasingly difficult subject areas that build upon one other in order to solve complex problems. Today’s machines simply don’t have that kind of ability. Yet.[/blockquote]

As I suggested last month, this type of work might dovetail nicely with efforts in the deep learning and broader neural network community to build models that can explain what’s happening in pictures or classify large sections of text with little human supervision. It doesn’t take much imagination to consider the benefits of machines that can not only see what’s in front of them, but also understand that the bear hovering over the child is bad news, or that the research papers they’re reading are starting to show positive trends.

And who knows, the new common-sense-based Turing test some experts are proposing might look like child’s play in a couple years’ time.