My idea was to combine computer vision with augmented reality to create a simple, streamlined UI that wouldn’t be possible without it.

Once I decided this, I narrowed my list down to a few concepts that met all of my criteria and ultimately landed on building a crossword puzzle solver. After exploring that for a few days I determined that it wasn’t going to be feasible with the tools I had available ( Vision 's image segmentation API wasn’t up to the task) and switched to building a Sudoku solver.

How adding computer vision to the equation changes things

Simpsons did it.

~ A lot of people

Most feedback has been positive. But the most common negative reaction I’ve gotten has been something along the lines of “Google Goggles has been doing this since 2011.” And yes, sudoku solvers have been available for a long time. The sudoku solver itself is not the cool part. Of the ~1 month of development time, writing the code that actually solves the puzzle took only an hour or two.

Technical people tend to understand why the app is cool. But it boils down to this: Magic Sudoku demonstrates a new model for human-computer interaction; computer vision is the input device and augmented reality is the output device.

Sidenote: changing or adding new “input/output” pairing combinations often provides new and better ways of doing things. Examples: Self Driving Cars (vision/motors), IOT (sensors/API), Google Translate (text/text), Instruments (touch/audio), Shazam (audio/text), Snapchat filters (image/image), Amazon Echo (voice/<many>). What other input/output pairings are there that haven’t yet been explored?

Several people have compared the app to “Terminator vision” and I think this is a good look at what’s possible when you combine CV+AR.

The Terminator doesn’t have to look at something, take a snapshot, feed it into a specific app to process it, and then look at the results. He simply looks at something and it transforms into a more useful state.

Luke Wroblewski describes this mode of interaction perfectly in his blog about how augmented reality headsets should work.