At a restaurant, you pull out your phone to check email. Without even thinking about it, you tap in a PIN to unlock your phone. Your back’s to the wall and nobody can see what you’re typing, so there’s no reason to worry that somebody could intercept your passcode.

Except, sadly, there is. Researchers at Syracuse University have demonstrated that hackers can guess PINs by analyzing video of people tapping on their smartphone screens -- even when the screen itself isn’t visible. Software used to analyze such video relies on “spatio-temporal dynamics” to gauge the distance from the fingers to the phone’s screen, and then approximate which characters the fingers tap on a keypad. “It’s like lip reading,” says Vir Phoha, an engineering and computer science professor at Syracuse and co-author of a paper on the technology. “Based on hand movement and the known geometry of the phone, we can see which keys are pressed.”

There don’t appear to be any known instances of hackers stealing PINs this way, but technologists think it’s only a matter of time. “We believe that it is very likely to be adopted by adversaries who seek to stealthily steal sensitive private information,” Phoha and three others Syracuse researchers wrote in their paper, published last year by the Association for Computing Machinery. The technology is fairly simple for anybody familiar with programming, and the exploding use of smartphones provides many millions of targets.

On top of that, the increased use of phones for banking and managing other financial accounts makes PINs a lucrative prize for hackers. And the same video-analysis technology can be used to infer PINs punched into ATMs, smart locks on the front doors of homes, garage door openers and other gizmos requiring similar codes.

Used by the good guys, too



Publicizing such black-hat technology through articles such as this one can obviously tip fraudsters to possible new methods of ripping people off. Security experts and some of their criminal foes already know about it, however, since such research has been published in technical journals. So Yahoo Finance decided it’s appropriate to alert consumers to this new form of hacking. National security and law enforcement agencies could also use it to keep track of bad guys; DARPA, the Pentagon’s technology skunk works, for instance, partly funded the Syracuse research.





The Syracuse experiments involved 50 volunteers typing PINs into HTC One smartphones, in a variety of different settings and postures. For each volunteer, researchers shot four different videos. The recordings were made using two off-the-shelf devices: a Google Nexus 5 smartphone camera and a Sony camcorder. All the videos were shot from the side or back of the phone, from 12 to 15 feet away. None of the videos captured the phone screen or explicitly showed what users were typing.

Software filled in the gaps, however, with a combination of image analysis and motion tracking algorithms being remarkably effective at “guessing” the PINs users typed in. On the first guess, software determined the correct password between 40% and 62% of the time, depending on the quality of the video and the zoom ratio. The highest-quality video produced an 82% accuracy rate after 5 guesses and 94% accuracy after 10 guesses. Using more than one video for each phone raises the odds of success even further.

“We can do it in about 30 minutes once we capture the video,” says Phoha. “We have almost 100% accuracy.” This graph lays out the results of computer guesswork for video shot using the Nexus smartphone and the Sony camcorder at zoom levels of 2x, 4x and 6x:

View photos Sources: Diksha Shukla, Rajesh Kumar, Abdul Serwadda and Vir V. Phoha of Syracuse University; Association for Computing Machinery More

Story continues