While *my* deep learning projects have been focused on boring topics like anomalous pattern detection in unstructured textual data sets, the cool kids have been out teaching drones to fly by themselves.

Autonomous drones in a Galaxy Far Far Away

Drone autonomy has been a much discussed topic, and vendors have commercialized around all sorts of traditional navigation techniques like GPS, dead reckoning, beacon navigation, and even cooler things like bluetooth, wifi or cellular enabled “follow me” functionality.

I’ll never forget seeing Star Wars for the first time, and the devious IT-O Interrogator Droid, and thinking that someday we would all have droids hovering around after us.

When you think about drone autonomy, you have essentially the same set of challenges that self-driving cars face:

Where am I in space? Where am I supposed to go? What is around me? How far is it until I run into something?

1 and 2 are solved problems with traditional technologies. The combination of GPS, mapping and route navigation and optimization software allows a car to basically always know where it is and where it is supposed to go. The complexity of aerial mapping makes this problem slightly harder for drones, but conversely, there are fewer things to run into up there.

What is around me?

Image classification is at the heart of understanding where it’s safe to drive or fly, and can unfortunately lead to dire consequences when it’s implemented with inferior hardware or software. For a successful autonomous drone, it becomes mission critical to have a real-time video based image classification system. Being able to follow and film a skier down the slopes or the edges of a farmer’s field at 20 mph requires not only a fast real time system, but also one that can run onboard — with limited camera, battery and processing capabilities. Like every other cognitive process, deep learning is coming to drone navigation to solve this problem.

To understand the technology players in this space, I started withGitHub. Tracing code bases on Github indicate that most paths to self-flying projects lead to Joseph Redmon, a PhD student at The University of Redmond. Joseph is the originator of Darknet, an open source neural network framework written in C and CUDA. Along with his co-contributors, Joseph published You Only Look Once: Unified, Real-Time Object Detection, from which came YOLO.

Traditional neural network image detection blankets an image with many different matching attempts simultaneously, with any high scoring ones being classified. YOLO applies a single neural network to the image, and uses edge detection to weight and predict what is in the image. The result is a new kind of image classifier that is lightweight and wicked fast. I’ve seen demos using a webcam and classifying tens of images simultaneously running at 30fps. Joseph claims YOLO can run at up to 150fps.

Computers watching computers…

Collision Avoidance

The other critical innovation that autonomous drones need to address is collision avoidance. Avoidance comes down to being able to calculate precise distances from objects and predict their direction.

You could argue that distance detection is a also a solved problem. After all, radar has been around for ever, and lidar is how those Google cars image the world around them. But those technologies are HEAVY and drone needs to be light.

Given that drones have on-board cameras, the solution becomes one of detecting distances in video in real-time. Here again, we see lots of efforts using traditional algorithms. Entrepreneur Adrian Rosenbrock published a nice tutorial in early 2015 on the subject titled “Find distance from camera to object/marker using Python and OpenCV”, and that work is based on research that goes back to at least 2012 (probably much further). But the implementation of a CNN to solve this problem really seems to be approaching current research efforts. When I hunted around for what team is really pushing the art of distance detection using deep learning in an aerial form factor, one name kept coming up: Pilot.ai.

Putting it all together: Pilot.ai

The 8 person team at Pilot.ai has shown some pretty compelling demonstrations of autonomous drone technology relying on deep learning frameworks. They’ve managed to demonstrate both real time object following and real time distance identification through the onboard drone webcam with both onboard and offboard processing.

The follow me technology looks to me to be a fork of YOLO, but the real time distance estimation below looks to be novel from my research.

Combined, you can see a future where all sorts of machines are able to navigate autonomously with simple cameras and onboard processing.

If you are trying to put your finger on which Silicon Valley startup is the next to have their technology and expertise acquired for millions, you can stop your search on Pilot.ai. Pilot was founded a year ago by a group of Stanford PhDs / Metamind refugees. The CEO, Jonathan Su, had a win with his previous company’s technology being acquired by ebay. He partnered up with two engineers who each had the founding year of Metamind under their belts to launch Pilot.

We know that the Metamind team has done proprietary research on image recognition but lacked a commercialization vector — and were consequently acquired by Salesforce earlier this year for about $35M. I’m speculating here, but with Pilot, I see a couple of aggressive engineers who wanted to commercialize this kind of tech, and found a willing leader in Su.

I am shocked that the more Silicon Valley investors are not also connected to Su on LinkedIn. Pilot has raised seed capital from Danhua, which is not at all surprising given the Stanford Machine Learning connection. And the Chen / Guo team at Greylock is at least casually aware, but beyond that Pilot seems to be easy pickings. (I’m sure a lot is happening behind the scenes that I am not able to discover from public links.)

Self-flying tech doesn’t have the mainstream mojo of self-driving tech. And I don’t expect Pilot to be a moonshot acquisition like Otto. But if I were a Series A firm, I’d send my partner to have a coffee with Pilot ASAP.