What’s your background and how did you come up with the idea for InstaSaber?

I’ve been programming since I was around 12 years old. I taught myself QBasic by looking at other DOS programs like Gorillas (before the internet made this kind of thing much easier!). I kept learning new languages and programming for fun or building websites after that. I graduated from UC San Diego with a degree in electrical engineering, but found it was easier to get a job with my programming skills. My first job out of college was for a small startup called Goowy. It was then that I quickly realized that being an entrepreneur was my calling and I have started several companies since.

I started my path to InstaSaber with virtual hand puppets (YoPuppet). I was using OpenCV at was having difficulty getting it working in various lighting conditions and backgrounds. Nevertheless, I was able to raise a small seed round with what I had built. I did not have a ton of experience with Machine Learning / CNN’s prior to that, but decided I needed to dive in if I wanted a product that worked well. I initially started training on just hand key points, and got some promising results. In an attempt to go from 2d points to 3d, I decided to train on a less complex, more uniform object and see what kind of results I could get. I chose a rolled up sheet of paper because of it’s uniformity along two dimensions. It didn’t take much time swinging it around to realize that this would make an awesome virtual saber! Hence, InstaSaber : )

What’s does your tech stack look like and what tools did you find helpful?

First thing I bought was parts to build a midrange, deep learning machine (essentially, a gaming rig). I use Keras / TensorFlow for the model training. I built my own annotation tools in Python and collected my own data. On the iOS side, just CoreML and Metal libraries. A lot of the work has been automating various pieces of my process.

What was the hardest part?

TensorFlow is complex. It took me more time to learn its ins-and-outs than any library I can think of. In addition to that, optimizing everything to run at 60fps on an iPhone SE was challenging. But really, the hardest part is trying to be a dad when running a startup — babies take a lot of time and energy, even when I’m doing the minority of the care!

Do you have any advice for other developers who are looking to get started with machine learning?

Be patient, it takes a lot of time to learn. That said, you don’t need a PhD to use the tools out there. Have a goal in mind, but start with something simple like CIFAR10 and work your way toward your desired result.