ML-Native: Rewriting the relationship between software and data science

And why production machine learning is just getting started

Disclaimer: The following is based on my observations of production machine learning teams — not an academic survey of the industry. For context, I’m a contributor to Cortex, an open source platform for deploying models.

Any time an advance in machine learning research bubbles up into mainstream media coverage, one of the first questions asked—particularly by people who aren’t in the field—is, “How will we actually use this?”

Embedded in that question is a skepticism fostered by a decade of sensational claims promising that fully autonomous cars were just months away, or that all business problems would soon be solved with Big Data™.

While that skepticism is deserved, the reality is that machine learning has already undergone something of a production revolution.

Over the last few years, virtually all of the most popular software products have incorporated machine learning functionality, so much so that you probably use products powered by machine learning every day.

In addition, the barrier to entry for machine learning has come down significantly.

Startups don’t need a team of PhDs, deep pockets, and access to huge datasets to train models. Techniques like transfer learning have made state-of-the-art performance available to teams with small data sets, open source infrastructure platforms have removed the infra bottleneck, and improvements in popular ML frameworks have made the entire field more approachable.

Thanks to this democratization, small startups and solo engineers are shipping a new generation of ML-powered products, most of which would have sounded like science fiction just five years ago.

At this point, production machine learning isn’t just viable—it is ubiquitous.

Machine learning powers all your favorite products

Apple’s top 10 most downloaded free iPhone apps from 2019 are: YouTube, Instagram, Snapchat, TikTok, Messenger, Gmail, Netflix, Facebook, Google Maps, and Amazon.

All of them leverage machine learning.

Netflix, TikTok, YouTube, Instagram, Facebook, Amazon, and Snapchat all use recommendation engines to show you relevant content/products.

Messenger and Gmail use natural language processing to write messages for you, like in Gmail’s Smart Compose and Messenger’s chatbots.

Google Maps uses machine learning to predict traffic patterns.

And that’s without mentioning the Snapchat/Instagram filters that use object detection to do things like this:

There is hardly a major tech company who does not incorporate machine learning functionality into its core product. There are, however, two things you’ll notice about machine learning in these examples:

It is largely powering features that improve products—products that could, at least in theory, exist without it.

It is still mostly the domain of large tech companies who have the resources to develop it.

Fortunately, both of those things are changing.

The new generation of ML-native products

A new generation of products is emerging, products whose core functionality relies entirely on machine learning.

Everyone is already familiar with autonomous vehicles and driver assist software. Similarly, you’ve probably come across facial recognition technology in apps like Apple’s Face ID.

These products are not enhanced by machine learning, they are completely reliant on it—and they’re part of a new generation of ML-native products.

For example, what if the “AI” in a video game was actually built with machine learning, allowing you to do anything you wanted within the game? Such a game exists, and it’s called AI Dungeon:

Or what if, instead of going to the doctor for various cancer screens, a machine could screen your entire body for cancer in an hour? Ezra is a medical startup that uses machine learning to identify cancers based on a single, full body MRI, and they’re already active in three states:

Both of these products fundamentally could not exist without machine learning. Interestingly, both of them are also developed by small teams, not tech giants with infinite resources.

AI Dungeon, for example, started as a hackathon project by a single engineer. Ezra, similarly, is a relatively small team that has only raised $4 million, and they’ve already brought a product to market.

This trend is important to note because it reflects a new critical development in machine learning:

For the first time, the machine learning ecosystem is mature enough for any engineer—not just well-funded PhDs—to build products.

Machine learning is finally accessible to engineers—not just researchers

In the past, developing a new machine learning product required a team of data scientists, large quantities of proprietary data, and model designed from scratch. Every new product was, in essence, a research experiment.

This created an obvious bottleneck. The only people capable of producing new machine learning products worked at companies or research labs with extensive resources.

Over the last few years, three major changes have made machine learning approachable to individual engineers:

The development of frameworks like PyTorch, TensorFlow, and Keras, which abstract away lower-level ML programming.

The open sourcing of state-of-the-art models by major companies (OpenAI, Facebook, Google, Microsoft, etc.)

The release of machine learning infrastructure tooling like Cortex and SageMaker.

As an example of the above, when Nick Walton built AI Dungeon, he didn’t design a new model, collect an enormous data set, conduct countless experiments, and then architect his infrastructure from scratch.

Instead, he took OpenAI’s open source GPT-2 model, finetuned it on a small dataset, and deployed it on Cortex.

Walton is a perfect example of a modern machine learning engineer—someone who uses machine learning to build products. A parallel can be drawn to web development, where developers:

Use frameworks like Node/React, Ruby on Rails, and Django to abstract away the low-level programming of web applications.

Rely on open source projects like PostgreSQL, webpack, and Yarn.

Automate their infrastructure with platforms like Heroku and Elastic Beanstalk.

Most web developers don’t write their own databases or architect their own deployment infrastructure—they build web applications. There are of course engineers who focus on things like building new databases, new frameworks, and new infrastructure, but they are specialized and exist in tandem with engineers who just want to build products.

As this dynamic is mirrored in the machine learning ecosystem, it is enabling a boom in the development of new ML-native products.

Machine learning is in the hands of developers—and that’s a good thing

In order to move the ball forward in machine learning, as with any field, it is important to have a critical mass of people working with it.

Not long ago, this wasn’t possible in machine learning. Individual engineers simply didn’t have the resources to build real, functional products. Now, things have changed.

Just last week, in a story similar to AI Dungeon’s, an engineer in Berlin published their DIY license plate detector, which they built using a Rasberry Pi along with open source models and infrastructure:

As open source models continue to become more capable, and as ML infrastructure continues to improve, we can expect the proliferation of more projects like this.

We’ve already seen production machine learning become commonplace—but now that anyone can build with ML, it looks as though we’ve only seen the tip of the iceberg.