Photo by Charlie Deets

The Five Pillars of Tesla’s Large-Scale Fleet Learning

How over half a million Teslas help train neural networks for autonomous driving

Here are the five pillars of Tesla’s large-scale fleet learning approach to autonomous driving, as I see them:

All the pillars except (1) use automatic labelling because all the techniques mentioned except for fully supervised learning use automatic labelling. Weakly supervised learning, self-supervised learning, imitation learning, and reinforcement learning all use automatic labels.

With ~1,000x more vehicles than Waymo and ~500x more vehicles than all U.S. autonomous vehicle fleets combined, Tesla can collect commensurately more automatically labelled training data for (2), (3), (4), and (5) and commensurately higher-quality manually labelled training data with (1).

For (2–5), if performance scales with data like top-1 error (i.e. correct on the first guess) on the ImageNet image classification dataset, that means Tesla should have roughly 10x better performance on these tasks than competitors. If performance scales like top-5 error (i.e. correct within the first 5 guesses), then it’s roughly 30x better performance. For (1), I’m not yet aware of any research that attempts to quantify anything analogous. The scaling rate for different tasks is different, anyway, so these numbers should be taken with a grain of salt.

Lidar is an ongoing topic of debate. Regardless of whether a company uses lidar, it needs accurate and robust computer vision. This is not just for redundancy, but also because there are certain tasks lidar can’t help with. For example, only cameras can determine whether a traffic light is green, yellow, or red. At any point in the future, Tesla can deploy a small fleet of test vehicles equipped with high-grade lidar, just as companies like Waymo and Cruise are doing today. These lidar-equipped Teslas would combine the benefits of lidar and large-scale fleet learning.

My prediction: in the next 1–3 years, Tesla will surprise many observers with how much progress it makes through its large-scale fleet learning approach. Many folks think Waymo is far ahead and Tesla has no chance of catching them. I think Tesla will pull ahead eventually.