The steady rise of mobile Internet traffic has provoked a parallel increase in demand for on-device intelligence capabilities. However, the inherent scarcity of resources at the Edge means that satisfying this demand will require creative solutions to old problems. How do you run computationally expensive operations on a device that has limited processing capability without it turning into magma in your hand?

The addition of TensorFlow Lite to the TensorFlow ecosystem provides us with the next step forward in machine learning capabilities, allowing us to harness the power of TensorFlow models on mobile and embedded devices while maintaining low latency, efficient runtimes, and accurate inference.

TensorFlow Lite provides the framework for a trained TensorFlow model to be compressed and deployed to a mobile or embedded application. Interfacing with the TensorFlow Lite Interpreter, the application can then utilize the inference-making potential of the pre-trained model for its own purposes.

In this way, TensorFlow Lite works as a complement to TensorFlow. The computationally expensive process of training can still be performed by TensorFlow in the environment that best suits it (personal server, cloud, overclocked computer, etc.).

TensorFlow Lite takes the resulting model (frozen graph, SavedModel, or HDF5 model) as input, packages, deploys, and then interprets it in the client application, handling the resource-conserving optimizations along the way.

<From TensorFlow to Client Application, From TensorFlow Lite Docs>

If you’re familiar with the TensorFlow ecosystem, you might ask “Doesn’t TensorFlow Mobile already address this use-case?” And you would be right… sort of. TensorFlow Mobile was the TensorFlow team’s first solution for extending model functionality to mobile and embedded devices.

However, TensorFlow Lite offers lower latency, higher throughput, and a generally lighter weight solution that will be the focus of the TensorFlow team for 2019 and beyond. Magnus Hyttsten, a Developer Advocate on Google’s TensorFlow team makes a clear statement on the difference between the two:

“TensorFlow Lite should really be seen as the evolution of TensorFlow Mobile and as it matures it will become the recommended solution for deploying models on mobile and embedded devices.”

TensorFlow Lite functionally differs from TensorFlow Mobile in the degree to which it has been optimized to support this process of transformation, deployment, and interpretation. TensorFlow Lite has been leveraged at every level from the model translation stage to hardware utilization to increase the viability of on-device inference while maintaining model integrity.

Below, we take a look at a few of the key optimizations across the components of TensorFlow Lite.