Tensor Flow Lite Interpreter

The TensorFlow Lite Interpreter follows the following steps in order to return the predictions based on the input.

1. Converting the model into a ByteBuffer

We must memory map the model from the Assets folder to get a ByteBuffer, which is ultimately loaded into the interpreter:

@Throws(IOException::class)

private fun loadModelFile(assetManager: AssetManager, filename: String): ByteBuffer {

val fileDescriptor = assetManager.openFd(filename)

val inputStream = FileInputStream(fileDescriptor.fileDescriptor)

val fileChannel = inputStream.channel

val startOffset = fileDescriptor.startOffset

val declaredLength = fileDescriptor.declaredLength

return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength)

}

2. Loading the labels classes into a Data Structure

The labels file consists of thousands of different classes from ImageNet. We’ll load those labels into an Array. In the end, the interpreter will return predictions based on these label strings.

@Throws(IOException::class)

fun loadLines(context: Context, filename: String): ArrayList<String> {

val s = Scanner(InputStreamReader(context.assets.open(filename)))

val labels = ArrayList<String>()

while (s.hasNextLine()) {

labels.add(s.nextLine())

}

s.close()

return labels

}

var labels = loadLines(context, "labels.txt")

3. Initializing Our Interpreter

Now that we’ve got our ByteBuffer and label list, it’s time to initialize our interpreter. In the following code, we’ve added the GPUDelegate in our Interpreter.Options() method:

In the above code, once the model’s setup is done in the interpreter, we retrieve the input tensor shape of the model. This is done in order to preprocess the Bitmap into the same shape that the model accepts.

The Callable interface is similar to Runnable but allows us to return a result. The ExecutorService is used for managing multiple threads from a ThreadPool.

The initialize method is called in the onCreate method of our MainActivity , as shown below:

private var tfLiteClassifier: TFLiteClassifier = TFLiteClassifier(this@MainActivity) tfLiteClassifier

.initialize()

.addOnSuccessListener { }

.addOnFailureListener { e -> Log.e(TAG, "Error in setting up the classifier.", e) }

4. Preprocessing the Input and Running Inference

We can now resize our Bitmap to fit the model input shape. Then, we’ll convert the new Bitmap into a ByteBuffer for model execution:

In the above code, the convertBitmapToByteBuffer masks the least significant 8 bits from each pixel in order to ignore the alpha channel.

Along with the ByteBuffer, we pass a float array for each of the image classes on which the predictions will be calculated and returned.

5. Computing Arg Max

Finally, the getMaxResult function returns the label with the highest confidence, as shown in the code snippet below:

private fun getMaxResult(result: FloatArray): Int {

var probability = result[0]

var index = 0

for (i in result.indices) {

if (probability < result[i]) {

probability = result[i]

index = i

}

}

return index

}

The classifyAsync method that runs in the Analyzer use case gets a string consisting of prediction and inference time via the onSuccessListener , thanks to Callable interface.

tfLiteClassifier

.classifyAsync(bitmap)

.addOnSuccessListener { resultText -> predictedTextView?.text = resultText }

In return, we display the predicted label and the inference time on the screen, as shown below:

Screengrab from our application

Conclusion

So that sums up this article. We used TensorFlow Lite and CameraX to build an image classification Android application using MobileNet while leveraging the GPU delegate—and we got a pretty accurate result pretty quickly. Moving on from here, you can try building your own custom TFLite Models and see how they fare with CameraX. CameraX is still in alpha stages, but there’s already a lot you can do with it.

The full source code of this guide is available here.

That’s it for this one. I hope you enjoyed reading.