A Flutter plugin for accessing TensorFlow Lite API. Supports image classification, object detection (SSD and YOLO), Pix2Pix and Deeplab and PoseNet on both iOS and Android.

Table of Contents #

Breaking changes #

Since 1.1.0:

iOS TensorFlow Lite library is upgraded from TensorFlowLite 1.x to TensorFlowLiteObjC 2.x. Changes to native code are denoted with TFLITE2 .

Since 1.0.0:

Updated to TensorFlow Lite API v1.12.0. No longer accepts parameter inputSize and numChannels . They will be retrieved from input tensor. numThreads is moved to Tflite.loadModel .

Add tflite as a dependency in your pubspec.yaml file.

In android/app/build.gradle , add the following setting in android block.

aaptOptions { noCompress 'tflite' noCompress 'lite' }

Solutions to build errors on iOS:

'vector' file not found" Open ios/Runner.xcworkspace in Xcode, click Runner > Tagets > Runner > Build Settings, search Compile Sources As , change the value to Objective-C++

'tensorflow/lite/kernels/register.h' file not found The plugin assumes the tensorflow header files are located in path "tensorflow/lite/kernels". However, for early versions of tensorflow the header path is "tensorflow/contrib/lite/kernels". Use CONTRIB_PATH to toggle the path. Uncomment //#define CONTRIB_PATH from here: https://github.com/shaqian/flutter_tflite/blob/master/ios/Classes/TflitePlugin.mm#L1

Create a assets folder and place your label file and model file in it. In pubspec.yaml add:

assets: - assets/labels.txt - assets/mobilenet_v1_1.0_224.tflite

Import the library:

import 'package:tflite/tflite.dart';

Load the model and labels:

String res = await Tflite.loadModel( model: "assets/mobilenet_v1_1.0_224.tflite", labels: "assets/labels.txt", numThreads: 1, // defaults to 1 isAsset: true, // defaults to true, set to false to load resources outside assets useGpuDelegate: false // defaults to false, set to true to use GPU delegate );

See the section for the respective model below. Release resources:

await Tflite.close();

GPU Delegate #

When using GPU delegate, refer to this step for release mode setting to get better performance.

Image Classification #

Output format:

{ index: 0, label: "person", confidence: 0.629 }

Run on image:

var recognitions = await Tflite.runModelOnImage( path: filepath, // required imageMean: 0.0, // defaults to 117.0 imageStd: 255.0, // defaults to 1.0 numResults: 2, // defaults to 5 threshold: 0.2, // defaults to 0.1 asynch: true // defaults to true );

Run on binary:

var recognitions = await Tflite.runModelOnBinary( binary: imageToByteListFloat32(image, 224, 127.5, 127.5),// required numResults: 6, // defaults to 5 threshold: 0.05, // defaults to 0.1 asynch: true // defaults to true ); Uint8List imageToByteListFloat32( img.Image image, int inputSize, double mean, double std) { var convertedBytes = Float32List(1 * inputSize * inputSize * 3); var buffer = Float32List.view(convertedBytes.buffer); int pixelIndex = 0; for (var i = 0; i < inputSize; i++) { for (var j = 0; j < inputSize; j++) { var pixel = image.getPixel(j, i); buffer[pixelIndex++] = (img.getRed(pixel) - mean) / std; buffer[pixelIndex++] = (img.getGreen(pixel) - mean) / std; buffer[pixelIndex++] = (img.getBlue(pixel) - mean) / std; } } return convertedBytes.buffer.asUint8List(); } Uint8List imageToByteListUint8(img.Image image, int inputSize) { var convertedBytes = Uint8List(1 * inputSize * inputSize * 3); var buffer = Uint8List.view(convertedBytes.buffer); int pixelIndex = 0; for (var i = 0; i < inputSize; i++) { for (var j = 0; j < inputSize; j++) { var pixel = image.getPixel(j, i); buffer[pixelIndex++] = img.getRed(pixel); buffer[pixelIndex++] = img.getGreen(pixel); buffer[pixelIndex++] = img.getBlue(pixel); } } return convertedBytes.buffer.asUint8List(); }

Run on image stream (video frame):

Works with camera plugin 4.0.0. Video format: (iOS) kCVPixelFormatType_32BGRA, (Android) YUV_420_888.

var recognitions = await Tflite.runModelOnFrame( bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required imageHeight: img.height, imageWidth: img.width, imageMean: 127.5, // defaults to 127.5 imageStd: 127.5, // defaults to 127.5 rotation: 90, // defaults to 90, Android only numResults: 2, // defaults to 5 threshold: 0.1, // defaults to 0.1 asynch: true // defaults to true );

Object Detection #

Output format:

x, y, w, h are between [0, 1]. You can scale x, w by the width and y, h by the height of the image.

{ detectedClass: "hot dog", confidenceInClass: 0.123, rect: { x: 0.15, y: 0.33, w: 0.80, h: 0.27 } }

SSD MobileNet:

Run on image:

var recognitions = await Tflite.detectObjectOnImage( path: filepath, // required model: "SSDMobileNet", imageMean: 127.5, imageStd: 127.5, threshold: 0.4, // defaults to 0.1 numResultsPerClass: 2,// defaults to 5 asynch: true // defaults to true );

Run on binary:

var recognitions = await Tflite.detectObjectOnBinary( binary: imageToByteListUint8(resizedImage, 300), // required model: "SSDMobileNet", threshold: 0.4, // defaults to 0.1 numResultsPerClass: 2, // defaults to 5 asynch: true // defaults to true );

Run on image stream (video frame):

Works with camera plugin 4.0.0. Video format: (iOS) kCVPixelFormatType_32BGRA, (Android) YUV_420_888.

var recognitions = await Tflite.detectObjectOnFrame( bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required model: "SSDMobileNet", imageHeight: img.height, imageWidth: img.width, imageMean: 127.5, // defaults to 127.5 imageStd: 127.5, // defaults to 127.5 rotation: 90, // defaults to 90, Android only numResults: 2, // defaults to 5 threshold: 0.1, // defaults to 0.1 asynch: true // defaults to true );

Tiny YOLOv2:

Run on image:

var recognitions = await Tflite.detectObjectOnImage( path: filepath, // required model: "YOLO", imageMean: 0.0, imageStd: 255.0, threshold: 0.3, // defaults to 0.1 numResultsPerClass: 2,// defaults to 5 anchors: anchors, // defaults to [0.57273,0.677385,1.87446,2.06253,3.33843,5.47434,7.88282,3.52778,9.77052,9.16828] blockSize: 32, // defaults to 32 numBoxesPerBlock: 5, // defaults to 5 asynch: true // defaults to true );

Run on binary:

var recognitions = await Tflite.detectObjectOnBinary( binary: imageToByteListFloat32(resizedImage, 416, 0.0, 255.0), // required model: "YOLO", threshold: 0.3, // defaults to 0.1 numResultsPerClass: 2,// defaults to 5 anchors: anchors, // defaults to [0.57273,0.677385,1.87446,2.06253,3.33843,5.47434,7.88282,3.52778,9.77052,9.16828] blockSize: 32, // defaults to 32 numBoxesPerBlock: 5, // defaults to 5 asynch: true // defaults to true );

Run on image stream (video frame):

Works with camera plugin 4.0.0. Video format: (iOS) kCVPixelFormatType_32BGRA, (Android) YUV_420_888.

var recognitions = await Tflite.detectObjectOnFrame( bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required model: "YOLO", imageHeight: img.height, imageWidth: img.width, imageMean: 0, // defaults to 127.5 imageStd: 255.0, // defaults to 127.5 numResults: 2, // defaults to 5 threshold: 0.1, // defaults to 0.1 numResultsPerClass: 2,// defaults to 5 anchors: anchors, // defaults to [0.57273,0.677385,1.87446,2.06253,3.33843,5.47434,7.88282,3.52778,9.77052,9.16828] blockSize: 32, // defaults to 32 numBoxesPerBlock: 5, // defaults to 5 asynch: true // defaults to true );

Thanks to RP from Green Appers

Output format: The output of Pix2Pix inference is Uint8List type. Depending on the outputType used, the output is: (if outputType is png) byte array of a png image (otherwise) byte array of the raw output

Run on image:

var result = await runPix2PixOnImage( path: filepath, // required imageMean: 0.0, // defaults to 0.0 imageStd: 255.0, // defaults to 255.0 asynch: true // defaults to true );

Run on binary:

var result = await runPix2PixOnBinary( binary: binary, // required asynch: true // defaults to true );

Run on image stream (video frame):

var result = await runPix2PixOnFrame( bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required imageHeight: img.height, // defaults to 1280 imageWidth: img.width, // defaults to 720 imageMean: 127.5, // defaults to 0.0 imageStd: 127.5, // defaults to 255.0 rotation: 90, // defaults to 90, Android only asynch: true // defaults to true );

Thanks to RP from see-- for Android implementation.

Output format: The output of Deeplab inference is Uint8List type. Depending on the outputType used, the output is: (if outputType is png) byte array of a png image (otherwise) byte array of r, g, b, a values of the pixels

Run on image:

var result = await runSegmentationOnImage( path: filepath, // required imageMean: 0.0, // defaults to 0.0 imageStd: 255.0, // defaults to 255.0 labelColors: [...], // defaults to https://github.com/shaqian/flutter_tflite/blob/master/lib/tflite.dart#L219 outputType: "png", // defaults to "png" asynch: true // defaults to true );

Run on binary:

var result = await runSegmentationOnBinary( binary: binary, // required labelColors: [...], // defaults to https://github.com/shaqian/flutter_tflite/blob/master/lib/tflite.dart#L219 outputType: "png", // defaults to "png" asynch: true // defaults to true );

Run on image stream (video frame):

var result = await runSegmentationOnFrame( bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required imageHeight: img.height, // defaults to 1280 imageWidth: img.width, // defaults to 720 imageMean: 127.5, // defaults to 0.0 imageStd: 127.5, // defaults to 255.0 rotation: 90, // defaults to 90, Android only labelColors: [...], // defaults to https://github.com/shaqian/flutter_tflite/blob/master/lib/tflite.dart#L219 outputType: "png", // defaults to "png" asynch: true // defaults to true );

Model is from StackOverflow thread.

Output format:

x, y are between [0, 1]. You can scale x by the width and y by the height of the image.

[ // array of poses/persons { // pose #1 score: 0.6324902, keypoints: { 0: { x: 0.250, y: 0.125, part: nose, score: 0.9971070 }, 1: { x: 0.230, y: 0.105, part: leftEye, score: 0.9978438 } ...... } }, { // pose #2 score: 0.32534285, keypoints: { 0: { x: 0.402, y: 0.538, part: nose, score: 0.8798978 }, 1: { x: 0.380, y: 0.513, part: leftEye, score: 0.7090239 } ...... } }, ...... ]

Run on image:

var result = await runPoseNetOnImage( path: filepath, // required imageMean: 125.0, // defaults to 125.0 imageStd: 125.0, // defaults to 125.0 numResults: 2, // defaults to 5 threshold: 0.7, // defaults to 0.5 nmsRadius: 10, // defaults to 20 asynch: true // defaults to true );

Run on binary:

var result = await runPoseNetOnBinary( binary: binary, // required numResults: 2, // defaults to 5 threshold: 0.7, // defaults to 0.5 nmsRadius: 10, // defaults to 20 asynch: true // defaults to true );

Run on image stream (video frame):

var result = await runPoseNetOnFrame( bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required imageHeight: img.height, // defaults to 1280 imageWidth: img.width, // defaults to 720 imageMean: 125.0, // defaults to 125.0 imageStd: 125.0, // defaults to 125.0 rotation: 90, // defaults to 90, Android only numResults: 2, // defaults to 5 threshold: 0.7, // defaults to 0.5 nmsRadius: 10, // defaults to 20 asynch: true // defaults to true );

Prediction in Static Images #

Refer to the example.

Refer to flutter_realtime_Detection.