Detecting object using TensorFlowSharp Plugin

I created the scripts in TF-Unity for running inferences using Unity TensorFlowSharp plugin. The scripts is tested with MobileNet model for image classification, and SSD MobileNet and Tiny YOLOv2 model for object detection.

To run object detection with SSD MobileNet model, we first need to initialize the detector. The model and label file are passed to the component as TextAsset.

[SerializeField]

TextAsset model; [SerializeField]

TextAsset labels; Detector detector = new Detector(model, labels, input: "image_tensor");

Everytime a new camera frame is captured, we pass the texture to the detector and run inference to get the detected objects.

var outputs = detector.Detect(m_Texture, angle: 90, threshold: 0.6f);

We then loop the outputs and look for a target object with class name “apple”.

for (int i = 0; i < outputs.Count; i++)

{

var output = outputs[i] as Dictionary<string, object>;

if (output["detectedClass"].Equals("apple"))

{

DrawApple(output["rect"] as Dictionary<string, float>);

break;

}

}

A single output is in the following format where x, y, w, h is the left, top, width and height of a box area containing the object. The values are ratio instead of actual size.