First, let’s walk through how to use the on-device Vision API and get familiar with how to use Fritz AI. Under the hood, the Vision APIs use ML models to run predictions on an image.

There are 4 main components:

FritzVisionPredictor — Manages model execution and processing.

— Manages model execution and processing. FritzVisionImage — Prepares an image for model prediction.

— Prepares an image for model prediction. FritzOnDeviceModel — Contains information to load the underlying model file.

— Contains information to load the underlying model file. FritzVisionResult — A user-friendly object to access the result from the predictor.

The FritzVisionPredictor class handles all the complicated processing that comes with running on-device models — usually this refers to handling pre-processing (any manipulations required to format the input for the model) and post-processing (handling the raw output from the model to create a developer-friendly result).

Each predictor contains a predict method that takes a FritzVisionImage object as an argument and returns a FritzVisionResult with methods that make it easy to visualize and understand the model results.

Predictors are initialized with a FritzOnDeviceModel . For Image Segmentation, there are 4 models that you can use.

People Segmentation Model (Fast) — Identifies and segments pixels belonging to people. The output resolution is 384 x 384.

Living Room Segmentation Model (Fast) — Identifies and segments pixels belonging to living room objects (chair, windows, floor, wall, etc). The output resolution is 384 x 384.

Outdoor Segmentation Model (Fast) — Identifies and segments pixels belonging to outdoor objects (buildings, cars, bikes, etc). The output resolution is 384 x 384.

People Segmentation Model (Accurate)— Identifies and segments pixels belonging to people. The output resolution is 768 x 768.

In this tutorial, we’ll use the People Segmentation Model (Small) and bundle it with our app.*

To download the model, include the following dependency in your app/build.gradle:

aaptOptions{

noCompress "tflite"

} dependencies {

implementation "ai.fritz:vision-people-segmentation-model-fast:4.+"

}

*Note: We use TensorFlow Lite models in our SDK. You’ll need to specify the noCompress option in your app’s build.gradle file to make sure the model is not compressed when it’s built.

Pulling down this dependency will provide a class called PeopleSegmentOnDeviceModelFast (a subclass of FritzOnDeviceModel ).

Putting this all together, here’s the complete code for running Image Segmentation:

All of our Vision APIs follow a similar format in order to run prediction on an image.

This is a high-level overview but if you’d like more details, you can take a look at our Image Segmentation documentation for Android.

With that in mind, let’s get started building our app for background replacement.

*In order to reduce the initial app size, you can also choose to lazy load the model. For more information, take a look at the documentation.