Creating a basic demo app

We will be creating a simple app to use style transfer to transform live video into van Gogh’s Starry Night. It will probably take about 30 minutes to get all the way through. The completed demo app is available here. For a more full-featured style transfer demo, you can check out the example in Fritz AI Studio.

Step 1: Setup your Xcode Project

First things first, create a new Xcode project. If you want, fork the Fritz Style Transfer Demo project. Each commit in the demo app will follow a step in this tutorial. Before going onto the next step, make sure your empty project builds (you will likely need to change the Bundle Identifier).

Step 2: Install Fritz

To use the Style Transfer Feature, you need to create a Fritz account and setup the SDK in your app. This is a quick process that shouldn’t take you more than 5 minutes. The Fritz SDK is available on CocoaPods.

Overview of Fritz Setup Steps:

Create an account here Create a project — projects group your apps together e.g. iOS and Android flavors Add an app to the project. You will need the bundle identifier you used in Step 1. After following the directions in the webapp, run your app with the Fritz SDK configured

Today we’re just using the Fritz SDK for style transfer, but you can also use the Fritz SDK to create apps that can label the contents of an image (think “not hotdog”) or detect the location of objects in an image. Check out the docs for all the machine learning powered features you can build.

After you’ve initialized your app per the instructions in the webapp, you’re ready to start building your Style Transfer app.

Step 3: Add PreviewView to Display Live Video

In this step, we’re going to use some helper code to display live video from the camera, to make sure that our app will be ready to display live style transfer code.

To do this, we’re going to add a UIImageView to our ViewController.

This way, the preview view will be rendered when the view controller launches.

Build the project here to make sure that all is well.

Step 4: Setup Camera and Display Live Video

Next, we’re going to add add the camera to our app and just display live video.

First things first, we need to add camera permissions to the Info.plist. As a key, add “Privacy — Camera Usage Description” with a description

To display frames from the camera, we need to configure AVCaptureSession and AVCaptureVideoDataOutput objects.

When we are configuring the AVCaptureSession, we need to use the 640x480 sessionPreset in preparation for sending the data to the style transfer model:

Next, configure the AVCaptureVideoDataObject and add that to the session:

To make sure we have everything hooked up correctly, let’s display the raw video coming from the camera on the subview. When you run the app, you should see normal video being displayed. Here we make sure to update the previewView asynchronously so we don’t block the UI Thread:

Run this and you should see live video in your app!

Step 5: Add some style

Once you have the video displaying to your preview view, it’s time to add the style transfer model!

First, update your Podfile to include the Style Transfer Model:

pod 'Fritz/VisionStyleModel/Paintings'

Initialize the style transfer model as a variable in the view controller and run the model on the captureOutput sample buffer:

Run the app to see a live working style transfer model! There are 11 different Style Transfer models included out of the box.

Congrats! You’ve made your first app using real-time Style Transfer!

Check out Fritz AI Studio for a more complex implementation of Style Transfer. It uses a gesture recognizer to switch the style transfer model when you tap the screen.

Flexible Core ML Models

With the release of Core ML 2, Apple added the ability to create models that accept flexible input and output sizes. Because our style models are fully convolutional, it’s possible to create your own styles can be applied to images of any size. To change the input size of the model, adjust flexibleModelDimensions option on FritzVisionStyleModelOptions

Build your own Styles

If you like what you see and want to build your own styles, check out 20-Minute Masterpiece: Training your own style transfer model.

Jameson Toole shows how you can use a Fritz AI training template to create a totally unique style from your own images.

Our goal is to make it easier for developers to get into mobile machine learning. With the Fritz SDK and the Style Transfer API, you can now take any image and transform it into a work of art. We can’t wait to see what you’ll create! Try it out and let us know what you think.