Using Core ML Raw

Let’s say you need to make an iOS app that predicts what’s in a photo. You do some googling, and you find out that you need to download a Core ML model and process the image through it. One approach you could take would be to modify an incoming CVPixelBuffer like so:

func resize(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {

let imageSize = 299

var ciImage = CIImage(cvPixelBuffer: pixelBuffer, options: nil)

let transform = CGAffineTransform(scaleX: CGFloat(imageSide) / CGFloat(CVPixelBufferGetWidth(pixelBuffer)), y: CGFloat(imageSide) / CGFloat(CVPixelBufferGetHeight(pixelBuffer)))

ciImage = ciImage.transformed(by: transform).cropped(to: CGRect(x: 0, y: 0, width: imageSide, height: imageSide))

let ciContext = CIContext()

var resizeBuffer: CVPixelBuffer?

CVPixelBufferCreate(kCFAllocatorDefault, imageSide, imageSide, CVPixelBufferGetPixelFormatType(pixelBuffer), nil, &resizeBuffer)

ciContext.render(ciImage, to: resizeBuffer!)

return resizeBuffer

}

After you get your buffer resized, you then pass it through your model, and you do what you want with your predictions:

DispatchQueue.global(qos: .userInteractive).async {

if let pixelBuffer = self.convert(image) {

guard let prediction = try? self.model.prediction(image: pixelBuffer) else {

return

}

DispatchQueue.main.async {

screen.updatePredictionLabel(prediction)

}

}

}

Not bad, right? In just a short amount of code, you take an image and leverage the GPU of an Apple device with native iOS code to predict the contents of that image.

This is great! However, this is also exactly why I made Lumina the way it is. Let’s look again at the singular line of code you need to enable Core ML with Lumina:

camera.streamingModelTypes = [MobileNet()]

It’s not that it was difficult before — rather, I saw an opportunity to make it easier, so I took a shot at it. Let’s start at the beginning by deriving an interface.

The Interface

When you design an application programming interface (API), it’s good practice to think in human terms about what you’re trying to accomplish. The goal of Lumina is to make using a camera in iOS as stress-free as possible. Thanks to iOS 11, the use of the camera includes using Core ML as well.

When designing this interface, I thought — how simple can I make this? In the sample app that comes with Lumina, I provide a file called MobileNet.mlmodel , which can be downloaded from Apple’s website here — we’ll refer to this model often.

Test Driven Development is a very widely discussed practice in software, but one basic tenet of the practice is to write your tests first, and then write code that makes your tests pass. Keeping this in mind, I decided to write the interface, and then write code to make the interface valid. I arrived at the snippet above that this article is based on, where you assign a collection of instantiated Core ML classes to camera.streamingModelTypes .

To utilize a Core ML model, Vision needs to access the property named model like so:

let model = MobileNet().model

Core ML is not compatible with iOS 10 or below. However, because I want Lumina to be accessible to a wider array of developers, and I like a little bit of masochism, I wanted to maintain one code base that works for developers with and without access to Core ML. This meant that I had to store my models in a more generic variable container:

open var streamingModelTypes: [AnyObject]?

My favorite video game of all time is Metal Gear Solid, and this image is appropriate:

Nothing to see here, LLVM, move right along.

It would have been a lot easier to make developers wrap all of their code in if #available (iOS 11.0, *) statements just to use my framework, but that wouldn’t be any fun, would it? Upon some reflection, I realized that I could work around this by using the Mirror class in Swift, like so:

for type in types {

let reflection = Mirror(reflecting: type)

for (name, value) in reflection.children where name == “model” {

guard let model = value as? MLModel else {

continue

}

modelsToSet.append((model, reflection.subjectType))

}

}

If you make a reflection of your object using Mirror , then you can loop through each property and match against the one you’re looking for. In this case, you’re looking for the model property, and if you can cast it to an object of type MLModel , then you’re in luck. I use the reflection.subjectType to keep track of which model is which.