Adding new layers to an existing Core ML model.

In the previous section, we dealt with converting unsupported custom layers. In this section, we’ll add entirely new layers directly to an existing Core ML model. Style transfer models sometimes suffer from edge artifacts due to padding on convolution layers.

If same padding is used, convolution windows on the edges of an image will have a disproportionate number of zero values and will distort style transfer results for those pixels. This problem can be remedied by using reflection padding to create a visually consistent border around the input image, then using cropping at the end to keep the original size consistent. Unfortunately, reflection padding isn’t supported by Keras, making it necessary to add the layer manually in Core ML. Here’s how:

Flexible input and output sizes

For simplicity and performance, machine learning models are often trained with fixed input and output dimensions. Input data that doesn’t conform to these dimension requirements is scaled and / or cropped to match. When integrated into an application, though, these fixed dimensions might not be convenient or performant.

In the case of artistic style transfer, models are configured to accept and produce images of a fixed size, like 256 by 256 pixels, to speed up training. During inference, though, we want to support high resolution images and images of different aspect ratios. We’d also like to avoid creating dozens of different models to support each configuration.

Thankfully, Apple added support for models with flexible input and output shapes in Core ML 2. As long as a model is fully convolutional, images of arbitrary sizes can be fed as input to our style transfer models. This allows us to support a wider variety of use cases and control performance across devices (e.g. by setting a maximum image input sizes for users on older, less powerful phones).

In this case, we added a continuous range of supported input sizes. It’s also possible to allow a discrete set of input sizes.

Quantize a model to shrink its size.

So far we’ve converted a model with custom layers, manually added reflective padding to fix edge artifacts, and made input sizes flexible for users. As a final step, lets compress our Core ML model to keep our bundle size low. We can do this with Core ML’s quantization tools. The function below quantizes Core ML models and is safe to use on any platform.

In practice, we’ve found little to no decrease in accuracy using 8-bit quantization to shrink models to 25% of their original size. You can dive deeper into quantization here.

Conclusion

This post covered a few advanced techniques for working with Core ML models. We fixed a broken model conversion process using custom layers. We added layers to an existing Core ML model by hand. We made use of flexible input and output sizes to give users more options and decrease the number of models we need to maintain. Finally, we used quantization to shrink a model’s size to keep our app bundle small.