Logging data

We’re finally ready to log data. Back in Xcode, let’s return to the breakpoints. You should have placed them before the model is initialized and after the predict function runs. We’ll use these to make sure we’re only logging function calls related to Core ML.

Set breakpoints before a model is run and after a prediction is made.

Build and run the application on your test device and perform any actions necessary to trigger the first break point. The next three steps should be performed in relatively quick succession so read ahead:

Start recording on the Counter Instrument. Click the Continue Program Execution button in the debut panel in Xcode and perform any actions needed run your model. Once the application hits the next breakpoint, stop recording in the Counter Instrument.

If you’ve done this properly, the Counters Instrument should have recorded data in it.

Identifying Runtime Location

We’ll sift through our logs to determine where the Core ML model was run. First, let’s look at a model that runs on the ANE. This is a simple MobileNet model.

In the Input Filter box found in the lower left corner of the Instruments window, type “ANE” to find calls that make use of Apple Neural Engine APIs.

A MobileNet model making use of the Apple Neural Engine.

Notice that the Responsible Library is often Apple Neural Engine and the thread name begins with H11ANE . The presence of these calls indicates that our model is making use of the Neural Engine for predictions. If we drill down on a specific call, we can start to make sense of what’s going on.

Select a row in the Sample List to see a stack trace. This is a trace for a prediction call. It reads from bottom to top. Midway through, we see a call to MLNeuralNetworkEngine predictionFromFeatures eventually leads to a call to the ANERuntimeEngine .

Now let’s look at a log from a model that doesn’t use the ANE.

Models that are incompatible with the ANE do not use any methods with it’s name.

Now calls to any functions containing the phrase ANE appear. However, if we search for the keyword Metal , the name for Apple’s GPU library, we find plenty of hits.

A model making use of the GPU not the ANE.

Metal is now the Responsible Library and the MPSEngine (short for Metal Performance Shaders) appears in the stack trace.

To summarize: models that run on the Apple neural engine will have many method calls with the acronym ANE . Models that only run on the GPU will have calls referencing Metal , MTL , or MPS .

Troubleshooting incompatible layers

Now that we have a method for reliably identifying which models run on the ANE, we can start isolating layers that might not be compatible. We do this by creating models with a single layer and testing each individually. Let’s consider two different types of convolution layers: straight forward 2D convolutions and the increasingly popular atrous (or dilated) convolutions.

We can create Core ML models quickly using Keras and coremltools.

Performing the steps in previous sections, we’ll see that standard convolutions run on the ANE, but dilated convolutions will not, despite being supported by Core ML. Armed with this information, we can change our model architecture and retrain without the incompatible layer to improve runtime performance.

Conclusion

Taking advantage of hardware acceleration via Apple’s Neural Engine is extremely important if you want your application to run fast and not drain a user’s battery. This post provides a process to determine if your Core ML model is compatible with the ANE so you can make any necessary changes.

PS — If you know of an easier or programatic way to get the same information, let us know in the Fritz AI Community Slack or on Twitter.