Notice the ‘Orientation’ data element. This tells the image viewer program that the image needs to be rotated 90 degrees clockwise before being displayed on screen. If the program forgets to do this, the image will be sideways!

Why does this break so many Python Computer Vision Applications?

Exif metadata is not a native part of the Jpeg file format. It was an afterthought taken from the TIFF file format and tacked onto the Jpeg file format much later. This maintained backwards compatibility with old image viewers, but it meant that some programs never bothered to parse Exif data.

Most Python libraries for working with image data like numpy, scipy, TensorFlow, Keras, etc, think of themselves as scientific tools for serious people who work with generic arrays of data. They don’t concern themselves with consumer-level problems like automatic image rotation — even though basically every image in the world captured with a modern camera needs it.

This means that when you load an image with almost any Python library, you get the original, unrotated image data. And guess what happens when you try to feed a sideways or upside-down image into a face detection or object detection model? The detector fails because you gave it bad data.

You might think this problem is limited to Python scripts written by beginners and students, but that’s not the case! Even Google’s flagship Vision API demo doesn’t handle Exif orientation correctly:

Google Vision’s API demo fails to rotate a portrait-oriented image captured with a standard cell phone.

And while Google Vision still manages to detect some of the animals in the sideways image, it detects them with a non-specific “Animal” label. This is because it is a lot harder for a model to detect a sideways goose than an upright goose. Here’s what Google Vision detects if the image is correctly rotated before being fed into the model:

With the correct image orientation, Google detects the birds with the more specific “Goose” label and a higher confidence score. Much better!

This is a super obvious problem if you can see that the image is sideways like in this demo. But this is where things get insidious —normally you can’t see it! Every normal program on your computer will only display the image in its properly rotated form instead of how it is actually stored sideways on disk. So when you try to view the image to see why your model isn’t working, it will be displayed the right way and you won’t know why your model isn’t working!

Finder on a Mac always displays images with Exif rotation applied. There is no way to see that the image data is actually sideways inside the file.

This inevitably leads to people posting issues on Github complaining that the open source projects that they are using are broken or the models aren’t very accurate. But the problem is so much simpler — they are feeding in sideways and/or upside-down images!

Fixing the Problem

The solution is that whenever you load images in your Python programs, you should check them for Exif Orientation metadata and rotate the images if needed. It’s pretty simple to do, but surprisingly hard to find examples of code online that does it correctly for all orientations.

Here is code to load any image into a numpy array with the correct rotation applied:

From there, you can pass the array of image data to any standard Python ML library that expects arrays of image data, like Keras or TensorFlow.

Since this comes up so often, I published this function as a library on pip called image_to_numpy. You can install it like this:

pip3 install image_to_numpy

You can use it in any Python program to load an image correctly, like this:

import matplotlib.pyplot as plt

import image_to_numpy # Load your image file

img = image_to_numpy.load_image_file("my_file.jpg") # Show it on the screen (or whatever you want to do)

plt.imshow(img)

plt.show()

Check out the readme file for more details.

Have fun!