With the introduction of iPhone XR, every phone in Apple’s lineup now supports depth capture. But the XR is unique: it’s the first iPhone to do it with a single lens. As we were starting to test and optimize Halide for it, we found both advantages and disadvantages.

In this post we’ll take a look at three different ways iPhones generate depth data, what makes the iPhone XR so special, and show off Halide’s new 1.11 update, which enables you to do things with the iPhone XR that the regular camera app won’t.

Depth Capture Method 1: Dual Camera Disparity

Humans perceive depth with the help of two eyes. Our eyes may only a few inches apart, but our brains detect subtle differences between image. The greater difference, or disparity, the closer an object.

Disparity illustrated. This image is from Professor David Heeger’s page on Perception.

The iPhone 7 Plus introduced a dual-camera system, which enables a similar way to construct depth. By taking two photos at the same time, each from a slightly different position, we can construct a disparity map.

There’s a lot of guesswork involved when matches images. Add video noise and things get even rougher. A great deal of effort goes into filtering the data, additional post processing that guesses how to smooth edges and fill holes.

Via WWDC 2017

This requires a lot of computation. It wasn’t until the iPhone X that iOS could achieve 30 frames a second. This all takes a lot memory. For a while, our top crash was running out of memory because the system was using most of it on depth.

Dual Camera Drawbacks

One limitation and quirk of this method is that you can only generate depth for the parts of the two images that overlap. In other words, if you have a wide angle and telephoto lens, you can only generate depth data for the telephoto lens.