# Stereo Photo Creation from Depth Images

[Jan Zimandl](https://www.strv.com/blog/authors/jan) iOS Engineer

---

### When I gave a talk about stereoscopy, 3D images and my [3D Photo](http://itunes.apple.com/app/id1364981409?ref=strv.ghost.io) app last year, I got a question from the audience about using the iPhone dual depth camera to create a 3D photo. For a while, I didn't give the idea much thought.

### To create a stereo image, you basically need two images. One for the left eye, one for the right. We do this by taking two photos from two different places, the distance between which is usually the same as the distance between the human eyes. But there is no direct access to both pictures from an iPhone camera, and their distance is also too small to use them directly to create a stereo photo.

### However, I later realized that there is access to raw depth data for every photo taken in portrait mode. At first, I didn't take time to experiment with it, because that would mean processing every pixel in the image - something I had no experience with. But it didn't leave my mind, until I finally decided to start playing and having fun with depth data and pixels in the picture.

## THEORIES OF STEREOSCOPY

If you want to create a 3D effect for your eyes and your brain, you need to work with parallax. It is the distance between images of one object in two pictures taken from different places. You can see an example in real life. When you move from left to right and focus on a specific object, other closer objects move from right to left relative to the focused object. This is called negative parallax. All distant objects move from left to right, which is positive parallax.

When we have a static image and its depth map (a grayscale image where black indicates far away, white indicates close objects), we can create pictures with different parallaxes for each pixel to generate one for the left eye and one for the right eye. Input consists of the original image, depth data, and a maximum shift in pixels.

To simulate parallax for the left eye, we shift each pixel based on its depth: black areas shift the maximum distance to the left, white areas shift maximal distance to the right, grayscale pixels shift proportionally. For the right eye, the shift direction is reversed.

## GET THE DEPTH DATA

The first task is to get depth data from the portrait image on your iPhone: create an `AVDepthData` from auxiliary info obtained via `CGImageSourceCopyAuxiliaryDataInfoAtIndex`. This involves creating a `CGImageSource` from the image's `CFData` or `CFURL`, then extracting a `CVPixelBuffer` for the depth map, which can be normalized and converted to a grayscale image for visualization or storage.

```swift
static func getDepthImage(imageData: Data) -> CIImage {
    let source = CGImageSourceCreateWithData(imageData as CFData, nil)!
    let auxDataInfo = CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeDisparity) as? [AnyHashable : Any]!
    var depthData = try! AVDepthData(fromDictionaryRepresentation: auxDataInfo)
    if depthData.depthDataType != kCVPixelFormatType_DisparityFloat32 {
        depthData = depthData.converting(toDepthDataType: kCVPixelFormatType_DisparityFloat32)
    }
    let pixelBuffer = depthData.depthDataMap
    pixelBuffer.normalize() // custom extension method
    let ciDepthImage = CIImage(cvPixelBuffer: pixelBuffer)!
    return ciDepthImage
}
```

## PROCESS IMAGE'S PIXELS

Thinking about pixel buffers, the approach involves iterating row by row, computing pixel shifts according to depth, and writing shifted pixels into a result buffer, handling overlaps by choosing the pixel with the closest depth, and filling gaps with background color when needed.

```swift
currentColor = imageBuffer[currentIndex]
//1
relativeShift = depthBuffer[currentIndex] / 255 / 2 - 1 //2
currentPixelShift = maxShift * relativeShift
//3
resultBuffer[currentIndex + currentPixelShift] = currentColor //4
```

Issues encountered:
- Multiple pixels might land on the same spot; the brightest pixel (closest depth) is kept.
- Gaps appear due to varying shifts; these are fixed by filling with background color.

## IDEA #1 - SHIFTING LOOP AND FIXING LOOP

Loop through pixels, shift them, and store depth info to decide whether to overwrite existing pixels based on depth. Fix gaps in a subsequent pass by propagating previous pixel colors.

## IDEA #2 - FIX GAPS IN ONE LOOP

Optimize by fixing gaps during the shift loop: detect when the shift difference exceeds the last, and fill in the gap with the last used color, moving in the correct direction based on shift sign.

## IDEA #3 - PROCESS FRONT PIXEL LAST

By processing from the object in front last (e.g., from right to left), the nearest pixels overwrite farther ones, simplifying processing. Gaps are filled with current pixel color, avoiding the need for complex gap filling.

Improvements consider precomputing a depth map buffer and balancing performance, noting that large shifts slow down processing significantly—impractical for video.

## FINAL SOLUTION WITH CIFILTER

Create a custom `CIFilter` with `CIKernel` or Metal shader code. Due to deprecation of `makeKernels`, rewrite the kernel in Metal Shading Language, compile it into a `.metallib` file, and load via `CIKernel` initializer:

```swift
class CIStereoShiftMetal: CIFilter {
    @objc dynamic var inputImage: CIImage!
    @objc dynamic var inputDepthImage: CIImage!
    @objc dynamic var inputIncrement: NSNumber!
    @objc dynamic var inputMaxShift: NSNumber!
    private let kernel: CIKernel = {
        let url = Bundle.main.url(forResource: "default", withExtension: "metallib")!
        let data = try! Data(contentsOf: url)
        return try! CIKernel(functionName: "stereoImageFromDepth", fromMetalLibraryData: data)
    }()
    override var outputImage: CIImage? {
        guard let img = inputImage, let depthImg = inputDepthImage, img.extent == depthImg.extent else { return nil }
        let imageSampler = CISampler(image: img)
        let depthSampler = CISampler(image: depthImg)
        return kernel.apply(extent: img.extent, roiCallback: { _, rect in rect }, arguments: [imageSampler, depthSampler, inputIncrement, inputMaxShift])
    }
}
```

The kernel code uses Metal language, with syntax adjusted (`float2`, `float4`, etc.). The full process involves subclassing `CIFilter`, specifying input parameters, and overriding `outputImage` to apply the kernel with upscaled depth data for stereo pair generation.

### Example function to create stereo pair:

```swift
static func createStereoPairFromImageAndDepthMask(image: CIImage, depthImage: CIImage) -> (leftImage: CIImage, rightImage: CIImage) {
    let upsampleFilter = CIFilter(name: "CIEdgePreserveUpsampleFilter")!
    upsampleFilter.setValue(depthImage, forKey: "inputSmallImage")
    upsampleFilter.setValue(image, forKey: kCIInputImageKey)
    let upsampledDepthImage = upsampleFilter.outputImage!
    let shiftF: Double = 0.01
    let increment = 1 / Double(image.extent.width)
    let shiftFilter = CIStereoShiftMetal()
    shiftFilter.setValue(image, forKey: "inputImage")
    shiftFilter.setValue(upsampledDepthImage, forKey: "inputDepthImage")
    shiftFilter.setValue(NSNumber(value: increment), forKey: "inputIncrement")
    shiftFilter.setValue(NSNumber(value: shiftF), forKey: "inputMaxShift")
    let leftImage = shiftFilter.outputImage!
    shiftFilter.setValue(NSNumber(value: -shiftF), forKey: "inputMaxShift")
    let rightImage = shiftFilter.outputImage!
    return (leftImage, rightImage)
}
```

## WE'RE DONE

Finally, we can generate a stereo image pair from a 2D image with its depth map in a fraction of a second, thanks to Metal and GPU acceleration. I've integrated this into my app, [3D Photo](http://itunes.apple.com/app/id1364981409?ref=strv.ghost.io), enabling users to create, view, and export stereoscopic images in various formats. Search for "stereo photo maker" in the App Store.

Hopefully, this inspires you to explore depth maps and Metal-based CIFilters further. I wanted to write this guide to encourage experimentation — inspiring others makes me very happy.

[We're hiring](https://www.strv.com/careers?ref=strv.ghost.io#current-openings)

---

*Don't miss anything*