Getting Depth Data from UIImagePickerController

Question

I am trying to get the depth data associated with an image in the PhotoLibrary. I can get the image, and the URL, but I can't seem to get the aux data associated with it. The call to CGImageSourceCreateWithURL returns a source, but the call to CGImageSourceCopyAuxiliaryDataInfoAtIndex returns nil for both kCGImageAuxiliaryDataTypeDisparity and kCGImageAuxiliaryDataTypeDepth. Is there something I am missing here?

func imagePickerController(_ picker: UIImagePickerController,    didFinishPickingMediaWithInfo info: [String : Any]) {
    let image = info[UIImagePickerControllerOriginalImage]
    let url = info[UIImagePickerControllerImageURL]
    print("url=",url)

    guard let source = CGImageSourceCreateWithURL(url as! CFURL, nil) else {
        return
    }
    guard let auxDataInfo = CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeDisparity) as? [AnyHashable : Any] else {
        return
    }
}

Answer 1

The image URL supplied by UIImagePickerController does not include any of the metadata associated with depth. To get this information, you must access the PHAsset using the PhotoBook API.

First, import the API:

import Photos

Before you display your image picker, request user access to the photo book. You'll need to add an info dictionary key for Photo Library Usage for this to work:

    switch PHPhotoLibrary.authorizationStatus() {
    case .notDetermined:
        PHPhotoLibrary.requestAuthorization { (status) in
            if status == .authorized {
                DispatchQueue.main.async {
                    // Display image picker here
                }
            }
        }
    case .authorized: // Display image picker here
    case .denied, .restricted: // Display appropriate error here
    }

Now, in your image picker delegate you can do this:

    if let asset = info[.phAsset] as? PHAsset {
        PHImageManager.default().requestImageData(for: asset, options: nil) { (imageData, dataType, orientation, info) in
            let url = info?["PHImageFileURLKey"] as? URL
            // Pass this URL to your existing code.
        }
    }

Note that the file may contain depth or disparity information. You can convert between these easily enough, but you may need to check which one you have using CGImageSourceCopyProperties(). Also look out for the new supplementary depth data, kCGImageAuxiliaryDataTypePortraitEffectsMatte, which gives a much higher resolution mask for just the subject person in portrait images and is great for doing greenscreen-style effects.

Answer 2

I struggled with this for a whole day! I finally figured it out, though, after watching the first half of the WWDC Video titled "Editing Images With Depth."

My problem was using a URL for the image that was not from the PHAsset.

Here is the link:

LINK TO WWDC VIDEO

If you don't feel like watching it, check out this function that I wrote that does pretty much exactly what is done in the video.

You have to provide the function the [info] that is returned from the DID_FINISH_PICKING_IMAGE_WITH_INFO function from the UIImagePickerDelegate.

Before using this function - note that is actually doesn't work! It is great to look at though, because it shows the steps clearly. But due to asynchronous behavior, the function will always return nil before it has a chance to set the local depth variable to the AVDepthData.

My solution was to break the function apart and use Grand Central Dispatch to create a Dispatch Group, enter it, retrieve the imageURL from the PHAsset, and then leave the Dispatch Group. Upon leaving the Dispatch Group, the DispatchGroup.NOTIFIED function then proceeded with the rest of the process.

I hope this helps!!!

     func returndepthdata(usingimageinfo: [UIImagePickerController.InfoKey : Any]) -> AVDepthData? {
          var depthdata: AVDepthData! = nil

          if let photoasset = usingimageinfo[.phAsset] as? PHAsset {
               let input = photoasset.requestContentEditingInput(with: nil, completionHandler: { (input, info) in
                    if let imageurl = input?.fullSizeImageURL {
                         if let source = CGImageSourceCreateWithURL(imageurl as CFURL, nil) {
                              if let imageproperties = CGImageSourceCopyProperties(source, nil) {
                                   if let disparityinfo = CGImageSourceCopyAuxiliaryDataInfoAtIndex(source, 0, kCGImageAuxiliaryDataTypeDisparity) {
                                        if let truedepthdata = try? AVDepthData(fromDictionaryRepresentation: disparityinfo as! [AnyHashable : Any]) {
                                             depthdata = truedepthdata
                                        }
                                   }
                              }
                         }
                    }
               })
          }
          return depthdata
     }