Convert a CGImage to MTLTexture without premultiplication

Question

I have a UIImage which I've previously created from a png file:

let strokeUIImage = UIImage(data: pngData)

I want to convert strokeImage (which has opacity) to an MTLTexture for display in an MTKView, but doing the conversion seems to perform an unwanted premultiplication, which darkens all the semitransparent edges.

My blending settings are as follows:

pipelineDescriptor.colorAttachments[0].isBlendingEnabled = true
pipelineDescriptor.colorAttachments[0].rgbBlendOperation = .add
pipelineDescriptor.colorAttachments[0].alphaBlendOperation = .add
pipelineDescriptor.colorAttachments[0].sourceRGBBlendFactor = .one
pipelineDescriptor.colorAttachments[0].sourceAlphaBlendFactor = .one
pipelineDescriptor.colorAttachments[0].destinationRGBBlendFactor = .oneMinusSourceAlpha
pipelineDescriptor.colorAttachments[0].destinationAlphaBlendFactor = .oneMinusSourceAlpha

I've tried two methods of conversion:

let stampTexture = try! MTKTextureLoader(device: self.device!).newTexture(cgImage: strokeUIImage.cgImage!, options: nil)

and the more elaborate dataProvider-driven method:

let image = strokeUIImage.cgImage!
    let imageWidth = image.width
    let imageHeight = image.height
    let bytesPerPixel:Int! = 4
    let rowBytes = imageWidth * bytesPerPixel        

    let texDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm_srgb,
                                                              width: imageWidth,
                                                              height: imageHeight,
                                                              mipmapped: false)

    guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else { return }

    let srcData: CFData! = image.dataProvider?.data
    let pixelData = CFDataGetBytePtr(srcData)

    let region = MTLRegionMake2D(0, 0, imageWidth, imageHeight)
    stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))

both of which yield the same unwanted premultiplied result.

The latter I tried, as there were some posts suggesting that the old swift3 method CGDataProviderCopyData() extracts raw pixel data from the image which is not premultiplied. Sadly, the equivalent:

let srcData: CFData! = image.dataProvider?.data

does not seem to do the trick. Am I missing something?

Any pointers would be appreciated.

Answer 1

After much experimenting, I've come to a solution which addresses the pre-multiplication issue inherent in CoreGraphics images. Thanks to Warren's tip regarding using an Accelerate function (vImageUnpremultiplyData_ARGB8888 in particular), I thought, why not build a CGImage using vImage_CGImageFormat which will allow me to play with the bitmapInfo setting that specifies how to interpret alpha...The result is not perfect, as demonstrated by the image attachment below:

Somehow, in the translation the alpha values are getting punched up slightly, (possibly the rgb as well, but not significantly). By the way, I should point out that the png pixel format is sRGB, and the MTKView I'm using is set to MTLPixelFormat.rgba16Float (app requirement)

Below is the full metalDrawStrokeUIImage routine I implemented. Of particular note is the line:

bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue)

which essentially unassociates the alpha (I think) without calling vImageUnpremultiplyData_ARGB8888. Looking at the resulting image certainly looks like an un-premultiplied image...

Lastly, to get back a premultiplied texture on the MTKView side, I let the fragment shader handle the pre-multiplication:

fragment float4 premult_fragment(VertexOut interpolated [[stage_in]],
                                 texture2d<float> texture [[texture(0)]],
                                 sampler sampler2D [[sampler(0)]]) {
  float4 sampled = texture.sample(sampler2D, interpolated.texCoord);

  // this fragment shader premultiplies incoming rgb with texture's alpha

  return float4(sampled.r * sampled.a,
                sampled.g * sampled.a,
                sampled.b * sampled.a,
                sampled.a );


} // end of premult_fragment

The result is pretty close to the input source, but the image is maybe 5% more opaque than the incoming png. Again, png pixel format is sRGB, and the MTKView I'm using to display is set to MTLPixelFormat.rgba16Float . So, I'm sure something is getting mushed somewhere. If anyone has any pointers, I'd sure appreciate it.

Below is the rest of the relevant code:

func metalDrawStrokeUIImage (strokeUIImage: UIImage, strokeBbox: CGRect) {

self.metalSetupRenderPipeline(compStyle: compMode.strokeCopy) // needed so stampTexture is not modified by fragmentFunction

let bytesPerPixel = 4
let bitsPerComponent = 8

let width = Int(strokeUIImage.size.width)
let height = Int(strokeUIImage.size.height)

let rowBytes = width * bytesPerPixel
//
let texDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm_srgb,
                                                             width: width,
                                                             height: height,
                                                             mipmapped: false)
guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else { return }

//let cgImage: CGImage = strokeUIImage.cgImage!
//let sourceColorSpace = cgImage.colorSpace else {
guard
  let cgImage = strokeUIImage.cgImage,
  let sourceColorSpace = cgImage.colorSpace else {
    print("Unable to initialize cgImage or colorSpace.")
    return
}

var format = vImage_CGImageFormat(
  bitsPerComponent: UInt32(cgImage.bitsPerComponent),
  bitsPerPixel: UInt32(cgImage.bitsPerPixel),
  colorSpace: Unmanaged.passRetained(sourceColorSpace),
  bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue),
  version: 0, decode: nil,
  renderingIntent: CGColorRenderingIntent.defaultIntent)
var sourceBuffer = vImage_Buffer()

defer {
  free(sourceBuffer.data)
}

var error = vImageBuffer_InitWithCGImage(&sourceBuffer, &format, nil, cgImage, numericCast(kvImageNoFlags))

guard error == kvImageNoError else {
  print ("[MetalBrushStrokeView]: can't vImageBuffer_InitWithCGImage")
  return

}

//vImagePremultiplyData_RGBA8888(&sourceBuffer, &sourceBuffer, numericCast(kvImageNoFlags))


// create a CGImage from vImage_Buffer
var destCGImage = vImageCreateCGImageFromBuffer(&sourceBuffer, &format, nil, nil, numericCast(kvImageNoFlags), &error)?.takeRetainedValue()


guard error == kvImageNoError else {
  print ("[MetalBrushStrokeView]: can't vImageCreateCGImageFromBuffer")
  return
}

let dstData: CFData = (destCGImage!.dataProvider!.data)!
let pixelData = CFDataGetBytePtr(dstData)

destCGImage = nil

let region = MTLRegionMake2D(0, 0, Int(width), Int(height))
stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))
let stampColor = UIColor.white
let stampCorners = self.stampSetVerticesFromBbox(bbox: strokeBbox)
self.stampAppendToVertexBuffer(stampLayer: stampLayerMode.stampLayerFG, stampCorners: stampCorners, stampColor: stampColor)
self.metalRenderStampSingle(stampTexture: stampTexture)
self.initializeStampArray() // clears out the stamp array so we always draw 1 stamp at a time


} // end of func metalDrawStrokeUIImage (strokeUIImage: UIImage, strokeBbox: CGRect)