Is there a way to stream a Metal Texture to Youtube on iOS?

Question

I was looking into replaykit here https://developer.apple.com/videos/play/wwdc2018/601/ and was interested in knowing whether it would be possible to stream the result of a Metal renderpass to youtube. The use case involves giving the user additional visuals that an audience should not be able to see. So I would start by rendering only the shared parts of my graphical scene first, stream THAT to youtube, and then finish rendering the user-only content. I think this would require VideoToolbox as well if replaykit does not accept custom video, which would be disappointing. I am trying to avoid writing my own network code.

After rendering, somehow I'd need to convert the first texture to a CVPixelBuffer, I think. Sort of the way it's described here though that's not quite right for my use case.

That's where I get kind of lost. I think I probably need to copy the pixel buffer as well since that probably gets overwritten after Metal renders a frame.

Insert some use of BlitEncoder somewhere.

May I have help getting started? The documentation is rather sparse, and I think it's unclear how to combine all these steps.

Answer 1

Looking at the API, I don't think it's possible to stream arbitrary content, at least not directly.

The API forces you to record the whole screen (while prompting user if he actually wants to do that). But you can post-process it afterwards using the handler you pass to the startCapture method. In that handler, however, you have accesss to CMSampleBuffer, and if it's a RPSampleBufferType.video, then you can grab your MTLTexture for the right timestamp and basically draw into it.

Here's a part where it gets really confusing. CVImageBuffer, CVPixelBuffer are just typealiases for CVBuffer.

To draw into CMSampleBuffer, you can grab CVImageBuffer from the CMSampleBuffer using its property imageBuffer, then CVPixelBufferLockBaseAddress it and then you have a linear bitmap, basically, in memory. From there, you just need to figure out a way to blit out the MTLTexture into this linear memory using the right formats, unlock the base address for the CVPixelBuffer and proceed to next one.