RTCVideoFrameBuffer to CVPixelBuffer in iOS. nativeHandle function was closed! Any other options?

[Giraff Wonbat]

Hi guys,

I would like to modify captured frames/buffers under RTCCameraVideoCapturer. I found the git below is simple and effective using RTCVideoCapturerDelegate and capturer(_ capturer: RTCVideoCapturer, didCapture frame: RTCVideoFrame).

https://gist.github.com/lyokato/d041f16b94c84753b5e877211874c6fc

However, I cannot get CVPixelBuffer from RTCVideoFrame. I found Chromium says nativeHandle function to get pixelBuffer is no more available (pls see below link). Also, frame.buffer gives me RTCVideoFrameBuffer but it does not contain pixel buffer. Looking at my framework>headers>RTCVideoFrameBuffer.h, pixelBuffer property was also gone!

https://codereview.webrtc.org/2990253002

So my questions are:

1) Any efficient ways to convert back and forth btw RTCVideoFrame and CVPixelBuffer. Or I shouldn't?

2) Any other good scheme to modify captured buffer under RTCCameraVideoCapturer?

Thank you very much for any helps.

Regards, GW

[Nagamuni reddy]

Hey Giraff Wonbat,

1. You can convert  RTCVideoFrame into CVPixelBuffer using the following code: 

  RTCCVPixelBuffer* remotePixelBuffer = (RTCCVPixelBuffer *)frame.buffer;

  CVPixelBufferRef pixelBuffer = remotePixelBuffer.pixelBuffer;

    frame is of type : RTCVideoFrame.

2. To convert CVPixelBuffer back to RTCVideoFrame : 

- (instancetype)initWithBuffer:(id<RTCVideoFrameBuffer>)buffer

                      rotation:(RTCVideoRotation)rotation

                   timeStampNs:(int64_t)timeStampNs

use that instance method to convert CVPixelBuffer back to RTCVideoFrame.

Hope this information is helpful.

[Sanjay Kumawat]

How to manage RTCI420Buffers?

[sanjay kumawat]

RTCCVPixelBuffer* remotePixelBuffer = (RTCCVPixelBuffer *)frame.buffer;

Every frame is not a RTCCVPixelBuffer, how do you manage if the frame is of type RTCI420Buffer?

[Juan Giorello]

Hey Sanjay, did you manage to find a solution?

[Sanjay Kumawat]

not yet. Do you have any solutions?

[Juan Giorello]

I did find a workaround, which might not be the most performant one:

Using  the method (UIImage *)convert:(RTCVideoFrame *)frame from: https://gist.github.com/shalyf/69880f70ca9470a1e91440196cb7d127

And then:

    func buffer(from image: CIImage) -> CVPixelBuffer? {

        let attrs = [kCVPixelBufferMetalCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary

        var pixelBuffer : CVPixelBuffer?

        let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.extent.width), Int(image.extent.height), kCVPixelFormatType_32BGRA, attrs, &pixelBuffer)

        guard (status == kCVReturnSuccess) else {

            return nil

        }

        return pixelBuffer   

    }

If you are going to use it in the Metal framework, then, make sure to put kCVPixelBufferMetalCompatibilityKey: kCFBooleanTrue in the attrs dictionary.

One thing to notice is that in the emulator I received RTCI420Buffers but in the real device I got RTCCVPixelBuffer. Unfortunately, somehow the RTCCVPixelBuffer is not compatible with the Metal view I'm using along with some video effects I'm currently using.

Hope it's helpfull!