Broadcast extension chopped video

[Neil Young]

Hi all,

I'm having a very weird problem with the video quality in a broadcast extension using WebRTC latest CocoaPod version 1.1.27724 

Step 1: Created a ReplayKit broadcast extension

Step 2: The core component is the forwarding the received sample buffer to the webRTC core

This is done for every single sampleBuffer using this sequence:

            let pixelBuffer =  CMSampleBufferGetImageBuffer(sampleBuffer)

            let rtcpixelBuffer = RTCCVPixelBuffer(pixelBuffer: pixelBuffer!)

            

            let timeStampNs: Int64 = Int64(CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer)) * 1000000000)

            let videoFrame = RTCVideoFrame(buffer: rtcpixelBuffer, rotation: rotation, timeStampNs: timeStampNs)

            

            self.webRTCClient?.feedCapturer(videoFrame: videoFrame)

The "webRTCClient?.feedCapturer" is finally just satisfying this delegate of an RTCVideoCapture descendant:

- (void)didCaptureVideoFrame:(RTCVideoFrame*)videoFrame;

The format of the sample buffer is kCVPixelFormatType_420YpCbCr8BiPlanarFullRange, so YUV. Width is 886 and height 1920 on my iPhoneXS.

https://developer.apple.com/documentation/corevideo/1563591-pixel_format_identifiers/kcvpixelformattype_420ypcbcr8biplanarfullrange?language=objc

Step 3: Video quality and latency good at receiving side. Since the default connection between a Chrome browser and the native device is using VP8 codec, the memory limit of the extension (50 MB) is reached relatively quickly and the extension is kicked off the memory then w/o mercy by Apple, I switched to H.264 by favouring the H.264 codecs in the SDP

Step 4: Here I noticed a chopped out video at receiving side the first time. Since it is ONLY that bad on my iPhone XS but not on an iPhone 8, iPad 201 and iPad mini 2013 second generation up to now I thought, it would be due to a particular issue with either the iPhone XS H.264 hardware encoder or the driver.

Step 5: Tried to limit the memory load by using a cropping/scaling initialiser for the RTCVideoFrame

           let rtcpixelBuffer = RTCCVPixelBuffer(pixelBuffer: pixelBuffer!, adaptedWidth: 640, adaptedHeight: 480, cropWidth: 640, cropHeight: 480, cropX: 0 , cropY: 0)

In fact: The video on the receiving side is now exactly 640x480 cropped out as this rectangle from (0,0). But now I'm having this chopped out effect on both codecs, VP8 and H.264.

So my theory is now, that the effect is NOT caused by a special issue of the iPhone XS hardware, but by some internal cropping/scaling or other handling applied to the video buffer by the WebRTC core, which can be provoked similarly by using the cropping/scaling initialiser for RTCCVPixelBuffer. But I have no clue, why the hell this happens.

Here is a video of the effect seen on the remote side:

https://www.youtube.com/watch?v=ZpClOu5F8PE

[Neil Young]

Nobody?

[Neil Young]

OK, today I found, that there is a good certainty to run into the same issue (chopped out video) with both codecs. So I'm 100 % convinced, that WebRTC has a defect in scaling/cropping YUV buffers. But nobody cares. So then - what can I do? I filed a bug, I have offered help - no reaction.

[Philipp Hancke]

Be patient, I've seen issues that went for weeks without being looked at. Your issue hasn't even been triaged yet.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/85a51865-4988-4762-8bf6-a703a7f57834%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Neil Young]

The reason of the problem is in nativeAdaptOutputFormat, especially the cropping part.

If I scale the entire YUV buffer to 640/480 by using just another constructor for RTCCVPixelBuffer

            let w = Int32(CVPixelBufferGetWidth(pixelBuffer!))

            let h = Int32(CVPixelBufferGetHeight(pixelBuffer!))

            let rtcpixelBuffer = RTCCVPixelBuffer(pixelBuffer: pixelBuffer!, adaptedWidth:640, adaptedHeight: 480 cropWidth: w, cropHeight: h, cropX: 0, cropY: 0)

and OMIT the call to 

        videoSource.adaptOutputFormat(toWidth: 640, height: 480, fps: 30)

then the aspect ratio gets lost, but since the ENTIRE capture information is just scaled and nothing is cropped, then there is NO FLICKER (no chopped out video).  Whenever a cropX or cropX of != 0 comes in, the problems start.

Again: This issue is only observable on iPhone XS

To unsubscribe from this group and stop receiving emails from it, send an email to discuss...@googlegroups.com.