EXC_BAD_ACCESS in RTCVideoEncoderH264 on MacOS

[Peter Bloomfield]

Hi folks,

I've recently taken over a project which uses WebRTC's native code on MacOS. It's reasonably simple, just capturing the screen and streaming it to a remote browser. Our code is based on RTCCameraVideoCapture, except it uses AVCaptureScreenInput to capture the screen in place of a camera device. However, we're seeing a fairly consistent bad access exception after the stream has been running for a short time (just a few seconds in many cases).

The bad access occurs in sdk/objc/components/video_codec/RTCVideoEncoderH264.mm, in a method called resetCompressionSessionIfNeededWithFrame. The problem is the line shown below, which eventually attempts to access a pixel buffer pool after it's been deallocated:

NSDictionary *poolAttributes =
        (__bridge NSDictionary *)CVPixelBufferPoolGetPixelBufferAttributes(_pixelBufferPool);

This seems to happen after the resolution of the video stream has increased. I think having a slightly unreliable network connection makes the issue more likely to happen. After the resolution change, the pixel buffer pool gets released and reallocated during a call to VTCompressionSessionEncodeFrame. However, RTCVideoEncoderH264 is still holding the old memory address in _pixelBufferPool. Next time it tries to use it, the bad access occurs.

I'm currently using a fairly simple fix to resolve the problem in our application. Near the top of RTCVideoEncoderH264::encode(), I've added this line to update the stored reference:

_pixelBufferPool = VTCompressionSessionGetPixelBufferPool(_compressionSession);

I suspect a better approach would be to avoid storing a reference to the pixel buffer pool at all. I think it should possibly be fetched from the compression session every time it's needed. There may be performance implications in doing that though.

Is anyone else able to confirm the issue please? I'm very new to MacOS so it's possible I've got something wrong.

Many thanks.

[Juan García]

Hi Peter,

I can confirm the issue on MacOS and I think it was introduced in this changeset: https://webrtc-review.googlesource.com/c/src/+/90403/7/sdk/objc/Framework/Classes/VideoToolbox/RTCVideoEncoderH264.mm . I have tried your solution and it works.

Many thanks for your solution.

[Peter Bloomfield]

Unfortunately, I encountered additional problems after working around this one. It seems like the Objective C SDK for WebRTC is only stable on iOS, not macOS.

I'm now using WebRTC's C++ code directly on macOS instead and it works well. It can be called from within an Objective C app without needing the SDK.

On Thursday, 26 March 2020 08:20:43 UTC, Rong shi wrote:

Hi,

I am now struggling on this problem at same line same error, but this time _pixelBufferPool has point to something. I am using 10.15.3 macos to build webrtc/72 farmework. 

[Kanon]

Hi Peter, 

Could you explain further how you solve the problem by using WebRTC's C++ code directly? I encountered the same problem but I have no idea how to fix it.

I'll appreciate it if you can show me the code.

Thanks a lot!!

在 2020年3月26日星期四 UTC+8下午5:44:36，Peter Bloomfield写道：

[Peter Bloomfield]

I didn't really solve the problem as such. I avoided it by reimplementing my whole program in C++. It doesn't use any Objective C at all (although you can mix Objective C and C++ code if you need to).

I'm sorry I can't share my code as it's part of a commercial product. I would recommend looking at the C++ example code which is included with WebRTC.

[Joey Huang]

Hi Juan,

Did you encounter any other issue with Peter's solution? Is this a known issue on MacOS? Has google fixed it in latest webrtc version?

Thanks in advance!

[Juan Garcia]

Hi,

the solution was working on M72 and if I'm not wrong it was included on codebase, as I can see looking at revision M90 in method resetCompressionSessionWithPixelFormat. I haven't tested it on this revision yet.

Regards.