iOS: renderFrame is never called - how to debug

[Eddie Sullivan]

Hi.

I'm working on an iOS app using the latest Cocoapods version of the WebRTC SDK, 1.1.27828 in Swift

Audio and data channels connect and transmit correctly, but the remote video is not rendering. While I couldn't get AppRTCMobile to compile, I copied the logic of setting up the video capturer and renderer from there.

I created a simple wrapper around RTCMTLVideoView and I can see that renderFrame is never getting called.

It's very possible I got something wrong in the logic of setting up the video transceivers. My question is, how can I debug this? Is there a way to tell if video data is even being received? I tried enabling verbose logging but that wasn't helpful.

What are some things that could cause video to not be received?

Thanks.

-Eddie Sullivan

[Eddie Sullivan]

So I've added in stats collection, copied over from AppRTC, and I can see that video frames are sent and received and decoded. And the video output RTCMTLVideoView has added to the remote video track via the Swift version of "- (void)addRenderer:(id<RTCVideoRenderer>)renderer".  And yet renderFrame never gets called by the video receiver.

Any clues on where these video frames went missing along the way?

Here's the stats output:

CN 6ms | local->local/udp | (s)627Kbps | (r)584Kbps
VS (input) 192x144@20fps | (sent) 192x144@20fps
VS (enc) 580Kbps/600Kbps | (sent) 602Kbps/1.03Mbps | 4ms | H264
AvgQP (past 21 encoded frames) = 13
VR (recv) 192x144@17fps | (decoded)18 | (output)17fps | 530Kbps/0bps | 4ms
AS 36Kbps | opus
AR 37Kbps | opus | 158ms | (expandrate)0

Thanks!

[Neil Young]

Are you forced to use Swift or is there a chance to use Objc for this particular part?

I'm asking because that kind of issue sounds somehow familiar and I had a similar problem last week. Every sample code ou can find on the internet which deals with forwarding a captured frame to WebRTC is more or less like so:

let localVideoSource = self.peerConnectionFactory!.videoSource()
let videoCapturer = RTCVideoCapturer()
localVideoSource.capturer(videoCapturer, didCapture: videoFrame!)




The problem: That might have been true last year, but right now the captured frame does not arrive. 

What I did is to create my own MyRTCVideoCapturer in objc, wich was derived from RTCVideoCapturer. Then I called an exported function and forwarded the videoFrame via the Objective C delegate

- (void)didCaptureVideoFrame:(RTCVideoFrame*)videoFrame {

    [self.delegate capturer:self didCaptureVideoFrame:videoFrame];

}

That did work. As if Swift was not reaching the correct target function... Don't know

Sorry if OT.

[Eddie Sullivan]

Thanks for the reply. I have been wondering if perhaps it was an issue with the Swift/Objective C bridge. I noticed there are a few places where the Objective C sources lack the proper nullability annotations that make things work with Swift. I don't think that in particular would cause the problem I'm seeing, but it's a sign that Swift usage may not get as much testing.

The example code you provided deals with the local capture of video frames. I am fairly confident that that side of things is working. It's on the other side where the video gets lost -- somewhere after the decoder but before the renderer.

There's a very good chance I'm doing something wrong, but I can't figure out what -- that part of the interface seems quite simple:  I find the remote video track and call addRenderer on it, passing in my video view, which is RTCMTLVideoView (or a wrapper of it -- I've tried both ways).

I've checked all the obvious things I can think of: I am finding exactly one remote video track, as expected, nothing is getting prematurely deallocated, etc.

Thanks.

-Eddie

[Eddie Sullivan]

If I create a custom wrapper around the Decoder object I can get the decoded frames and pass them straight to the video renderer, and that works, although the timestamps are all zero and the rotation is not preserved. So there's a clue.

For now I can make it work like this, but it would be wonderful to be able to use the standard frame pipeline.

Thanks to anyone who read the thread.

-Eddie

[Eddie Sullivan]

To clarify, the timeStamp field is set in the frame buffer, but timeStampNs is 0. I don't know what are the different use cases for the two timestamps, but the renderer checks timeStampNs.

[Marián Dvorský]

I've had a problem with similar symptoms. I was hooking up the renderer in my RTCPeerConnectionDelegate's didAddReceiver function, like this:

(receiver.track! as! RTCVideoTrack).add(renderer)

However, the RTCVideoTrack was deallocated shortly after this call which caused the renderer link to be removed. I have to hold on to a reference to the video track to avoid this, like so:

self.videoTrack = receiver.track! as? RTCVideoTrack
self.videoTrack!.add(renderer)

This is likely a bug in the objc SDK, but this workaround worked well.

Marian

[X Hogan]

Thanks, save my life.