Android rendering slowdown for 3 minutes then super smooth

[Tinfoil_Trousers]

Hello,

I've inherited a webrtc client and I'm doing 1-way video from a camera. The original client uses an old customized version of webrtc (pre-h264) and exoplayer. I'm updating the client to use "vanilla" webrtc and dropping exoplayer. I'm running in to a strange problem:

Things render great for a few seconds and then the rendering frame-rate drops down to something like 1 frame every 4-5 seconds. After ~3 minutes (consistently) something happens and I get >10 fps playback for as long as I let it play.

I have added logging to the WebRTC library and I can see that in the generic_decoder.cc frames are getting decoded and added to the ring buffer at the correct rate, but the "decoded" callback that reads things out of the ring buffer and sends to the renderer is what slows down to many seconds between calls. By the time it does get called, the frame it is looking for is long gone, causing very long periods of frozen video. Again, decoded frames are being added at the correct rate.

On the app side I'm using a TextureView that I use to create a SurfaceEglRenderer that is added to the mediaTrack as the VideoSink. I feel like there must be a threading issue with how I'm setting things up because if I create a VideoSink wrapper class and force the "onFrame()" call on the main thread I pretty much always get smooth video. But this approach seems wrong, and I end up getting an "Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 26171 (EvoEglRenderer), pid 25401 (enceapp_android)" error after a few minutes of streaming (sometimes as long as 20 minutes).

The RTC demo app adds a renderer with "addRenderer" to the PeerConnection object but that approach isn't available anymore in the version I'm using. I'll be eternally grateful if anyone has some advice for things I can look regarding this issue.

Thank you!

[Tinfoil_Trousers]

To add a bit more information:

I'm still trying to stream H264

I have the same issue on all tested devices, modern and old.
I have the same issue on multiple releases of WebRTC; I'm sure it is an issue with my integration.

I am attaching a log (test_output.txt) that shows the reported frames rendered and other stats over time. You will see that it starts okay, drops to almost nothing for some period of time, and then all of the sudden works great!

I have also attached a log showing the full stacktrace of the low-level Signal 11 crash (crash_snippet.txt)

[Christoph Eggert]

Any chance to see more of what you are doing? Very hard to say much with the information we have, most likely too many sinks are added or something along those lines. Are you using the same client for sending and receiving, so is the call Android<->Android with the same app? Are you using unified plan or plan b? If you are on unified plan, did you fully change your code to only listen to the onTrack feedback, rather than also using the legacy methods? (perhaps you are adding tracks multiple times or something along those lines). The more you can show or explain from the solution, the better.

And out of curiosity since you mentioned H264: Does this not happen in VP8 or did you just not test that?

[Tinfoil_Trousers]

Christoph, thanks so much for responding. Here are some additional details:

There is a sending device, a camera running custom firmware (I did not write) that can only do H.264.The android side I have set up only to receive a stream, not send one. In the onTrackAdded callback is when I add the video sink call:

        @Override
        public void onAddTrack(RtpReceiver rtpReceiver, MediaStream[] mediaStreams) {
            Log.d(TAG, "onAddTrack: new track added." + log);
            if (mediaStreams.length > 0 && mediaStreams[0].videoTracks.size() > 0) {
                Log.d(TAG, "onAddTrack: Adding video sink");
                mediaStreams[0].videoTracks.get(0).addSink(videoSink);
            }
        }

I believe that is part of the legacy callbacks? I have also tried doing a similar thing in onAddStream but the results are the same. From what I can tell there is only one video sink being added to the media stream (the above logs only happen once for a connection and streaming session).

I've been playing around with the rendering views a bit to see if something is related to that but I have not been successful figuring that out. I'll explain this as best I can because I have a feeling something here may be the key:

Layout contains a TextureView. I add a surfaceTextureListener and when the surface is ready (callback) I send that to my ProxyVideoSink where I create a target:

private class ProxyVideoSink implements VideoSink {

    private SurfaceEglRenderer target = null;

    public synchronized void setSurfaceTexture(SurfaceTexture surfaceTexture) {
        if (surfaceTexture != null) {
            if (target == null) {
                target = new SurfaceEglRenderer("Test");
                target.init(eglContext, EglBase.CONFIG_PLAIN, new GlRectDrawer());
                target.createEglSurface(surfaceTexture);
            }
        }
    }

    @Override
    public synchronized void onFrame(VideoFrame videoFrame) {
        if (target != null) {
            target.onFrame(videoFrame);
        }
    }
}

If I change the "onFrame" callback to the following:

@Override
public synchronized void onFrame(VideoFrame videoFrame) {
    executor.execute(() -> {
        if (target != null) {
            target.onFrame(videoFrame);
        }
    });
}

...then I get smooth playback. But as mentioned above I get a low level crash from EglRenderer at some point. And the "onFrame()" call itself has no delay even without the executor - if I put a log statement immediately above and below that they both log essentially at the same time, even when I'm getting a single frame every few seconds. But the next "onFrame()" is delayed.

And the thing that really boggles my mind is that when I run it without that modification to force it on the executor thread, it is bad for a long time and then *poof*! It just because super smooth at some point. I can't explain what happens there, but once it starts working after the couple minutes of bad-ness it stays good indefinitely.

I've analyzed the stream with wireshark and packet capture and the data ingest rate is normal. As I've mentioned, I added logging to WebRTC itself and I can see that the frames are decoded and added to the ring buffer (from generic_decoder.cc) at the normal rate. It is just whatever is supposed to be triggering the "decoded" method to pull the frame out of the ring buffer and add it to the renderer is what is not happening.

[Christoph Eggert]

Hm unfortunately I don't have any experience with that approach - I'm actually not sure if calling "createEglSurface" is the way to go in this example. I personally use the SurfaceViewRenderer instead, which seems to handle this without doing the createEglSurface call. If all else fails and noone else is able to help, it might be worth a try switching to a SurfaceViewRenderer instead. In my case, I use a Proxy for the sink as well which has a SurfaceViewRenderer as the target which I directly added to a layout. Perhaps this is less prone for errors since it's more high level. 

Also, my proxy does not have the synchronized key at its onFrame method, not sure if that might be relevant?

[Tinfoil_Trousers]

Thank again Christoph. I've attempted what you suggest without any difference.

However, I do have an interesting update about this - my camera received a firmware update last night and I can no longer reproduce this issue on it! I'm working with our firmware guys to figure out which change they made could be causing / fixing this issue on my end, but at this point it doesn't appear to be app-side.

Thank you again for all of your suggestions. They were helpful for me to double check things on my end.

[Christoph Eggert]

Well, as long as it's solved :). Please let us know in case you get a meaningful update about this. Cheers!

[Bo Zhou]

Hi all, is this problem being solved yet ?