[Android] Proper way to release inactive receiver sources?

[eggert.c...@googlemail.com]

Hey everyone,

let's assume we have a use-case of a call where videos are switched frequently. With Unified Plan, this now leads to an increasingly large SDP with as many mids as videos were previously shared and them just being marked as "inactive". Feeding the Android implementation of WebRTC with that, this leads to a lot of decoders that have been initialized, but never released.

This can be problematic on not-so-powerful devices. For example, HMDs like the Vuzix M400 or the RealWear HMT-1 will be unable to initialize decoders when reaching ~14 tracks in total (I suppose because all the egl stuff of inactive tracks is still loaded and they go out of memory).

My question is: Is it possible to somehow force a release here? Removing them from the SDP does not work as this leads to my favorite m-line consistency errors. I tried disposing/releasing tracks and transceivers without any luck as this either leads to errors when closing the peerconnection or it simply has no effect and still does not trigger "release" calls in the decoder.

I feel like I am missing the proper way to deal with this. As the tracks will never be there again, there should be somehow to release the decoders here and with that free up the egl resources etc., right? My only workaround solution at the moment is releasing the peer connection and creating a new one, but that's a bit ugly as it leads to an obvious connection loss.

Any advice/ideas appreciated, thanks!

[Henrik Boström]

Have you tried RTCRtpTransceiver.stop() (https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/sdk/android/api/org/webrtc/RtpTransceiver.java;l=234?q=RtpTransceiver.java&originalUrl=https:%2F%2Fcs.chromium.org%2F, https://w3c.github.io/webrtc-pc/#dom-rtcrtptransceiver-stop ?) It releases the resources of that m= section (in both directions, so make sure to have recvonly transceivers if you want to only do this for the receive side).

It also makes it so that this m= section can be recycled by future transceivers, ensuring the the SDP does not grow indefinitely.

There's also an experiment to make decoder initialization to prevent creating inactive decoders which hopefully gets rolled out soon.

[Henrik Boström]

It's also possible to re-use transceivers (i.e. recvonly to inactive and then back to recvonly and so on) instead of creating new transceivers every time videos switch. Or if it's a case of switching between front and back camera you don't even need to re-negotiate, you can simply call sender.replaceTrack()

[eggert.c...@googlemail.com]

Thanks Henrik! I tried it with stopping the transceiver, and while that definitely causes the decoder to be released, this unfortunately seems to lead to issues with the peer connection's setup. Whenever I stop the transceiver before applying the remote offer again, a new transceiver seems to be created. So for each stopped transceiver, the peerConnection.getTransceivers() has the first entry with isStopped = true and mid = null, and a second transceiver referencing the same track id with isStopped = false and mid = the actual number. Consequently any kind of track references are screwed up. These "ghost transceivers" also don't seem to go through the onTrack methods and are just silently created. Any idea what could cause this? All I'm doing is basically setting the original transceiver to inactive, stopping it and then going through the setOffer/createAnswer with the new content again.

[eggert.c...@googlemail.com]

...upon further investigation these "ghost transceivers" seem to be an old issue, I was on an April 2020 build for test comparisons and they don't appear on the newest versions. It seems like the "isStopped()" information changes after applying new SDPs so that old inactive tracks are marked as isStopped=false again, but based on current tests the native memory remains released either way. So it's looking good so far with that approach. Thanks again.

[eggert.c...@googlemail.com]

In case anyone stumbles upon this: Using the stop() method for the transceivers, I ran into followup issues where the connection was simply lost and didn't recover. A typical error was "Failed to setup RTCP mux". To avoid this you need to make sure you don't call stop() for the peer connection's very first transceiver with mid 0. I have no idea why that is the case, but putting an exception in for mid 0 it looks like the memory is cleared so that crashes are avoided and the connection stays active.