Firefox 52 receiving unexpected hangup events

[Ancor Gonzalez Sosa]

Jangouts seems to be incompatible with very recent versions of Janus Gateway. I bisected the problem to this commit
https://github.com/meetecho/janus-gateway/commit/db66959e4e462a3185f8f64ebd2fb5e5d014f0b2

After a few seconds (maybe less than a second) in the room with Firefox 52.1.1, I can see a "Got RTCP BYE on stream..." in Janus, so the browser get a hangup and everything goes downhill from there.

With any Janus version prior to that commit, Jangouts works just fine.

Before you ask, none of the commits introduced in master until the time of writing this has solved the issue (I see some commits related to hangup after the offending one).

The problem is rather easy to reproduce (if we consider installing Jangouts to be easy enough).

Any clue what Firefox 52 + Jangouts are doing that can be interpreted as RTCP BYE?

Thanks

https://github.com/jangouts/jangouts/

[Lorenzo Miniero]

If you see that, Firefox sends an RTCP BYE which closes the PeerConnection. We had to add that because apparently Firefox was not sending DTLS alerts anymore when things were done, and so we had no way to detect a PeerConnection was destroyed if signalling didn't tell us explicitly. RTCP BYE provides an alternative to DTLS alert.

Firefox seems to work just fine for me, so not sure what's happening in your case. I only have Firefox 53 and 55 installed, though.

L.

[Marcel]

Hi Anor & Lorenzo

I'm running into the same problem with Firefox 52

For us it's quit interesting to get Firefox 52 working. Because Firefox 52 is the current ESR (Extended Support Release). Which is a froze version of Firefox which only receive security fixes for about a year and has some extra features for mass deployments.

So, quit interesting for big companies and some of our clients are using Firefox 52 ESR 

Does one of you have a solution for this problem or some more information?

Kind Regards,

Marcel

[Marcel]

Hi,

I took al look at the janus code. Respect for building this code!

When I remove this block from the code https://github.com/meetecho/janus-gateway/blob/master/ice.c#L2240-#L2244

I got Firefox 52 running again. The only thing I tested was adding a few users to the videoroom. So I'm not 100% I hit the use case of the RTCP BYE. 

Lorenzo mayby you can tell if I hit the RTCP BYE use case?

Tested successfully with (all on a mac):

Chrome 59.0.3071.115

Firefox 52.2.1 ESR

Firefox 53.0.3

Firefox 54.0.1 

Janus 0.2.3

Kind Regards

Marcel

On Wednesday, 7 June 2017 07:37:29 UTC+2, Ancor Gonzalez Sosa wrote:

[Lorenzo Miniero]

I don't think the lines you linked to are correct, removing those would break the compilation.

L.

[Marcel]

You're right Lorenzo.

https://github.com/meetecho/janus-gateway/blob/master/ice.c#L2230-L2234

I removed the if statement with the janus_rtcp_has_bye call and everything in the if statement

But I'm not sure if I tested correctly for RTCP BYE use case.

Kind Regards

Marcel

[Lorenzo Miniero]

So you're basically not reacting to an incoming BYE. We added that because Firefox does not always send a DTLS alert, and a BYE gives us an alternative to detect the PeerConnection has gone. Without that you risk keeping a PeerConnection alive when it really isn't, so I'm not going to remove it from master. Not sure why for Firefox 52 you get this behaviour.

L.

[Marcel]

Hi Lorenzo,

You're right. This isn't the right way to fix it. Now I understand the case of the BYE in Firefox a bit better.

I the mean time I have done some more debugging with Janus and Firefox 52.

I noticed the wrong BYE in Firefox 52 is always in the first receiver report (201) compound RTCP packet and only occurs on video stream (based on the SSRC) 

[Thu Aug  3 13:40:35 2017] [5137722984666930]  Got an RTCP packet (bundled stream)!

[Thu Aug  3 13:40:35 2017] [5137722984666930] Got RTCP BYE on stream 1 (component 1), closing...

[Thu Aug  3 13:40:35 2017] [5137722984666930] Incoming RTCP, bundling: this is video (remote SSRC: video=2898910065, audio=987153488, got 2898910065)

[Thu Aug  3 13:40:35 2017]    Parsing compound packet (total of 68 bytes)

[Thu Aug  3 13:40:35 2017]      #1 RR (201)

[Thu Aug  3 13:40:35 2017]        RTCP PT 201, length: 8 bytes

[Thu Aug  3 13:40:35 2017]      #2 SDES (202)

[Thu Aug  3 13:40:35 2017]        -- SSRC: 2898910065

[Thu Aug  3 13:40:35 2017]        RTCP PT 202, length: 52 bytes

[Thu Aug  3 13:40:35 2017]      #3 BYE (203)

[Thu Aug  3 13:40:35 2017]        -- SSRC: 2898910065

[Thu Aug  3 13:40:35 2017]        RTCP PT 203, length: 8 bytes

[Thu Aug  3 13:40:35 2017]   End of compound packet

When the connections closes Firefox sends 2 receiver report (201) compound RTCP packet with a bye. One for video and one for audio, based on the SSRC. 

Tested with Firefox 52 ESR, 53 and 54

[Thu Aug  3 13:40:45 2017] [5137722984666930]  Got an RTCP packet (bundled stream)!

[Thu Aug  3 13:40:45 2017] [5137722984666930] Got RTCP BYE on stream 1 (component 1), closing...

[Thu Aug  3 13:40:45 2017] [5137722984666930] Incoming RTCP, bundling: this is video (remote SSRC: video=2898910065, audio=987153488, got 2898910065)

[Thu Aug  3 13:40:45 2017]    Parsing compound packet (total of 68 bytes)

[Thu Aug  3 13:40:45 2017]      #1 RR (201)

[Thu Aug  3 13:40:45 2017]        RTCP PT 201, length: 8 bytes

[Thu Aug  3 13:40:45 2017]      #2 SDES (202)

[Thu Aug  3 13:40:45 2017]        -- SSRC: 2898910065

[Thu Aug  3 13:40:45 2017]        RTCP PT 202, length: 52 bytes

[Thu Aug  3 13:40:45 2017]      #3 BYE (203)

[Thu Aug  3 13:40:45 2017]        -- SSRC: 2898910065

[Thu Aug  3 13:40:45 2017]        RTCP PT 203, length: 8 bytes

[Thu Aug  3 13:40:45 2017]   End of compound packet

[Thu Aug  3 13:40:45 2017] [5137722984666930]  Got an RTCP packet (bundled stream)!

[Thu Aug  3 13:40:45 2017] [5137722984666930] Got RTCP BYE on stream 1 (component 1), closing...

[Thu Aug  3 13:40:45 2017] [5137722984666930] Incoming RTCP, bundling: this is audio (remote SSRC: video=2898910065, audio=987153488, got 987153488)

[Thu Aug  3 13:40:45 2017]    Parsing compound packet (total of 68 bytes)

[Thu Aug  3 13:40:45 2017]      #1 RR (201)

[Thu Aug  3 13:40:45 2017]        RTCP PT 201, length: 8 bytes

[Thu Aug  3 13:40:45 2017]      #2 SDES (202)

[Thu Aug  3 13:40:45 2017]        -- SSRC: 987153488

[Thu Aug  3 13:40:45 2017]        RTCP PT 202, length: 52 bytes

[Thu Aug  3 13:40:45 2017]      #3 BYE (203)

[Thu Aug  3 13:40:45 2017]        -- SSRC: 987153488

[Thu Aug  3 13:40:45 2017]        RTCP PT 203, length: 8 bytes

[Thu Aug  3 13:40:45 2017]   End of compound packet

Now it works like if there is one BYE received the PeerConnection is closed. If this is changed to when both the BYE for the audio and video is received and then the PeerConnection is closed.

So, when firefox 52 ESR send the wrong bye for the video in the beginning of the connections the PeerConnection won't be closed because there is no bye for the audio yet.

And when the connection is closed firefox send the the audio and video BYE packets and the PeerConnection will be closed.

I think this could be done by adding 2 booleans (one for audio and one for video) to the janus_ice_stream struct. Which keeps in account if a bye for the audio or video is received and if both are set the PeerConnection is closed

Have a nice holiday!

Kind Regards

Marcel

[Lorenzo Miniero]

BYEs can be lost, and so can alerts. Waiting for both may end up in never getting to the right status. The real issue is why you're getting a BYE at the beginning in the first place, or if it's indeed a a false positive (something we think is a BYE and really isn't). Firefox recently added a way to save decrypted traffic, so you may use that to check the PCAP for RTCP BYE: https://blog.mozilla.org/webrtc/debugging-encrypted-rtp-is-more-fun-than-it-used-to-be/

Apart from that, if it's 52 only, I'm personaly not interested in modifying Janus for that, as 52 is very far from being the stable version (it shouldn't even be around anymore). We might add that as an option in janus.cfg (e.g., ignore BYEs) but that would break other versions of Firefox (which IIRC don't sent DTLS alerts but only BYEs, which is why we added the feature in the first place).

L.

[Marcel]

Hi Lorenzo,

I hope you had a good vacation!

I did build a fix based on above. I seems to work for us for now. But lost BYEs could an issue.

https://github.com/marcelrouw/janus-gateway/tree/firefox-52

I wish you where right and 52 wan't used any more. But unfortunate this is not the case with 52. This is because 52 is an ESR release of Firefox and will be around until June next year. This is a special release for mass deployment. So interesting for big businesses.

Please take a look at this link:

https://www.mozilla.org/en-US/firefox/organizations/faq/

We see some of our customer are using it. Especially the big businesses.

Kind Regards

Marcel

[Lorenzo Miniero]

Can you check if this commit fixes it for you? Getting master will include this too:

https://github.com/meetecho/janus-gateway/commit/0c4aa227f24e452428e1181e6c771364f7a59f64

L.

[Marcel]

Hi Lorenzo,

Thank you for the fix!

Did some tests and it looks fine to me. Will do some more testing later on.

[Ancor Gonzalez Sosa]

El martes, 24 de octubre de 2017, 19:14:03 (UTC+2), Lorenzo Miniero escribió:

Can you check if this commit fixes it for you? Getting master will include this too:

https://github.com/meetecho/janus-gateway/commit/0c4aa227f24e452428e1181e6c771364f7a59f64

Unfortunately, not enough for me in my test setup: Firefox 51 (yes, it's also affected) on Linux in a local network. I had to multiply by four that timeout (from 500000 to 2000000) for it to work reliably. Of course, that opens the door to false positives. :(

There is no way to detect that broken BYE in a more reliable way than a timeout?

[Lorenzo Miniero]

Actually, looks like browsers also send a BYE when an ssrc is replaced by another (e.g., camera switch achieved via removeTrack addTrack and renegotiation). This isn't something we support yet (renegotiation) but still means we'll have to figure something out. Waiting for both byes to arrive (audio and video ssrcs, for instance) won't work if you only have one audio or one video stream in the PC. At the same time, getting rid of the bye processing altogether brings us back to the missing alert scenario.

Try disabling byes interception in your setup to see if you can notice any wrong behaviour. I'll be abroad for one more week so nothing I can do until I come back.

L.

--
You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ancor Gonzalez Sosa]

El sábado, 18 de noviembre de 2017, 10:54:21 (UTC+1), Lorenzo Miniero escribió:

Actually, looks like browsers also send a BYE when an ssrc is replaced by another (e.g., camera switch achieved via removeTrack addTrack and renegotiation). This isn't something we support yet (renegotiation) but still means we'll have to figure something out. Waiting for both byes to arrive (audio and video ssrcs, for instance) won't work if you only have one audio or one video stream in the PC. At the same time, getting rid of the bye processing altogether brings us back to the missing alert scenario.

Try disabling byes interception in your setup to see if you can notice any wrong behaviour. I'll be abroad for one more week so nothing I can do until I come back.

I deployed a version with this patch some days ago in https://jangouts.tk and seems to be working ok (although the traffic is really low on that server).
https://build.opensuse.org/package/view_file/network:jangouts/janus-gateway/no_bye_handling.patch?expand=1

Today I deployed the same patched version in SUSE's internal server which is used quite intensively by people with different browsers/setups. So if something gets broken by that patch, I assume we will know soon. :-)

Cheers.

L.

To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

[Lorenzo Miniero]

Il giorno giovedì 23 novembre 2017 17:06:38 UTC+1, Ancor Gonzalez Sosa ha scritto:

El sábado, 18 de noviembre de 2017, 10:54:21 (UTC+1), Lorenzo Miniero escribió:

Actually, looks like browsers also send a BYE when an ssrc is replaced by another (e.g., camera switch achieved via removeTrack addTrack and renegotiation). This isn't something we support yet (renegotiation) but still means we'll have to figure something out. Waiting for both byes to arrive (audio and video ssrcs, for instance) won't work if you only have one audio or one video stream in the PC. At the same time, getting rid of the bye processing altogether brings us back to the missing alert scenario.

Try disabling byes interception in your setup to see if you can notice any wrong behaviour. I'll be abroad for one more week so nothing I can do until I come back.

I deployed a version with this patch some days ago in https://jangouts.tk and seems to be working ok (although the traffic is really low on that server).
https://build.opensuse.org/package/view_file/network:jangouts/janus-gateway/no_bye_handling.patch?expand=1

Today I deployed the same patched version in SUSE's internal server which is used quite intensively by people with different browsers/setups. So if something gets broken by that patch, I assume we will know soon. :-)

Any news on this? I guess I can put the BYE management in janus.cfg so that it can be disabled (but keeping it TRUE as it is now by default) and behave like your patch does.

L.

[Lorenzo Miniero]

Decided to drop the behaviour eventually, and only log RTCP BYE when it appears:

https://github.com/meetecho/janus-gateway/commit/b0b4bc6358f369b03c47bf546271d83ce8bf16df

L.

[Ancor Gonzalez Sosa]

El lunes, 27 de noviembre de 2017, 11:11:09 (UTC+1), Lorenzo Miniero escribió:

Il giorno giovedì 23 novembre 2017 17:06:38 UTC+1, Ancor Gonzalez Sosa ha scritto:

I deployed a version with this patch some days ago in https://jangouts.tk and seems to be working ok (although the traffic is really low on that server).
https://build.opensuse.org/package/view_file/network:jangouts/janus-gateway/no_bye_handling.patch?expand=1

Today I deployed the same patched version in SUSE's internal server which is used quite intensively by people with different browsers/setups. So if something gets broken by that patch, I assume we will know soon. :-)

Any news on this? I guess I can put the BYE management in janus.cfg so that it can be disabled (but keeping it TRUE as it is now by default) and behave like your patch does.

Nobody complained or reported any problem in one week. So I guess nobody miss the BYE management. :-) 

Cheers.

[Marcel]

A slightly different way of implementing BYE management as suggested before. To check on the two BYE's for audio and video only in the case if there is a video and audio stream available. Maybe based on the SDP negotiation.

Kind Regards

Marcel 

[Lorenzo Miniero]

We already got rid of it weeks ago. BYEs are also sent when you renegotiate and SSRCs are replaced, so we can't rely on them now that we do support renegotiations.

L.