Chrome/Firefox/Licode REMB issue

[Jeremy Noring]

Problem: when streaming through licode, if Chrome and Firefox are both present, Firefox ultimately tanks Chrome's video quality by sending very low REMB values, causing the chrome stream to look poor.

Configuration:

1. Our version of Licode passes RR/SR and other related RTCP traffic through, changing SSRCs and payload types where necessary
2. Chrome SDP is munged with a b=AS line at 300 kbps

Here's what I know:

• Network is not an issue, I can reproduce this with 100% localhost setup; network is effectively unlimited, no loss, very little latency.
• If both clients are Chrome, no problem
• If both clients are Firefox, no problem
• Examining RTCP packets flowing through licode, the problem arises when Receiver Reports are allowed to flow through
• Disabling Chrome's b=AS handling seems to prevent firefox from tanking Chrome's bandwidth
• If Chrome is only sending and firefox only receiving, it's good.  If firefox is only sending and Chrome only receiving, it's good.  It's only when Firefox is both sending and receiving that stuff goes south.

I realize this is a pretty questionable use of RTCP; unfortunately, though, it's the situation I'm in.  Anyone ideas welcome.

[Jeremy Noring]

I think I got to the basic source of the problem.  It has to do with FEC.

Licode supports FEC, using fec_receiver_impl contained in the webrtc project (see https://github.com/ging/licode/blob/master/erizo/src/erizo/WebRtcConnection.cpp#L297).  A while ago I'd added code to licode to convert the RED payload type to the VP8 payload type Firefox was expecting, knowing that firefox didn't support FEC.

Unfortunately what I didn't realize is that the incoming sequence numbers include sequence numbers dedicated solely to FEC packets (which I see in the WebRTC project, it makes specific note not to NACK), so the resulting stream sent to firefox is missing sequence numbers entirely.  Firefox interprets this as packet loss, starts NACKing like crazy, and in response to the frequent NACKs Chrome tanks the bandwidth.

Does this sound like a reasonable explanation?

If yes, I see two options:

1. Disable FEC entirely

2. Maintain a sequence number mapping, and rewrite sequence numbers as they go out.  When I receive a NACK from Firefox, I have to translate sequence numbers before forwarding on to Chrome.  And translate the resulting RTX when it passes back through licode.

Any other ideas?

[Jeremy Noring]

Okay, it's more than FEC; I can still get in a situation where firefox causes a bitrate crash in Chrome.  I have not been able to isolate why this is.  I tried Firefox 45, hoping that the updated WebRTC codebase might correct this problem, but no such luck.  I can also reproduce with Jitsi videobridge (one person in Chrome, one in Firefox, open chrome://webrtc-internals, then wait....watch your available send bitrate crater after a minute or so), so it isn't a problem that only affects licode.

I can "correct" the problem by rewriting REMB serverside, although that has a whole host of issues I'd like to avoid.

[Jeremy Noring]

Talking to myself here :)

One more issue resolved: publishers to licode were sendrecv, receivers were recvonly; this caused publishers to send RR, which I think was giving Firefox recvonly fits.  At this point it's fairly stable, but I notice firefox is still _really_ aggressive about pulling the bitrate down and not recovering for quite some time, even when Chrome detects no such bandwidth issue.  Are there major differences in REMB between Firefox and Chrome that I should be aware of?

[Alvaro Gil]

Jeremy your updates and insights are very welcome and useful!

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/99dcc09d-8788-4af8-8330-52ebd522aa20%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Alvaro

[mkh]

Hi Jeremy,

I'm experiencing exactly the same issue, tested with Firefox 44, 45 and 49. For me it happens on "Chrome => Firefox". Could you find any fix for that?

[Jeremy Noring]

I have not.  Firefox/chrome interop still seems to result in quality in Chrome being absolutely tanked; this seems to happen in nearly every SFU I've tried.

[mkh]

Jeremy I read your post here: https://groups.google.com/d/msg/lynckia/vZdirnQ3kdQ/LCHfWOolDQAJ 

I believe it's Licode issue since it doesn't happen for me at least on one other SFU and there is no similar complaint from other SFUs.