Janus, NACKS, and REMBs

[Pete Andrews]

Lorenzo, etc...

Can you (or anyone else) walk me through how Janus is handling NACKS from peers that are experiencing poor link conditions?

It's my understanding that upon seeing a certain # of NACKS from one side, it will call the slow_link function of the plugin (videocall for example) which will issue a REMB to the sender? so Janus is acting as a middleman to throttle the sender? will that cap be lifted ever? is there a timeout on it?

what i am seeing here in my lab is Janus getting a bunch of NACKs from one peer, a REMB is issued and the OTHER peer's bitrate goes to the floor (65k), never to return.

thanks!!

-pete

[Wilbert Jackson]

We see the same for the videoroom plugin. For some reason the lowbit rate condition as you describe happens if we set a config file based bitrate. If we do not set a config file bitrate, then we do not see the automatic drop to a very low bitrate. When we start seeing NACKS, or buffer to small messages, we manually adjust the cap lower, and observe that bandwidth and bytes sent are adjusted and the system runs at the adjusted bitrate. We have tested adjusting caps higher and lower and have not seen the very low bitrate issue occur. If go back and set a bitrate in the config file, no matter what rate, the rate goes to the floor (65k), never to return the same as you experience.

wilbert jackson

[Lorenzo Miniero]

It depends on what plugin you're using. Normally, Janus automatically sends retransmissions when receiving NACKs, and sends NACKs when incoming packets are missing. When too many of those are seen, a slow_link is notified to plugins, which can handle them if they want. In the echotest and videocall, this is done by halving the bitrate cap, using properly REMB feedback towards the browser; in particular, for the videocall plugin if a user has trouble receiving media and so is sending too many NACKs, its peer is notified and so its his bitrate that is reduced. In the videoroom, this is just notified at the application level, which might or might not want to do something about it (it currently doesn't in our demos). Same thing in the recorder plugin, which reduces the bitrate.

L.

[Pete Andrews]

yes - understood. 

but once the bitrate is halved via the REMB, will it ever climb again?

or is this something left to the developer to implement in the plugin? 

-pete

[Lorenzo Miniero]

It's up to developers to implement on the client side, whether that's in the browser (javascript) or in a server wrapping Janus: that's what the feedback is for. Once the reduced bitrate works fine, with no excessive NACKs, Janus doesn't notify plugins anymore, and so there's no component in there that tries and raise it again.

L.

[Pete Andrews]

ah ha!

insteresting. that expains why once Janus REMBs a guy, he never comes back.

got it. i'll look into implementing something on our videocall plugin then to lift that cap.

thanks Lorenzo!

[Lorenzo Miniero]

Now that I think about it, I think the echotest and videocall don't notify the application side about the slow link as the videoroom and record&play plugins do. As such, you might just see the bitrate drop without any feedback on why. I'll try and add this to future revisions of the plugin, so that the plugin can notify you of the issue and about the new bitrate it is forcing now.

L.

[Lorenzo Miniero]

Have a try with this latest commit:

https://github.com/meetecho/janus-gateway/commit/83e230eddb7fc45071dd18f05537226dc500c23d

This will notify your videocall user about a slow link, by also providing the new target bitrate, no matter what you tried to force it to previously.

L.

[Pete Andrews]

would it be appropriate to also add code to age the REMB sent, such that Janus would "lift the cap" once it's out of the condition where it's seeing more than 16 NACKs per 2 seconds (or whatever the window is)? Or is this getting in BWE territory that Janus doesn't need to be in? this is where i get a bit fuzzy on the "middle-man" role that the Janus Server is playing and how it's throttling the peers. In normal WebRTC peer to peer operation, the peers are handling this kind of thing themselves. No?

-pete

[Lorenzo Miniero]

Yes, but this is not peer-to-peer. The Janus core has no idea what happens with a PeerConnection when it reaches a plugin. It may end there (echo test, record&play), be attached to another PeerConnection (videocall) or to many (videoroom), or be fed with external media (streaming, SIP), and so on; that's plugin-specific, and opaque to the core. As such, there's nothing specific the core can do if not to handle/genertae NACKs and notify plugins when it detects issues. Plugins and applications then need to cooperate to cope with that, which is what the new event I wrote is for.

L.

[Pete Andrews]

ok - i see. thanks. 

i'll do a new build and give this a go.

thanks!

pete