NACK Issues with Janus

696 views
Skip to first unread message

Kannan Murali

unread,
May 9, 2017, 3:52:34 AM5/9/17
to meetecho-janus
Hi:

I assume that the NACK issue has been discussed before and recommend that the issue could be on the network side - either packets 
drop or disorder or lack of bandwidth(slow link). 

To support WebRTC clients to our existing conferencing solution, we are using Janus Gateway. We are in the process of doing performance 
test with 800 RTP sessions (each session with either one audio m= line or video m= line). We are testing this using two Janus Gateway 
server with different H/W specifications.

1. With 2 Cores VM and 8GB memory:

    This works fine with the throughput of ~14MBps, and 80% to 85% CPU load and 1.70Gb memory usage.
    There is no NACK issues at all.


2. With 4 Cores VM and 8GB memory:

   The throughput and the CPU utilization seems to be ok when compared to 2 core VM testing. However, after some time (few minutes) 
   we experience lot of NACK message and the memory usage slowly claims up and finally the system becomes unusable.


With this 2 different test scenarios every thing else (same WebRTC clients automation server, same TURN server, Same network setup, 
same set of our native conferencing clients, same set of video and audio streams - 1 h264 video and 4 OPUS audio channels) only the 
H/W system specification is different for these two servers.

From the WebRTC clients we generate the offer SDP to Janus Gateway and our plugin in Janus produces the answer SDP. In the offer 
SDP the WebRTC clients provides all the NACK related attributes. In the answer SDP, we don't negotiate the NACK related attributes.

However, we see tons of NACKs and we end up in a situation when the 4 core VM becomes unusable. The following is the snippet of 
the offer/answer SDP for WebRTC clients that we are using in our testing:

Offer SDP from the WebRTC Client:
==========================

v=0^M
o=- 5899939495561709681 2 IN IP4 127.0.0.1^M
s=-^M
t=0 0^M
a=group:BUNDLE audio video^M
a=msid-semantic: WMS^M
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 126^M
c=IN IP4 0.0.0.0^M
a=rtcp:9 IN IP4 0.0.0.0^M
a=ice-ufrag:P5lW^M
a=ice-pwd:FSIesPwv9vSDBe+zR
[INFO] 2017-05-08 10:54:49.515 lksRyez^M
a=fingerprint:sha-256 A0:8D:56:B7:6A:49:57:09:B6:45:37:EC:6F:C2:21:57:47:6F:88:5C:FC:D0:B3:B0:8D:77:03:D4:BE:59:66:68^M
a=setup:actpass^M
a=mid:audio^M
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level^M
a=recvonly^M
a=rtcp-mux^M
a=rtpmap:111 opus/48000/2^M
a=rtcp-fb:111 transport-cc^M
a=fmtp:111 minptime=10;useinbandfec=1^M
a=rtpmap:103 isac/16000^M
a=rtpmap:104 isac/32000^M
a=rtpmap:9 G722/8000^M
a=rtpmap:0 PCMU/8000^M
a=rtpmap:8 PCMA/8000^M
a=rtpmap:106 cn/32000^M
a=rtpmap:105 cn/16000^M
a=rtpmap:13 cn/80
[INFO] 2017-05-08 10:54:49.515 00^M
a=rtpmap:126 telephone-event/8000^M
m=video 9 UDP/TLS/RTP/SAVPF 100 101 107 116 117 96 97 99 98^M
c=IN IP4 0.0.0.0^M
a=rtcp:9 IN IP4 0.0.0.0^M
a=ice-ufrag:P5lW^M
a=ice-pwd:FSIesPwv9vSDBe+zRlksRyez^M
a=fingerprint:sha-256 A0:8D:56:B7:6A:49:57:09:B6:45:37:EC:6F:C2:21:57:47:6F:88:5C:FC:D0:B3:B0:8D:77:03:D4:BE:59:66:68^M
a=setup:actpass^M
a=mid:video^M
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset^M
a=extmap:4 urn:3gpp:video-orientation^M
a=extmap
a=recvonly^M
a=rtcp-mux^M
a=rtcp-rsize^M
a=rtpmap:100 VP8/90000^M
a=rtcp-fb:100 ccm fir^M
a=rtcp-fb:100 nack^M
a=rtcp-fb:100 nack pli^M
a=rtcp-fb:100 goog-remb^M
a=rtcp-fb:100 transport-cc^M
a=rtpmap:101 VP9/90000^M
a=rtcp-fb:101 ccm fir^M
a=rtcp-fb:101 nack^M
a=rtcp-fb:101 nack pli^M
a=rtcp-fb:101 goog-remb^M
a=rtcp-fb:101 transport-cc^M
a=rtpmap:107 H264/90000^M
a=rtcp-fb:1
[INFO] 2017-05-08 10:54:49.515 07 ccm fir^M     ===>  We disbale this in the answer SDP. Is there way to disable this from the WebRCT Client???
a=rtcp-fb:107 nack^M
a=rtcp-fb:107 nack pli^M
a=rtcp-fb:107 goog-remb^M
a=rtcp-fb:107 transport-cc^M
a=fmtp:107 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f^M
a=rtpmap:116 red/90000^M
a=rtpmap:117 ulpfec/90000^M
a=rtpmap:96 rtx/90000^M
a=fmtp:96 apt=100^M
a=rtpmap:97 rtx/90000^M
a=fmtp:97 apt=101^M
a=rtpmap:99 rtx/90000^M
a=fmtp:99 apt=107^M
a=rtpmap:98 rtx/90000^M
a=fmtp:98 apt=116^M


Answer SDP from Our Janus Plugin:
===========================
v=0^M
o=- 899939495561709681 2 IN IP4 172.16.102.237^M
s=-^M
t=0 0^M
a=gr
[INFO] 2017-05-08 10:54:49.537 oup:BUNDLE audio video^M
a=msid-semantic: WMS janus^M
m=audio 9 UDP/TLS/RTP/SAVPF 111 13^M
c=IN IP4 172.16.102.237^M
a=sendonly^M
a=mid:audio^M
a=rtcp-mux^M
a=ice-ufrag:klIU^M
a=ice-pwd:ANVqVI1nYjt9Lwfw7vGMV2^M
a=ice-options:trickle^M
a=fingerprint:sha-256 DD:10:5D:C6:18:74:28:1F:8C:A6:AF:1D:DC:D3:56:44:CC:A7:C3:2A:AF:12:54:82:7F:C2:3B:A7:3A:D9:F5:60^M
a=setup:active^M
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level^M
a=rtpmap:111 opus/48000/2^M
a=rtcp-fb:111 transport-cc^M
a=fmtp:111 minptime=10;maxaveragebitrate
[INFO] 2017-05-08 10:54:49.537 =24000;usedtx=1;^M
a=rtpmap:13 CN/8000^M
a=ssrc:2340513383 cname:janusaudio^M
a=ssrc:2340513383 msid:janus janusa0^M
a=ssrc:2340513383 mslabel:janus^M
a=ssrc:2340513383 label:janusa0^M
a=candidate:1 1 udp 2013266431 172.16.102.237 20457 typ host^M
m=video 9 UDP/TLS/RTP/SAVPF 100 101 107^M
c=IN IP4 172.16.102.237^M
a=sendonly^M
a=mid:video^M
a=rtcp-mux^M
a=ice-ufrag:klIU^M
a=ice-pwd:ANVqVI1nYjt9Lwfw7vGMV2^M
a=ice-options:trickle^M
a=fingerprint:sha-256 DD:10:5D:C6:18:74:28:1F:8C:A6:AF:1D:DC:D3:56:44:CC:A7:C3:2A:AF:12:54:8
[INFO] 2017-05-08 10:54:49.537 2:7F:C2:3B:A7:3A:D9:F5:60^M
a=setup:active^M
a=extmap:2 urn:ietf:params:rtp-hdrext:toffset^M
a=extmap:4 urn:3gpp:video-orientation^M
a=rtpmap:107 H264/90000^M
a=fmtp:107 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f^M
a=ssrc:3783097010 cname:janusvideo^M
a=ssrc:378
[INFO] 2017-05-08 10:54:49.537 3097010 msid:janus janusv0^M
a=ssrc:3783097010 mslabel:janus^M
a=ssrc:3783097010 label:janusv0^M
a=candidate:1 1 udp 2013266431 172.16.102.237 20457 typ host^M


We are trying to tackle(or troubleshoot)  this issue from two perspective. 

1. How the 4 core VM is different from the 2 core VM and it's associated network related resources like physical NIC, virtual NIC and etc.

2. Is there is a way to disable the NACK related attributes from the WebRTC offer SDP??? Basically, we want to disable the NACK from 
   both side completely. Our WebRTC client uses the janus.js and adapter.js files that are provided by Janus.

-KMurali. 

Lorenzo Miniero

unread,
May 9, 2017, 5:44:32 AM5/9/17
to meetecho-janus
First of all, [pedant jerk MODE ON], please don't paste huge bunches of text, as explained in the guidelines on this page, use gist/pastebin instead [pedant jerk MODE OFF].

You can disable NACKs setting max_nack_queue=0 in janus.cfg.

L.

Ju Ju

unread,
May 9, 2017, 10:37:56 AM5/9/17
to meetecho-janus
Hi Lorenzo,

Maybe they are tackling the same issue we did (mutex contention in libnice) and you could propose your workaround ?

J-

Lorenzo Miniero

unread,
May 9, 2017, 10:43:11 AM5/9/17
to meetecho-janus
That may be that, true. For those unaware of this, libnice has a global lock for all outgoing packets, which obviously makes it underperform when you significantly increase the load, up to a point where it slows everything down. NACKs then occur because Janus can't send as fast as it would like due to this global lock, and so users fail to receive packets in a timely manner. More info here: https://phabricator.freedesktop.org/T93
That report contains a reference to a user-provided branch which tries to fix that (switch to a per agent-lock), although that's of course experimental. We tried something similar and it helped us a lot, although it might be even better.

Fixing this is part of our Janus optimization plans in the very near future, whether it means trying to fix libnice ourselves or switch to a different library.

L.
Reply all
Reply to author
Forward
0 new messages