How best to debug stuttering in streaming plugin

[aiden....@codethink.co.uk]

We are sending video RTP and RTCP packets to Janus over UDP on localhost (GStreamer and Janus running on same physical machine); broadcasting the video using the Streaming Plugin with the following settings (https://pastebin.com/URRvtPN8). For POC purposes, we are using the demo streamingtest.html/js files to test performance.

When the consumer of the stream is on the local network, everything seems to work as expected (no noticeable artifacts or discontinuities in the playback); however when connecting from outside the local network, the video becomes choppy and stutters. Looking at the admin interface for Janus, we appear to be getting a *lot* of `video_nacks`; which I assume explain the choppiness and are due to packet loss.

How should I go about debugging/fixing this? I've added elements on the GStreamer side to re-transmit RTP packets; however, I assume that this would require for Janus to notify GStreamer (using RTCP?) that it needs certain packets again. Is there some configuration in Janus that I'm missing? Should the demo html/js files be modified in some way?

Apologies for the general nature of this question. I'm happy to provide more detail; but I'm mainly looking for pointers do that I can diagnose and fix this issue myself.

[lmin...@gmail.com]

Missing packets on the viewer side doesn't necessarily mean Gstreamer is not sending them all. Maybe it is, and the viewer is not getting them. As a side note, Janus does retransmissions on its own, no need to ask Gstreamer for those.

If it works locally and not on the Internet or with other machines, either bandwidth is not enough, or your packets exceed the MTU and are dropped somewhere between Janus and the viewer. If so, send smaller packets.

Lorenzo

[Aiden Jeffrey]

Hi Lorenzo,

I've tried different packet sizes and still get packet loss / video_nacks. The only thing that eliminates the stutter is if I lower the bitrate of the h264 compression from 20Mbps to say 10Mbps. Packet sizes are the same (1200 bytes) from gstreamer -> janus.

Any ideas?

Thanks,

Aiden

[Aiden Jeffrey]

Hi,

I am still struggling with stutter / packet loss when using the streaming plugin.

This must be a Janus configuration error, or a limitation with some component of Janus not being able to handle 20Mbps h264 4k video. I am able to stream the rtp packets between the same machines, using gstreamer udpsink/src; with no stutter. Also, preliminary testing of the `webrtcbin` in gstreamer doesn't show stutter at the same bitrates between the same machines. The packet loss only happens when Janus is introduced into the picture. Also, stutter / video_nacks increase when more client peers are connected to the streaming plugin.

Janus is a really great project, with first class documentation; if you could give me some pointers on how to debug this, I would very much appreciate it.

Ignore the red herring in my first post about it working over local network, but not over larger network distances; we see the packet loss on localhost -> localhost at the bitrates above (20Mbps).

Thanks in advance,

Aiden

[Tristan Matthews]

Hi,

On Thu, May 16, 2019 at 1:29 PM Aiden Jeffrey
<aiden....@codethink.co.uk> wrote:
>
> Hi,
>
> I am still struggling with stutter / packet loss when using the streaming plugin.
>
> This must be a Janus configuration error, or a limitation with some component of Janus not being able to handle 20Mbps h264 4k video. I am able to stream the rtp packets between the same machines, using gstreamer udpsink/src; with no stutter. Also, preliminary testing of the `webrtcbin` in gstreamer doesn't show stutter at the same bitrates between the same machines. The packet loss only happens when Janus is introduced into the picture. Also, stutter / video_nacks increase when more client peers are connected to the streaming plugin.
>
> Janus is a really great project, with first class documentation; if you could give me some pointers on how to debug this, I would very much appreciate it.
>
> Ignore the red herring in my first post about it working over local network, but not over larger network distances; we see the packet loss on localhost -> localhost at the bitrates above (20Mbps).
>

Have you tried looking at your OS' udp buffers, i.e. with netstat
-suna? You may also want to check out:

https://jvns.ca/blog/2016/08/24/find-out-where-youre-dropping-packets/
https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-performance-51d167bb1360

[Lorenzo Miniero]

Have you tried adding a helper thread to your Streaming mountpoint already? That should help if the cause is the plugin struggling to queue outgoing packets AND staying up-to-date while receiving a very high bitrate/frequency incoming stream at the same time.

L.

[Alessandro Amirante]

Also make sure you're not hitting the infamous global lock issue of libnice, as it is known to affect the throughput. Which version of libnice are you using? 

A.

--
You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/meetecho-janus/bbfc09a2-02de-45e9-b497-494f59e0ba04%40googlegroups.com.

[Aiden Jeffrey]

Hi,

I had tried increasing these settings yes, but increasing it didn't have any impact.

Thanks,

Aiden

[Aiden Jeffrey]

I just tried adding helper threads now, thanks for the tip. They appear to make the issue worse (tried adding 2, 5 and 10 threads - progressively worse).

[Aiden Jeffrey]

This may be it. I uninstalled libnice and built it from source before building janus (as per the README), but in apt installing gst-plugins-bad, I've inadvertently reinstalled the stock ubuntu version (0.1.14-1). Not sure which it's linking against at runtime.

If you're interested, here is a gst-launch stripped down version of our pipeline, which produces similar stutter in the janus demo playback page:

gst-launch-1.0 -v  rtpbin name=rtpbin ! videotestsrc ! \
 'video/x-raw, width=(int)3840, height=(int)2160, format=(string)I420, framerate=(fraction)30/1' ! \
 videoconvert ! queue ! vaapih264enc bitrate=25000 ! \
'video/x-h264, stream-format=(string)byte-stream, profile=baseline' ! \
rtph264pay config-interval=10 pt=96 ! \
rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! \
udpsink host=127.0.0.1 port=5000 rtpbin.send_rtcp_src_0 ! \
udpsink host=127.0.0.1 port=5001 sync=false async=false

To unsubscribe from this group and stop receiving emails from it, send an email to meetech...@googlegroups.com.

[Alessandro Amirante]

Version 0.1.14 is definitely affected by the global lock. You should properly set the LD_LIBRARY_PATH env variable before launching Janus in order to make it pick the right version of libnice.

A.

To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/meetecho-janus/ad4558a6-086a-422c-b55e-54498999e0e4%40googlegroups.com.

[Aiden Jeffrey]

I tried that last night, then tried purging my system of libnice, building and installing from master, then editing the .deb file for gst-plugins-bad to strip out libnice dependency, installing that. Still getting packet loss / stutter. I'd be interested to know if anyone else is seeing a similar issue when running the following pipeline:

gst-launch-1.0 -v  rtpbin name=rtpbin ! videotestsrc ! \
 'video/x-raw, width=(int)3840, height=(int)2160, format=(string)I420, framerate=(fraction)30/1' ! \
 videoconvert ! queue ! vaapih264enc bitrate=25000 ! \
'video/x-h264, stream-format=(string)byte-stream, profile=baseline' ! \
rtph264pay config-interval=10 pt=96 ! \
rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! \
udpsink host=127.0.0.1 port=5000 rtpbin.send_rtcp_src_0 ! \
udpsink host=127.0.0.1 port=5001 sync=false async=false

With streaming plugin mountpoint config:

h264-sample: {
  type = "rtp"
  id = 1
  description = "H.264 WebRTC Stream"
  audio = false
  video = true
  videoport = 5000
  videortcpport = 5001
  videopt = 96
  videortpmap = "H264/90000"
  videofmtp = "profile-level-id=428028;packetization-mode=1"
}

We're seeing this on a range of hardwares.

A.

To view this discussion on the web visit https://groups.google.com/d/msgid/meetecho-janus/ad4558a6-086a-422c-b55e-54498999e0e4%40googlegroups.com.

[Lorenzo Miniero]

To be honest I never tried pushing 20mbps to a single user via Janus, so no idea if we have bottlenecks that might prevent that.

L.

[Alessandro Amirante]

I won't be surprised other people experience the same behavior since, again, libnice < 0.1.15 is know to have throughput issues.

A.

To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/meetecho-janus/a892385e-c7f5-46d3-bd5a-4f758970fc55%40googlegroups.com.

[Aiden Jeffrey]

OK. Thanks for the tips and pointers anyway.

PS: re: libnice, as of this morning I can definitively say that I see this issue when Janus is definitely linking against libnice built from master.

To view this discussion on the web visit https://groups.google.com/d/msgid/meetecho-janus/a892385e-c7f5-46d3-bd5a-4f758970fc55%40googlegroups.com.

[Paul Homlish]

I'm seeing stuttering also with just ~ 500Kbits/sec.  Also linking against libnice master (libnice.so.10).

Any other tips on debugging the stutter?

[Alessandro Amirante]

I'm not familiar with vaapih264enc. I made a few tests and it doesn't seem to take into account the bitrate option: it sends ~40Mbit/s even if I set bitrate=500.

OTOH I tested with x264enc and it works just fine without stuttering. My machine is not powerful enough to encode a 4k video @ 30fps, but I can send a 2704x1520 @ 30fps stream with a 25Mbit/s bitrate flawlessly. My gstreamer pipeline looks like this.

A.

Il giorno ven 24 mag 2019 alle ore 07:21 Paul Homlish <phom...@gmail.com> ha scritto:

I'm seeing stuttering also with just ~ 500Kbits/sec.  Also linking against libnice master (libnice.so.10).

Any other tips on debugging the stutter?

--
You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/meetecho-janus/fbfd0af6-7ff6-4bcb-86bd-d0a3aa61b294%40googlegroups.com.

[Alessandro Toppi]

Same issue with your pipeline, but something weird is happening on the bitrate with that 4K resolution. I see VERY high bitrate on the webrtc internals (like 80Mbps) that might explain the stuttering, while turning down the resolution to 1080p the stream become perfect and the bitrate locks to the nominal 25Mbps. Maybe this is an issue with the encooder when handling such high resolutions?

[Maestro M.]

Perhaps try feeding the stream over a more reliable protocol (something using TCP) and having two instances of GStreamer (one to broadcast over the reliable transport, and one on the Janus server to receive the feed which then converts it to UDP over localhost).

That being said, on an unrelated note, I'm experiencing packet drops over localhost using purely ffmpeg (no Janus involved). I've pretty much gone crazy by this issue.
I found out that Wireshark has excellent RTP support! I recommend you try running Wireshark on the Janus host to verify that the RTP packets are actually making it all the way.
Install WireShark, Go to menu Analyze, select Enabled Protocols. Search for RTP. Tick 'rtp_udp'.
Start the packet capture on the right interface, and then go to the menu Telephony, select RTP and analyze the stream that is available.
You can check that all packets arrived by simply looking at the sequence numbers. The analysis tool will also inform you of other errors such as invalid timestamps etc.