Major stuttering and laggy playback with VideoRoom

[lizardpeter]

Hello,

I have managed to use WHIP to stream OBS WebRTC video to a Janus VideoRoom with the help of a very nice person in this group. While the image quality itself is good at high bitrates, when playing this content I get the error messages "Got 40 SRTP/SRTCP errors in the last few seconds (last error: srtp_err_status_replay_old)". These errors coincide with major stutters and frame drops on the playback side. Using the WebRTC internals in Chrome, I can see that the frame rate skips around majorly instead of staying at a locked 30 FPS (I want to use 60, but I am using 30 for testing purposes).

How can I fix this issue? My main goal is very high quality streaming (15000 mbps and 60 FPS). This is over a gigabit fiber connection, and so far I have only tested locally, so I doubt this is an issue with the connection itself.

Thanks,

Peter

[Alessandro Toppi]

Check the admin API to analyze the publisher (OBS) inbound feed (loss, jitter, selected candidates etc.)

[lizardpeter]

Thanks for the suggestion. These are the notable things I see in the ADMIN API.

For the out stream stats: 

"in_stats": { "audio_packets": 0, "audio_bytes": 0, "audio_bytes_lastsec": 0, "do_audio_nacks": false, "video_packets": 0, "video_bytes": 0, "video_bytes_lastsec": 0, "do_video_nacks": true, "video_nacks": 641, "video_retransmissions": 0, "data_packets": 2, "data_bytes": 815 }, 

 "out_stats": { "audio_packets": 24194, "audio_bytes": 653238, "audio_bytes_lastsec": 1350, "audio_nacks": 0, "video_packets": 257289, "video_bytes": 302276507, "video_bytes_lastsec": 572350, "video_nacks": 0, "data_packets": 3, "data_bytes": 2033 }

For the in stream stats:

"in_stats": { "audio_packets": 37820, "audio_bytes": 869860, "audio_bytes_lastsec": 1150, "do_audio_nacks": false, "video_packets": 384306, "video_bytes": 447421645, "video_bytes_lastsec": 10067, "do_video_nacks": true, "video_nacks": 0, "video_retransmissions": 888, "data_packets": 3, "data_bytes": 2278 }, "out_stats": { "audio_packets": 0, "audio_bytes": 0, "audio_bytes_lastsec": 0, "audio_nacks": 0, "video_packets": 0, "video_bytes": 0, "video_bytes_lastsec": 0, "video_nacks": 891, "data_packets": 3, "data_bytes": 2014 }

It seems like there are a decent amount of video retransmissions on the ingest side. 

[lizardpeter]

I used a much more powerful PC to stream from and it seems like the playback is much better. However, I am still getting this kind of periodic behavior.  

Interestingly, Chrome only reports a single dropped frame. How can this be? On the encoding side, OBS is reporting less than 0.1% of frames being dropped... 

[Alessandro Toppi]

If the stream OBS -> Janus has some loss there is not much Janus can do (aside from requesting retransmissions with NACKs as it should already do) and you will always have troubles on the viewer side.

Some other things you can still check:

- lower the stream quality (bitrate, fps etc.)

- check the admin api for loss and jitter

- test in a local network

- try to record the publisher video through the admin api (there is a recording feature). Then post-process the generated .mjr file with janus-pp-rec to visually inspect the video quality

[lizardpeter]

Do you know why the ingest stream reports a lot of retransmissions but no nacks? I've been trying now with 15000 mbps and 60 FPS, and at times the playback is perfect. However, at other times the playback drops to 20 FPS on the client side. These drops seem to coincide mostly with scene changes (like the video transitioning from being indoors to outdoors). Additionally, as I know Janus just moves bytes around and should output a 1:1 version of the ingested media, my ultimate goal is to stream at 120 FPS using 120 FPS source media. I have tried this, with Chrome occasionally outputting the full 120 FPS, but unfortunately I am experiencing the same kinds of drops as I posted in the image above. Firefox reports an average stream FPS of about 90 FPS using the 120 FPS stream. 

Also, I don't see where in the Admin API I can see loss and jitter. Where can that information be found?

Lastly, I seem to only be able to use VP8 for these tests. Trying to use OBS WebRTC with H264 or VP9 results in only the audio stream being ingested by Janus. Are there any specific OBS settings that must be used for the greatest compatibility? I believe I am using the baseline encoding setting with NVIDIA NVENC (new) and the low latency performance mode with no b-frames. The encoding PC is using an i9 9900k, RTX 2080 Ti, and 32 GB of 3600 MHz CL 16 RAM. The server PC is on the same network and is only using a 4th generation i3, GTX 970, and 8 GB of RAM, but the Janus CPU usage is only around 10-20%, so I don't think the hardware on either side should be an issue. The PCs are connected through the local network (one passes through an unmanaged network switch), and neither PC has ever seemed to have any issues with packet loss or anything like that, so I'm truly confused as to why this configuration doesn't work flawlessly with a network stream of only 15000 mbps when both computers and the local network can easily handle 1 gbps.

Thanks,

Peter

[Alessandro Toppi]

Do you know why the ingest stream reports a lot of retransmissions but no nacks?

Nacks are being sent. Check "video_nacks" in out stats for the ingest stream.

 

I've been trying now with 15000 mbps and 60 FPS, and at times the playback is perfect. However, at other times the playback drops to 20 FPS on the client side. These drops seem to coincide mostly with scene changes (like the video transitioning from being indoors to outdoors).

This seems to suggest an issue at the encoder. As I said try to record the video received by Janus.

Are you sure this is not a normal behavior in case of static/blank image ?

 

Additionally, as I know Janus just moves bytes around and should output a 1:1 version of the ingested media, my ultimate goal is to stream at 120 FPS using 120 FPS source media. I have tried this, with Chrome occasionally outputting the full 120 FPS, but unfortunately I am experiencing the same kinds of drops as I posted in the image above. Firefox reports an average stream FPS of about 90 FPS using the 120 FPS stream. 

Yes, Janus does not touch media.

 

Also, I don't see where in the Admin API I can see loss and jitter. Where can that information be found?

Loss and jitter should be reported in rtcp stats, if you can't find them just paste the whole admin api info on pastebin.

 

Lastly, I seem to only be able to use VP8 for these tests. Trying to use OBS WebRTC with H264 or VP9 results in only the audio stream being ingested by Janus. Are there any specific OBS settings that must be used for the greatest compatibility? I believe I am using the baseline encoding setting with NVIDIA NVENC (new) and the low latency performance mode with no b-frames. The encoding PC is using an i9 9900k, RTX 2080 Ti, and 32 GB of 3600 MHz CL 16 RAM. The server PC is on the same network and is only using a 4th generation i3, GTX 970, and 8 GB of RAM, but the Janus CPU usage is only around 10-20%, so I don't think the hardware on either side should be an issue. The PCs are connected through the local network (one passes through an unmanaged network switch), and neither PC has ever seemed to have any issues with packet loss or anything like that, so I'm truly confused as to why this configuration doesn't work flawlessly with a network stream of only 15000 mbps when both computers and the local network can easily handle 1 gbps.

Sorry, I am not familiar with OBS and its WHIP implementation.

[lizardpeter]

Sorry for the delay. I will try the recording now. I have managed to get it fairly stable at around 120 FPS playback, but there are still intermittent stutters. These stutters seem to correlate with the error message "Got 6 SRTP/SRTCP errors in the last few seconds (last error: srtp_err_status_replay_old)." Does this suggest the error is on the encoding side?

[lizardpeter]

Alright, so I tried using the recording. I found that the recording DID NOT have the stutters seen in the actual live replay. Now I am really confused.

[lizardpeter]

The RTCP stats show the following:

For the inbound handle (from encoder): I believe that is the one because it is the session with only one handle

Session 5915053391906627 and handle 5815880603399852

"video": {

                    "base": 90000,

                    "rtt": 0,

                    "lost": 4872,

                    "lost-by-remote": 0,

                    "jitter-local": 0,

                    "jitter-remote": 0,

                    "in-link-quality": 100,

                    "in-media-link-quality": 100,

                    "out-link-quality": 0,

                    "out-media-link-quality": 0

                }

For the outbound (?) handles:

First: Session 1760421852357769 and handle 8823115287184788

"video": {

                    "base": 90000,

                    "rtt": 0,

                    "lost": 9,

                    "lost-by-remote": 0,

                    "jitter-local": 1,

                    "jitter-remote": 0,

                    "in-link-quality": 100,

                    "in-media-link-quality": 100,

                    "out-link-quality": 0,

                    "out-media-link-quality": 0

                }

Second: Session 8823115287184788 and handle 8823115287184788

"video": {

                    "base": 90000,

                    "rtt": 1,

                    "lost": 0,

                    "lost-by-remote": 2356,

                    "jitter-local": 0,

                    "jitter-remote": 3,

                    "in-link-quality": 0,

                    "in-media-link-quality": 0,

                    "out-link-quality": 100,

                    "out-media-link-quality": 100

                }

So I guess it does appear to be an issue from the encoder. I am not certain, however.

[Alessandro Toppi]

The SRTP error (srtp_err_status_replay_old) shows up when Janus is trying to send RTP packets and libsrtp (the underlying rtp library) rejects the operation because it discards packets with too old sequence numbers.

My guess is that the packets arrives from the encoder with some jitter (it could be due to the encoder or the network) and since the number of packets sent per second is very high, it's very likely that a delayed packet goes outside of the "safe sequence number window" used by libsrtp to accept/reject it.

This seems to be confirmed by the recording being okay (the post-processor works offline by sorting all the packets before generating the file) and the stats you shared (delayed packets could be considered as lost).

[lizardpeter]

Thanks again for the information. I have since tried the "sendmmsg" branch of Janus and the performance improved a very little bit. I have opened up a ticket with OBS WebRTC, and they will be looking into it within a few weeks. It doesn't seem like the issue is isolated to my implementation, however, as other users have now reported issues with high bitrates.