Error: only sent -1 bytes

[lizardpeter]

Hello,

I am having an issue where using the streaming plugin with an RTP GStreamer pipeline is resulting in playback with major stutters. The output shows many errors that read something along the lines of 

[ice.c:janus_ice_outgoing_traffic_handle:4494] [1174500298337307] ... only sent -1 bytes? (was 1328).

I have verified using VLC that the RTP stream itself works flawlessly with no stutters. Does anyone have any potential solutions. I'm thinking it has to be related to something with the packets or MTU. Are there any settings I can adjust in the Janus configs or Linux itself to maximize performance?

Thanks,

Peter

[lizardpeter]

Interestingly, this is what the in and out stats look like.

[lizardpeter]

I should also add my GStreamer script itself.

gst-launch-1.0 -v file:///opt/janus3/share/janus/streams/beat.mp4 ! qtdemux ! queue ! h264parse ! rtph264pay mtu=1300 config-interval=1 pt=96 ! udpsink host=192.168.1.229 port=8004

[Alessandro Toppi]

There are a lot of nacks received by janus, so I guess you should try to understand why the client is not receiving so many packets.

[lizardpeter]

Thanks. One of the things I'm confused about is whether it's an ingestion issue or not. Why I am getting the error ":janus_ice_outgoing_traffic_handle"? Is that just saying that Janus didn't send any packets out? Also, this process is occurring on the same PC, so I'm not sure why any packets wouldn't be received. I've tested this exact pipeline with VLC as the player for the RTP stream and it worked perfectly fine.

[lizardpeter]

Thanks to Lorenzo's comment to another user about a similar issue, I have managed to find something that affects the issue. Changing these values in Linux can impact the amount of errors I see in the console. With something low like 10000, there is a vast increase in the number of errors. I have increased it to 500000 for default and 1000000 for max, and it seems to have solved most of the problems. Do you have any idea what I should set it to for maximum performance in Janus? Why does Janus not use the maximum value when it's needed? I tried only increasing the maximum value, but that did not fix the problem. The default value was the one that needed to be changed. I don't know if changing this value can have other harmful impacts on the system, however.

The default for me was 262144 for all four

# sysctl -w net.core.rmem_default=500000 

# sysctl -w net.core.wmem_default=500000 

# sysctl -w net.core.rmem_max=1000000 

# sysctl -w net.core.wmem_max=1000000

This was with 50000 kbps H264 video at 1080p 60 FPS. I will have to run other tests and report back with whether or not this fixed the other high bitrate issues I have encountered before.

[Lorenzo Miniero]

Will you guys PLEASE STOP saying it works with VLC? WebRTC and VLC do NOT work the same way, and do NOT share the same capabilities!

L.

[Alessandro Toppi]

50000 kbps is a 50 mbps video stream!

I'd expect a tuning in the socket buffers like you did in order to have decent performance.

Also keep in mind that Janus is not tested against such high bitrate streams.

[lizardpeter]

Yes, it was almost pointless to say it works with VLC. I was just saying that to show that I didn't think the problem lied with the RTP stream itself because it had the ability to have smooth playback, but in the end that was flawed too because, as you said, VLC could be doing things behind the scenes to make its own kind of buffer or something.

It took me a decent amount of time to figure out that the socket buffers needed to be tuned. Since you obviously both know a lot about this, do either of you have any other recommendations for anything that should be tuned for absolute maximum performance in Janus (especially in terms of high bitrate/bandwidth)? I'm already on the sendmmsg branch.

[Ravi Srivastava]

I am facing this issue on one of the servers in google cloud

[Tue Mar  1 13:06:47 2022] [janus.plugin.recordplay-0x7f22ec0090f0] WebRTC media is now available
[Tue Mar  1 13:06:47 2022] [4579198889511125] Sending event to transport...; 0x7f23380020a0
[Tue Mar  1 13:06:47 2022] Joining playout thread
[Tue Mar  1 13:06:47 2022] [4579198889511125] DTLS already set up, disabling retransmission timer!
[Tue Mar  1 13:06:47 2022] Got a Janus API request from janus.transport.http (0x7f22ec00cb50)
[Tue Mar  1 13:06:47 2022] Session 5500552827237382 found... returning up to 10 messages
[Tue Mar  1 13:06:47 2022] Got a keep-alive on session 5500552827237382
[Tue Mar  1 13:06:50 2022] [4579198889511125] Retransmitted 1 packets due to NACK (video stream #0)
[Tue Mar  1 13:06:55 2022] [4579198889511125] Retransmitted 9 packets due to NACK (video stream #0)
[Tue Mar  1 13:07:02 2022] [4579198889511125] Retransmitted 1 packets due to NACK (video stream #0)
[Tue Mar  1 13:07:07 2022] [4579198889511125] Retransmitted 4 packets due to NACK (video stream #0)
[Tue Mar  1 13:07:12 2022] [4579198889511125] Retransmitted 1 packets due to NACK (audio stream #0)
[Tue Mar  1 13:07:18 2022] Got a Janus API request from janus.transport.http (0x7f22ec00cb50)
[Tue Mar  1 13:07:18 2022] Session 5500552827237382 found... returning up to 10 messages
[Tue Mar  1 13:07:18 2022] Got a keep-alive on session 5500552827237382
[Tue Mar  1 13:07:23 2022] [4579198889511125] Retransmitted 1 packets due to NACK (video stream #0)
[Tue Mar  1 13:07:28 2022] [4579198889511125] Retransmitted 1 packets due to NACK (video stream #0)
[Tue Mar  1 13:07:33 2022] [4579198889511125] Retransmitted 2 packets due to NACK (video stream #0)
[Tue Mar  1 13:07:39 2022] [4579198889511125] Retransmitted 6 packets due to NACK (video stream #0)
[Tue Mar  1 13:07:41 2022] [4579198889511125] Component state changed for component 1 in stream 1: 5 (failed)
[Tue Mar  1 13:07:41 2022] [WARN] [4579198889511125] ICE failed for component 1 in stream 1, but let's give it some time... (trickle received, answer received, alert not set)
[Tue Mar  1 13:07:41 2022] [4579198889511125] Creating ICE state check timer with ID 667
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 111)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 1084)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 1085)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 1085)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4512] [4579198889511125] ... only sent -1 bytes? (was 64)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4512] [4579198889511125] ... only sent -1 bytes? (was 64)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 120)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 121)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 962)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 962)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 962)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 962)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 121)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 122)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 1199)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 1199)
[Tue Mar  1 13:07:41 2022] [ERR] [ice.c:janus_ice_outgoing_traffic_handle:4663] [4579198889511125] ... only sent -1 bytes? (was 1199)

[Ravi Srivastava]

I solved it with new version of libnice (= 1.18).