RTP forward and videoroom plugin

[Juju Trankilou]

Hello,

I was able to RTP forward the webcam flow of one publisher to a fixed audio and video port but I would like to know if I could use theses flow as input of another videoroom ?

I know I can use the streaming plugin but I would like to have an SFU because the streaming plugin use many CPU.

thx

J-

[Lorenzo Miniero]

You cannot pass them to another VideoRoom, that plugin only gets WebRTC publishers. Not sure what you mean by "use may CPU", instead.

L.

[Juju Trankilou]

Hello Lorenzo,

I mean when I'm using streaming plugin I can handle "only" 400 users (viewers) per 8 VCPU VM but when I'm using videoroom plugin I can go to 1000 users (viewers). The ice threads seems to take little more CPU in streaming plugin.

However 400 per VM is OK atmI will just have to add the right number of VM

However you should know this: when I was testing videoroom plugin with headless firefox clients (no deco with firefox driver of selenium), I was keeping 1 client with a screen to check the video rendering :

everything was ok with 300 viewers (CPU of janus server was around 20% so quiet low). After 300 to 350 the video starting to stuttering. After 350 the video was stuck, audio was OK.

With 400 viewers Janus CPU was near 40%, no thread seems to take too much CPU. I was still receiving UDP packets on my "screened" client. I don't think it is a network bandwidth issue because I was in local on a 10Gbits network.

So something is limiting janus numbers of clients for videoroom plugin around 350, not the transport network nor the CPU usage: Do you have an idea what it could be ?

Did you test the videoroom plugin with more than 350 viewers ?

br,

J-

[Lorenzo Miniero]

I guess the problem is that the same thread is queueing outgoing packets for all users. Maybe having workers to do that may improve things there, but I'm not sure. We'll have to make some tests, maybe in a separate branch: if you have ways to quickly stress test as you're doing, it should be easy enough to figure out when we've done it. Not sure when we'll be able to start working on that, though.

L.

[Julien Godier]

Hi lorenzo

First I thought of a limitation regarding pps ( packet per second) as you don't use sendmmsg (http://man7.org/linux/man-pages/man2/sendmmsg.2.html) which is usefull to avoid 7usec overhead each packet you send. 

But in this case I would have had my  CPU to 100%

So I going to :

1/ check pps with ipstat to have an idea of hiw many pps I have on the interface

2/ check in your source code (SO_SNDBUF) if you don't override system default buffer size

3/ check if one socket is well created per clients

4/ make a capture and check hole in seq num at client side

5/ try to tweak socket buffer values

Julien

--
You received this message because you are subscribed to a topic in the Google Groups "meetecho-janus" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meetecho-janus/HR0f3XN-_i8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to meetecho-janu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lorenzo Miniero]

Any detail you can provide that can help us improve this would be precious, thanks!

[julien Godier]

Hello lorenzo,

Is it possible to easily remove the cipher part for the media stream (I mean the DTLS) to get something « in clear »  ?

kr,

J-

[Juju Trankilou]

I think I found the answer by myself

http://webrtc-security.github.io/

Encryption is a mandatory feature of WebRTC, and is enforced on all components, including signaling mechanisms. Resultantly, all media streams sent over WebRTC are securely encrypted, enacted through standardised and well-known encryption protocols. The encryption protocol used depends on the channel type; data streams are encrypted using Datagram Transport Layer Security (DTLS) and media streams are encrypted using Secure Real-time Transport Protocol (SRTP).

To unsubscribe from this group and all its topics, send an email to meetecho-janus+unsubscribe@googlegroups.com.

[Juju Trankilou]

Hi some updates here :

I asked a colleague to help me (a black belt in C programming and unix system ) :) so we could sucessfully done what I wanted to

1/ check pps with ipstat to have an idea of how many pps I have on the interface

-> quiet low < 10k

 tcpstat -i eth0 -o "Time:%S\Pckt-per-sec=%p\UDP=%U\tcp=%T\tbps=%b\n" 1

Time:1466640527 Pckt-per-sec=3783.00 UDP=3776  TCP=6  bps=23327216.00

2/ check in your source code (SO_SNDBUF) if you don't override system default buffer size

none but i found "setsockopt" in videoroom.c (come back on this later)

3/ check if one socket is well created per clients

yes it is

4/ make a capture and check hole in seq num at client side

Done: We could sucessfully analyse both capture and retrieve seq number and so on (i'm making the short story)

Verdict: losses are on the janus

-> it seems you are doing none blocking writing (what is good) but, when the problem occurs,  it seems janus is trying to write to a full buffer and OS answer him it is not possible janus just drop the packet away

5/ try to tweak socket buffer values

tomorrow my colleague will write a patch for videoroom.c

Besides he told me you are not checking the answer when you make your "sendto" -> when the buffer is full you should have a specific answer

[Lorenzo Miniero]

VideoRoom doesn't send anything by itself, it just passes packets to be sent to the core. It's the ICE send thread in the core that physically sends the packets, and it does that using the libnice primitives, so nothing we can manipulate socket-wise, I think.

L.

[julien Godier]

Hi lorenzo,

Sorry i’m really bad in C but what is this in video room.c ?

2024                         if(video && rtp_forward->is_video) {

2025                                 sendto(participant->udp_sock, buf, len, 0, (struct sockaddr*)&rtp_forward->serv_addr, sizeof(rtp_forward->serv_addr));

2026                         }

2027                         else if(!video && !rtp_forward->is_video) {

2028                                 sendto(participant->udp_sock, buf, len, 0, (struct sockaddr*)&rtp_forward->serv_addr, sizeof(rtp_forward->serv_addr));

2029                         }

kr,

J-

[Lorenzo Miniero]

That's the RTP forward feature, to forward media of a publisher to an external component that is not WebRTC based (e.g., a gstreamer pipeline or an ffmpeg script). 

L.

[julien Godier]

Hi lorenzo,

Thx for your answer. I forgot the rap feature… even I’m using it ..stupid hobbit I am :D

My colleague has a look on libnice and told me that: 

- nice_agent_get_selected_socket => return a  GSocket

- g_socket_set_option => apply a setsockopt() on it.

The tricky part is is the "selected » of nice_agent_get_selected_socket : it seems you will have to identify the right sub stream using  streamid and components.

So can use please add in your code  a SO_SNDBUF of 10MB (we will manage to change its size if we need) ?

J-

[Lorenzo Miniero]

Sorry but I didn't understand what you asked about... if you've identified the part of the code where you think something can be done, please consider filing a pull request with the change you need.

L.

To unsubscribe from this group and all its topics, send an email to meetecho-janus+unsubscribe@googlegroups.com.

[Juju Trankilou]

Hi Lorenzo,

We are trying to do the job as I think it is a strong limitation to the actual janus and I can't wait as I have to produce my report before end of summer.

So  we started to find ou the right place in the ice.c but with no success

At the end of the janus_ice_setup_local function, we called this:

  if (handle->video_stream) {

  

   GSocket *gsok;

  

   gsok = nice_agent_get_selected_socket( handle->agent , handle->video_stream->stream_id,    

    handle->video_stream->rtp_component->component_id );

  

   JANUS_LOG(LOG_WARN, "SOCKETPATCH(%p) stream=%d component=%d return=%p\n",

    handle->agent,handle->video_stream->stream_id,    

    handle->video_stream->rtp_component->component_id,gsok);

  }

But we keep getting gsok = NULL :

[Fri Jun 24 16:17:34 2016] [WARN] SOCKETPATCH(0x7f2d34006c00) stream=2 component=1  return=(nil)

Clearly we must be too early in the socket's lifecycle; can you help us find a better spot to do this ?

kr,

J-

[Juju Trankilou]

Let it go, we did it: find the right place and write down the patch.

First test with few clients are ok but

Now I have to test it with many clients now

stay tuned ...

[Lorenzo Miniero]

Cool, keep us posted! Although I don't believe it's socket related, as a single user is supposed to receive the same amount of data no matter how many viewers are on a mountpoint. I think it's more probably related to the fact that, with more viewers, it takes more time for the plugin to relay every single packet, and it can't keep up, also from a source perspective. I'm thinking of ways to overcome this for both VideoRoom and Streaming, with a per-publisher thread that doesn't keep that one busy for the VideoRoom, and multiple relaying threads for both plugins.

L.

[Juju Trankilou]

Hi lorenzo,

 You were right, the patch did nothing more than set up directly to /proc the socket buffer

at least I'm sure now all sockets buffer videoroom has set up was at 12 MB.

But problem is still here. I'm sure now it is related to a feedback issue ( FIR / NACKS & PLI)

I noticed when I set up NACKS queue to high ( around 1000), the gateway loose her selfcontrol when users > 300 and go to 100% CPU. How to find out the right value in general ?

Something I noticed : If I use H264, janus quicly  set up to (about) 56k all headless clients , even I set up in cfg file much more (1 Mbit/s); to avoid that I have to use VP8 codec. With VP8 codec I can get close to 40K pkts /s , 250 Mbits with a 2048 bits 720p webcam flow in input. 

why these difference between botch codec ?

I want a behaviour of "good connection" viewers are not impact by bad ones. It seems it is not the case, even if I set up NACKs queue to 0 and FIR to 0.

Can you explain exactly to me what happen with janus when I set up the NAKs queue to 0 ? Will he ignore all demands regarding retransmission packets ? 

I read somewhere (from you)  browser still send FIR every 100 sec, still the case ?

I would like to understand, why janus start dropping packets to my "screened clients" when he have to deal with bad clients (I mean : clients which drops packets because they are too busy = 100% CPU), do you have an idea where I should start to look  ?

kr,

[Lorenzo Miniero]

Il giorno domenica 26 giugno 2016 02:39:35 UTC+2, Juju Trankilou ha scritto:

Hi lorenzo,

 You were right, the patch did nothing more than set up directly to /proc the socket buffer

at least I'm sure now all sockets buffer videoroom has set up was at 12 MB.

But problem is still here. I'm sure now it is related to a feedback issue ( FIR / NACKS & PLI)

I noticed when I set up NACKS queue to high ( around 1000), the gateway loose her selfcontrol when users > 300 and go to 100% CPU. How to find out the right value in general ?

The problem are not NACKs or the NACK queue. The problem, as I said in earlier posts, is that with too many listeners for the same source, the source probably can't keep up. The for loop to iterate on all listeners probably take more than the interval between packets, with too many listeners, and so you miss packets to relay. This coauses losses on the receivers end (they all miss those packets) so they all start NACKing. Multiply this to all the packets you start missing and it causes that issue. The solution is definitely not in disabling NACKs (you need those) but to fix the relaying problem for too many users. This is what I meant by multiple relaying threads and clustering of receivers for the same publisher: this would make sure the source is never blocked, and different threads would take the burden of relaying to multiple users at the same time. I'll probably have a look at this and prepare a PR next week, if I have time to do that.

In the meantime, you can verify it's not an issue of Janus with too many streams by just using multiple publishers or multiple mountpoints you feed with the same stuff (e.g., a gstreamer pipeline that sends to two or more mountpoints at the same time, or multiple publishers originated by the sae webcam/mic), and adding a limited number of receivers (e.g., 100-200) on each of them, which would be an application-level way of doing what I want to do within Janus itself.

 

Something I noticed : If I use H264, janus quicly  set up to (about) 56k all headless clients , even I set up in cfg file much more (1 Mbit/s); to avoid that I have to use VP8 codec. With VP8 codec I can get close to 40K pkts /s , 250 Mbits with a 2048 bits 720p webcam flow in input. 

why these difference between botch codec ?

I'm not familiar with the H.264 implementation in Chrome, you'll have to ask them. I guess they have a more limited access to the encoding properties with H.264 than they do with VP8, and so REMB feedback is lesse effective, but tha's just guessing.

 

I want a behaviour of "good connection" viewers are not impact by bad ones. It seems it is not the case, even if I set up NACKs queue to 0 and FIR to 0.

Can you explain exactly to me what happen with janus when I set up the NAKs queue to 0 ? Will he ignore all demands regarding retransmission packets ? 

I read somewhere (from you)  browser still send FIR every 100 sec, still the case ?

I would like to understand, why janus start dropping packets to my "screened clients" when he have to deal with bad clients (I mean : clients which drops packets because they are too busy = 100% CPU), do you have an idea where I should start to look  ?

See above.

L.

[Julien Godier]

Le 26 juin 2016 à 13:20, Lorenzo Miniero <lmin...@gmail.com> a écrit :

Il giorno domenica 26 giugno 2016 02:39:35 UTC+2, Juju Trankilou ha scritto:

Hi lorenzo,

 You were right, the patch did nothing more than set up directly to /proc the socket buffer

at least I'm sure now all sockets buffer videoroom has set up was at 12 MB.

But problem is still here. I'm sure now it is related to a feedback issue ( FIR / NACKS & PLI)

I noticed when I set up NACKS queue to high ( around 1000), the gateway loose her selfcontrol when users > 300 and go to 100% CPU. How to find out the right value in general ?

The problem are not NACKs or the NACK queue. The problem, as I said in earlier posts, is that with too many listeners for the same source, the source probably can't keep up. The for loop to iterate on all listeners probably take more than the interval between packets, with too many listeners, and so you miss packets to relay. This coauses losses on the receivers end (they all miss those packets) so they all start NACKing. Multiply this to all the packets you start missing and it causes that issue. The solution is definitely not in disabling NACKs (you need those) but to fix the relaying problem for too many users. This is what I meant by multiple relaying threads and clustering of receivers for the same publisher: this would make sure the source is never blocked, and different threads would take the burden of relaying to multiple users at the same time. I'll probably have a look at this and prepare a PR next week, if I have time to do that.

OK I get it. Can you tell me where (what source file, what line number) you are doing this loop ?

In the meantime, you can verify it's not an issue of Janus with too many streams by just using multiple publishers or multiple mountpoints you feed with the same stuff (e.g., a gstreamer pipeline that sends to two or more mountpoints at the same time, or multiple publishers originated by the sae webcam/mic), and adding a limited number of receivers (e.g., 100-200) on each of them, which would be an application-level way of doing what I want to do within Janus itself.

OK I will do the test. 

 

Something I noticed : If I use H264, janus quicly  set up to (about) 56k all headless clients , even I set up in cfg file much more (1 Mbit/s); to avoid that I have to use VP8 codec. With VP8 codec I can get close to 40K pkts /s , 250 Mbits with a 2048 bits 720p webcam flow in input. 

why these difference between botch codec ?

I'm not familiar with the H.264 implementation in Chrome, you'll have to ask them. I guess they have a more limited access to the encoding properties with H.264 than they do with VP8, and so REMB feedback is lesse effective, but tha's just guessing.

Actually I m not using chrome selenium client as he s bugged (deconnection) but firefox selenium's one. Is it the same ?

[Lorenzo Miniero]

I created a quick'n'dirty pull request for the Streaming plugin that does what I anticipated (more details on the PR text)

https://github.com/meetecho/janus-gateway/pull/568

Please try that version and let me know if it improves the quality of streaming/experience with high numbers, or if I went the wrong way with this.

Lorenzo

To unsubscribe from this group and all its topics, send an email to meetecho-janus+unsubscribe@googlegroups.com.