IMPORTANT: Improving performances

[Lorenzo Miniero]

Hi all,

as you may or may not know, the master version of libnice suffers from a throughput limitation, due to a global lock when sending packets: you can find more info in a post I wrote on their mailing list some time ago [1]. This basically means that, while under normal circumstances everything works great, when you start increasing traffic a lot, the libnice bottleneck will kick in and as a result Janus will struggle even though CPU/mem/bw are available.

The libnice developers are aware of this, and there are two patches you can use for removing those limitations:

1. an older patch, which we tested successfully ourselves, but that will never be merged [2];
2. a newer patch they're working on, which we didn't try yet, but will eventually be merged to master [3].

The older patch we refer to we tested intensively in several stress tests, which we also used to collect data for a paper we'll publish next month. It proves that, without this limitation, we manage to saturate the resources while we previously couldn't, thus serving MANY more sessions than we could before. As I was saying, although it works in our scenarios that older patch won't be merged as it has some issues the libnice developers aren't comfortable with, which explains why they're trying to solve it the right way in the other patch I mentioned.

If you know your server will serve a lot of traffic and you want to make sure you won't get into those bottlenecks, get rid of the libnice library you're using, and use one of the experimental branches I linked to instead. As anticipated, we only tried the older patch but not the other, so I can't give feedback on both: anyway, while the older patch is possibly a "safer" choice for production (but only because we stress-tested it in some scenarios), I encourage you to test the newer one and provide feedback to the libnice developers as well, as that will accelerate the merging process.

Hope this helps,

Lorenzo

[1] https://lists.freedesktop.org/archives/nice/2016-October/001328.html

[2] https://gitlab.freedesktop.org/libnice/libnice/merge_requests/12

[3] https://gitlab.freedesktop.org/libnice/libnice/merge_requests/13

[Mirko Brankovic]

Thanks for the tips Lorenzo,

We'll make sure to test the upstream libnice PR :)

--
You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Mirko

[Ben Mark]

A few inputs that I assume will help many others in improving performances

After removing the current libnice we had on the machine and replaced it with https://gitlab.freedesktop.org/libnice/libnice/merge_requests/12

then rebuilt janus version 0.4.4 release

We've done a load test of 1 publisher & 500 viewers 

The load test:

 the publisher refreshes the browser, reconnecting/republishing to janus 

every viewer reports how many seconds it took to after the refresh to get back the publisher's stream

We experimented 2 load tests 

1st load test config every client a turn server was configured janus was not configured with a turn server 

2nd load test config: every client a turn server was configured, janus was also configured with a turn server

the turn server we used:"coturn"

and the results were stunning.

without turn configured to janus: 7 seconds

with turn configured to janus: 50 seconds

number of seconds = the last viewer (from all 500) to receive the publisher's stream

Even though this is off topic, I believe this still can help others improve their performance in no time after reading this:

The client and janus turn configuration is abit obscured to me, if the client has turn configured , is it necessary to configure the janus as well???

I've read that a client can have a list of turn servers (to pick the closest one) why is the janus also configured with a turn server? and why is there only 1 turn server configure option

note: on the case where janus was configured with turn server, it seemed all viewer connections were bounded to some global lock exactly as described in the issue Lorenzo found in libnice 

[Alessandro Amirante]

https://janus.conf.meetecho.com/docs/FAQ.html#turn

--

[Béranger Kabbas]

Hi Lorenzo,

I tried janus gateway weeks ago but couldn't make it work past 8 RTSP streams (but the network was working fine, and the 8 video were smooth as baby skin), my conclusion based on the logs I saw was it was related to the signaling server having so much trouble answering calls, it this possible that by removing libnice may have fixed it ?

I am using https://github.com/mpromonet/webrtc-streamer in the meantime which doesnt have those kinds of problems (i got up to 64 video streams with this one) but it has other that I would like to avoid (like reencoding)

I will probably try anyways with this version but maybe you know already ?

[Mirko Brankovic]

Theoretically you don't need turn on Janus server side not even a stun sometimes, unless your server is behind public IP that will change over time.

If this happens then you should change cloud provider :d

--

You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Mirko

[Lorenzo Miniero]

We've managed to do thousands of streams on a single box, so you're likely doing something wrong. As Alessandro and Mirko told you, TURN on the Janus side is absolutely unneeded. That said, please post questions on other posts: this thread is for hints and annoucements on the libnice thing, and I'd rather not see its purpose diverged by unrelated matters.

Lorenzo

[Lorenzo Miniero]

Just FYI, the two branches I mentioned are now together in a new merge request by the libnice developers:

https://gitlab.freedesktop.org/libnice/libnice/merge_requests/17

Please test that, if you're interested in performance, as that's the patch that will end up in master, if it works as expected.

Lorenzo

[Lorenzo Miniero]

This PR has just been merged, which means that if you now get libnice master, it has this fix built-in: no need to pull from other branches anymore.

Kudos to the libnice developers for the huge effort on getting this issue sorted out!

Lorenzo

[Mirko Brankovic]

Oh good news.

Congrats on great work and efforts both on yours and libnice side.

Do you happen to know if they made a tag for official release? Like witch version we should count on?

--

[Lorenzo Miniero]

I don't know, they just merged it, so you can check the commit I guess. I know they wanted to release a new version soon, though.

Lorenzo

To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

[Mirko Brankovic]

Perfect, thanks :)

To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,

Mirko

[Ben Mark]

Great, will start testing it,

[Tristan Matthews]

On Thu, Nov 1, 2018 at 4:02 AM Ben Mark <bm73...@gmail.com> wrote:
>
> Great, will start testing it,

If anyone has benchmarks they'd be willing to share, we'd love to see
them. Also, I've found mutrace to be quite helpful in profiling lock
contention (it's available in distros):
http://0pointer.de/blog/projects/mutrace.html

Best,
-t

[Lorenzo Miniero]

You can find some info here: https://webrtchacks.com/sfu-load-testing/

We have other data in a different paper we published at the same conference (I don't think there's a video of the talk yet).

L.

>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

>>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

>>> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Regards,
>> Mirko
>>
>> --
>> You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

[Tristan Matthews]

On Thu, Nov 1, 2018 at 12:10 PM Lorenzo Miniero <lmin...@gmail.com> wrote:
>
> You can find some info here: https://webrtchacks.com/sfu-load-testing/

Thanks, I see a link to a certain thesis in there too ;)

> We have other data in a different paper we published at the same conference (I don't think there's a video of the talk yet).

Do you have a link for that paper?

Best,
t

>> >>>>> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

>> >>>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

>> >>> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Mirko
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

>> >> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

>> > To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

[Lorenzo Miniero]

Il giorno giovedì 1 novembre 2018 21:43:49 UTC+1, Tristan Matthews ha scritto:

On Thu, Nov 1, 2018 at 12:10 PM Lorenzo Miniero <lmin...@gmail.com> wrote:
>
> You can find some info here: https://webrtchacks.com/sfu-load-testing/

Thanks, I see a link to a certain thesis in there too ;)

> We have other data in a different paper we published at the same conference (I don't think there's a video of the talk yet).

Do you have a link for that paper?

I don't think it's available in the online proceedings yet.

These are the slides Simon presented though: https://prezi.com/aulm7gkvb7c5/measuring-janus-temperature-in-ice-land/

L.

 

>> >>>>> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

>> >>>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

>> >>> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Mirko
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

>> >> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

>> > To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups "meetecho-janus" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.

[be...@newrow.com]

libnice's community is restoring global locks.

master since nov 20 has the global locks back again

https://gitlab.freedesktop.org/libnice/libnice/commit/a05e9dcff179c2c4c6fc3feb422c20b07e936576

[lmin...@gmail.com]

Looks like it may be just for TURN/TCP and not everything. I've added a comment on their repo, let's see what happens.

Lorenzo

[Burak Senyuva]

Hi,

Are you planning to share Janus ICE Library which mentioned here: https://prezi.com/aulm7gkvb7c5/measuring-janus-temperature-in-ice-land/

Thanks,

Burak

[lmin...@gmail.com]

If you mean jice, it was already released for a long time: it was a pull request [1] that we closed in the meanwhile, since libnice has now been fixed. Just use libnice master and you'll be fine.

Lorenzo

[1] https://github.com/meetecho/janus-gateway/pull/1297