Kurento Vs. Jitsi Videobridge Scalability and Performance

[Ayaan]

We are using Kurento as a SFU and on a server that has 40 vCPUs (Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz), it tends to max out 100% CPU when there is around 500-700 user sessions in a room. 

Looking at Jitsi Videobridge https://jitsi.org/jitsi-videobridge-performance-evaluation/ performance, it claims that is can handle "1056 Streams; Bitrate mean: 550.4Mbps; CPU usage mean: 20.3%", which means it can handle 1056 streams with just 2 CPUs (their test server machine is equipped with a quad-core Intel ® Xeon® E5-1620 v2 @ 3.70GHz CPU)

So, why Kurento is very CPU intensive even when we use it as a SFU? Is there anything special we need to do to make it more scalable like Jitsi?

In our last 3 years of Kurento in production, it was great for one-o-one sessions and small group conferences up to 5-10. When we use it to deliver webinars with over 500 attendees, it usually crashes or maxes out the CPU cores. We usually reserve high-end servers with minimum 40 cores, but no luck. 

Kurento clustering is another puzzle. There is no official ElasticRTC support and latest builds available in AWS after Twilio's acquisition of Kurento. I am not sure anyone else have successfully clustered Kurento using Hazelcast (or Redis, not sure??) and optimized it for enterprise grade service for end users. 

AWS is not cheap. Even if we go with C5 compute optimized instances, we need to provision servers to meet capacity in advance. Let's say you are a SaaS provider, and you have 2 x c5.2xlarge instances which brings to total 16 vCPUs. Each core may handle up to 20 sessions, so total 320 sessions. As a SaaS provider, the traffic is unpredictable. Either you have to spin multiple instances in advance to meet a 1000 concurrent session scenario or rely on auto scaling to do that. With auto scaling, there is warm up time for the new instance to come alive, and until then new users joining the rooms may have a bad experience. 

Ideally, number of sessions per core needs to be higher so we can save on AWS billing cost in the longer run and capacity can be reserved in advance for a predictable workload. However, Kurento clustering is not something readily available and it may be possible by reaching out to Kurento professional services, but the performance of Kurento being a SFU is again questionable when we compare the numbers with Jitsi Videobridge. 

Hope someone came across these concerns before. Any feedback or thoughts on this will be highly appreciated.

[Micael Gallego]

Hi Ayaan,

We are continuosly improving Kurento Media Server performance. As other media servers also are doing. There are six good webrtc SFU media servers out there with good performance characteristics always improving.

Kurento has many features like transcoding and the ability to use media filters that impose restrictions on how is architected. But we are working in ways to improve this limitations to improve performance.

Regarding to horizontal scalability, ElasticRTC is no longer available, but the current dev team is working on a new similar service, but based on OpenVidu platform. In that way, we cover 95% of use cases of Kurento. 

As a final note, in the future we plan to make OpenVidu multi-media server. In that way, you can use any compatible media server selecting it in a config file. And then, when a media servers evolve, you could always pick the best.

Regards

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/2151fc4c-86b0-4598-906d-a277c8afbaf1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ayaan]

Thanks Michael. Do you know the timeline for OpenVidu multi-media server release? Our management is pushing to migrate from Kurento to Jitsi for scalability reasons, but personally I like Kurento as we have been using it for nearly 3 years. :)

[Micael Gallego]

I can't tell you a concrete deadline for that. But we hope to have it in Q1 of 2019 with other advanced management solutions. 

We plan to have performance improvements in Kurento Media Server in the following weeks.

Regards

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/e3995121-6f04-471a-a7de-e6a7a49a7a1c%40googlegroups.com.

[Ayaan]

Thanks Micael. Looking forward to see the performance improvements.

[Jon Ruddell]

You may be interested in this recent post comparing the available SFUs on the market and how well they handle load: https://webrtchacks.com/sfu-load-testing/ . Make sure to read carefully to understand how they tested (multiparty rooms of n=7 so 1 user is usually 7 peer connections).  The full paper is linked at the bottom of the post, as are the slides which show the charts broken down by each SFU.

Best,

Jon

[Micael Gallego]

We are working hard to improve Kurento numbers on that performance comparison. It seems there is a problem with libnice.

Please stay tuned to an updated post in webrtchacks blog

--

You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/e2233de9-7276-4df2-aa4f-cc9a4e2e7b6f%40googlegroups.com.

[Ayaan]

I know it's old thread. Do we have any published metrics to compare Kurento 6.13.1 with Jitsi bridge?

As per https://jitsi.org/jitsi-videobridge-performance-evaluation/, Jitsi can handle the following with Intel ® Xeon® E5-1620 v2:

53 conferences with 5 people

However, Kurento is well over 100% CPU in our tests with no transcoding with just 10 people streaming both audio and video.

On Thursday, 25 October 2018 03:01:11 UTC-4, Micael Gallego Carrillo wrote:

We are working hard to improve Kurento numbers on that performance comparison. It seems there is a problem with libnice.

Please stay tuned to an updated post in webrtchacks blog

El mié., 24 de octubre de 2018 23:42, Jon Ruddell <jonatha...@gmail.com> escribió:

You may be interested in this recent post comparing the available SFUs on the market and how well they handle load: https://webrtchacks.com/sfu-load-testing/ . Make sure to read carefully to understand how they tested (multiparty rooms of n=7 so 1 user is usually 7 peer connections).  The full paper is linked at the bottom of the post, as are the slides which show the charts broken down by each SFU.

Best,

Jon

On Thursday, October 18, 2018 at 7:33:19 PM UTC-7, Ayaan wrote:

We are using Kurento as a SFU and on a server that has 40 vCPUs (Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz), it tends to max out 100% CPU when there is around 500-700 user sessions in a room. 

Looking at Jitsi Videobridge https://jitsi.org/jitsi-videobridge-performance-evaluation/ performance, it claims that is can handle "1056 Streams; Bitrate mean: 550.4Mbps; CPU usage mean: 20.3%", which means it can handle 1056 streams with just 2 CPUs (their test server machine is equipped with a quad-core Intel ® Xeon® E5-1620 v2 @ 3.70GHz CPU)

So, why Kurento is very CPU intensive even when we use it as a SFU? Is there anything special we need to do to make it more scalable like Jitsi?

In our last 3 years of Kurento in production, it was great for one-o-one sessions and small group conferences up to 5-10. When we use it to deliver webinars with over 500 attendees, it usually crashes or maxes out the CPU cores. We usually reserve high-end servers with minimum 40 cores, but no luck. 

Kurento clustering is another puzzle. There is no official ElasticRTC support and latest builds available in AWS after Twilio's acquisition of Kurento. I am not sure anyone else have successfully clustered Kurento using Hazelcast (or Redis, not sure??) and optimized it for enterprise grade service for end users. 

AWS is not cheap. Even if we go with C5 compute optimized instances, we need to provision servers to meet capacity in advance. Let's say you are a SaaS provider, and you have 2 x c5.2xlarge instances which brings to total 16 vCPUs. Each core may handle up to 20 sessions, so total 320 sessions. As a SaaS provider, the traffic is unpredictable. Either you have to spin multiple instances in advance to meet a 1000 concurrent session scenario or rely on auto scaling to do that. With auto scaling, there is warm up time for the new instance to come alive, and until then new users joining the rooms may have a bad experience. 

Ideally, number of sessions per core needs to be higher so we can save on AWS billing cost in the longer run and capacity can be reserved in advance for a predictable workload. However, Kurento clustering is not something readily available and it may be possible by reaching out to Kurento professional services, but the performance of Kurento being a SFU is again questionable when we compare the numbers with Jitsi Videobridge. 

Hope someone came across these concerns before. Any feedback or thoughts on this will be highly appreciated.

--
You received this message because you are subscribed to the Google Groups "kurento" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kur...@googlegroups.com.

[Hemrajsinh Gharia]

I am keenly interested in knowing the performance metrics of the latest kurento. Along with that, I would like to share some metrics from our production. We are using Kurento for audio-only commentary. We have 2 Kurento. One is for commentators, where 2-3 commentators join and their audio gets through Composite unit and the combined stream goes out to another Consumer Kurento. Let me share the Consumer kurento stats on which at peak around 600 users/consumers get connected through WebRTC. Refer below stats:

We have hosted on aws EC2.

- Commentator Kurento: c5.large, MCU mode

- Consumer Kurento on: c5.2xlarge, SFU mode

Commentator kurento stats are not very encouraging. The Composite unit is very unstable and I advise to never use it. Our Commentator instance can anytime go up to 100-200% with only 2 commentators joined. As you can see in the above graph (of yesterday's), around at 22:22, we needed to restart the whole application as a broadcaster instance was having issues. So that's a different story and currently, I am working on switching to SFU mode on the commentator side as well, with hope to stabilize the things. That's the reason I am interested in the latest performance metrics. 

I would love to hear from Kurento team. How does the above stats look with audio-only (c5.2clarge have 8 vCPUs)? And are there any numbers which indicate the max capacity on a single instance/machine, kind of a guide which illustrates best practice to follow?

Thanks,

Hemraj

[Ayaan]

Hi Hemrajsinh - May I know what tool you are using to generate these performance graphs with Kurento? Thanks.

[Hemrajsinh Gharia]

We are using Prometheus for metrics. And you can collect the WebRTC stats from kurento, here is the example https://github.com/lulop-k/kms-monitoring-java/blob/master/src/main/java/org/kurento/tutorial/kmsmonitor/KmsMonitor.java. And for the presentation we use Grafana 

Thanks,

Hemraj

[Ayaan]

Great, thanks for the info.