want to use bbb with 500

[Med Majdi Ben aoune]

Hello everyone, please I want to use bbb with 500 user, which one and the server configuration to establish this number of users !!!

[Fred Dixon]

We recommend running a standard BigBlueButton server with 150-200 users, see

 https://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support

There is no configuration change that will get you to 500 users on BigBlueButton 2.2 server.  We're working on improving that limit in BigBlueButton 2.3 (currently under development).

In your other post, you referenced FreeSWITCH.  If you have all users joining with a microphone, then FreeSWITCH may be the bottleneck that the server reaches.  Fortunately, FreeSWITCH uses multiple cores, so moving to a 16 core server (over the 8 cores recommended by the docs) will enable you to support more users with microphones.

Regards,... Fred

On Mon, Jan 18, 2021 at 5:41 AM Med Majdi Ben aoune <mmba.p...@gmail.com> wrote:

Hello everyone, please I want to use bbb with 500 user, which one and the server configuration to establish this number of users !!!

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/59365e30-9190-4489-a706-428d0789c207n%40googlegroups.com.

--

BigBlueButton Developer

http://bigbluebutton.org/

Like BigBlueButton?  Tweet us at @bigbluebutton

[basisbit]

We have a bunch of Hetzner AX61 servers, which regularly reach 450 concurrent voice participants (real-world users) split up in 60 sessions (including bunch of breakout rooms), but that dedicated server has a 12 core (24 threads) CPU running at ~4,1 GHz and optimizing NodeJS to not occasionally hit some limit with that user count is quite tiresome. We expect to get these servers to ~ 750 real world users once 2.3.x is stable. Our setup is very much optimized for handling lots of concurrent users, so recording and logging and some of other stuff is disabled.

I'd suggest you to set up a couple of BBB servers instead and use Scalelite for horizontal scaling by distributing new sessions to the different servers - see https://github.com/blindsidenetworks/scalelite . Also make sure to use the current stable version of BBB.

[bellmse...@gmail.com]

Same here, 12 Cores on Intel CPU, about 420concurrent users in about 30 session, with recording and logs available. Expecting about 500 users on the Server.

[basisbit]

Just a short update: after some changes the last weekend, we had 550 users (freeswitch audio + kurento listen-only count) on each of our AX61 servers a bunch of times without total CPU or NodeJS single thread going above 80%, and hitting 600 concurrent users a few times when load balancing was a bit unfortunate.

[sd...@distancelearning.cloud]

Nice Graphs,

 

Can you track how many were full audio in freeswitch vs kurento listen only.  Would like to see 400 freeswitch users cpu utilization on your amd cores?

 

Regards,

Stephen

 

From: bigblueb...@googlegroups.com <bigblueb...@googlegroups.com> On Behalf Of basisbit
Sent: Friday, January 22, 2021 6:25 PM
To: BigBlueButton-dev <bigblueb...@googlegroups.com>
Subject: Re: [bigbluebutton-dev] want to use bbb with 500

 

Just a short update: after some changes the last weekend, we had 550 users (freeswitch audio + kurento listen-only count) on each of our AX61 servers a bunch of times without total CPU or NodeJS single thread going above 80%, and hitting 600 concurrent users a few times when load balancing was a bit unfortunate.

 

bellmse...@gmail.com schrieb am Freitag, 22. Januar 2021 um 23:03:23 UTC+1:

Same here, 12 Cores on Intel CPU, about 420concurrent users in about 30 session, with recording and logs available. Expecting about 500 users on the Server.

basisbit schrieb am Montag, 18. Januar 2021 um 16:31:31 UTC+1:

We have a bunch of Hetzner AX61 servers, which regularly reach 450 concurrent voice participants (real-world users) split up in 60 sessions (including bunch of breakout rooms), but that dedicated server has a 12 core (24 threads) CPU running at ~4,1 GHz and optimizing NodeJS to not occasionally hit some limit with that user count is quite tiresome. We expect to get these servers to ~ 750 real world users once 2.3.x is stable. Our setup is very much optimized for handling lots of concurrent users, so recording and logging and some of other stuff is disabled.

 

I'd suggest you to set up a couple of BBB servers instead and use Scalelite for horizontal scaling by distributing new sessions to the different servers - see https://github.com/blindsidenetworks/scalelite . Also make sure to use the current stable version of BBB.

Fred Dixon schrieb am Montag, 18. Januar 2021 um 15:59:24 UTC+1:

We recommend running a standard BigBlueButton server with 150-200 users, see

 

 https://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support

 

 

There is no configuration change that will get you to 500 users on BigBlueButton 2.2 server.  We're working on improving that limit in BigBlueButton 2.3 (currently under development).

 

In your other post, you referenced FreeSWITCH.  If you have all users joining with a microphone, then FreeSWITCH may be the bottleneck that the server reaches.  Fortunately, FreeSWITCH uses multiple cores, so moving to a 16 core server (over the 8 cores recommended by the docs) will enable you to support more users with microphones.

 

 

Regards,... Fred

 

On Mon, Jan 18, 2021 at 5:41 AM Med Majdi Ben aoune <mmba.p...@gmail.com> wrote:

Hello everyone, please I want to use bbb with 500 user, which one and the server configuration to establish this number of users !!!

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/59365e30-9190-4489-a706-428d0789c207n%40googlegroups.com.

 

--

BigBlueButton Developer

http://bigbluebutton.org/

 

Like BigBlueButton?  Tweet us at @bigbluebutton

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/ba022024-3f81-4e17-8648-99741db7380fn%40googlegroups.com.

[basisbit]

sure, approximately 90% of attendees were also connected to freeswitch. Approximately 8% of users were connected listening-only to Kurento. The missing ~2% are probably users connecting with a second device for whiteboard drawing or screen share or just failing audio setup, I guess.

Those numbers seem to be rather stable and the high percentage of freeswitch users is probably caused by most users at those peak times being students from schools.

Regarding CPU usage, we see approximately 1,75% of CPU usage in htop per user on a system with 2 threads per CPU core. So we see during peak (600 attendees) ~4 and a half of the 12 CPU cores being used up by freeswitch.

[Florian Erfurth]

Hello Stefan,

did you install one BBB instance on that AX61 or did you "split" by using virtualisation and multiple root servers on that machine?

How did you optimize the NodeJS?

Best regards

Floh

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/dadc5032-bac6-4e1e-9a41-3a699d9c18dfn%40googlegroups.com.

[sd...@distancelearning.cloud]

Thanks for the info… makes sense 540 full audio take about 12 cores.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/dadc5032-bac6-4e1e-9a41-3a699d9c18dfn%40googlegroups.com.

[Stefan L.]

@Florian No virtualization, it's only customized BBBs installed on bare metal.

@sdame, 540 freeswitch users take up approximately 9,5 CPU threads, not CPU cores. There still is enough CPU time for Kurento to handle for example 180 webcam users (numbers from today's grafana stats), for NodeJS and for the listen-only users to still have decent audio. And there is still enough spare CPU time so that uploaded presentation can be converted or smaller DDoS can be handled and also so that the CPU does not throttle down too much because of it hitting the TDP.

You received this message because you are subscribed to a topic in the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bigbluebutton-dev/tLOeBFyIWiU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/01f701d6f41a%244519e3e0%24cf4daba0%24%40distancelearning.cloud.

[hyunho]

How are you digesting 180 user users? 16 core CPU server 30 people can not digest. Have you tuned node.js? I would appreciate if you can tell me how.  

2021년 1월 27일 수요일 오전 5시 35분 15초 UTC+9에 basisbit님이 작성:

[basisbit]

> How are you digesting 180 user users?

Dedicated bare metal server with high single-threading speed.  

[Fred Dixon]

Hi Med,

We're working on scalability improvements in 2.3-dev -- specifically, the bottleneck of having a single node process handle all the messages has been refactored into front-end and back-end node processes.  See

   https://groups.google.com/g/bigbluebutton-dev/c/MvpY7S_n60s/m/_DiovzflAAAJ

We would be very interested if your able to test the latest build (see above link) in a non-production environment and let us know if you see an increase in scalability.

Regards,... Fred

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/fcb2f242-1bf0-4f3e-9d8b-237665decd99n%40googlegroups.com.

[hyunho]

3.7Ghz 16 threads 32 gigabyte memory. Lack of specifications. It doesn't work anymore, and digestion of more than 200 people is okay.
 There seems to be a fundamental problem with kurento and node.js, how do you solve this??  

2021년 2월 24일 수요일 오후 6시 29분 48초 UTC+9에 basisbit님이 작성:

[Fred Dixon]

Hi,

> There seems to be a fundamental problem with kurento and node.js, how do you solve this??  

The work we're doing in 2.3-dev should get you more users on a server.  We recommend trying out the latest release (2.3-alpha7) on a non-production server to see if you can get a larger number of users on the server.

Regards,... Fred

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/7b86637b-438f-40b2-8c07-8037c49463aan%40googlegroups.com.

[sd...@distancelearning.cloud]

Hi Stefan,

 

Question on Ryzen,   are you running your AX61 with hyper threading disabled?    Just provisioned a 3900XT with 2.3 alpha-7 for testing.  Was going to run some stress testing

 

Any suggestions or lessons learned on optimizations of ubuntu 18.04 would be much appreciated.   

 

Regards,

Stephen

 

From: bigblueb...@googlegroups.com <bigblueb...@googlegroups.com> On Behalf Of basisbit
Sent: Wednesday, February 24, 2021 4:30 AM
To: BigBlueButton-dev <bigblueb...@googlegroups.com>
Subject: Re: [bigbluebutton-dev] want to use bbb with 500

 

> How are you digesting 180 user users?

 

Dedicated bare metal server with high single-threading speed.  

hyunho schrieb am Mittwoch, 24. Februar 2021 um 04:31:27 UTC+1:

How are you digesting 180 user users? 16 core CPU server 30 people can not digest. Have you tuned node.js? I would appreciate if you can tell me how.  

2021년 1월 27일 수요일 오전 5시 35분 15초 UTC+9에 basisbit님이 작성:

@Florian No virtualization, it's only customized BBBs installed on bare metal.

@sdame, 540 freeswitch users take up approximately 9,5 CPU threads, not CPU cores. There still is enough CPU time for Kurento to handle for example 180 webcam users (numbers from today's grafana stats), for NodeJS and for the listen-only users to still have decent audio. And there is still enough spare CPU time so that uploaded presentation can be converted or smaller DDoS can be handled and also so that the CPU does not throttle down too much because of it hitting the TDP.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/fcb2f242-1bf0-4f3e-9d8b-237665decd99n%40googlegroups.com.

[basisbit]

Hello Stephen,

so far we are running current BBB 2.2.x with the NodeJS optimizations mentioned in the linked issues (which will be part of 2.2.32), but with default ubuntu-16.04-hwe config. No network card optimizations (those will only help when your uplink is congested -> better solution is to lower video bitrates if necessary so that you never get close to congested uplink), no disabling of Hyperthreading. However, I suggested to our team to assign two CPU threads of one CPU core to only be used by NodeJS (plus its garbage collector) and then pin NodeJS to those. That should help in case of total CPU usage being above 70% or so, where without this optimization the CPU core resources where NodeJS is running on might be used up by another process/thread running on the other CPU thread of that CPU core.

Didn't have the free time to test this in production yet, but should have no disadvantages on a system with more than 8 CPU cores.

3900XT should do nicely, probably supports a bit higher number of users than our servers thanks to the ~30W higher max power consumption / real TDP limit. Maybe 700 users with 2.2.32 or 2.2.31 with the mentioned NodeJS garbage collector optimizations. At that point, you'll notice that the next bottleneck currently is Kurento.

Best regards,

Stefan

[basisbit]

One major problem however with 2.2.x in production is when NodeJS is already at medium load (for example 400 concurrent users), and then a big session moderator decides to open up a bunch of breakout rooms. That will overwhelm the process with too much work, so that for a few minutes, events like drawing on the whiteboard or chat messages don't get processed in time by the server and then don't appear. If that overload takes too long, some users will disconnect, try reconnecting and then even further overload the server.

However, that problem should be solved with the multiple-kurento approach from 2.3.x.

It would be nice if in the mean time, someone figures out a way to disable Breakout-Room creation for big sessions - maybe a check for session-size > 100 attendees could be implemented in JavaScript in bbb-html5 and added to the upcoming 2.2.x release?

[sd...@distancelearning.cloud]

Thanks for info,

 

I justed updated and enabled HWE for 2.3-alpha7

 

sudo apt-get install --install-recommends linux-generic-hwe-18.04 

 

Going to do some tests with multiple html5 front/back end’s configured

 

Also enabled kurento round-robin to push kurento some to see how many users with cams can get into a meeting reliability.

Going to limit client to 5 cams with pagination.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/e4ec242b-e2a5-4269-9e20-b98dc9817743n%40googlegroups.com.