BigBlueButton could'n not handle 1000 online users in 1 room
Shukurillo Baikhanov
BigBlueButton could'n not handle 1000 online users in 1 room ans started freezing and kicking participants
But server have enough resources and wheh there was 1000 user online server resouse usage was only 50%
Any advice? solution? why this happened
We have 10 BBB physical server connected over Scalelite
each BBB server has:
48 CPU core
64 GB Memory
10 GB network
This is graph server usage when there was 1000 online user
This is monitroing of BBB when there was 990 0nline user is stopped send metrics of BBB
Anton Georgiev
Shukurillo Baikhanov
BigBlueButton Server 2.7.6
Yes default installation via script
Anton Georgiev
Anton Georgiev
|
--
You received this message because you are subscribed to a topic in the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bigbluebutton-dev/Il9v8OhoL1E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/143ab2e8-f05f-48a7-82ce-9b514bc5b74en%40googlegroups.com.
Shukurillo Baikhanov
I changed NUMBER_OF_FRONTEND_NODEJS_PROCESSES=8 but still freezes, and still kicking participants
Is there any other config to apply for large room mettings? or solution
Here is meeting where 200 online users
100 voice
100 camera on
and in graph u can see it kicked about 100 particpants
Server resource at that time frame only 25%
Marcel Hellkamp
Hi there,
did you enable video pagination? Because 100 video participants
in a single meeting sounds like a very unrealistic test scenario
and will absolutely crash your clients (not necessary the server)
if pagination is disabled. If all your test clients join as
moderator, they have no pagination limit by default and all will
try to display 100 videos at once. That would be 10.100 Videos
streams, plus 10.000 more for the audio-only participants. Plus
audio streams.
Oh, and how did you test this? Are you sure your test-resources are not the limiting factor? Freezing and videos and disappearing participants sounds a lot like your clients did not respond anymore, or not quickly enough. Do your test clients have enough CPU and RAM available? How about the network connection between test clients and server?
If that's sorted out, did you check your logs for error messages?
Did you check which processes consumed the most CPU time during
your test or reached a significant limit? Some processes are
single-core, so for those processes reaching roughly 1/48 (0.02%)
CPU utilization for multiple seconds is a clear sign that this
particular process is overwhelmed and needs attention.
Start monitoring your connection counts (number of open sockets). If you see a plateau there, it means that either your network stack can't handle the connection count, or you need to increase certain file descriptor limits or process counts on your BBB server.
That said, 1000 active participants in a single meeting is currently unrealistic on any hardware and ten times the officially recommended limit. I have seen meetings with ~500 participants, but only one or two webcam streams and most of the listeners joining in listen-only mode. That works with a carefully configured and beefy server.
VG, Marcel
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/3157dce4-dad3-42be-a702-6aebcac70497n%40googlegroups.com.
Shukurillo Baikhanov
Firstly Thanks for brief explanation.
It was real meeting where 108 cameras turned on and presentation
1) about video pagination - in that room video and presenation was enabled
4) Connection on server via 10GB and input output reached max 200MB
But why there is freeze, lag, delay - even there is enough resources?
Marcel Hellkamp
It was real meeting where 108 cameras turned on and presentation.
Why 108 cameras? There is no scenario (I know of) where you need
to look at 108 spamp-sized pictures of bored faces at the same
time. For large meetings only the presenter should have their
camera enabled.
1) about video pagination - in that room video and presenation was enabled
I talked about video pagination (limiting how many videos are visible at the same time). Showing 100 videos in a browser window is insane. The default of 48 (grid layout) is also way too much for large meetings. We deployed a limit of 24 videos per page.
2) about the crash Presentation did showed up to participants after several try it showed slide, voice was clear, participants joined as normal
Sounds like the internet connections (or CPUs) of the individual
participants were overwhelmed, not the BBB server. Probably
because of too many video streams displayed at the same time.
Audio is prioritized, so that often works while video stutters or
freezes.
3) About background processes I did not checked which process taking resource because server has enough resource.
As I already explained, some processes are limited to a single
core and need tuning even if your server has dozens of idle cores.
You won't see that in your dashboard.
4) Connection on server via 10GB and input output reached max 200MB
As I already explained, the number of connections is also
relevant. As well as latency, package loss, jitter, and the
throughput to individual participants. Also, most hosting services
sell 'up to' 10GB. That does not mean anything. Measure, don't
just assume.
Shukurillo Baikhanov
Anton, as you mentioned
I set NUMBER_OF_FRONTEND_NODEJS_PROCESSES=8
also I set NUMBER_OF_BACKEND_NODEJS_PROCESSES=4
Why there is only Max 8 and 4? I'm trying to set higher number 12/16 for better performance, but bbb not getting start after restart
Marcel,
We deployed a limit of 24 videos per page.