High CPU-Usage from kurento-media-server

[Florian Erfurth]

Hello together,

today I watched an international meeting with 10 participants. All webcams were enabled and video bitrate is set to 1,2-1,8 MBit/s (depends on which quality users picked). So everybody could see each other. All of them were deaf, that is why I increased the video bitrate.

While meeting was running, I noticed that the 4-Core CPU had 25% Load on all cores and there were no other running meeting. The load was caused by kurento-media-server. Is that normal? I don't think so, since many are talking about 200-2000 Users... and I'm talking about 10 users.

Someone suggested to Update Bigbluebutton to 2.2.3. Now I'm not sure if I looked in right place but do I still have Version 2.2.0 if I read following correct?

~$ apt show bigbluebutton

Package: bigbluebutton
Version: 1:2.2.0-5
Priority: extra
Section: default
Maintainer: ffd...@bigbluebutton.org
Installed-Size: 7.168 B
Depends: bbb-config
Homepage: http://bigbluebutton.org/
License: unknown
Vendor: BigBlueButon
Download-Size: 3.830 B
APT-Manual-Installed: yes
APT-Sources: https://ubuntu.bigbluebutton.org/xenial-220 bigbluebutton-xenial/main amd64 Packages
Description: Open source web conferencing platform (bbb)

Is 2.2.3 still beta? Since I want to avoid beta-versions in production environment.

Best regards

Floh

[basisbit]

Hello,

to get the actual bbb version number, you can use 'bbb-conf --check' and the first line of the response will contain the BigBlueButton server version number. Anyways, the debian/ubuntu package 2.2.0-5 is actually 2.2.5, which is the most current version. Anyways, to answer your response: 1,8 Mb/s times 10 users each sending their data to 9 other users results in roughly 162Mb/s of traffic which kurento will have to forward, which explains the evenly spread 25% of CPU load. I'd guess that the CPU is also currently not at its maximum CPU frequency because of energy saving stuff, and we do not know your setup, thus can't know if your system is running on bare metal hardware or a VM (which would increase required CPU performance).

Anyways, the 200-2000 users which you mentioned are probably using a bunch of BigBlueButton servers plus using Scalelite (also open source) as session/room based load-balancer. If all users have their camera turned on, that will result in significantly less users you can serve with one physical server than when all or most users only participate using audio only. Increasing the video bitrate makes this even much more of a problem, because of the server load growing exponentially (each video-stream has to be forwarded to each other user of the same session/room).

BigBlueButton supports using speech recognition for writing and displaying captions. That might help a lot for your use case. A quick search only showed where the code was added, but not how to use that. Maybe someone else here or from commercial BBB support can help you with that. One of the related settings is 'enableDictation'

I hope this already help you a bit.

Best regards,

basisbit

[Florian Erfurth]

Hello  Basisbit,

thank you for your reply. It's KVM-Based RootServer with 4-Core (based on Intel Xeon Gold 6230 @ 2,1GHz) and 16GB RAM. I need to know how many users this one RootServer can handle if each room contains 30 and all Webcam is enabled (max 0,8MBit/s Videorate in that case).

I'll have a project with 880 peoples and 30 each room at same time. That is why I want to order several servers and will setup Scalelite.

Thank for telling me about speech recognition, I'll see into that for other cases. But these 10 persons were all deaf and none talked since they were signing. ;)

Best regards

Floh

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/b77a9f72-3b57-4fbc-829b-4d838261183a%40googlegroups.com.

[basisbit]

That sounds like a Netcup RS 2000 G8. Very affordable and nice for the price..... But: I would worry about the "guaranteed average" outbound network speed of "above" 200Mb/s. 30 people sharing webcam at the same time will be maybe (30 people * 0,8Mb/s video * 29 other people to send videostreams to) + (30 people *0,1Mb/s audio) = 700Mb/s of outbound traffic. So, I'd guess that the Netcup machines might not have the required network bandwidth all the time. (Also, does every participant have an internet connection which provides the required 24 Mb/s of download speed constantly?)

Also, I'd guess that the 4 CPU cores will have a difficult time handling those 696Mb/s of video data forwarding while also doing all the other stuff.

If you split those 30 people rooms into 2x 15 people rooms each, that would be 400Mb/s of outbound traffic per two rooms, which I'd be quite confident to handle fine with the mentioned netcup server.

I hope this answers your questions a bit. You might want to reconsider if really all those 880 people are going to be sharing their webcams with very high video quality at the same time. For a typical school class, you'll be absolutely fine with 0,2Mb/s video quality streams or lower, especially when you are going to see 30 people on one screen and each webcam is only a small part of the screen. You can also have the teacher use a higher quality video setting while the students are on low quality video.

PS: I already did sessions with 130 audio participants within that one session (no webcam sharing + disabled room recording) on a netcup R8 2000 and it worked okay.

To unsubscribe from this group and stop receiving emails from it, send an email to bigblueb...@googlegroups.com.

[Florian Erfurth]

Hello Basisbit,

Your reply is very valuable to me, thank you very much.

Yes, that *is* the Root Server I'm using since it has the best cpu/price-ratio. I assume it can deliver around 300MBit/s so I'm planning to test with 6x BBB (6x 300MBit/s = 1,8GBit/s) and add some more if not sufficient. They were telling me 3-5 Webcam connections for each room with 30 participants is sufficient. So I thought 6 BBB-Servers should be fine, but then the director told me "I want sometime that *everybody* turns on their webcam since we need to see each other sometimes for social contact because of exit restrictions due to the corona-situation.". That is why I'm calculating the worst case scenario with 30 webcams for each room for 880 users. :D but I'm sure it won't happen for all at the same time. I defined 0,3MBit/s as default and hope not everybody ramps up the  quality.

But my concern is the CPU if it raised to 25% with 10 webcams only (ok it was probably* 1,8MBit/s) so with 880 it would be 2200% => 22 RootServers are needed which means 22*4 = 88 cores. :-O

*) Is there a way to find out by API-Call which bitrate one user choosed? I have Postman here for API-calling for information, very useful.

I remember, when BBB was around Version 0.93 with flash-client, there I could define Video-Quality for attendee and moderator separately so I'd like to limit attendee to 0,3MBit/s and Moderator to 0,8MBit/s. Is that still possible, and how/where? I remember the setting was in video-module-settings (some xml file) of flash-client.

Best regards

Floh

To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/376282ab-546b-449b-a6b1-d8c2953cd6da%40googlegroups.com.