performance issues

[Alex Kovalev]

Hello all.

We use the scalelite cluster (3 BBB nodes, Scalelite+Greenlight, NFS and TURN) for the team behind the NAT. It was installed in January 2021. All works well except the performance. When number of users in one room more then 40 peoples we see the performance degradations with audio, screenshare (black screen) and session breaks down for some people. They can not join to this room after that once again. All BBB nodes have the same resources: 8 CPU, 16Gb RAM.  We enabled multiplekurento settings to split streams and improve the performance. But this didn't help us much. Also we reduced video quality for all. To catch the issue I installed all-in-one monitoring where I see that the nodes have a lot of  hardware resources in peak hours and number of users per node not prevent 50. It is the strange situation, because I know that the typical number of users per BBB node is 200.  We don't record sessions often. During the day we have 4-10 sessions per node.  Next step we thinking of is to migrate from SFU to MCU, but we are not sure that it will help.

Could you please help us to find root cause of performance issue? Any help is appreciated.

Thanks,

Alex 

[sd...@distancelearning.cloud]

Team behind the Nat.    are you hosting all this on premise behind your enterprise routers and firewalls?

What is the symmetrical bandwidth?

 

Sounds like network related if after 40 users have issues.

 

Regards,

Stephen

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-Setup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/2c4ee69c-fdd3-4614-936c-0853ba80ede9n%40googlegroups.com.

[Fabrice Rouillier]

This is a known problem that has been corrected in bbb 2.2.33 (see https://groups.google.com/g/bigbluebutton-setup/c/lIX-GVrdZO8)

Just update  by simply running the bbb-install.sh script 

Then , in addition, you should set the variable  networkInterfaces to your default network interface (probably ens2 or ens3) in the file  /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini

[Alex Kovalev]

Thank you Stephen!

Yes, we use on premise installation behind enterprise routers and firewalls.  The symmetrical bandwidth is 1.05 Mbits/sec.

понедельник, 29 марта 2021 г. в 22:30:16 UTC+3, distancelearning.cloud:

[Alex Kovalev]

Thank you Fabrice!

It seems this is our case. I will update.

вторник, 30 марта 2021 г. в 11:28:14 UTC+3, Fabrice Rouillier:

[Alex Kovalev]

Hi Fabrice, Stephen. I updated all BBB nodes to the latest version (2.2.36)  and added CPU, but still has the performance issues: ~40 users can be connected simultaneously.

вторник, 30 марта 2021 г. в 14:14:39 UTC+3, Alex Kovalev:

[Alex Kovalev]

I updated   /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini  with  variable  networkInterfaces=ens160.

But seems it not help for us. I can't see the errors in logs. 

Any help will be appreciated.

понедельник, 19 апреля 2021 г. в 22:38:17 UTC+3, Alex Kovalev:

[Eugene Vlasov]

You say that "The symmetrical bandwidth is 1.05 Mbits/sec." It is very low. You need 250 Mbits/sec bandwidth (symmetrical) or more to run BigBlueButton in production.

See https://docs.bigbluebutton.org/2.2/install.html#minimum-server-requirements and https://docs.bigbluebutton.org/support/faq.html#bandwidth-requirements

Eugene V.

LMS hosting and BigBlueButton services

[Alex Kovalev]

Yes, I know. I provided to my client this info. However, he want to a proof.  Do you know the tool can do stress test for BBB where I can see the bandwidth issues?
Is it possible that BBB v.2.2.36 hasn't fixes that exist in 2.2.33 for this issue (not more than 40 users per room)? 

вторник, 20 апреля 2021 г. в 11:47:04 UTC+3, eugen...@gmail.com:

[Fabrice Rouillier]

Le mardi 20 avril 2021 à 10:59:13 UTC+2, avkov...@gmail.com a écrit :

Yes, I know. I provided to my client this info. However, he want to a proof. 

with 1 Mbps ... it a joke ?

Do you know the tool can do stress test for BBB where I can see the bandwidth issues?

with a unique connexion, just share a screen and observe the bandwidth with htop and you will see it at 100%

 

[Alex Kovalev]

I checked bandwidth again between load balancer and BBB nodes:

TCP bandwidth: 

root@P-SCL-LOADBAL:~# iperf -s

------------------------------------------------------------

Server listening on TCP port 5001

TCP window size: 85.3 KByte (default)

------------------------------------------------------------

[  4] local 10.101.221.200 port 5001 connected with 10.101.221.202 port 35562 (peer 62719.64774.65524-unk)

connect failed: Connection refused

[  6] local 10.101.221.200 port 5001 connected with 10.101.221.202 port 35680

[ ID] Interval       Transfer     Bandwidth

[  6]  0.0-146.2 sec  10.1 GBytes   591 Mbits/sec

UDP bandwidth:

^Croot@P-SCL-LOADBAL:~# iperf -s -u -p 15000

------------------------------------------------------------

Server listening on UDP port 15000

Receiving 1470 byte datagrams

UDP buffer size:  208 KByte (default)

------------------------------------------------------------

[  3] local 10.101.221.200 port 15000 connected with 10.101.221.202 port 40532

[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams

[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec   0.227 ms 2147481862/2147482755 (1e+02%)

Do I understand correctly, we need to estimate UDP bandwidth for audio, video and screenshare instead of TCP bandwidth?

I specify UDP transport  in /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini  for TURN server connection like this:

turnURL=someuser:some...@xx.xx.xx.192:3478?transport=udp

вторник, 20 апреля 2021 г. в 12:33:36 UTC+3, Fabrice Rouillier:

[Paulo Lanzarin]

I'll be honest: I did not read the full thread. But if your server bandwidth is 1 mbps, forget it. It just won't work.
Look at the recommended bandwidth specs in the docs: https://docs.bigbluebutton.org/support/faq.html#what-are-the-minimum-bandwidth-requirements-for-the-bigbluebutton-server

Estimating bandwidth is a complicated endeavor. You can make raw calculations on bandwidth of cameras, audio and screen via the configured values.

Thing is: bitrate for all of those is configurable, so it also depends. Stock values are generally (number-of-cameras²) * (200kbps) + (48kbps)[audio] + (1.5mbps)[screen].

And those are generally (as almost always, unless you have something very specific in mind) UDP in the server side

But then again that estimate is skewed because you'll need to account for packet loss and re-transmission which generally generate noticeable upticks in bandwidth

usage. And the aforementioned formula changes drastically if you have video pagination enabled or dynamic camera profiles.
And then you have to account for client data transfers. And [...].

So, my 2cts: 1 mbps is not nearly enough. Best way to estimate is actually put a server in the wild with plenty of bandwidth available, install a monitoring

tool (there's a great prometheus exporter for BBB out in the wild; search github). Let it run for a week. Get the high average percentile of bandwidth usage

for that week. Put your +x% safety net over that and come up with an actually battle tested value.

> I specify UDP transport  in /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini  for TURN server connection like this:

> turnURL=someuser:some...@xx.xx.xx.192:3478?transport=udp

Advice: unless you have a very specific setup, you should never have to set a TURN server in Kurento. And please don't do

so if you don't need it.

s,

prlanzarin

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-Setup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-s...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/37586eaf-5a11-466e-8e32-9830e437f503n%40googlegroups.com.

[Vasyl Ostrovskyi]

I think that iperf lies. I tried the same commands from my home to my bbb server and got the same 1.05 Mbits/sec for UDP, while I have 100 mbit/s uplink and my bbb works stable. Try something else, e.g. cli version of speedtest.

Regards,

 Vasyl

20.04.21 14:56, Alex Kovalev пише:

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-Setup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-s...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/37586eaf-5a11-466e-8e32-9830e437f503n%40googlegroups.com.

[Alex Kovalev]

Iperf set default bitrate for UDP to 1Mbits/sec. I experimented with  bitrate values ( -b) from 10M to 5000M and got  result ~814 Mbits/sec for UDP.

It seems to be true and as I understand this value meets bandwidth requirements.

We use turnURL because our TURN/Scalelite/BBB nodes behind the NAT. 

I installed "All-in-one" monitoring on all BBB nodes and see that number of users not prevent of 40.

Now I want to understand how to generate workload (stress testing) and get performance bottleneck. 

вторник, 20 апреля 2021 г. в 15:52:31 UTC+3, vasy...@gmail.com: