BigBlueButton stress test with all users video sharing

[Joy Banerjee]

Hi,

We are trying to do stress testing (all users - with video conference, with/without audio conference) on our BigBlueButton server. 

For one bbb server, I want to calculate how many meetings can be held concurrently with each meeting having 1 moderator and 40 attendees with all users having webcam on, such that CPU utilization doesn't exceed 80%. 

If I configure scalelite, how many bbb servers should I add for running 25 such concurrent meetings?

I tried the following load testing approaches:

• I wrote a js script which generates multiple tabs in browser (similar to bbb-test). But with webcam on, one person can open max 3 or 4 tabs (dependent on bandwidth). Also in each tab the video sharing and joining audio conference needs to done manually. Thus, manual stress testing is not feasible.
• I used testRTC (https://testrtc.com/) to join multiple meetings on a remote browser. I tried couple of ways such that the audio and video conference can be started without manual intervention on joining meeting, but didn't work. Thus, test failed.

PFB my doubts.

• Did your team do any stress testing with all user sharing video? If yes, could you please share the results.
• Can I do load testing directly with websockets or webrtc apis by using any tool like JMeter? If yes, could you please help me identify these apis?
• for bbb 2.2, can I configure all users to join audio and video conference automatically on joining meeting? I tried using params (as in https://docs.bigbluebutton.org/2.2/customize.html#passing-custom-parameters-to-the-client-on-join) and modifying PhoneModule in /var/www/bigbluebutton/client/conf/config.xml for autojoining audio conference but it didnt work. I didnt find any params for autojoining video conference.

Our bbb server config:

• version: bigbluebutton 2.2
• installed in EC2 - c5.xlarge instance which is configured using minimum server requirements as in https://docs.bigbluebutton.org/2.2/install.html#minimum-server-requirements
• However, we used shared instance in AWS instead of bare metal
• we have also configured a TURN server in t2.micro instance
• We'll create, join meeting through API calls from another application
• PFB output of sudo bbb-conf --check

BigBlueButton Server 2.2.16 (1987)

                    Kernel version: 4.4.0-1107-aws

                      Distribution: Ubuntu 16.04.6 LTS (64-bit)

                            Memory: 7814 MB

                         CPU cores: 4

/usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties (bbb-web)

       bigbluebutton.web.serverURL: https://vc.skillprep.co

                defaultGuestPolicy: ASK_MODERATOR

                 svgImagesRequired: true

/etc/nginx/sites-available/bigbluebutton (nginx)

                       server name: vc.skillprep.co

                              port: 80, [::]:80

                              port: 443 ssl

                    bbb-client dir: /var/www/bigbluebutton

/var/www/bigbluebutton/client/conf/config.xml (bbb-client)

                Port test (tunnel): rtmp://vc.skillprep.co

                              red5: vc.skillprep.co

              useWebrtcIfAvailable: true

/opt/freeswitch/etc/freeswitch/vars.xml (FreeSWITCH)

                       local_ip_v4: 172.31.29.111

                   external_rtp_ip: 3.7.178.84

                   external_sip_ip: 3.7.178.84

/opt/freeswitch/etc/freeswitch/sip_profiles/external.xml (FreeSWITCH)

                        ext-rtp-ip: $${external_rtp_ip}

                        ext-sip-ip: $${external_sip_ip}

                        ws-binding: :5066

                       wss-binding: :7443

/usr/local/bigbluebutton/core/scripts/bigbluebutton.yml (record and playback)

                     playback_host: vc.skillprep.co

                 playback_protocol: https

                            ffmpeg: 4.2.2-1bbb1~ubuntu16.04

/etc/bigbluebutton/nginx/sip.nginx (sip.nginx)

                        proxy_pass: 3.7.178.84

/usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml (Kurento SFU)

                        kurento.ip: 172.31.29.111

                       kurento.url: ws://127.0.0.1:8888/kurento

                    kurento.sip_ip: 172.31.29.111

                    localIpAddress: 172.31.29.111

               recordScreenSharing: true

                     recordWebcams: true

                  codec_video_main: VP8

               codec_video_content: VP8

/usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml (HTML5 client)

                             build: 934

                        kurentoUrl: wss://vc.skillprep.co/bbb-webrtc-sfu

                  enableListenOnly: true

# Potential problems described below

# IP does not match:

#                           IP from ifconfig: 172.31.29.111

#   /etc/nginx/sites-available/bigbluebutton: vc.skillprep.co

# Warning: API URL IPs do not match host:

#

#                                IP from ifconfig: 172.31.29.111

#  /var/lib/tomcat7/demo/bbb_api_conf.jsp: vc.skillprep.co

# Warning: defaultGuestPolicy is set to ASK_MODERATOR in

#    /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties

# This is not yet supported yet the HTML5 client.

#

# To revert it to ALWAYS_ALLOW, see

#

#    sed -i s/^defaultGuestPolicy=.*$/defaultGuestPolicy=ALWAYS_ALLOW/g /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties

#

# Warning: The setting of 3.7.178.84 for proxy_pass in

#

#    /etc/bigbluebutton/nginx/sip.nginx

#

# does not match the local IP address (172.31.29.111).

# (This is OK if you've manually changed the values)

# Warning: The API demos are installed and accessible from:

#

#    https://vc.skillprep.co

#

# and

#

#    https://vc.skillprep.co/demo/demo1.jsp

#

# These API demos allow anyone to access your server without authentication

# to create/manage meetings and recordings. They are for testing purposes only.

# If you are running a production system, remove them by running:

#

#    apt-get purge bbb-demo

[Matias Silva]

Hi,

There is already a tool called bigbluebot that does this very well. I
will say that you are unlikely in any case to get more than 30 webcam
users on your server per meeting. You will also find it very hard to get
more than 15 test users using headless browsers.

> If I configure scalelite, how many bbb servers should I add for
> running 25 such concurrent meetings?

Assuming a more reasonable 20 cams, maybe you could run 2 or 3 per
server?

> I didnt find any params for autojoining
> video conference.

These work, I have tried them. You can't get around asking for the
browser's permission, that's basic privacy :)

---
Thanks, Matias

On 2020-07-01 08:51, Joy Banerjee wrote:
> Hi,
>
> We are trying to do stress testing (all users - with video conference,
> with/without audio conference) on our BigBlueButton server.
> For one bbb server, I want to calculate how many meetings can be held
> concurrently with each meeting having 1 moderator and 40 attendees
> with all users having webcam on, such that CPU utilization doesn't
> exceed 80%.

>

> I tried the following load testing approaches:
>

> * I wrote a js script which generates multiple tabs in browser

> (similar to bbb-test). But with webcam on, one person can open max 3
> or 4 tabs (dependent on bandwidth). Also in each tab the video sharing
> and joining audio conference needs to done manually. Thus, manual
> stress testing is not feasible.

> * I used testRTC (https://testrtc.com/) to join multiple meetings on

> a remote browser. I tried couple of ways such that the audio and video
> conference can be started without manual intervention on joining
> meeting, but didn't work. Thus, test failed.
>
> PFB my doubts.
>

> * Did your team do any stress testing with all user sharing video? If

> yes, could you please share the results.

> * Can I do load testing directly with websockets or webrtc apis by

> using any tool like JMeter? If yes, could you please help me identify
> these apis?

> * for bbb 2.2, can I configure all users to join audio and video

> conference automatically on joining meeting? I tried using params (as
> in
> https://docs.bigbluebutton.org/2.2/customize.html#passing-custom-parameters-to-the-client-on-join)
> and modifying PhoneModule in
> /var/www/bigbluebutton/client/conf/config.xml for autojoining audio
> conference but it didnt work. I didnt find any params for autojoining
> video conference.
>
> Our bbb server config:
>

> * version: bigbluebutton 2.2
> * installed in EC2 - c5.xlarge instance which is configured using

> minimum server requirements as in
> https://docs.bigbluebutton.org/2.2/install.html#minimum-server-requirements

> * However, we used shared instance in AWS instead of bare metal
> * we have also configured a TURN server in t2.micro instance
> * We'll create, join meeting through API calls from another
> application
> * PFB output of sudo bbb-conf --check

> --
> You received this message because you are subscribed to the Google
> Groups "BigBlueButton-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bigbluebutton-...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bigbluebutton-dev/9704b220-0924-42e2-933d-d0e0c35642bao%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/bigbluebutton-dev/9704b220-0924-42e2-933d-d0e0c35642bao%40googlegroups.com?utm_medium=email&utm_source=footer

[Joy Banerjee]

Thank you Matias for your suggestion.

I tried bigbluebot but maximum 36 users with webcam could join successfully. For the bot, video join would FAIL for subsequent users. Even through UI, any user who joins the same meeting after that and turns on video, an error appears as 'Media could not reach the server(error 1020)'. 

I understand it's recommended to have overall streams below 200 but is there any option to tune up bigbluebutton server and TURN server so that it could support more users with video.

I need to configure bigbluebutton such that it can support:

1. 50 users in a meeting with all video

2. 150 users in a meeting with all video (under discussion)

Could you or anyone suggest the server requirements for achieving this?

Or is there a limitation that bigbluebutton would no way support 50 users with video in a meeting?

Thanks,

Joy

> an email to bigblueb...@googlegroups.com.

[Matias Silva]

Hi Joy,

This is not an issue with BBB itself. This is an issue with your many
instances of the HTML5 client on headless browsers. These naturally take
up lots of CPU and memory and eventually will cap out depending on the
specs of your testing server. I recommend you read Pedro's comments on a
similar thread in this forum on how he has optimized it for his testing.

---
Thanks, Matias

>> [1])

>>> and modifying PhoneModule in
>>> /var/www/bigbluebutton/client/conf/config.xml for autojoining
>> audio
>>> conference but it didnt work. I didnt find any params for
>> autojoining
>>> video conference.
>>>
>>> Our bbb server config:
>>>
>>> * version: bigbluebutton 2.2
>>> * installed in EC2 - c5.xlarge instance which is
>> configured using
>>> minimum server requirements as in
>>>
>>
> https://docs.bigbluebutton.org/2.2/install.html#minimum-server-requirements

>> [2]

>>> * However, we used shared instance in AWS instead of bare
>> metal
>>> * we have also configured a TURN server in t2.micro
>> instance
>>> * We'll create, join meeting through API calls from
>> another
>>> application
>>> * PFB output of sudo bbb-conf --check
>>>
>>> BigBlueButton Server 2.2.16 (1987)
>>> Kernel version: 4.4.0-1107-aws
>>> Distribution: Ubuntu 16.04.6 LTS (64-bit)
>>> Memory: 7814 MB
>>> CPU cores: 4
>>>
>>> /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
>> (bbb-web)
>>> bigbluebutton.web.serverURL: https://vc.skillprep.co
>>> defaultGuestPolicy: ASK_MODERATOR
>>> svgImagesRequired: true
>>>
>>> /etc/nginx/sites-available/bigbluebutton (nginx)

>>> server name: vc.skillprep.co [3]

>>> port: 80, [::]:80
>>> port: 443 ssl
>>> bbb-client dir: /var/www/bigbluebutton
>>>
>>> /var/www/bigbluebutton/client/conf/config.xml (bbb-client)

>>> Port test (tunnel): rtmp://vc.skillprep.co [3]
>>> red5: vc.skillprep.co [3]

>>> useWebrtcIfAvailable: true
>>>
>>> /opt/freeswitch/etc/freeswitch/vars.xml (FreeSWITCH)
>>> local_ip_v4: 172.31.29.111
>>> external_rtp_ip: 3.7.178.84
>>> external_sip_ip: 3.7.178.84
>>>
>>> /opt/freeswitch/etc/freeswitch/sip_profiles/external.xml
>> (FreeSWITCH)
>>> ext-rtp-ip: $${external_rtp_ip}
>>> ext-sip-ip: $${external_sip_ip}
>>> ws-binding: :5066
>>> wss-binding: :7443
>>>
>>> /usr/local/bigbluebutton/core/scripts/bigbluebutton.yml (record
>> and
>>> playback)

>>> playback_host: vc.skillprep.co [3]

>>> playback_protocol: https
>>> ffmpeg: 4.2.2-1bbb1~ubuntu16.04
>>>
>>> /etc/bigbluebutton/nginx/sip.nginx (sip.nginx)
>>> proxy_pass: 3.7.178.84
>>>
>>> /usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml
>> (Kurento
>>> SFU)
>>> kurento.ip: 172.31.29.111
>>> kurento.url: ws://127.0.0.1:8888/kurento

>> [4]

>>> kurento.sip_ip: 172.31.29.111
>>> localIpAddress: 172.31.29.111
>>> recordScreenSharing: true
>>> recordWebcams: true
>>> codec_video_main: VP8
>>> codec_video_content: VP8
>>>
>>>
>>
> /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml
>>
>>> (HTML5 client)
>>> build: 934
>>> kurentoUrl:

>>> wss://vc.skillprep.co/bbb-webrtc-sfu [5]

>>> enableListenOnly: true
>>>
>>> # Potential problems described below
>>> # IP does not match:
>>> # IP from ifconfig: 172.31.29.111

>>> # /etc/nginx/sites-available/bigbluebutton: vc.skillprep.co [3]

>>> # Warning: API URL IPs do not match host:
>>> #
>>> # IP from ifconfig: 172.31.29.111

>>> # /var/lib/tomcat7/demo/bbb_api_conf.jsp: vc.skillprep.co [3]

>>>
>>> # Warning: defaultGuestPolicy is set to ASK_MODERATOR in
>>> # /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
>>> # This is not yet supported yet the HTML5 client.
>>> #
>>> # To revert it to ALWAYS_ALLOW, see
>>> #
>>> # sed -i
>>> s/^defaultGuestPolicy=.*$/defaultGuestPolicy=ALWAYS_ALLOW/g
>>> /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
>>> #
>>>
>>> # Warning: The setting of 3.7.178.84 for proxy_pass in
>>> #
>>> # /etc/bigbluebutton/nginx/sip.nginx
>>> #
>>> # does not match the local IP address (172.31.29.111).
>>> # (This is OK if you've manually changed the values)
>>>
>>> # Warning: The API demos are installed and accessible from:
>>> #
>>> # https://vc.skillprep.co
>>> #
>>> # and
>>> #

>>> # https://vc.skillprep.co/demo/demo1.jsp [6]

>>> #
>>> # These API demos allow anyone to access your server without
>>> authentication
>>> # to create/manage meetings and recordings. They are for testing
>>> purposes only.
>>> # If you are running a production system, remove them by running:
>>> #
>>> # apt-get purge bbb-demo
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>
>>> Groups "BigBlueButton-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>> send
>>> an email to bigblueb...@googlegroups.com.
>>> To view this discussion on the web visit
>>>
>>
> https://groups.google.com/d/msgid/bigbluebutton-dev/9704b220-0924-42e2-933d-d0e0c35642bao%40googlegroups.com

>> [7]

>>> [1].
>>>
>>>
>>> Links:
>>> ------
>>> [1]
>>>
>>
> https://groups.google.com/d/msgid/bigbluebutton-dev/9704b220-0924-42e2-933d-d0e0c35642bao%40googlegroups.com?utm_medium=email&utm_source=footer

>> [8]

>
> --
> You received this message because you are subscribed to the Google
> Groups "BigBlueButton-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to bigbluebutton-...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/bigbluebutton-dev/4bbe9035-da7d-4160-a725-cf2f2ff36adco%40googlegroups.com
> [9].
>
>
> Links:
> ------
> [1]
> https://docs.bigbluebutton.org/2.2/customize.html#passing-custom-parameters-to-the-client-on-join
> [2]
> https://docs.bigbluebutton.org/2.2/install.html#minimum-server-requirements
> [3] http://vc.skillprep.co
> [4] http://127.0.0.1:8888/kurento
> [5] http://vc.skillprep.co/bbb-webrtc-sfu
> [6] https://vc.skillprep.co/demo/demo1.jsp
> [7]
> https://groups.google.com/d/msgid/bigbluebutton-dev/9704b220-0924-42e2-933d-d0e0c35642bao%40googlegroups.com
> [8]
> https://groups.google.com/d/msgid/bigbluebutton-dev/9704b220-0924-42e2-933d-d0e0c35642bao%40googlegroups.com?utm_medium=email&utm_source=footer
> [9]
> https://groups.google.com/d/msgid/bigbluebutton-dev/4bbe9035-da7d-4160-a725-cf2f2ff36adco%40googlegroups.com?utm_medium=email&utm_source=footer

[Joy Banerjee]

Hi Matias,

I used 4 instances each with 36 cpu cores and 60GB ram to run the load test. Each test run connects to the demo meeting of our BBB server. Just for checking purpose, I have run these test server bots with 10, 10, 10, 5 users respectively. Also, I joined the meeting through UI (- total 36 users). The CPU utilization of test servers is not much affected by it. 

Also, the CPU utilization of BBB server reached max 55% while running the test (where I found out max 36 users could join out of which 32 has joined successfully with video).

Thus, CPU maxing out doesn't seem to be the problem here.

We have used the min server requirements for BBB server. If we scale up, can we have at least 50 users with video in a meeting? If yes, could you please suggest the specifications?

On a different note, others (say google meet) developed a browser extension to facilitate gallery view of N no of users. 

Did anyone come up with anything similar for BBB?

 

Thanks,

Joy

[Lonesome Walker]

Funny to read all the people which try to simulate by machines instead of running a real-world scenario...

This is one of our servers:

The issue is mostly NOT the server.
Main problem is the internet connection of the participants.
This will multiply with the users webcams, so most of them will drop by ~25 users with webcam in a session.

Basically our server would be able to handle 150 video participants in one session, but this will never happen because then the participants would all need to have a fiber connection :-)
(but yes, we tried that!)

[Anton Meyburgh]

Hi 

Thanks for the graphs

What was the server specs for the below Real-world Test?

Regards

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/3edbcea0-81c4-4311-a1d0-fb6009bfa95do%40googlegroups.com.

[Joy Banerjee]

Hi,

Thanks a lot. Seeing the graphs of a load test is really helpful. Could you possibly share the specs of your BBB server?

But really sad to hear about the real world test results.

This means even using a higher specification of Bigbluebutton server, 40-50 users wont be able to join a meeting with video. And as this is not dependent on the server then using Scalelite would make no difference. Is my understanding correct? 

We are very much interested with Bigbluebutton but this seems to be blocker. This point alone is a decision factor for Go/No-Go.

@All,

If there is still any way to support more video users, kindly let us know your expert opinion.

[If there's a possibility for scheduling a call for discussing more on this, then that would be great.]

Thanks,

Joy