Audio stress Test: Results

[Fred Dixon]

Hi Everyone,

Thanks for participating in the audio stress test this morning at 9am.
Here's a summary of the results.

Caveat: Each server is different -- virtual vs dedicated, # of cores,
hyperthreadig, and memory -- so take this test as a data point. Your
mileage will vary.

Server:
Core2Duo E8400 Dual Core running 4G (dedicated server). This server
has Hyperthreading, so it reports 8 cores.

Software:
FreeSWITCH 1.0.6
red5 0.91
Internal build of BigBlueButton 0.8 using 16khz wide-band speex
(encode quality of 6).

Heres a breakdown of the sequence of test. Note: We actually started
the testing at 9am EST, but the time on the graph is skewed, so I'll
summarize the results based on the graph (see attached image).

7:48 -- We had 20 users in voice with ~30% CPU
7:57 -- We had 40 users in voice with ~45% CPU
8:05 -- We had 60 users in voice with ~70% CPU
8:08 -- We had 80 users in voice with ~90% CPU <-- Audio stared to degrade

We roughly had about 30 real users running BigBlueButton, spread all
over the world, so to get to 60 and 90 BigBlueButton sessions, we
asked everyone to open additional browser tabs. To the BigBlueButton
server, each browser tab is a distinct user.

At 80 users the overall CPU (average of all 8 cores) was at 90% CPU.
At this point, FreeSWITCH starts to have challenges decoding/encoding
speex, and the audio started to degrade. We dropped the number of
users back to 60 (by closing some browser tabs), and the audio
returned to normal. We then asked everyone to open a webcam .

8:09 -- We had 80 users in voice with 20 webcams with ~70% CPU

At this point, we had 60 users with 20 simultaneous webcams (see
attached screen shot), audio remained good. This makes sense because
there isn't much additional CPU processing required for the video
packets: they are just broadcasted out to users. We then started
desktop sharing, which triggered a crash of red5. We've opened an
issue for this at

http://code.google.com/p/bigbluebutton/issues/detail?id=910

We're definitely going to look into why desktop sharing caused red5 to
crash with 60 users.

In summary: the purpose of this stress test was to test the audio, and
remote users reported the audio was good for 60 simultaneous users,
and it remained good when we started up 20 webcams.

In observing the bandwidth with small number of users, we estimate
each audio connection requires roughly 60 Kbits/sec to transmit to the
server, and 60 Kbits/sec to receive. So, here's how the bandwidth
scales for audio.

1 60 0.06
2 120 0.12
3 180 0.18
4 240 0.24
5 300 0.30
6 360 0.36
7 420 0.42
8 480 0.48
9 540 0.54
10 600 0.60
20 1200 1.20
30 1800 1.80
40 2400 2.40
50 3000 3.00
60 3600 3.60
70 4200 4.20
80 4800 4.80

The first colum is # of users, the second is Kbits/sec bandwidth, the
third colum is Mbits/sec. So, for 60 users the server had roughly 3.6
Mbits/sec transmission and 3.6 Mbits/sec receiving.

Thanks again to everyone who participated. Here's some locations for the users:

Chahira - 09:13 : Bonn, Germany
Chris K - 09:13 : Dearborn, Michigan
Dave - 09:13 : Ottawa, Canada
David Butler_Chrome 2 - 09:13 : Switzerland
Dip - 09:13 : Near Boston
Fred - 09:13 : Ottawa, Ontario, Canada
Gerry - 09:13 : Ottawa
HostBBB - 09:13 : northern maine
Kako - 09:13 : La Serena, Chile
Leonardo 2 - 09:13 : Brazil, Porto Alegre
Nelson - 09:13 : Toronto, Canada
Ophileonb - 09:13 : Netherlands that is
Pascal St-Jean - 09:13 : Ottawa, Ontario
Ravi - 09:13 : Abuja , Nigeria
RobertPlummer - 09:13 : Midwest - USA
Shylesh - 09:13 : Hong Kong
Suzanne Cadwell - 09:13 : durham, nc, usa
Séverin TERRIER - 09:13 : Toulouse, France
Tom C - 09:13 : Waskom Texas
hans - 09:13 : Netherlands
james - 09:13 : Winston-Salem NC
kimberly - 09:13 : Ottawa
ranjeev - 09:13 : Nice, France
susana2 - 09:13 : Ottawa
trevor - 09:13 : Ireland

Regards,... Fred

[Andrew E]

Awesome. Thank you for posting the results. Looks like a great
test. I wish I had been able to participate.

Did any users note a delay in audio, and if so how much?

Also, I'm confused about the server's CPU. Is it a dual-socket server
(meaning it has two E8400's)? If not, I'm not sure how you had 8
logical cores.

Thanks,

Andrew

>  stress_test.PNG
> 69KViewDownload
>
>  many_users2.PNG
> 1520KViewDownload

[Fred Dixon]

> Did any users note a delay in audio, and if so how much?

I'll let some of the other users post their response.

> Also, I'm confused about the server's CPU.

1 CPU, 4 cores with hyper-threading, so the OS reports 8 cores. See

http://en.wikipedia.org/wiki/Hyper-threading

Regards,... Fred

> --
> You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
> To post to this group, send email to bigblueb...@googlegroups.com.
> To unsubscribe from this group, send email to bigbluebutton-...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/bigbluebutton-dev?hl=en.
>
>

[Andrew E]

Oh ok. I must have found the wrong link then, the CPU spec page I
found said it only had two cores. Thanks.

I'm very interested in what the users have to say about the delay.

Keep up the great work!

Andrew

[Josh]

Does this mean that we cannot use BigBlueButton for a large
university / school with thousands of students and multiple rooms? Is
there any workaround to use it in such a scenario?

Thank you.

[Séverin TERRIER]

Hello,

i think that if you really intend to use BBB for a big university,
with lot of simultaneous users, and make it a key service, you can
then buy a dedicated server for that.
You can already find (at least) some with 2 CPU with 4 (or 6) core
each, and lot of RAM (16 to 144 Gb or more).
After that, if think that programs (red5...) will have to be tweaked
to make benefit of that.

Hope this helps.
I would be interested to know if someone uses such server, and how
many users it can handle...

Séverin

[Fred Dixon]

Hi Andrew,

I was thinking about your question early this morning ... the server
was reporting 8 cores, but the specs for E8400 are dual core. Turns
out you were right!

After double-checking, I realized my mistake. The server we used for
the voice testing yesterday was running a Xenon 3450 quad core running
2.66 Ghz. It has hyper-threading, and thus the OS reports 8 cores.

Thanks Andrew!

Regards,... Fred

[Fred Dixon]

Josh,

There is a solution.

Putting on my Blindside Networks hat for a moment, we have developed a
commercial load balancing server for BigBlueButton that we are
currently using to scale BigBlueButton in the way you described.

If a university or college want to use BigBlueButton for thousand of
students, they should be achieving a significant cost savings over
commercial solutions. (Just like Moodle and Sakai help them reduce
their costs over commercial LMS systems)

For a university or college to use BigBlueButton for thousands of
students, they need three components (I'm simplifying here, but these
are the core):

1) BigBlueButton must have the features they need (such as record and
playback) and it must be solid, really solid
2) Load balancing
3) Commercial support

Yourself Josh, and everyone else who uses BigBlueButton, are counting
on us to deliver (1). Along with other very talented open source
developers, we've been working hard on BigBlueButton for over three
years to achieve that goal. We're not going to let you down.

(2) and (3) are funding our efforts to achieve (1).

I want to emphasize the BigBlueButton ecosystem is not just Blindside
Networks. Today, if you need commercial support and hosting, you can
contact any one of the companies listed at
http://bigbluebutton.org/support.

Others may develop open source solutions for load balancing. And
that's OK as well (just don't ask us to help you do it :-). The
developers working on BigBlueButton -- whether from Blindside Networks
or other companies -- are all entrepreneurs. We are working hard to
create a complete solution for real-time distance education, and if we
can achieve that, everyone wins.

Regards,... Fred

[Andrew E]

Thanks for the update about the CPU, Fred. That is very helpful
information for our implementation of BBB.

We just ordered the parts to build a dual Xenon E5620 2.4Ghz (16
logical cores!), 24GB DDR3 1333 server solely for BBB.

We have a couple companies that we'll be sharing the server with, so
once they start using it, I'll try to report some stats on how it
performs.

If anyone is interested, you can compare processor specs on this page
to see how yours will compare to the X3450:

http://ark.intel.com/ProductCollection.aspx?familyId=594

Andrew

On Mar 25, 6:42 am, Fred Dixon <ffdi...@gmail.com> wrote:
> Josh,
>
> There is a solution.
>
> Putting on my Blindside Networks hat for a moment, we have developed a
> commercial load balancing server for BigBlueButton that we are
> currently using to scale BigBlueButton in the way you described.
>
> If a university or college want to use BigBlueButton for thousand of
> students, they should be achieving a significant cost savings over
> commercial solutions.  (Just like Moodle and Sakai help them reduce
> their costs over commercial LMS systems)
>
> For a university or college to use BigBlueButton for thousands of
> students, they need three components (I'm simplifying here, but these
> are the core):
>
> 1) BigBlueButton must have the features they need (such as record and
> playback) and it must be solid, really solid
> 2) Load balancing
> 3) Commercial support
>
> Yourself Josh, and everyone else who uses BigBlueButton, are counting
> on us to deliver (1).   Along with other very talented open source
> developers, we've been working hard on BigBlueButton for over three
> years to achieve that goal.  We're not going to let you down.
>
> (2) and (3) are funding our efforts to achieve (1).
>
> I want to emphasize the BigBlueButton ecosystem is not just Blindside
> Networks. Today, if you need commercial support and hosting, you can

> contact any one of the companies listed athttp://bigbluebutton.org/support.

[Andrew E]

Oops... meant Xeon*

> ...
>
> read more »

[Walter Tak]

Hi Fred,

Just checking ;

did you monitor the CPU usage using 'top' or did you check every core using 'mpstat -P ALL' from the 'sysstat' package (debian) ?

I haven't played around with the latest BBB but not every tool is fully multithreaded which means it will just use 1 core, perhaps 2 cores at most. In that case you can have 4 cores, 6 cores or 3700 cores ; the application will only run on 1 core and when that one is at 100% the application has reached its limits.

Besides that the E8400 is a desktop-CPU and a decent Xeon with some more cache will probably work better than that CPU so for servers the scenario might look a bit better.

IF one of the modules/apps is core-bound then you could install multiple VM/VPS (virtual machine) on that dedicated server (as much as you have cores) and then you'd increase the productive power by 2, 4 or even 12 (if you have 2 CPU's with each 6 cores).

Decoupling modules and spreading them over multiple servers might (read will) influence greatly as well ; the internal webserver can easily handle 1000's of users, it's just the audio-mixing that eats time.

If you want to identify the real bottleneck you have to breakdown the cpu-usage per process. Is it Red5 doing transcoding stuff (is the Red5Phone module still in use ? sorry for asking, didnt read anything here for , cough , half a year) ? So is the java process eating any cpu ? Or is it purely freeswitch ?

If freeswitch is the culprit ; how about this info here ? http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Some_random_results

There is a lot of information how to tune freeswitch.

My 2 cents,
Walter

(I'm going to investigate 'back' into BBB because of the upcoming Flash 10.3 with the echo cancelling feature which suddenly makes Flash again interesting to use as an audio-interface).

[Sly]

Hi,

I think, the overhead problem is in design.
Right now, the streams are custom generated for each user for their
voice cancellation.
And even costly server have huge problem with only a handfull of
users.

Why not create, on conference creation (server side) separate global
sip stream that would be used for users that only listen.
By default joining the conference would be in listening mode (muted).
When un-muted, the existing logic would kick in, and vice-versa.
Just until you are not speaking, you dont have your own stream, you
use global sip stream.

Than CPU and memmory problem is out ;)
What Do You think, should not be hard to implement at all?

Sly

>  stress_test.PNG
> 69KZobaczPobierz
>
>  many_users2.PNG
> 1520KZobaczPobierz

[Richard Alam]

That's indeed we plan to do next.

If you can help implement it, patches and pull request
(http://help.github.com/pull-requests/) are welcome.

Richard

> --
> You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
> To post to this group, send email to bigblueb...@googlegroups.com.
> To unsubscribe from this group, send email to bigbluebutton-...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/bigbluebutton-dev?hl=en.
>
>

--
---
BigBlueButton
http://www.bigbluebutton.org
http://code.google.com/p/bigbluebutton

[César Javier García López]

Hi Fred!

It is very interesting the information that you share about the stress testing, can you detail more about the badwith conditions of the test? I have a similar computing infratucture with 8 cores and 16 GB RAM, and i think that 35 users will be perfect, but i want to figure out how much bandwith and network conditions do i require. Im planning to use BBB to help a health institution's psychologist to collaborate and share their experience and knowledge between them.

all the best!

[Fred Dixon]

Hi 

Based on our experiences, you have a machine that can definitely handle 35 users, assuming there is sufficent bandwidth for the server.

For the bandwidth requirements, see

    http://code.google.com/p/bigbluebutton/wiki/FAQ#What_are_the_bandwidth_requirements_for_running_a_BigBlueButton 

You can also test yourself.  Try running a tool such as bmon on the server, then join with one or two users with webcams.  You can see how the webcam bandwidth usage is adaptive -- in increases and decreases with movement.  

By doing one or two users with desktop sharing, webcams, and audio, you can easily get some real-world numbers for your server.  The above FAQ entry will also give you guidance on the bandwidth usage as well.

Regards,... Fred

-- 
BigBlueButton Developer

http://bigbluebutton.org/
http://code.google.com/p/bigbluebutton

BigBlueButton on twitter: @bigbluebutton

--

You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To view this discussion on the web visit https://groups.google.com/d/msg/bigbluebutton-dev/-/BE2RWD8sXcIJ.

[gigo]

here are some stats from my real server running real user loads 

37 users = ~1.5-2MiB/s  upload to clients 170-224KiB/s download from clients

35 users = ~3.5MiB/s with changing slides

only one or two webcams active at a time, and no desktop sharing

also you can ignore the cpu load as the server was reprosesing all its recorded vids while the meeting was running

btw iv gotten to 48 real users with 6 webcams and it runs, but i dident log the system usage :(

[César Javier García López]

Hi everyone!

Thank you very much for sharing this info, we runned testing and we confirmed that our server works just fine for our needs, the only restriction is bandwith and this issue will be on the money field, the solution works great!

We have 4 MB sync on Central site (where the server is) and on remote sites we have 515k and we are using Satelital link, if we run the app without desktop sharing the solution works ok, but desktop sharing runs slow and users see it with approx 30 secs delay.

All the best

César García

[César Javier García López]

Hi Fred;

Thank you for the info, you are right, the server works without a problem as you said, we performed testing and analized network traffic with wireshark and now we have a good idea about the bandwith requirements.

Thank you very much for your help

All the best

César García

[Patrick]

Hi!

We tested with 24-25 people with videos on BBB .81 , Red5 1.0.rc2

When we get to about 25 people the videos freeze (the audio keeps running ok), but the CPU isn't maxed out. Could you look at the CPU info and suggest what might be the problem.

thanks 

[Fred Dixon]

Hi Patrick,

BigBlueButton 0.81 to still in development, so you should have better results once we reach beta.  When running with 25 simultaneous webcams is putting 625 streams through red5, which is pretty good.

Not sure why yours is failing.  If you want to help, can you do the following:

1.  Run 

   sudo bbb-conf --clean

2.  Test with 25 webcams again (hopefully it fails)

3.  Post the output of 

   sudo bbb-conf --debug

 

Regards,... Fred

-- 
BigBlueButton Developer

http://bigbluebutton.org/
http://code.google.com/p/bigbluebutton

BigBlueButton on twitter: @bigbluebutton

--

You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To view this discussion on the web visit https://groups.google.com/d/msg/bigbluebutton-dev/-/pyp3HI3b8jsJ.

[HostBBB.com]

Patrick, couple of things to look at..

bbb-conf --watch will show the number of connections being made as
users are added. To scale to very large meeting sizes you will need
to optimize the OS, Tomcat to handle larger number of open files,
etc....

ulimit is your friend here, and there are some changes that can be
made so red5 and tomcat have more resources to handle this increased
load.

Usually when you run into issues you will see the too many files
open, socket error issues in logs.

regards,
Stephen
hostbbb.com

On Jan 14, 10:00 pm, Fred Dixon <ffdi...@gmail.com> wrote:
> Hi Patrick,
>
> BigBlueButton 0.81 to still in development, so you should have better
> results once we reach beta.  When running with 25 simultaneous webcams is
> putting 625 streams through red5, which is pretty good.
>
> Not sure why yours is failing.  If you want to help, can you do the
> following:
>
> 1.  Run
>
>    sudo bbb-conf --clean
>
> 2.  Test with 25 webcams again (hopefully it fails)
>
> 3.  Post the output of
>
>    sudo bbb-conf --debug
>
> Regards,... Fred
> --

> BigBlueButton Developerhttp://bigbluebutton.org/http://code.google.com/p/bigbluebutton

[ofer kruzel]

Hi Fred,

Can you share info on which process is taking the larger portion of the resources?

FreeSwitch / Red5 or both?

is it CPU or RAM?

Thanks,

Ofer

[Fred Dixon]

Hi Ofer,

In almost all cases, the component that uses the most CPU is FreeSWITCH (real-time mixing of audio).  See

  http://code.google.com/p/bigbluebutton/wiki/FAQ#Some_users_are_experiencing_audio_problems

Next would be red5.  For example, running the unix command 'top' on a test BigBlueButton server shows the following two entries 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

  725 freeswit  20   0  473m 114m 5608 S   36  1.6   2638:36 freeswitch

 1502 red5      20   0 1170m 729m  13m S   14 10.2   1120:14 java

 

When I took the above sample the server had 30 users spread over 10 sessions.  You can see FreeSWITCH using 36% of the CPU while red5 uses 14%.

Regards,... Fred

-- 
BigBlueButton Developer

http://bigbluebutton.org/
http://code.google.com/p/bigbluebutton

BigBlueButton on twitter: @bigbluebutton

To view this discussion on the web visit https://groups.google.com/d/msg/bigbluebutton-dev/-/XF68755ebzQJ.

[ofer kruzel]

I understand.

 

sorry if this a basic question, but,

why do you need to mix audio streams?

couldn't we just stream the audio streams to all users as separate streams thru a server and render them in parallel on the client?

similar to the way you do the video thru red5, but for audio, I guess it require another type of server to do that. as far as I know red5/wowza/FMS can't do without video in the stream. 

Thanks,
Ofer

[Fred Dixon]

Hi Ofer,

> why do you need to mix audio streams?

Consider a session where you have ten people collaborating.  By mixing the audio on the server, each user has two streams for their audio: sending and receiving.  There is a total of twenty (20) streams.  For each user, the two audio streams pass through red5 and are forwarded to FreeSWITCH, which mixes the audio together in the voice conference.

If you didn't have any voice conference bridge, each user would need to have a stream of audio from each other user.  That would require 10 x 10 or 100 streams of data, either going through the BigBlueButton server.

By having an audio conference in place, we significantly reduce the bandwidth requirements.

Regards,... Fred

-- 
BigBlueButton Developer

http://bigbluebutton.org/
http://code.google.com/p/bigbluebutton

BigBlueButton on twitter: @bigbluebutton

To view this discussion on the web visit https://groups.google.com/d/msg/bigbluebutton-dev/-/LHKqv4C1oscJ.

[ofer kruzel]

got it.

thanks,

ofer

Ofer Kruzel
M: +972 (54) 4516627

[Nader]

hi

there is no conference-flags parameter in conference.conf.xml in BBB 0.81.

i want to know why ?

can u help me?  

 

 

[HostBBB.com]

you can add conference-flags in the freeswitch dialplan as users join bbb conference with the [+flags],  the freeswitch wiki for mod_conference has some good examples.  for example you can start everyone muted.

<action application="conference" data="[bridge:]confname[@profile][+[pin][+flags{mute|deaf|...}]]">

These also can be added to to the conference profile, is BBB's case the wideband profile.

Regards,

Stephen

hostbbb.com