Converting BigBlueButton into a Webinar Solution

[Jon]

I searched through the archives, and it seems that several people have
danced around this question, but I thought I would ask it directly:

What would it take to convert BigBlueButton into a webinar solution
that can support 1000+ concurrent attendees?

From what I gather, the main bottleneck is the CPU usage from
freeswitch synchronizing the audio, and the solution to that
bottleneck is integrating it with Icecast and creating a separate
stream for attendees who are listening but not talking. I'm planning
on trying this over the next few days.

But are there any other bottlenecks?

In case it matters, I'm not from higher education, nor am I a casual
user. I'm a developer, and I have staff of programmers working for me.
Right now, I have one full-time Java programmer assigned to this
project, and I can assign more, if need be.

My goal is to create a software package capable of supporting 1000+
people per webinar with as many as 50 or 100 webinars running at a
time. I know that's not what BBB was designed for, but it seems like
the foundation is here, and it would be easier to use that foundation
than starting from scratch.

If you can give me some guidance, I'd really appreciate it. I'm
willing to do the work.

Thanks!

Jon

[Fred Dixon]

Hi Jon,

Thanks for your interest!  We (the core developers) have tried to be pretty clear about the limitations about BigBlueButton (no dancing around!).  

See this FAQ entry

   http://code.google.com/p/bigbluebutton/wiki/FAQ#How_many_simultaneous_users_can_BigBlueButton_support 

which states

   As a rule of the thumb, without knowing anything about your server, 

  we recommend you use BigBlueButton with on-line classes totaling 

  twenty-five (25) users or less. 

And 

   

   https://groups.google.com/d/msg/bigbluebutton-setup/shrIZ7lBK5M/2cH8BuJ1IfYJ

and

  https://groups.google.com/forum/#!topic/bigbluebutton-setup/a2tfffuJrBU

In the last post we emphasized our commitment to making sure BigBlueButton works well for groups of 25 users.  Our goal is to make the current system solid before we scale; otherwise, what's the point of scaling?

We have not tried using icecast.  There are no built-in configuration options in BigBlueButton to support it (but others on these forums have used it).  

> Right now, I have one full-time Java programmer assigned to this

> project, and I can assign more, if need be.

We would welcome your help!!  To understand the process for contributing to BigBlueButton, see

  http://code.google.com/p/bigbluebutton/wiki/FAQ#Contributing_to_BigBlueButton

At the moment, we are focused on stabilizing BigBlueButton 0.8 for release.  The current release is BigBlueButton 0.8-beta-3 and there will be (at least) one more beta release before final.  

If you have a full-time developer, we would recommend having them understand the core system through working on bugs (pick ones that are easy at first).  You can see the open bugs for stability here

  http://code.google.com/p/bigbluebutton/issues/list?can=2&q=label:Stability&x=priority&y=component&mode=grid&cells=tiles

If you want to help us with this release, you can see the 0.8 bugs here

   http://code.google.com/p/bigbluebutton/issues/list?can=2&q=milestone=Release0.8

Jon, we're interested in your thoughts on the above.   Helping us QA BigBlueButton 0.8, report bugs, and, if possible, contribute fixes would be the shortest path to making immediate contributions that benefit all BigBlueButton users.  

It's our belief that the faster we make BigBlueButton solid, the faster we can work on the stability.  

Regards,... Fred

-- 
BigBlueButton Developer
http://bigbluebutton.org/
http://code.google.com/p/bigbluebutton 

Jon

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To post to this group, send email to bigblueb...@googlegroups.com.
To unsubscribe from this group, send email to bigbluebutton-...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/bigbluebutton-dev?hl=en.

[HostBBB.com]

Jon,

we have implemented icecast with BBB and freeswitch for a few
customers. the delay is 4-7 seconds depending on how the client
buffering is setup. For real small implementations you can have
icecast running on same server as red5, and freeswitch, for larger you
can implement a cluster of icecast servers and integrate to BBB.

We have also recently distributed BBB .8 over 3 servers, running 1)
freeswitch stand alone, 2) presentation/chat on second 3) video/
deskshare on third all seperate which allows for some modest scaling.
This might allow for 100+ in a meeting with larger CPUS/MEMORY, our
testing will find out soon.

But spreading a single 1000 person webinar across multiple servers
with standard hardware would be a real challenge if you needed to use
deskshare, video cameras, presentation and chat...

The biggest issue is generating the test scripts is to do real load
testing, and finding a friendly customer that has access to large user
base for beta testing to really work thru the issues.

The business case is also a real issue, you can build it, but
getting enough revenue to run it profitably is a question. The big
guys with goto meeting, and webex have good commericial webinar
solutions.

Would be glad to work on this project if it benefits back to the core.

regards.
Stephen
http://hostbbb.com

[Felipe Cecagno]

Hi guys,

Some developers of the Mconf development team, including me, are working on scaling BigBlueButton for a large number of users. As the first step on this task we have developed a Java application (based on the core of Mconf-Mobile) that is able to create a meeting and join N users (as moderators or viewers) with video both ways (receiving and transmitting) and audio one way (just receiving). The application is very simple and flexible, we can make whatever we want.

But it is not a finished work, there's no deskshare and presentation, and we still want to implement the audio transmitting. As Stephen said, we think it's important to have a "script" to test a scalability solution, and this is the reason we developed this bot. The source code is available on https://github.com/mconf/bbbot. Any thoughts about it would be appreciated.

The biggest issue is generating the test scripts is to do real load
testing, and finding a friendly customer that has access to large user
base for beta testing to really work thru the issues.

If anyone could help developing a methodology to do a real load testing, it could be simulated with this application.

Best regards,

--

   Felipe Cecagno

   Mconf Development Team

   https://mconf.org

[HostBBB.com]

Felipe how is audio, just one way with your app... does each user
still connect to conference direct with speex16, and you just don;t
allow for speaking? There are some call generation techniques using
yate/gulp... I'm also looking at using another freeswitch server to
bridge a bunch of calls into freeswitch to see if can simulate a heavy
workload on conference.

We have successfully got BBB broken down into 3 servers... 1) red5
audio/freeswitch 2) video/deskhare/whiteboard 3) chat/api/
presentation

I tried different combinations, some require very little modifications
to get working, others wont work without changes. LIke running the
whiteboard, and presentation on totally different red 5 servers is
pretty neat.

I'm working now to try and put the bbb-client fully on the LMS, so
custom branding, and config options could be deployed for my shared
hosting customers with no server mods.

I know the core team is heads down on final .8, maybe after the
release we could have another developers summit to try and work
together to get a solution that works itself back into the core.

regards,
Stephen
hostbbb.com

On Jan 23, 8:05 am, Felipe Cecagno <fceca...@gmail.com> wrote:
> Hi guys,
>
> Some developers of the Mconf development team, including me, are working on
> scaling BigBlueButton for a large number of users. As the first step on
> this task we have developed a Java application (based on the core of
> Mconf-Mobile) that is able to create a meeting and join N users (as
> moderators or viewers) with video both ways (receiving and transmitting)
> and audio one way (just receiving). The application is very simple and
> flexible, we can make whatever we want.
> But it is not a finished work, there's no deskshare and presentation, and
> we still want to implement the audio transmitting. As Stephen said, we
> think it's important to have a "script" to test a scalability solution, and

> this is the reason we developed this bot. The source code is available onhttps://github.com/mconf/bbbot. Any thoughts about it would be appreciated.

[Yariv Asulin]

Hello Fred and all,

I have been following you guys with development and setup of BBB, and wanted to share my admiring to this project. I am trying to learn section of the code so I can start helping with bugs, hopefully soon as I am really eager to help.

Regarding the bandwidth issue.

Are u saying that a server running all BBB elements( red5, freeswitch,etc) can only serve 25 people? Does it make a difference if it 2 classes of 12 each? Or 12 classes of 2 persons each class?

For example, private tutoring, will have normally 1 moderator and one viewer, so how many classes can a strong server with 8gig of Ram and strong CPU can serve? ( considering video, document and whiteboard ONLY sharing)?

I would like to use BBB for online foreign exchange students teaching each other different languages

Thanks again

Yariv

[Fred Dixon]

Hi Yariv,

If you can help fix bugs then you are our new BFF.  :-)

Regarding server capacity, see

   http://code.google.com/p/bigbluebutton/wiki/FAQ#How_many_simultaneous_users_can_BigBlueButton_support

If you stress test a given server and find that it supports 50 simultaneous users (ideally with real-world usage), then you can have one room of 50 users, 2x25, 5x10, 25x2, and every variation in between -- as long as you don't exceed 50 simultaneous users.

But wait, you say, don't we always state that we recommend 25 users or less.  Yes, we do, largely because it takes some skill to stress test a server and most people will grab the nearest server and expect to support X number of users, where X is the highest number they read in these forums, regardless of the server's capacity.  So we are conservative and always say 25 users or less.  

Better to delight than to disappoint.   

In general, if you have a dedicated quad-core CPU, it should be able to handle more than 25 users.  

Regards,... Fred

-- 
BigBlueButton Developer
http://bigbluebutton.org/
http://code.google.com/p/bigbluebutton

[Séverin TERRIER]

If i really understand that you prefer to tell 25 people, even if BBB
can handle (a lot ?) more, i also think it would be interesting and
useful to know, based on real use of BBB, how much people some types
of server can handle.
For exemple :
- a 2 core 2 Ghz CPU with 2 GB RAM
- a 4 core 2,6 Ghz CPU with 4 GB RAM
- a bi 4 core 3,6 Ghz CPU with 16 GB RAM

That would allow people wanting to use BBB to have a better idea of
how they should calculate the server, based on the expected use...

If people have this kind of information, please share it !
Séverin Terrier

> In general, if you have a dedicated quad-core CPU, it should be able to
> handle more than 25 users.

[Fred Dixon]

Hi Séverin,

Benchmarking is not easy, and there is a very strong tendency for first-time administrators of BigBlueButton -- and even experienced BigBlueButton administrators -- to get it wrong.

For example, see this thread

   

  https://groups.google.com/forum/#!topic/bigbluebutton-dev/zhZrlzG9ri0

The above poster had their hear in the right place, but their conclusions were *way* off.  

It's interesting to look at how other open source projects handle the questions of "how many concurrent users".   For example, in the FreeSWITCH FAQ: 

   http://wiki.freeswitch.org/wiki/FreeSwitch_FAQ#Q:_How_many_concurrent_calls_can_it_support.3F__Any_benchmarks.3F

they write

   This all depends on your application. You will need to load test with your application to know your limits, your mileage will vary depending on your specific requirements.

   Please do not ask this question on the mailing lists as you will always get the same official response from the FreeSWITCH project; "we only perform benchmarking and confirm these results per FreeSWITCH deployment, as each deployment will result in varying figures. Commercial support is available from the project for this task. The project has learned from experience the dangers of entertaining such questions and its policy is to not do so over the free public forum."

 

We all want to believe that BigBlueButton will scale to large numbers of users, but as developers we're grounded in cold, hard reality.  As the FreeSWITCH FAQ points out, "it all depends on your specific requirements".  

Based on our development and testing of BigBlueButton over the past four years, we're comfortable enough saying that on a *DEDICATED* *QUAD-CORE* CPU with +2G of memory, BigBlueButton should be able to support 25 concurrent users with *GENERAL USAGE*.

Even the above statement is dangerous as people want just want absolute numbers.  There are some that will read the above and think "Thank you Fred.   25 is the minimum number." and proceed to install BigBlueButton alongside their existing web application running on a shared VPS server (not easy to do, but some will eventually figure it out) and then hold a session with 25 user joining from various internet connections, have everyone share webcams, and have the presenter sharing their desktop and slides.  When they do try it, they will find it doesn't support 25 concurrent users.  (If you read the previous sentence and are thinking 'we'll, it sounds like it should', then your getting a sense of the danger in posting absolute numbers). 

For BigBlueButton, we believe the best benchmark is real-world usage.  If you've just conducted a session with BigBlueButton for 15, 20, 25, 30, 35, etc. concurrent users and it worked well, then let the community know about it!  Give as much information as you can about your hardware, bandwidth, BigBlueButton version, and usage of BigBlueButton during the session.

When your reading such posts, we encourage you to be a bit sceptical and look for holes in their data.  Was everyone on a LAN?  Was the machine dedicated or virtualized?  Did they do the test at 2am in the morning?  How many times did they do the test?  Did they test a few users and then extrapolate the numbers?  Did they restart the server after tests?  How did the monitor CPU and bandwidth usage?  How did they conclude that the test went 'well' or 'not well'. Did they get feedback from users on their experience?  Was the feedback qualitative or quantitate?  Etc.

We're not looking for perfect tests (there are none).  But with the community sharing their experiences, readers can get a blended sense of BigBlueButton's scalability without the danger of absolute numbers.  

To help out, we'll be doing another community stress test of BigBlueButton 0.8 before it's final release, and will share all our results.

Regards,... Fred

-- 
BigBlueButton Developer

http://bigbluebutton.org/
http://code.google.com/p/bigbluebutton

BigBlueButton on twitter: @bigbluebutton

[KLam]

I am just thinking how to put 200 people in a room... With some
recently testing, I noticed that freeswitch is eating up CPU even
everyone is mute.
With about 10 users online mute, freeswitch is taking about 35% CPU on
one core. (just eye ball on the results from top )
If the freeswitch cpu usage is linear, we might be able to get 10 x 2
x 8 = 160 users on a 8 core cpu running at 70% CPU over all.
(This is just a possibility)

So, I think using freeswitch as a webinar solution is not good.

I just hold a webinar on Windows Media Server, we had about 250 users
online watching ,streaming about 400kpbs. The 100 M bandwidth was
maxed out,
and the CPU on the server is still very low usage.

One possible solution to support a large group is to get the audio
source and stream it with icecast or red5 to those that just want to
listen without joining them to freeswitch.
And if people want to participate and talk, they can click on the
Mic, and join the freeswitch conference.

In order to scale, I am not sure if running each of the red5 app on
its own JVM matters or not, assuming no limit on memory.

On Jan 24, 5:25 am, Fred Dixon <ffdi...@gmail.com> wrote:
> Hi Séverin,
>
> Benchmarking is not easy, and there is a very strong tendency for
> first-time administrators of BigBlueButton -- and even experienced
> BigBlueButton administrators -- to get it wrong.
>
> For example, see this thread
>
>  https://groups.google.com/forum/#!topic/bigbluebutton-dev/zhZrlzG9ri0
>
> The above poster had their hear in the right place, but their conclusions
> were *way* off.
>
> It's interesting to look at how other open source projects handle the
> questions of "how many concurrent users".   For example, in the FreeSWITCH
> FAQ:
>

> http://wiki.freeswitch.org/wiki/FreeSwitch_FAQ#Q:_How_many_concurrent...

> BigBlueButton Developerhttp://bigbluebutton.org/http://code.google.com/p/bigbluebutton

> BigBlueButton on twitter: @bigbluebutton
>
> On Tue, Jan 24, 2012 at 4:22 AM, Séverin TERRIER

> <severin.terr...@gmail.com>wrote:

[Pablo Sotomayor]

Great idea to the next release of BBB. I think the focus now is solve the current bugs of 0.8 version.

[KLam]

Recently, I was trying out a tool to do connection load testing on my
server.

This tool is pretty helpful. It allows you open up many connection to
your server with rtmp:
http://jakehilton.com/?q=node/64

In my testing, I put down 1000 connections. And I start seeing errors
on my server after a few hundred successful connection maded. The
errors was
"Too many open files"
in my red5.log.

If you haven't fine tune your ubuntu server, by default you get 1024
open file for a user. And I tried different things to change this, but
it didn't work.

Finally, I found this post, and followed the instruction. It worked.
And I was able to put more simultaneous connections on my server.

http://blog.thecodingmachine.com/content/solving-too-many-open-files-exception-red5-or-any-other-application

I was also able to run red5 with a higher number of threads for rmtp.

On Jan 26, 3:58 pm, Pablo Sotomayor <pa...@sotomayor.com.br> wrote:
> Great idea to the next release of BBB. I think the focus now is solve the current bugs of 0.8 version.
>
> On 26/01/2012, at 16:42, KLam wrote:
>
>
>
>
>
>
>
> > I am just thinking how to put 200 people in a room... With some
> > recently testing, I noticed that freeswitch is eating up CPU even
> > everyone is mute.
> > With about 10 users online mute, freeswitch is taking about 35% CPU on
> > one core. (just eye ball on the results from top )
> > If the freeswitch cpu usage is linear, we might be able to get 10 x 2
> > x 8 = 160 users on a 8 core cpu running at 70% CPU over all.
> > (This is just a possibility)
>
> > So, I think using freeswitch as a webinar solution is not good.
>
> > I just hold a webinar on Windows Media Server, we had about 250 users
> > online watching ,streaming about 400kpbs. The 100 M bandwidth was
> > maxed out,
> > and the CPU on the server is still very low usage.
>
> > One possible solution to support a large group  is to get the audio

> > source and stream it withicecastor red5 to those that just want to