Users work fine on Zoom, but not on BBB?

[Lee Cujes]

Hi team

We are a large Australian training organisation (70,000+ students per year) and researched, selected and implemented BBB as our virtual classroom platform solution.

We selected BBB primarily for:

a) the feature set

b) not needing to have users download anything to use it

We run on a dedicated m5.large AWS instance and have been live for about two months now. Presumably we're running the latest version of everything since it's all brand new.

We have done our own exhaustive testing of the system internally, and now several pilots with real students. Typically our courses run for anywhere from 1 day to 2 weeks in duration.

Our internal testing initially provided mixed results (poor audio for some users, freezing screensharing and video sharing, some video issues). We managed to address these issues by shifting the AWS region from Singapore (where all our other stuff is hosted) to Sydney. I hope this one point helps some users.

Our first pilot could not run on BBB at all because inexplicably, video would not work at all for any participant. Our external support partner added a STUN server (whatever that is) to resolve the issue (why wasn't this done as part of the original setup, who knows, and why everything was working prior to the pilot, again, who knows...)

Our latest pilot program ran last week.

This was the feedback from the trainer:

"We commenced day 1 using BBB as instructed however only 3 of my 8 participants could activate their cameras. The system was providing feedback to them to:

a] update camera software

b] incompatible camera

c] no camera found.

They all could connect via audio. 

We utilised the polling function. This provided good results, however one student could not participate in the polling, claiming nothing was appearing on his system.

The shared documents function was used, however it froze several times, requiring a restart of the process to regain access to the document shared.

At 10:30am, I transferred the class to Zoom. All students were able to video connect without issues. I will continue with Zoom and await the issues with BBB being sorted out."

As company CIO having made the the BBB decision, and having spent a huge amount of time (including time and investment with external support partners) I really am feeling a level of frustration at this point.

I don't expect anyone to be able to diagnose the above user issues. However, if all those users who were struggling to get their video working were immediately able to log in to Zoom without any problems whatsoever, then we really have a problem.

Our business and these courses rely on a smooth, hassle-free user experience.

If we can't sort this out, we'll need to decommission BBB and change our platform to Zoom, Google Meet, or similar. At this point, we'd give up features to stop the pain.

I would love to hear from others who may be in a similar boat (many students, many courses per week), especially if you have suggestions for "making the problems go away". How have you managed to smooth the user experience?

Kind Regards

Lee Cujes

Pinnacle Safety and Training

[dse...@gmail.com]

Dear Lee,

First of all in your experiment you haven't mention about several things:

1) As i understand you run service on one server ( m5.large AWS ) However you didn't mention number of session (classes) and number of users per session

2) Have you look https://docs.bigbluebutton.org/support/faq.html#what-are-the-minimum-hardware-requirements-for-the-bigbluebutton-server and https://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support

3) Have you made any fine tuning on server (OS, BBB etc) 

According to my experience (Scalelite cluster more than 1200 user over servers) BBB Work excellent. Voice problems most of depend network side. 

So my advice:

1) You can reduce webcam video BW requirement smaller so in this case you can eliminate BW related problems.

2) If number of  concurrent users much if you have option you can use dedicated server not Cloud server or you can divide servers on different size of machines ( Freeswitch on diffirent machine or scalelite and several machines. 

3) You can get support from  BBB professionals 

Good luck.

Devrim

[Lee Cujes]

Hi Devrim, thanks for your response.

In answer to your question, we're still piloting the system so we have never had more than 1 class at a time and around 15 users max, at this point. We will scale to several concurrent classes with a user total of ~50 max in the future. You've asked if we performed any fine tuning on the server - the answer is no.

In terms of server, we have now upgraded our m5.large instance to c5.2xlarge (8 CPUs and 16GB memory) as recommended by BBB, however nearly all the issues I mentioned do not appear to be infrastructure related. Users struggling to get their cameras operating is not related to the infrastructure, it's the software. The same users struggling to operate in BBB being able work easily in Zoom is not ideal. Our experience thus far shows that Zoom is a more hassle-free user experience. We want to make BBB hassle-free for our users.

I am most interested in how organisations who are using BBB in a large scale have managed to smooth the user experience.

Kind Regards

Lee Cujes

[Hasan Hüseyin Pay]

Hi, last month we had 30000+ students and 1000+ teachers in our system. I think the biggest issue is the browser because when users want to open the camera or microphone, the browser asking permission(per subdomain). It's one time ask but generally, users click the deny button. Since BBB is an in-browser app it cannot handle permissions related issues properly (e.g. some users disable the camera for chrome). But the Zoom is a native app so it can handle all permissions related issues. We prepared a step by step guide and FAQ with screenshots and videos it solved most of the issues. 

Hasan Hüseyin PAY

Lee Cujes <lee....@pinnaclesafety.com.au>, 9 Eyl 2020 Çar, 05:13 tarihinde şunu yazdı:

--
You received this message because you are subscribed to the Google Groups "bigbluebutton-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-users/f06f4bae-9532-440b-9b85-0ce7ed992f32n%40googlegroups.com.

[TeachReo]

Hi Hasan

Just for inscrease my knowledge how many and what kind of servers do you have ?
Could you describe your system ?
I think it could be interesting for a lot of people

Regards

Fabrice

[Lee Cujes]

Hi Husan

Thanks for your comments.

Is your guide something that is publicly accessible for your many students? Is it something you're able to share with us?

Kind Regards

Lee Cujes

[Hasan Hüseyin Pay]

Fabrice, I think our use case is a bit unique. We had 4 sessions for 90 minutes a day. In the most crowded session, there were 8000+ attendees, every meeting had 8 or 15 attendees, and 1 or 2 cameras opened(but in some meetings all cameras had closed). We developed our load balancer that can scale on the cloud. We used c5.2xlarge. Generally, there were 140-150 attendees per server but they can handle about 180 attendees. After 180, despite the server's CPU usage is not high(about %60), BBB is getting laggy. However, these numbers highly depend on the use case.

Lee, I'm attaching the FAQ (It's Turkish, you can use google translate). 

9 Eylül 2020 Çarşamba tarihinde saat 10:08:03 UTC+3 itibarıyla Lee Cujes şunları yazdı:

[Hasan Hüseyin Pay]

It's mentioning in the FAQ but I want to emphasize that most of the Samsung phone users confuse Samsung Internet with Chrome. Samsung Internet is a different browser and does not support BBB.

9 Eylül 2020 Çarşamba tarihinde saat 14:25:45 UTC+3 itibarıyla Hasan Hüseyin Pay şunları yazdı:

[Lee Cujes]

This is fantastic Husan - thank you so much!

Kind Regards

Lee Cujes

[Fred Dixon]

Hi Hasan and Lee,

I am the product manager for BigBlueButton. 

> We want to make BBB hassle-free for our users.

We want that too.  

Each release we (the core developers) work towards making the product more robust, easier to use, and more scalable.  We have been working on BigBlueButton 2.2 (the current release) for over two years, and overall over 10 years on the product itself.

We appreciate you sharing your feedback.   We prioritize stability over features, and we encourage you to run the latest version of BigBlueButton 2.2.

Also, to help diagnose issues, your welcome anytime to use our test server

 

  https://test.bigbluebutton.org/

which is running the latest build of BigBlueButton 2.2 along with a TURN server to help users through firewalls.  

Regards,... Fred

--

You received this message because you are subscribed to the Google Groups "bigbluebutton-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-users/7b7abf3b-924f-416f-9553-c4f21fc24ef5n%40googlegroups.com.

--

BigBlueButton Developer

http://bigbluebutton.org/

Like BigBlueButton?  Tweet us at @bigbluebutton

[TeachReo]

Hi hasan,

you're not alone and unique ;) and thanks for your feedback

Regards

Fabrice

[mabras]

Hi Hasan,

We have a similar issue here, but with sound. Most attendess complian about delay and cracky sound. If they switch to jitsi or Zoom, there will be no problem for them.

Hi Fred,

> We prioritize stability over features

Sorry if I am hijacking the thread, but I would like to draw you attention to this issue:

https://github.com/bigbluebutton/bigbluebutton/issues/8510

I thinks this will make a diffrence toward stability of BBB. This issue prevent us from hard limit the maxUsers of sessions. This a real problem. Our rooms keep crashing because of this. We know that BBB can't handle more than 100 users per session (soft limit) but in reality we need the hard limit to prevent room from crash.

Thanks for all your hard work.

Regards, Mohammad

[Fran Pereyra]

Hi Lee, hi all. We are in the exact same boat.. or well said, we were. Because we had to leave BBB behind for the exact same reasons you mentioned. We spent so much time, so many hours on setting and testing new configurations, server setups, etc. but nothing worked. 

Here are some lines I wrote a few weeks ago in other similar discussion, about the reasons of our decision to leave BBB:

Unfortunately these issue made BBB unreliable from our users standpoint and they won't ever look back into it again. We did everything possible from configurations, to server types and locations, TURN setups, but nothing really worked (in reply to @bayden about his suggestions on TURN, location, etc..) these are NOT the causes of the techincal issues.

We have spent so much time on learning the BBB workflow, setup, customizations and logic.. etc. but the system doesn't work 100% of the times. There are so much issues and variables (for example users from pc/mobile having different type of issues...) that made BBB unreliable -for us too-, unfortunately the conclusion is we have to put it to the side and just wait until we can check BBB is really stable. In the meantime we will help and continue testing everything we can during this process. 

This has been a hard decision but our institution decided to leave behind BBB because this issues were interfering in the educational process and in the students experience.

Our users just compared their experience on other web conferencing systems like Zoom or Google Meet (even when they miss lots of functionalities that BBB has), but at the end of the day these issues (audio/video, etc) are the very basic functionalities that should work smoothly.. and they're not in BBB.

We (administrators) hope one day BBB will be realiable as it was one day, but that is not happening right now.

Discussion link: https://groups.google.com/forum/#!topic/bigbluebutton-users/UbEYa5ieogA

Regards,

[Ghazi TRIKI]

Hi Guys,

I think the comparison is not 100% fair. Comparing a commercial product to an open-source platform is not a fair job 😁. While BigBlueButton makes its code and packages available for everyone, Zoom does not.

On the other hand, you pay a commercial license for any commercial platform to get a ready-to-use platform with a mastered infrastructure and on the opposite, you get an open-source platform where you need to master your deployment and learning how to ding things properly.

And if you want to have your deployment of BigBlueButton you need to have the minimum knowledge that is already mastered by the competitor commercial platform, you need to be aware of latency, VoIP configuration, bandwidth, networking issues, load balancing, servers capacity and so on...

BigBlueButton is very reliable and there are millions of people using it around the world while I am writing this line.

And, to have a production-grade deployment that fits your needs you can rely on the knowledge acquired by of the commercial support providers, not because they have unique knowledge, in contrary all their knowledge is scattered through the forums, but because they have concentrated experience thought long-term contributions and you can easily get in touch with any of them through this link https://bigbluebutton.org/commercial-support/

With Regards,

Ghazi

Le mardi 15 septembre 2020 à 04:39:18 UTC+1

[Fran Pereyra]

Hi Ghazi, thanks for your response.

Our servers and BBB deployments have been configurated and installed by experts. As I said, that is not the cause and we have checked it exhaustively. We have been using BBB for a long time. I repeat this over and over..

I don't think the point of this discussion is the comparison with commercial products. (BTW: Jitsi is an open source project and someone used it to compare its performance with BBB as well..)

On the other hand.. how "open" is a project, if the only ones who can set it up, or "handle" it correctly (according to your suggestion) are commercial-support companies..? 

I think the point is the reliability we expect from a big open source project like Bigbluebutton, as it was some time ago. We (engineers, developers and networking experts) choose open sorce projects like BBB (and many others..) because we know how to handle it and/or we take the responability to get the necessary knowledge to use it and test it before making a judgement like this in a community forum. 

You can find more complaints messages like Lee's or mine, and the number of cases keep growing.. Please take a deeper look at this thread as an example: the author is Jim Craig, Senior Product Manager of Red Hat. I think he probably checked twice their BBB configuration before posting some complaints here....... don't you think? 

Plese don't get me wrong, but before arguing how "open" is an open source project, I would consider how accesible is it to deploy, customize or configure without the need of commercial support... This should be a priority specially if you are aware of the millions of users using it..

Anyways, we will continue to contribute all we can, from betatesting and giving feedback in the next versions, etc. and we really appreciate the hard work the core BBB dev team have been doing for so many years. But right now we must wait until we confirm that BBB is reliable to use it in production again.

[sd...@distancelearning.cloud]

Collectively we “all” are running thousands of servers, and 100,000+ class sessions a day in “Production”.   I’ve Been doing this since 2011

 

I have customers that have just migrated from Zoom to BBB, because it crashes daily for their large school district,  scaling on saturated        public internets is always a challenge even for Commercial providers.

 

There are some older IPADS that crackle, updating IOS to latest solved this. IMHO don’t mess with the default opus/freeswitch settings (you cause more issues tweaking)

If you have a specific device problem open an issue.

 

If you run 2-5 cams a meeting with 2.2.23+ and limit total user to 200 on a server should never have a problem.

 

Most issues my customers have are trying to push a bunch of video cams down to a user on marginal wifi or cell connection, on underpowered laptops or mobile devices.   Audio will always breakup in this case.

 

If you need to move around talking heads… Zoom is the way the go currently.

 

We are dependent on the browsers for webrtc..   Zoom’s ability to have you download an optimized binary for a more robust solution to help with spotty network/reconnections is a given.

 

Good news is  95% of installs work thanks to bbb-install, and major universities and enterprises use daily.

 

And running a private dedicated cluster keeps all your students data in your control… Not the case with Zoom or other commercial platforms..

 

Sorry the current BBB in your configuration did not meet the customers expectations. 

My opinion- Less than 100 in class, 200 in server, limit cams 2-5 and it runs all day long.

 

Regards

Stephen

--

You received this message because you are subscribed to the Google Groups "bigbluebutton-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-users/b0f0b225-8277-44f0-a4f7-e2c7670af9e2o%40googlegroups.com.

[Adrienne Fitzer]

Same. Well said.

--

You received this message because you are subscribed to the Google Groups "bigbluebutton-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-users/2d4e0407-53db-4a6f-9290-8b696b998cfbo%40googlegroups.com.

[Fran Pereyra]

Hi Stephan, congratulations for serving so many users with your servers. We'd love to do that as well, but the results / UX with BBB made impossible to argue with our +6000 users (teachers and students) to move from Zoom to BBB for this causes. Unfortunately we cannot afford the risk anymore either.

And as you can see, I am not the only one. You can take a look on these forums and see that many users are facing the same issues. So congratulations for making it work, maybe you could share more details about your big deployment and how you made it to work in a stable-reliable fashion..

We have opened dozens of issues in github and the community has always kindly helped a lot, but at each release we found new issues or problems. I want to say (again) that we haven't had so many issues in earlier versions of BBB, and you can check this by looking at the number of people talking about this issues in these forums recently.

I will repeat.. this issues are not in our side (third time I need to repeat this in this thread). 

Maybe these cases should be taken more seroiusly (instead of just saying there's an issue in the setup..) and be analyzed a bit deeper to take a look for common factors of several BBB operators all around the globe at 1000 of Km's of distance bewteen each other having the exact same issues................

[Paulo Lanzarin]

Maybe these cases should be taken more seroiusly (instead of just saying there's an issue in the setup..) and be analyzed a bit deeper to take a look for common factors of several BBB operators all around the globe at 1000 of Km's of distance bewteen each other having the exact same issues................

You're (tentatively) implying we do not look nor take those reports seriously.

--
You received this message because you are subscribed to the Google Groups "bigbluebutton-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-users/ab0ecaa2-a3a0-4e95-bab0-024705b57dcao%40googlegroups.com.

[Fran Pereyra]

Hi Paulo, thank you for coming across this thread.

In this context (forums) my saying was referred for those who are finger pointing to our ""defective"" server deployment as a probable cause (even after explaining several times this has been checked) without wanting to understand the real reason of our complains or neither taking a deeper look for other situations than their own "effective" servers.. 

I'm sure you (developer team) are working hard to solve these, and we are very much thankful for that and for sharing your work with the world. 

We are BBB fans as you know, but due to the mentioned reasons, we had to leave BBB behind for now. We really want the day in wich we can move back to BBB confidently to come soon. We'll continue to share our feedback and help all as we can in that meantime.

[sd...@distancelearning.cloud]

Fran,

 

Do you have BBB infrastructure still running?   did you load balance the turn servers as well as the bbb servers for 6000 users? How many cameras and users were running on the servers when you had issues.

 

The use models vary so widely from customer to customer..  Agree if your outside of these constraints below you will most likely see issues and miss customer expectations.

 

My take “Less than 100 in class, 200 in server, limit cams 2-5 and it runs all day long” ..   I turn away all use cases out side this currently.   Looking forward as we improve 2.2.23 and 2.3 and the reliability and scaleability improves.

 

Sure it will run with higher numbers, but as you push, users will see issues, the biggest being client side bottlenecks and reconnections.

 

Would be glad to look at any specific issue in tracker and try to help debug, but it appears there are a number of reasons your customer base switched to Zoom..

 

 

Regards,

Stephen

 

 

 

 

From: bigbluebu...@googlegroups.com <bigbluebu...@googlegroups.com> On Behalf Of Fran Pereyra
Sent: Tuesday, September 15, 2020 10:19 AM
To: bigbluebutton-users <bigbluebu...@googlegroups.com>

--

You received this message because you are subscribed to the Google Groups "bigbluebutton-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-users/ab0ecaa2-a3a0-4e95-bab0-024705b57dcao%40googlegroups.com.

[br...@live.com]

+1 same here

Audio on jitsi more reliable than BBB and I tested that with our team before but I do not remember in which BBB version. 

regarding test on demo we can not ask our users to test the link because https://demo.bigbluebutton.org/ they consider it as third party  which is not allowed. 

I would like to ask Stephan to share his experience with us on this common issue: 

I would ask Stephan and Ghazi about this issue 8133 what did you say to your customers once they inform you that some students can not listen on some ios?   Of course for 100,000 customer at least you have one facing that issue....

This issue confirmed on BBB side not from webrtc or Apple side

(By the way Fran, even if all your setup are good with this issue, some of ios once they open their mic you will notice crackling noise )

This issue might fix this month and I would suggest to give it a try after fixing this issue

I knew BBB built as educational tool it's not a webinar or anything else. but what if a random student take a our BBB link meeting and share to public 

that will allow to more than 150 to join the meeting then BBB start lagging and stopped everything. You might suggest to allow joining by permission. but actually this is not helpful! One of our teacher said to me " I have to focus on teaching a lesson, not to waste time for operating the class by accept and deny students! " There's a bug on Max Participant which make BBB not hassle free even for lowest attendees number

I love BBB and I wish if core developer just focusing on reliability and make it hassle free. we do not need more features we need fixing core features. 

Thank you for everything

[sd...@distancelearning.cloud]

8133, there is a community developer (Mario) currently working on debugging some IOS edge cases. Not sure what the current status is on his work, but any specific data you have regarding your issues, with the IOS version, device etcs, and how to replicate should be added to that ticket.

 

Regards.

Stephen

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-users/cec12fa2-95d9-4bf3-ab25-c388a42dab49n%40googlegroups.com.

[br...@live.com]

Hi Stephan, 

Previously you said in the latest versions, it's easily  to open 5 camera  per 100-200 participants. However, I saw this picture attached it was up to 50 camera open with 88 participants. I wonder if something changed in the KMS made that not possible to happen again? And if it's not, how to make it happen without facing troubles in audio and video ? 

[HostBBB.com]

this is not a practical daily use of BBB.   since each user needs the bandwidth and cpu processing for their browser to decode the 50 streams.   if your all on an internal network with enterprise grade i7 cpus in your network you still would have issues since the number of kurento streams needed to send to each user 50+ would be a challenge

Also 1 meeting per server would be the max most likely.

2-5 cams per meeting...  is what works well in the real scenarios. IMHO.

regards,

Stephen

[Kenren Taisho]

Hi,

The picture seems only to be a screenshot of the online conference, normally used as attendance or for whatever reason it may serve. In my opinion, I don't think the conference was held with all these cams open.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-users/a78a7a17-85b0-4ce8-87a9-b20281ca0da3n%40googlegroups.com.