chopping when running 8 webcams

[Kent B]

Hi 

We have experienced chopping, when running a conference with 8 webcams running at the same time and 7 other logged in users with webrtc audio. Audio becomes occasionally very choppy and users complains over not hearing well. personally I think it has to do with the fact that the webcams consumes bandwidth and resources and those with lowest bandwidth suffer.

The thing is, they moved over to adobe connect and says they have no such problems there with the same set up.

Our setup is BBB 0.9.1 mconf 075 and server in facility with 1000 mb/s. 16 gb ram and 8 core cpu.

Is this something that you knew or is it something in our set up.

No one else that uses our service runs so many webcams at the same time and no one has any problems with audio.

How come adobe connect has solved this?

Regards

Kent

[Fred Dixon]

Hi Kent,

Can you post information about what browsers the users are using?  

To give suggestions on how to improve the audio, a bit of background first.  The audio and video from the BigBlueButton client actually runs through separate channels.  both media are, at the base level, simply data packets sent and received from the remote BigBlueButton server.

The key to understanding choppy audio is to look at it from three different perspectives: (a) how those data packets are sent/received (TCP/IP vs. UDP), (b) how much bandwidth is available for sending/receiving the packets, and (c) how well can the client/server compensate when (b) is not good.

Let's start with how the packets are sent/received.

IE and Safari

When a user runs BigBlueButton in IE or Safari, they will use the built-in Flash audio.  It's pretty good and has been around for years.  The audio is sent using Flash's  real-time message protocol (RTMP) audio and are encoded using SPEEX (a very good audio codec).  However, the packets are sent via TCP/IP, which means the underlying transport protocol will resend missing packets. 

When your browsing a web page on a wireless device, this resending works great.  Whether the page renders in 100ms or 200ms, you don't care -- the network stack is ensuring that *every* packet gets through to your client.

However, the human ear is very attuned to audio gaps.  When sending audio via TCP/IP, if bandwidth is sufficient to re-transmit dropped/missing packets  -- which tends to occur when a user is on a wireless network -- you don't notice anything.  The resending is occurring so quickly, there are no gaps that are picked up the ear.

However, if the user's bandwidth is low and/or there are many dropped packets, then you start to get scenarios where (1) the audio packets can't get through fast enough to keep up with the stream of audio (which causes gaps), or (2) the resending of TCP/IP packets starts to exacerbate (1).

FireFox and Chrome

When a user runs the BigBlueButton client in FireFox or Chrome, then, by default, the BigBigBlueButton client will attempt to send the audio via the browser's built-in support for WebRTC connection. 

I must stress "attempt" because if there is a firewall between the browser and the BigBlueButton server (specifically, between the browser and FreeSWITCH), then the firewall may prevent the user's browser from negotiating a WebRTC connection or establishing a real-time protocol (RTP) stream for the media itself.  In which case the BigBlueButton client will automatically fall back to sending audio via RTMP (over port 1935).  

This ability to fall back means BigBlueButton can support all major browsers (Safari, IE, Edge, Opera, FireFox and Chrome) with the best possible means to transport audio supported by each browser.

(Side note: If the user is behind a firewall that blocks port 1935, it the BigBlueButton client fall back even further to tunnelling everything through port 80.  This is not ideal, but if there is a good network connection, the user might not even notice).

WebRTC uses the opus codec, which is even better than the speex codec.

UDP doesn't attempt to resend data packet, making it ideal for streaming.  When browsing a web page, you obviously want your data going through TCP/IP packets -- the browser will always get correct data from the underlying TCP/IP stack (regardless of how many times the packets needed to be resent).  But with streaming media, the human ear is listening for a coherent stream of audio in real-time.  Time is of the essence.  A UDP packet bypasses the overhead of resending and simply tries to get through the network quickly as possible.

Unlike TCP/IP, UDP packets are not guaranteed to arrive.  This introduces other issues: the packets may arrive late, out of order, or simply not at all.  This brings us to the third perspective of how analyze choppy audio: how well can the client/server compensate when these issues occur?

We need to look at this question from two scenarios: (scenario 1) the speaker has poor upstream bandwidth, or (scenario 2) the listener has poor downstream bandwidth.

The best way to determine which scenario you encountered is to ask the following question: Did all the listeners in the session hear choppy audio from the speaker (this scenario 1) or only some of the listeners (this scenario 2)?

In scenario 1, for each incoming audio channel FreeSWITCH maintains a "jitter buffer" that is like a holding window for allow packets to arrive late or out-of-order.  If packets arrive within this window, FreeSWITCH can re-oder the packets to produce a steady stream for the human ear.  It can also compensate a bit for dropped packets to smooth out the missing gaps.  The jitter buffer works much better with UDP than TCP/IP as the resending might cause the packet to arrive so late is misses the window.

In this regard, if the presenter has poor upstream bandwidth, they can also turn off their webcam to free up as much precious bandwidth as possible for sending audio packets.

You might be wondering "well, how much bandwidth should the presenter have?".  In general, we recommend 0.5 Mbit/s upstream and 1.0 Mbit/s downstream.  If the presenter is going to share his or her desktop, then we recommend, at a minimum, 1.05 Mbit/s upstream.

Be careful when a user say "But I have X Mbit/s sec from my service provider".  A service provider does not guarantee the same bandwidth at all time. We recommend using speed test (http://speedtest.net) to check for actual bandwidth.  It's the actual bandwidth that counts.   Have them try a speed test to a server that close to the network location BigBlueButton server.  Speed test will also show ping time, which is how long a packet takes to reach the server.  The lower the number is better (if your under 50 that's pretty good).

Hard numbers are challenging as they imply hard boundaries, and they are not.  Low numbers don't mean the user experience is going to be poor, but they do help the user understand what might be causing the issue.  Most people never look at the number of bars their phone is displaying for signal strength, but if their audio call is going poorly, the can use the bars to help determine if the signal strength (network connectivity) is poor.  

It's hard to ask the user "Do you have many dropped packets or is your latency too high?"   This is why the echo test in BigBlueButton is so important.  Rather than give the user numbers (or attempt to determine the "signal strength" on the network, which very tricky), the BigBlueButton client puts the user into an echo test and asks a simple question "Say a few words.  Can you hear yourself?".

What the user hears is their full round-trop audio going from their client, through their upstream connection, hitting the BigBlueButton server, and returning back to them through their downstream connection.   In broad terms, they hear what others will hear (this isn't entirely accurate because others may be on a much poorer network condition).  The important point is if the echo test yields a poor audio, the first place to look is at their network connection.

In scenario 2, if all listeners are hearing good audio from the presenter except a particular user, then we would recommend looking closely at how that particular user is connecting to the BigBlueButton session.  Are they using FireFox or Chrome?  If so, do they see the words

  [ WebRTC Audio ]

in the lower right-hand corner of their BigBlueButton client?  If not, there may be a firewall.  If they see the words "[ Tunnelling ]", then they are very likely behind a restrictive firewall and will find a better internet connection.

BigBlueButton does give feedback as "client notification" which appear in the bottom right-hand corner as a tool-tip when they first connection.  

- If they don't connect with FireFox or Chrome, then BigBlueButton suggests "Try using FireFox or Chrome for better audio" in the configuration notifications (seen as a warning icon in the lower right-hand corner).  

- If they are tunnelling, BigBlueButton notifies them stating "A firewall is preventing your client from connecting directly on port 1935 to the remote server. Recommend joining a less restrictive network for a more stable connection".

- If they are running an older version of FireFox or Chrome, BigBlueButton will recommend they upgrade.  

There are two other benefits from using WebRTC for audio.  First, the implementation of acoustic echo cancellation (AEC) in FireFox and Chrome is better than that of Flash. This means there is less likely that a user connecting via WebRTC without a headset will cause echo for others in the session.  And second, the built-in support in FireFox and Chrome for auto-levelling of the speakers microphone helps automatically adjust the gain to pick up a speaker's audio if their microphone's sensitivity is initially too low.   Both FireFox and Chrome have different implementation of AEC and auto-levelling, so if the echo test does not sound good in one, or users are stating that the presenter's audio is too low, have them try switching between FireFox and Chrome.

Summary

Assuming the user's microphone and speaker (or headset) are working properly, then if they are experiencing choppy audio, try the following 

1.  Recommend they use FireFox or Chrome (especially if they are going to be the presenter).

2.  Ask if they see the words "[ WebRTC Audio ]" in the lower right-hand corner?  If not, try switching to FireFox and Chrome.

3.  Ask if they hear themselves clearly during the echo test?  If not, try switching *between* FireFox and Chrome.  (They both implement their own methods to establish WebRTC connectivity, we've seen network cases where one works where the other one doesn't.)

4.  Ask if they received any client notifications (warning icon in the lower right-hand corner) on startup or during the session?  In particular, ask if they getting notifications during the session that their is reconnecting (unstable network) or tunnelling (restrictive network).  If so, try switching to a different network location, or, if they can do a hotspot with their phone, try connecting via the hot spot and see if everything suddenly improves (which immediately isolates the network as the issue).  

5.  Ask if they are on a wireless connection?  If so, try getting closer to the base station, try a different wireless network, or connect via an ethernet cable to remove wireless from the equation.

6.  Have them do a speed test (http://speedtest.net/) to determine their actual network speed.

7.  Ask if there are many webcams open?  If so, try closing the webcams.

There are other general questions to help narrow down the cause.  

- Have they had good audio in the past?  If so, try to figure out if anything is different in their setup.  For example, a travelling instructor presenting one evening from their hotel room might not have as good audio as the previous classes.

- Did the audio start off good and then degrade over time?  If so, is it possible that many people have started to connect to their same network.   You start to get edge cases a person is in an on-line class and as other family members come home and start using a *lot* of bandwidth for file sharing their audio starts to degrade.

- Are they using a webcam to pickup their audio?  It many cases this is fine, but the user will *always* have a better audio experience with a built-in headset.

- Have they been running their browser for a few days (I do this all the time).  If so, try restarting FireFox or Chrome and login again.

You stated upfront Kent what version of BigBlueButton you were running and that your server had good network connectivity (which is great).  If your group of users can consistently reproduce the poor audio issues with your server, you are always welcome to have them try our demo server (currently running a developer build of BigBlueButton 1.1)

  http://demo.bigbluebutton.org/

For more privacy, have them use one of the rooms here

  http://demo.bigbluebutton.org/demo/demo2.jsp

and see if simply switching servers had any effect.  If it does, we'd be interested to hear about it.

Regards,... Fred

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-dev+unsubscribe@googlegroups.com.
To post to this group, send email to bigbluebutton-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/bigbluebutton-dev.
For more options, visit https://groups.google.com/d/optout.

--

BigBlueButton Developer

http://bigbluebutton.org/

@bigbluebutton

[Kent B]

Thank you Fred for a fast and extensive answer

I will seriously go through this and get back to you when I know more about the setups for the attendants on the specific meeting.

Thanks again

Kent

[Kent B]

Hi Fred

Thanks for your extensive post.

I have been reading it and have some thoughts.

For our situation I think that some do not follow the suggestion we have done to use Firefox or chrome.

If they use internet explorer as you mentioned will lead them to flash and a not as good sound as with webrtc.

I saw these links from adobe connect:

https://helpx.adobe.com/adobe-connect/tech-specs.html

http://www.makeuseof.com/tag/heres-make-internet-explorer-safari-work-webrtc/

As it seems they have webrtc support for internet explorer.

That shows why they say that problem where solved when they move to adobe connect.

Some older users are so used to internet explorer so they do not know any thing else.

The second link I posted was suggesting a plugin to explorer and safari to make them work with webrtc.

Of course some modifications might be needed to BBB.

So our problem was probably related to some using explorer as you mentioned.

Thanks for helping

Regards

Kent

Den lördag 8 oktober 2016 kl. 02:21:47 UTC+2 skrev Kent B:

[Fred Dixon]

Hi Kent,

> For our situation I think that some do not follow the suggestion we have done to use Firefox or chrome.

> If they use internet explorer as you mentioned will lead them to flash and a not as good sound as with webrtc.

Flash actually provides pretty good audio, but WebRTC is better (especially under lower bandwidth and/or higher packet loss conditions).

Definitely have them try a session again using either FireFox or Chrome.   And again feel free to user the demo.bigbluebutton.org server to test with them.

> The second link I posted was suggesting a plugin to explorer and safari to make them work with webrtc.

Thanks Kent.  Personally, in many cases, I believe its preferable to ask users to try a different browser than have them install a plugin.  Over the years, Google and Mozilla have done an amazing job providing support for WebRTC 1.0 specification.  Both browsers are frequently updated and provide for automatic updates.   

We'll check out the Temasys plugin.  Looking ahead, it looks like Microsoft Edge will have support for WebRTC 1.0 (which is what Chrome and FireFox support).

  https://developer.microsoft.com/en-us/microsoft-edge/platform/status/webrtcwebrtcv10api/

Regards,... Fred

--

You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-dev+unsubscribe@googlegroups.com.
To post to this group, send email to bigbluebutton-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/bigbluebutton-dev.
For more options, visit https://groups.google.com/d/optout.