BigBlueButton WebRTC connection quality between Continental USA and SE Asia

[Kirk Pathumanun]

Hello,

I have been using Bigbluebutton locally in Thailand on hardware servers with 1Gbp upload and download without any issue on the WebRTC quality for the past 3 years within the entire country, unless of course there were poor connections from clients which were anticipated technically. However as we scale out the system to the US between San Francisco and Texas, I have notices a high level of sound degradation from the US to Thailand side. As of now we have two Bigbluebutton servers set up in Thailand IDC, so I was wondering if  we have to set up additional TURN server or anything load balancing on the US side ? I'm thinking of using Digital Ocean droplets to implement this. Any suggestion on the theoretical or prototype implementation would be greatly appreciated. Thank you in advance. 

Regards,

[Kirk Pathumanun]

12 hours later and no response on this ?  I believe this is a fundamental issue that need to be addressed for BigBlueButton since majority of real-time communication covers transcontinental meeting. I have found this article in relation my question below:

Add support for configuring stun/turn servers with authentication.

https://github.com/bigbluebutton/bigbluebutton/issues/2611

Need an official comment from the good people at BBB. Thanks in advance.

[Kirk Pathumanun]

Further research into GitHub,

It seems this is hard coded into this xml file so I assume deploying additional TURN server might not be possible ? 

https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-web/grails-app/conf/spring/turn-stun-servers.xml

On Thursday, August 3, 2017 at 11:30:31 PM UTC+7, Kirk Pathumanun wrote:

[Chad Pilkey]

I can't help with any routing or packet loss issues unfortunately. In regards to the the STUN and TURN configuration file though, it can be edited on your server to put in your own TURN servers. It's a server-wide configuration though.

[Fu Jiantao]

Kirk, could you try using MTR to check the link delay and loss, https://www.digitalocean.com/community/tutorials/how-to-use-traceroute-and-mtr-to-diagnose-network-issues

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-dev+unsubscribe@googlegroups.com.
To post to this group, send email to bigbluebutton-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/bigbluebutton-dev.
For more options, visit https://groups.google.com/d/optout.

[Fred Dixon]

Hi Kirk,

Having a direct connection using WebRTC audio is the best audio.  It's 48 Khz wide-band audio using OPUS codec over UDP (low latency).  If your users are using FireFox or Chrome, the should (barring a firewall) be able to use WebRTC.

If they are using WebRTC (they see the words '[ WebRTC Audio ]' in the lower right-hand corner), then the audio will be as good as it will be.

However, if there is a firewall, and  their browsers are falling back to Flash (which is TCP/IP based and tends to have higher latency), a TURN server may help users who are behind NAT to relay the UDP packets and use WebRTC.

Regards,.. Fred

--

BigBlueButton Developer

http://bigbluebutton.org/

@bigbluebutton

[Kirk Pathumanun]

Hello,

Thank you for the insight, I'm pretty sure 100% that all of the users in the US office used chrome for BB communication here in Bangkok and there was no firewall. Yes all users were using WebRTC for audio which we have no problem at all when communicating. It is just from the US and Thailand that we usually experienced loss of sound quality periodically, anyway I will attempt to add additional TURN server and if this works I'll be more than happy to share my result with the group. Thank you.

Regards,

 

On Sunday, August 6, 2017 at 1:29:27 AM UTC+7, Fred Dixon wrote:

Hi Kirk,

Having a direct connection using WebRTC audio is the best audio.  It's 48 Khz wide-band audio using OPUS codec over UDP (low latency).  If your users are using FireFox or Chrome, the should (barring a firewall) be able to use WebRTC.

If they are using WebRTC (they see the words '[ WebRTC Audio ]' in the lower right-hand corner), then the audio will be as good as it will be.

However, if there is a firewall, and  their browsers are falling back to Flash (which is TCP/IP based and tends to have higher latency), a TURN server may help users who are behind NAT to relay the UDP packets and use WebRTC.

Regards,.. Fred

On Fri, Aug 4, 2017 at 7:41 PM, Fu Jiantao <fuj...@gmail.com> wrote:

Kirk, could you try using MTR to check the link delay and loss, https://www.digitalocean.com/community/tutorials/how-to-use-traceroute-and-mtr-to-diagnose-network-issues

2017-08-04 9:55 GMT-07:00 Chad Pilkey <capi...@gmail.com>:

I can't help with any routing or packet loss issues unfortunately. In regards to the the STUN and TURN configuration file though, it can be edited on your server to put in your own TURN servers. It's a server-wide configuration though.

On Friday, August 4, 2017 at 1:09:25 AM UTC-4, Kirk Pathumanun wrote:

Further research into GitHub,

It seems this is hard coded into this xml file so I assume deploying additional TURN server might not be possible ? 

https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-web/grails-app/conf/spring/turn-stun-servers.xml

On Thursday, August 3, 2017 at 11:30:31 PM UTC+7, Kirk Pathumanun wrote:

Hello,

I have been using Bigbluebutton locally in Thailand on hardware servers with 1Gbp upload and download without any issue on the WebRTC quality for the past 3 years within the entire country, unless of course there were poor connections from clients which were anticipated technically. However as we scale out the system to the US between San Francisco and Texas, I have notices a high level of sound degradation from the US to Thailand side. As of now we have two Bigbluebutton servers set up in Thailand IDC, so I was wondering if  we have to set up additional TURN server or anything load balancing on the US side ? I'm thinking of using Digital Ocean droplets to implement this. Any suggestion on the theoretical or prototype implementation would be greatly appreciated. Thank you in advance. 

Regards,

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To post to this group, send email to bigblueb...@googlegroups.com.

Visit this group at https://groups.google.com/group/bigbluebutton-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To post to this group, send email to bigblueb...@googlegroups.com.

Visit this group at https://groups.google.com/group/bigbluebutton-dev.
For more options, visit https://groups.google.com/d/optout.

[Kirk Pathumanun]

Hello,

I've just MTR one of our BBB server in Bangkok IDC to google.com. Here is the result.

   My traceroute  [v0.86]
QUEST (::)                                                                                        Sun Aug  6 10:15:02 2017
Resolver error: Received reply from unknown source: 203.113.5.250   quit
                                                                                  Packets               Pings
 Host                                                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. vlan-113.cwt-idcgw-01.totisp.net                                             0.0%    50    0.3   2.0   0.3  38.3   6.8
 2. node-12vwzw6qbo0ovpxlhd.ipv6.totisp.net                                      0.0%    50    0.7   0.6   0.6   1.0   0.0
 3. 2001:df0:62:88bb:0:3:8040:1                                                  0.0%    50    1.0   0.9   0.8   1.5   0.0
 4. 2001:df0:62:c1::2                                                            4.0%    50   16.6  16.2  16.0  17.4   0.1
 5. 2001:4860:1:1:0:9498:0:d                                                     0.0%    50   27.8  27.9  27.2  46.8   2.8
 6. 2001:4860:0:116c::1                                                          0.0%    50   23.8  23.8  23.7  24.0   0.0
 7. 2001:4860:0:1::2791                                                          0.0%    50   23.0  23.1  23.0  23.2   0.0
 8. kul08s06-in-x0e.1e100.net                                                    0.0%    50   23.0  23.0  23.0  23.1   0.0

On Saturday, August 5, 2017 at 5:41:08 AM UTC+7, jeromy wrote:

Kirk, could you try using MTR to check the link delay and loss, https://www.digitalocean.com/community/tutorials/how-to-use-traceroute-and-mtr-to-diagnose-network-issues

2017-08-04 9:55 GMT-07:00 Chad Pilkey <capi...@gmail.com>:

I can't help with any routing or packet loss issues unfortunately. In regards to the the STUN and TURN configuration file though, it can be edited on your server to put in your own TURN servers. It's a server-wide configuration though.

On Friday, August 4, 2017 at 1:09:25 AM UTC-4, Kirk Pathumanun wrote:

Further research into GitHub,

It seems this is hard coded into this xml file so I assume deploying additional TURN server might not be possible ? 

https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-web/grails-app/conf/spring/turn-stun-servers.xml

On Thursday, August 3, 2017 at 11:30:31 PM UTC+7, Kirk Pathumanun wrote:

Hello,

I have been using Bigbluebutton locally in Thailand on hardware servers with 1Gbp upload and download without any issue on the WebRTC quality for the past 3 years within the entire country, unless of course there were poor connections from clients which were anticipated technically. However as we scale out the system to the US between San Francisco and Texas, I have notices a high level of sound degradation from the US to Thailand side. As of now we have two Bigbluebutton servers set up in Thailand IDC, so I was wondering if  we have to set up additional TURN server or anything load balancing on the US side ? I'm thinking of using Digital Ocean droplets to implement this. Any suggestion on the theoretical or prototype implementation would be greatly appreciated. Thank you in advance. 

Regards,

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To post to this group, send email to bigblueb...@googlegroups.com.

[Kirk Pathumanun]

Hello,

This is MTR from our server in Bangkok to the Digital Ocean Server in San Francisco.

QUEST (0.0.0.0)                                                                                   Sun Aug  6 10:20:34 2017
Resolver error: Received reply from unknown source: 203.113.7.249   quit

                                                                                  Packets               Pings
 Host                                                                           Loss%   Snt   Last   Avg  Best  Wrst StDev

 1. vlan-113.cwt-idcgw-01.totisp.net                                             0.0%    28    0.3   2.3   0.2  54.1  10.2
 2. ten-0-0-0-1.cwt-idcgw-02.totisp.net                                          0.0%    27    0.5   0.5   0.5   0.7   0.0
 3. in-addr.net                                                                  0.0%    27    1.6   1.6   1.4   2.8   0.2
 4. in-addr.net                                                                  0.0%    27   39.9  40.1  39.8  43.5   0.6
 5. gi-eth-0-1-0-10.hyi-core-01.net                                              0.0%    27   44.1  43.7  41.4  45.1   0.8
 6. xe-0-0-0-30.r00.sngpsi05.sg.bb.gin.ntt.net                                   7.4%    27   86.0  84.9  78.9  87.5   1.7
 7. ae-10.r20.sngpsi05.sg.bb.gin.ntt.net                                        18.5%    27   86.8  87.1  81.2 101.9   4.3
 8. ae-8.r22.snjsca04.us.bb.gin.ntt.net                                          0.0%    27  270.6 271.7 268.6 281.0   2.8
 9. ae-19.r01.snjsca04.us.bb.gin.ntt.net                                         0.0%    27  266.2 265.8 262.6 267.0   1.0
10. ae-4.r05.plalca01.us.bb.gin.ntt.net                                          0.0%    27  268.6 267.6 263.2 269.0   1.4
11. ae-1.digital-ocean.plalca01.us.bb.gin.ntt.net                                0.0%    27  265.6 264.7 260.3 273.7   2.3
12. ???
13. 192.241.200.49                                                               0.0%    27  261.7 264.2 259.1 266.1   1.9

[Kirk Pathumanun]

Hello,

Ironically I've just MTR from my server in Digital Ocean San Francisco to the hardware server in Bangkok IDC the packet loss is a lot less. Perhaps I might need to either a TURN Server to manage the packet from the US. Or try out a BigblueButton server that reside from the United States and test out the connectivity.

engwithme (0.0.0.0)                                                                                            Sun Aug  6 04:32:11 2017
Keys:  Help   Display mode   Restart statistics   Order of fields   quit

                                                                                               Packets               Pings
 Host                                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev

 1. 107.170.236.253                                                                           0.0%  2549    0.5   0.4   0.3  29.0   1.4
 2. 138.197.248.214                                                                           0.0%  2549    0.9   1.0   0.2  48.2   5.6
 3. ae58.bar2.SanFrancisco1.Level3.net                                                        0.0%  2549    0.6   0.6   0.4  50.0   3.7
 4. ae-2-13.edge1.SanJose2.Level3.net                                                         0.0%  2549    2.6   2.5   2.4  39.4   2.3
 5. ae9-10G.edge1.SanJose.Level3.net                                                          0.0%  2549    2.1   2.0   2.0  30.2   2.1
 6. xe2-1-6.singapore2.sin.seabone.net                                                        0.0%  2549  172.9 172.8 172.7 205.5   1.6
 7. tot.singapore2.sin.seabone.net                                                            1.7%  2549  215.2 219.3 208.9 237.4   1.7
 8. in-addr.net                                                                               2.2%  2549  232.5 231.9 222.2 253.3   1.9
 9. in-addr.net                                                                               2.0%  2549  259.3 257.4 248.3 279.1   1.5
10. in-addr.net                                                                               1.6%  2549  261.5 260.5 252.8 285.6   1.4
11. ???
12. ???
13. 1.179.247.244                                                                             1.7%  2548  266.3 265.4 257.9 289.4   1.2

[Fred Dixon]

Hi Kirk,

Do you have a user who can regularly repeat their sound loss.  Recommend testing with them using

  https://demo.bigbluebutton.org/

and see if you get sound loss as well.  This server is running BigBlueButton 1.1.

Regards,... Fred

To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-dev+unsubscribe@googlegroups.com.
To post to this group, send email to bigbluebutton-dev@googlegroups.com.

Visit this group at https://groups.google.com/group/bigbluebutton-dev.
For more options, visit https://groups.google.com/d/optout.

[Kirk Pathumanun]

Hello Fred,

I'm schedule to have a meeting with people in Palo Alto, CA Wednesday 9th August and People in Dallas, Texas on Friday 11th August using our production server in Bangkok  (BigBlueButton 1.1). The connection has always been setup with no firewall and all highspeed (30mb up and download) from clients to our server and we only carry out team meeting from 2 points communication between office in the US and Bangkok with 4 people maximum per meeting.  Please advised what kind of tools I need to record the packet loss on  https://demo.bigbluebutton.org/ testing ? Since I will repeat the sound loss via both our production and your demo server. At the moment I will prepare MTR, Bmon and tracert. would you recommend using the BBB click check as well or any others ?  I will keep you posted with the data.

Regards,

[Fred Dixon]

Hi Kirk,

> Please advised what kind of tools I need to record the packet loss on  https://demo.bigbluebutton.org/ testing ? 

Not sure.  I was thinking if you have a user who is experiencing poor audio on your server, you could try https://demo.bigbluebutton.org/ and see if the problem persists.  There will be one of two cases.

A. If the problem persists, it's likely the user's internet connection which is insufficient for WebRTC audio.

B. If the problem only occurs with your BigBlueButton server and not demo, then there may be configuration changes you can make to improve the response of your server.

Try the above and let us know the results.

Regards,.. Fred

To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-dev+unsubscribe@googlegroups.com.
To post to this group, send email to bigbluebutton-dev@googlegroups.com.

Visit this group at https://groups.google.com/group/bigbluebutton-dev.
For more options, visit https://groups.google.com/d/optout.

[vn...@yandex.com]

I also spend too much time on this problem!  I live in north Vietnam, and we had 2 cable cuts.  Both are fixed, but the routing tables are not stable.  Rainy nights and weekends I see pack loss bursting to 40% and latency of 700 ms.  You will need to change hosting providers on your Asian end.  My studies on this boil down to Singapore, and HongKong.  These have major international internet connectivity.  You want to get a provider that can provide guaranteed bandwidth.  Best effort is not very good.  The problem is everybody downloads FROM the USA youtube, etc.  The links are saturated.  Thailand also has mediocre international connectivity.  Most of that is through Singapore, anyway..    You can still run BBB local, but I think you should move freeswitch offshore.  

Leaseweb.com is one I am considering, Also OVH.com.  OVH owns its own fiber.  Hard to beat that, and they do sell guaranteed bandwidth.

Spend some time on this page   http://sgp.smokeping.ovh.net/smokeping?target=APAC.AS7552.  Check their connections to USA, Thailand, etc

On Thursday, August 3, 2017 at 11:30:31 PM UTC+7, Kirk Pathumanun wrote:

[Walter Tak]

Rent another vps in US westcoast and setup port forwarding or an stunnel connection, or a ssh tunnel.

When people would connect to the server in the US they'd be rerouted to the server in thailand.

Experiment which provider in the westcoast has the best STABLE connect.

Mtr is like a photo, you need a movie. So run a ping inside a screen, detach screen, let it run for 24 hrs, stop it and check packetloss, min, avg and max latency.

Even better setup SMOKEPING which can create graphs of latency and runs as a daemon.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Kirk Pathumanun]

Hello,

I will initiate test this week as per instructed by Fred on my own server in Thailand and from the demo site on Bigbluebutton. My goal is to record the loss in packet via MTR and bmon. However yest and Walter suggest I will reconfigure my vps server in either NYC or San Francisco as suggested by Walter and observe the results. If this works I'll be more than happy to share this with everyone on how-to. Since I believe BigBlueButton is a great product which could be improved on definitely.

Regards,

[Walter Tak]

You don't need to setup BBB in another server, just use another server as "access point" do US users would connect to the IP in US westcoast and then your server redirects all that traffic through a tunnel to your own main BBB server in Thailand.

Sort of a private route. But manual.

You'd have to check all ports you'd have to reroute of course and instruct the BBB client for US users to connect to the US hostname/ip.

On your "login" you'd let users choose to connect to the "US" server or to the "Thai" server but in the end it's the same server which means easier maintenance and of course all users can chat with each other.

You have to know a bit about tunnels and or port redirects and how to deal with disconnected tunnels etc.

[Kirk Pathumanun]

Hello,

We have found a robust way to avoid future loss on sound quality between the two continents, I was going to approach this problem with ssh tunneling and found out it would be rather cumbersome for our users to set proxy as well as we wanted the system to as seamless as possible on the backend. The culprit of the problem came from poor international connections from the IDC between Bangkok and NTT switches in Asia with average 12% packet loss tested with MTR for few days. So I don't really recommend anyone deploy BBB servers within Thailand. Our solutions came with deployment of Digital Ocean VPS in San Francisco (you want to use the 160USD per month and up really for production). Also we have changed the STUN and TURN config on https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-web/grails-app/conf/spring/turn-stun-servers.xml as well as default STUN on Freeswitch at bbb-voice-conference/config/freeswitch/conf/vars.xml to xirsys for global redudancy. At the end we ended up with BBB servers between 2 points with better quality WebRTC STUN. So I thought it would be good to share with your kind folks and once again thank you for the the tips.

Regards,

[vn...@yandex.com]

Great catch on Xirsys.  I may set up a free developer account.

Another benefit of your new setup is of course US users connect to a US server.  But each of your Thai users will use their own ISP's international bandwidth quotas.  Before, they all hit your Thai server, and all that bandwidth was funneled through the international gateway of your hosting provider.  You eliminated a single point of failure there.  Better still, Thai ISP's will get many complaints if Youtube breaks, so they are basically forced to guarantee OK connectivity to San Jose, etc.  People gotta have those cat videos!  Also, Amazon AWS, etc. 

Free bandwidth is another nice plus!  Bandwidth prices in Asia are a SCAM..

Did you go with a dedicated server or a possibly overloaded VPS server?  Noisy neighbors, etc.

How many concurrent users to you have?  How many are sharing video?

Great job, always good to see success stories!

[Kirk Pathumanun]

Hello vn...@yandex.com,

Thank you for the reply and to answer your question to benefit others in the future. I have 2 1U hardware servers set up in Bangkok for quite sometime now in Bangkok both has 1Gb up and download speed with unlimited bandwidth. Of course as you have stated unlimited bandwidth here in Thailand only guaranteed the quality of service within Thailand only which geographically speaking the size of Texas. As our business requirements grows to accommodate people in the United States the system failed due to poor connectivity between Thailand, Japan and United States. So by setting up BBB in the US, yes indeed we have eliminated the point of failure without interrupting the user experience.  To answer your question:

1 In America we used the VPS from Digital Ocean with the plan of 160USD/month, the system works flawlessly with people getting on no more than 20 connections with video sharing. You can just follow the instruction on BBB website on how to setup on VPS and it will work fine the first time around.

2 In Thailand we have bare metal 1U 2 servers with i7 Xeon and 16Gb ram each, we have tested out with about 80 people connecting to the system simultaneously within the country, but of course anywhere far as United States or Europe that's whole another trouble shooting story.

Anyway just letting the community knows you can definitely go production with BBB on Digital Ocean servers of 4CPU and over with WebRTC provided by Xirsys.

Regards,