Screensharing/Webcam issue 1020 in Chrome behind Firewall. Works in Firefox at the same location on same client. Maybe problem in the code, also on test.bigbluebutton.org?

[Martin]

Since our upgrade from 2.2.29 to 2.2.31 we have problems in using the Screensharing/webcam behind our firewall with Chrome which we didn't have before. Before it worked in Firefox and Chrome with Audio and Screensharing/Webcam.

Our own Stun and Turn server is configured and working, Audio is working in Firefox and Chrome (via TURN). Screensharing/Webcam only working in Firefox.

Saw the change while updating in /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini and configured the file. ExternalIPv4 is set and my own stun server is set, connections are coming in to Stun. Seems to be fine.

As a test: on current test.bigbluebutton.org it is even worse. Audio 1007 and Webcam 1020 in Chrome and no Screensharing. Both working in Firefox and also Screensharing. Everything on the same client with the same network connection, only different browser.

What is the difference between Firefox and Chrome since the update? Should I use TURN instead of STUN for /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini because Chrome gets correct Audio via TURN?

Test from outside our firewall shows everything works like expected. So for me our configuration seems to be right because Firefox is working in both environments. Only Chrome doesn't. And Chrome doesn't work on test.bigbluebutton.org, so it seems to be a generic problem.

Maybe someone has a hint?

[Martin]

Noone using it behind a firewall to test with me? As said I think it is not a problem with our configuration since the behaviour is similar on test.bigbluebutton.org...

[Paulo Lanzarin]

Hey,

Couple of suggestions/questions:

  - Removing externalIPv4 from Kurento, leaving only a working STUN server set up there. Be sure the STUN server is working from inside the BBB instance (ie check the stun-client package, available on xenial via apt). Rationale: try to see if your firewall needs some sort of hole punching.

  - Did your firewall change recently?

  - Do you have two separate BBB instances (2.2.29 vs 2.2.31) with the exact same general network setup so that you can, in fact, compare to see if that's a regression? I highly doubt it is because changes around that area were minimal, if so.

  - About using TURN in Kurento: no, that should never be needed.

  - Can you describe the network topology a bit better? Firewall: does it have UDP ports open? Is it a 1:1 NAT? Does it have outbound NAT? Does it require hole punching from either of the ends (BBB, clients)?

  - Which STUN server are you setting up in bbb-web? Is it your own coturn instance? 3rd party?
  - "Noone using it behind a firewall to test with me": I do have a setup behind a firewall. I see none of what you are mentioning. Mainly because firewall and network topologies vary so wildly it is pretty hard to make such general comparisons.

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-Setup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/3af65db7-e233-40af-9f56-ec87cedfebe3n%40googlegroups.com.

[Martin]

Hi,

thanks for your reply! Fun fact (or to cry with): without changing anything I can use the test instance AND our own instance today with Chrome and camera/screensharing... Didn't record the chrome version since my problem :/

But to see if we can optimize it further I will answer your questions/suggestions:

 - Removing externalIPv4 from Kurento, leaving only a working STUN server set up there. Be sure the STUN server is working from inside the BBB instance (ie check the stun-client package, available on xenial via apt). Rationale: try to see if your firewall needs some sort of hole punching. 

-> STUN server is working, test from my browser with Trickle ICE works and from inside my BBB. But from inside only with "stunclient --mode full --localport 30000 turn.xyz.chat 80" and not with "stun turn.kolping.chat -p 80".

Trickle ICE:

Is answering with  rtp | srflx like described in the test.

First command output:

Binding test: success

Local address: 87.106.xxx.xxx:30000

Mapped address: 87.106.xxx.xxx:30000

Behavior test: success

Nat behavior: Direct Mapping

Filtering test: success

Nat filtering: Endpoint Independent Filtering

Second command output:

STUN client version 0.97

Primary: Blocked or could not reach STUN server

Return value is 0x00001c

Tested other STUN Servers where the second command is also successful. Don't know why my own COTURN server isn't working with this client. Maybe there is a hidden problem?

Other thing to your suggestion: why should the externalIPv4 be removed, I thought it is the best way to communicate directly to the server and STUN should only be used behind the firewall?

 - Did your firewall change recently?  

-> No changes to it before and since then except the automatic pattern updates which are delivered from Sophos

- Do you have two separate BBB instances (2.2.29 vs 2.2.31) with the exact same general network setup so that you can, in fact, compare to see if that's a regression? I highly doubt it is because changes around that area were minimal, if so. 

-> no, sadly not. Only have two vServer for BBB and Turn.

- About using TURN in Kurento: no, that should never be needed.

-> OK, great so I dismiss that thing.

- Can you describe the network topology a bit better? Firewall: does it have UDP ports open? Is it a 1:1 NAT? Does it have outbound NAT? Does it require hole punching from either of the ends (BBB, clients)?  

-> My BBB/turn servers are NOT inhouse, they are hosted, so for them I use ufw with the documented ports (no changes recently). Inouse firewall (behind my client is sitting in office): All ports are closed in default. Only opened for the used applications and default ports like 80/443 tcp. Changed STUN Port to 80 UDP because this one is open, too. Because of the closed UDP Ports it needs the TURN Server which is working like expected on TCP 443. NAT is used, normal setup like every home router. Only one public IP.

- Which STUN server are you setting up in bbb-web? Is it your own coturn instance? 3rd party?

-> It is our own coturn instance on a vServer. Setup with BBB script.

- "Noone using it behind a firewall to test with me": I do have a setup behind a firewall. I see none of what you are mentioning. Mainly because firewall and network topologies vary so wildly it is pretty hard to make such general comparisons.  

-> Yeah, you are very right. That is said too short/simple but I just thought there should be some other persons with the problem or at least with a client behind a firewall which could help me or even test my server with a test from my side. And it even helps me if you say you are using clients behind a firewall and don't have this issue.

Thanks for your answer and maybe we find out what the problem was (because at the moment it is not there) :( and what we could optimize along the way.

[Paulo Lanzarin]

> without changing anything I can use the test instance AND our own instance today with Chrome and camera/screensharing

I'd treat this as an occasional incident for the time being, then, and keep eyes open about it. This almost asserts that it's not a regression of any sorts.
If it does happen again, the most important thing is to get a packet capture client side (and server side if possible) for further inspection.
More likely causes are, then, undocumented changes to your network. For instance: botched fw pattern updates, unnoticed changes in MTU size (which might
cripple DTLS cert exchange), unnoticed changes in UDP fragmentation policies (also cripple DTLS), faulty hop, ... list goes on. Packet capture always makes
that kind of stuff clearer.

Keep in mind that DTLS issues are far more likely, for instance, since it only happened with Kurento-based features (video, screenshare)

in one of the testing ends. Depending on the setup, audio and the rest might use different certificates for the DTLS stuff, of different sizes, hence generating different
fragmentation behaviours.

You may ask why it failed in test.bigbluebutton.org for everything (audio + video + ss), then? Test server uses the same cert all across the board deliberately.

> Other thing to your suggestion: why should the externalIPv4 be removed, I thought it is the best way to communicate directly to the server and STUN should only be used behind the firewall?

Just a testing measure. It is the recommended way, but server-side STUN sometimes helps uncovering server-side hole punching conundrums. You should keep it set :).

s,

prlanzarin

--

You received this message because you are subscribed to the Google Groups "BigBlueButton-Setup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-s...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/c17c9a03-3031-4c59-896a-f30ecb28de18n%40googlegroups.com.

[Martin]

Ok, was just a small hope. Problem is back. Firefox is working like expected behind our firewall, Chrome doesn't.

We did two changes since then. As we had problems on IOS devices I updated our coturn from Ubuntu 18 to 20 and fixed the IOS devices with adding port 443 to the STUN server line. Since then these devices don't get 1020 anymore. Better but didn't fix Chrome issue.

On test.bigbluebutton.org it is also again like before, no audio with 1007 and no media with 1020. On my plattform audio works within Chrome.

If I take the captures from both sides, could you analyze it? I can take them and I can use my basic wireshark knowledge but I think that doesn't help me for solving this problem...

[Martin]

Before leaving here behind the firewall I did a tcpdump/wireshark from my client and both servers while opening sessions.

What I did:

- started session from both browsers, first Chrome, then Firefox

- switched on Audio (worked on both) with echo test

- tried to switch on and off camera (only working with Firefox)

- shared screen and switched it off again (only working with Firefox)

- closing conference

Captures from alle three sides are here: https://kolping.cloud/s/eRb2s4ST8N3gT7f

Maybe someone can "see" the problem.

[Paulo Lanzarin]

I'll look into the pcaps.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/d9f0c7f5-2cd9-4ffa-a385-09fcfb73dd5en%40googlegroups.com.

[Martin]

Could you "see" something in it? We have another BBB instance in our company for a special group of users. These one is hosted at a commercial provider and there we also have problems with chrome behind the firewall. Funny one: there the audio is not running where I don't have problems in any constellation but video works... In Firefox both works like expected there, too.

[Paulo Lanzarin]

Oops, sorry Martin. Forgot about this.

Did a very quick scan (2 min, so I might be wrong) over the Chrome pcaps.

Remember what I said about DTLS fragmentation?

>or instance: botched fw pattern updates, unnoticed changes in MTU size (which might
cripple DTLS cert exchange), unnoticed changes in UDP fragmentation policies (also cripple DTLS),

It looks like to be it. Some cert exchanges seem to be borked due to fragmentation.

I'll look into the pcaps more carefully later today. In the meanwhile you can try re-generating

a new dtls-srtp cert to be used in FS and Kurento that is smaller in size and see if that fixes your problem.

Alternatively, if coturn and BBB are in the same network, please force the MTU size between them to be

jumbo frames as a testing measure.

s,

prlanzarin

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/5aca7103-4e2f-4ddc-a0a6-f02caa82e64fn%40googlegroups.com.