IP address selection with FreeSWITCH and bbb-webrtc-sfu

[Brent W. Baccala]

Hi -

I'm still working towards a pull request to handle private addresses better.

Currently, we've got a long standing bug in the FreeSWITCH address selection logic that always prefers an IP address on the local subnet over everything else.  This means that if the server's private IP address space overlaps with the client's private IP address space, they won't be able to communicate.  A client and a server both on 192.168.1.0/24 subnets (for example), but not the same subnet, rather the same numbers on two different private networks connected over the public network, won't work.  They'll try to use their private addresses instead of their public addresses.

If the client and the server are actually on the same private network, then they can communicate either directly if they're on the same subnet, due to that same logic.  If they're on different subnets of the same private network, they can communicate if their NAT device supports hairpin NAT, i.e, the NAT device relays the packets back and forth between two devices, both on its private network.

Also, if the user hasn't given permission to use the microphone, then it's likely that private IPv4 addresses will be suppressed in favor of mDNS ".local" domain names.  This works best if the client, the server, and the mDNS server are all on the same subnet, since mDNS is a link local protocol.  I've found a mdns-repeater that relays mDNS traffic between subnets, but this requires configuration, so mDNS is not totally reliable on more complicated private networks.

IPv6 clients have other issues.  Web browsers are filtering out auto-configured IPv6 addresses (Stateless Address Auto-configuration, SLAAC) that expose MAC addresses embedded in the IPv6 addresses.  This is the old way of doing SLAAC (RFC 4291).  The new way (RFC 7217) obfuscates the MAC address, but the old way is still the default on Linux unless you specify "ipv6-address-generation: stable-privacy" in your netplan configuration, and I think you also need "renderer: NetworkManager" for that to work.

Based on all that, I'm proposing that we vary our address selection logic based on the source address of the client.  Specifically,

• connections from private IPv4 addresses get the server's private IPv4 address and fall back on public IPv4 addresses (hoping for hairpin NAT)
• connections from public IPv4 addresses get the server's public IPv4 address and fall back on private IPv4 addresses (hoping for a miracle)
• connections from IPv6 addresses get the server's IPv6 address and fall back on public IPv4 addresses (there's no way to know if an IPv6 client shares a private IPv4 network with the server)

Maybe the IPv4 addresses should also fall back on IPv6 addresses.

I've gotten FreeSWITCH working by having it listen on different ports for different types of addresses (5066 for public addresses; 5068 for private addresses; 5070 for IPv6 addresses, and another set of ports 7443/7445/7447 for wss instead of ws), then having nginx route to these different ports depending on where the connection came in from.

I've figured out enough of bbb-webrtc-sfu to understand that we could either do it that way (have bbb-webrtc-sfu listen on different ports), or add an extra HTTP header (probably X-Forwarded-For) and let bbb-webrtc-sfu announce different IP addresses depending on what kind of address the connection came in on.

An extra HTTP header would probably be more in the spirit of how bbb-webrtc-sfu is written, since it already uses extra headers to get authentication data.

Any comments?

    agape

    brent

[Paulo Lanzarin]

I've figured out enough of bbb-webrtc-sfu to understand that we could either do it that way (have bbb-webrtc-sfu listen on different ports), or add an extra HTTP header (probably X-Forwarded-For) and let bbb-webrtc-sfu announce different IP addresses depending on what kind of address the connection came in on.

An extra HTTP header would probably be more in the spirit of how bbb-webrtc-sfu is written, since it already uses extra headers to get authentication data.

I'm not sure I understand the need to add such logic in bbb-webrtc-sfu since - with mediasoup at least - you can advertise multiple server-side IPs simultaneously.

Going from that I don't see how it would have a problem in the scenario you're posing - remote candidates (from bbb-webrtc-sfu) would be probed

in priority order until a connectivity check locks down, which should happen as long as you're configuring the necessary IP addresses.

What am I missing?

PS.: Just as a curiosity: there is already a mechanism in bbb-webrtc-sfu that is similar in spirit triggered by the `ipMappings` configuration key.
It mangles and/or duplicates local (as in the server's) candidates based on the remote (as in the client's) origin IP,

but it's solely for plain RTP endpoints and based on SDP o=/c= header IPs. Hasn't been used in BBB as far as I know as

I only implemented it to address a very specific intra+internet deployment that needed to interop with legacy RTP bridges.

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/506706ac-1c1a-47cc-bac0-2d9bf3b634f9n%40googlegroups.com.

[Brent W. Baccala]

On Wed, May 18, 2022 at 10:13 PM Paulo Lanzarin <prlan...@mconf.com> wrote:

I've figured out enough of bbb-webrtc-sfu to understand that we could either do it that way (have bbb-webrtc-sfu listen on different ports), or add an extra HTTP header (probably X-Forwarded-For) and let bbb-webrtc-sfu announce different IP addresses depending on what kind of address the connection came in on.

An extra HTTP header would probably be more in the spirit of how bbb-webrtc-sfu is written, since it already uses extra headers to get authentication data.

I'm not sure I understand the need to add such logic in bbb-webrtc-sfu since - with mediasoup at least - you can advertise multiple server-side IPs simultaneously.

Going from that I don't see how it would have a problem in the scenario you're posing - remote candidates (from bbb-webrtc-sfu) would be probed

in priority order until a connectivity check locks down, which should happen as long as you're configuring the necessary IP addresses.

What am I missing?

Maybe I'm missing something!

I'm thinking that bbb-webrtc-sfu works like FreeSWITCH, by picking one candidate from a prioritized list.  Also, I only see one "announcedIp" listed in my production.yml file.  Are you saying that it announces multiple addresses?  How are they configured?

I've figured out that FreeSWITCH picks the client's IP address according to the "apply-candidate-acl" in its SIP profile, and I figure that the SIP code in our client is responsible for picking a server IP address (how I don't know), and the mediasoup client (also in our client) must also have some logic for picking the server IP address, but I don't know how that works.

Could you explain how it works, or point me to some documentation?

Thanks.

    agape

    brent

[Paulo Lanzarin]

Brent,

> I'm thinking that bbb-webrtc-sfu works like FreeSWITCH, by picking one candidate from a prioritized list. 
> Also, I only see one "announcedIp" listed in my production.yml file. 
> Are you saying that it announces multiple addresses?  How are they configured?

Yeah, the mediasoup.webrtc.listenIps configuration key is an array so you can specify as many IPs as you want.

Each array is an object of the following format:
{ ip: <address used to bind UDP/TCP ports>, announcedIp: <advertised IPs you'll see on mediasoup candidates> }

For example: in a BigBlueButton server with a public IPv4 address `$pubIPv4` and a private IPv4 `$privIPv4`, the configuration responsible for specifying the external addresses in mediasoup should be of the following format (YAML syntax, `/etc/bigbluebutton/bbb-webrtc-sfu/production.yml`):

```yaml

mediasoup:

webrtc:

listenIps:

- ip: 0.0.0.0

announcedIp: $pubIPv4        - ip: 0.0.0.0          announcedIp: $privIPv4        - ip: <IPv6s if you want to set those>

```

I've started adding this to docs, see https://github.com/bigbluebutton/bigbluebutton.github.io/pull/391
(Updating mediasoup section). Feel free to suggest improvements on areas that are not clear enough.

> and the mediasoup client (also in our client) must also have some logic for picking the server IP address, but I don't know how that works.

> Could you explain how it works, or point me to some documentation?

As far as mediasoup generating candidates is concerned, it's pretty straightforward: for every configured entry in listenIps,

it binds a different UDP/TCP socket to `ip`. If `announcedIp` is set, it will overwrite the generated candidate IP for that entry
with the value in announcedIP. That's only for signaling/advertisement purposes.

As far as establishing a candidate pair between client-and-server is concerned, RFC 8445 is what you are looking for, tbh.
It's better than any second-hand explanation I could give. Anyways: mediasoup is an ICE-lite implementation, which basically
means it doesn't really generate connectivity checks but only responds to them. What I think should happen is that by
advertising multiple IPs, candidate pair nomination would be eventually achieved through connectivity checks stemming
from the client (browser) - following the procedures in RFC 8445.

That's all theoretically, though, which is why I would appreciate if you could validate that scenario by configuring

multiple IPs as mentioned above and letting us know whether it works (I'm a bit short on time to set a lab scenario to

try this out). The reason why I'm littering this reply with shoulds and maybes is two-fold: 1) I have not tested that scenario

yet 2) having multiple IPv4 candidates (same IP family, but mainly IPv4) is a bit of a gray area for ICE-lite. It isn't a MUST NOT,

but it is discouraged - and it's not really explained in detail why.

So, please, do test the above and let us know if it works.