Problem with net.inet.tcp.path_mtu_discovery=1

[Christos Chatzaras]

Hello,

I manage some servers hosting websites.

Using the APN "vpn-internet" (this apn provides public IP to my home connection) on a 5G router, I receive a public IP address from the range 149.210.xxx.xxx, but I notice that some websites do not load.

The same problem has been reported by other users who also receive an IP from the same range (149.210.xxx.xxx).

In contrast, when I connect using the APN "internet" (this apn provides CGNAT IP) and get an IP from the range 5.203.136.xxx, all websites load normally.

Eventually, I found that the issue is related to Path MTU Discovery.

When the net.inet.tcp.path_mtu_discovery setting is enabled on the operating system, the problem occurs. If I disable it, then the problem goes away. This setting had been enabled for years on my servers, so something must have changed recently with my ISP's apn "vpn-internet".

Also enabling net.inet.tcp.pmtud_blackhole_detection didn't help.

Does anyone know if PMTUD blackhole detection is broken in FreeBSD?

Kind regards,
Christos Chatzaras

[Dave Cottlehuber]

On Wed, 4 Jun 2025, at 16:36, Christos Chatzaras wrote:
> Hello,
>
> I manage some servers hosting websites.

What does tcpdump/wireshark show for traffic, particularly icmp? Wireshark is very helpful in explaining some issues.

What is the actual MTU on the working net vs the failing one?

Is there a local MTU where the failing websites start working again?

see ping(8) and use -v -D -s …. together to find a working MTU and cross check with tcpdump to find where things seem to break.

On a recent cloud environment I needed to add ‘ set reassemble yes no-df’ to my pf.conf to address MTU issues between VNET jails and the internet.

Happy hunting
Dave

[Christos Chatzaras]

First, I reverted the server settings to their defaults:

sysctl net.inet.tcp.path_mtu_discovery=1

sysctl net.inet.tcp.pmtud_blackhole_detection=0

Next, I set the MTU on my local computer to 1460 and everything worked as expected:

tcpdump: listening on en0, link-type EN10MB (Ethernet), snapshot length 524288 bytes

20:15:05.651375 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)

    192.168.2.18.65322 > 94.130.217.87.443: Flags [S], cksum 0x293e (correct), seq 3503095669, win 65535, options [mss 1420,nop,wscale 6,nop,nop,TS val 639376397 ecr 0,sackOK,eol], length 0

20:15:05.705913 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 60)

    94.130.217.87.443 > 192.168.2.18.65322: Flags [S.], cksum 0x9c22 (correct), seq 3647364942, ack 3503095670, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1782053626 ecr 639376397], length 0

However, when I set my local computer’s MTU back to 1500 (the default), the issue reappeared:

tcpdump: listening on en0, link-type EN10MB (Ethernet), snapshot length 524288 bytes

20:17:45.662993 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)

    192.168.2.18.65333 > 94.130.217.87.443: Flags [S], cksum 0x4a07 (correct), seq 3674289142, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 681359835 ecr 0,sackOK,eol], length 0

20:17:45.726988 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 60)

    94.130.217.87.443 > 192.168.2.18.65333: Flags [S.], cksum 0x9b1d (correct), seq 1443843488, ack 3674289143, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2890559459 ecr 681359835], length 0

So, with local computer MTU 1460, everything works, but with MTU 1500, the problem persists.

[Christos Chatzaras]

On 4 Jun 2025, at 22:06, Michael Tuexen <michael...@lurchi.franken.de> wrote:

The difference is that you announce a smaller MSS in SYN segment you
sent. This means that the peer can only send you smaller TCP segments.

So there seems to be a problem if the peer sends too large TCP segments.
That means that the peer must do PMTUD or TCP blackhole detection, not
the local node.

Best regards
Michael

Hello Michael,

sysctl net.inet.tcp.path_mtu_discovery=1

sysctl net.inet.tcp.pmtud_blackhole_detection=1

With these settings, is the connection supposed to work even if an intermediate router is dropping the ICMP messages required for Path MTU Discovery? I tried this configuration, but it didn’t resolve the issue.

[Michael Sierchio]

On Wed, Jun 4, 2025 at 3:58 PM Christos Chatzaras <ch...@cretaforce.gr> wrote:
>
> With these settings, is the connection supposed to work even if an intermediate router is dropping the ICMP messages required for Path MTU Discovery? I tried this configuration, but it didn’t resolve the issue.

No. Path MTU Discovery breaks if you drop the ICMP "fragmentation
needed and DF set" packet is sent.

[Michael Tuexen]

> On 4. Jun 2025, at 19:29, Christos Chatzaras <ch...@cretaforce.gr> wrote:
>
>
>