Broken IPv6 / IPv4 configuration, or Gmail brokenness?

[Celejar]

Hello,

My Debian Sid system, including its networking system, has been
working fine for a while. Recently, Gmail has not been working properly
on the system: sending (via SMTP with SSL) times out, and receiving
(via POP3 and IMAP) takes abnormally long. Today I finally made some
efforts to understand what's going on. Here's what I know /
understand / have been able to establish:

1) My ISP provides me with only IPv4, not IPv6.

2) Trying to send email through Gmail using "smtp.gmail.com", via
Sylpheed and Swaks, times out without getting anywhere. For example:

$ swaks -t cel...@gmail.com -s smtp.gmail.com:587 -tls -a LOGIN
Username: celejar
Password: ********
=== Trying smtp.gmail.com:587...
*** Error connecting to smtp.gmail.com:587:
*** IO::Socket::INET6: connect: timeout

3) Forcing the use of IPv4 fixes the timeout problem. In particular,
both of the following work (in the second command, the IPv4 address is
the one returned by "dig" and "host" for "smtp.gmail.com"):

$ swaks -t cel...@gmail.com -4 -s smtp.gmail.com:587 -tls -a LOGIN
$ swaks -t cel...@gmail.com -s 142.251.163.109:587 -tls -a LOGIN

So, what's going on here? I suppose I don't really need IPv6 on my
system, but everything I've read says that it should be left enabled and
not disabled. Is something misconfigured on my system, is this a Gmail
problem, or am I missing something else?

--
Celejar

[Dan Ritter]

Your system expects to be able to use IPv4 and IPv6. Google
handles both. Your ISP does not. DNS returns both IPv4 A records
and IPv6 AAAA records for smtp.gmail.com.

In this particular case, you should change your system
preference to IPv4 first.

In /etc/gai.conf:

UNcomment

precedence ::ffff:0:0/96 100

and that should solve the issue immediately, at the cost of
making IPv6 unusable until you re-comment that.

-dsr-

[Celejar]

Thank you very much - this seems to have worked!

I understand now that the reason I only had the problem with Gmail and
not with the other major commercial email provider I use, GMX, is that
"smtp.gmx.com" has only an IPv4 address record and no IPv6 address
record. And I suppose the reason I wasn't seeing this problem with web
browsing is that web browsers (on their default settings?) try both
IPv4 and IPv6 simultaneously, and use whatever responses they get
without waiting.

Shouldn't this be included somewhere prominently in the Debian
documentation, in the form of a Big Fat Warning that the standard
dual-stack condiguration used by Debian can cause serious breakage if
one's ISP doesn't support IPv6? (I'd be happy to add a warning to, say,
https://wiki.debian.org/DebianIPv6, if I thought I understood the
issues well enough to get it right.)

--
Celejar

[Tim Woodall]

On Sun, 22 Jan 2023, Celejar wrote:

> On Sun, 22 Jan 2023 16:28:55 -0500
> Dan Ritter <d...@randomstring.org> wrote:
>
>> Celejar wrote:
>>> Hello,
>>>
>>> My Debian Sid system, including its networking system, has been
>>> working fine for a while. Recently, Gmail has not been working properly
>>> on the system: sending (via SMTP with SSL) times out, and receiving
>>> (via POP3 and IMAP) takes abnormally long. Today I finally made some
>>> efforts to understand what's going on. Here's what I know /
>>> understand / have been able to establish:
>>>
>>> 1) My ISP provides me with only IPv4, not IPv6.
>>>
>>> 2) Trying to send email through Gmail using "smtp.gmail.com", via
>>> Sylpheed and Swaks, times out without getting anywhere. For example:
>>>
>>> $ swaks -t cel...@gmail.com -s smtp.gmail.com:587 -tls -a LOGIN
>>> Username: celejar
>>> Password: ********
>>> === Trying smtp.gmail.com:587...
>>> *** Error connecting to smtp.gmail.com:587:
>>> *** IO::Socket::INET6: connect: timeout
>>>

>> Your system expects to be able to use IPv4 and IPv6. Google
>> handles both. Your ISP does not. DNS returns both IPv4 A records
>> and IPv6 AAAA records for smtp.gmail.com.
>>
>> In this particular case, you should change your system
>> preference to IPv4 first.
>>
>

> Shouldn't this be included somewhere prominently in the Debian
> documentation, in the form of a Big Fat Warning that the standard
> dual-stack condiguration used by Debian can cause serious breakage if
> one's ISP doesn't support IPv6? (I'd be happy to add a warning to, say,
> https://wiki.debian.org/DebianIPv6, if I thought I understood the
> issues well enough to get it right.)
>

I think, although I'm not 100% sure as I now have fully working IPv6
everywhere, that your problem might be that something is providing a
default route.

If the box trying to connect to gmail knows there's no route then it
will use IPv4. If it thinks there's a route it will use it.

At a guess, your router is sending out RA messages that say the router
is a default route and then dropping packets when you actually try to
send.

radvdump will let you see who is broadcasting advertisements.

For completeness, in case you're using radvd, the magic setting to
disable the default route is
AdvDefaultLifetime 0;

Easy when you know but I remember many frustrations trying to work out
how to disable the default route while still providing specific routes.

[Celejar]

Thank you - the plot thickens. I tried radvdump, and I was indeed
receiving IPv6 advertisements. I inspected my router (an OpenWrt box)
more carefully, and lo and behold, the router *thinks* that it has IPv6
connectivity: it reports that it has configured its WAN interface via
DHCPv6, that it has an IPv6 prefix delegation (a /56), and that it has
accordingly handed out an IPv6 address via DHCPv6 to my Debian system
(which indeed shows having its network interface configured with that
address). The problem is that IPv6 is not actually working at all:
ping6 to IPv6 hosts, even from the router itself, get no response, and
traceroute to IPv6 hosts show replies from only the first two hops, and
just asterisks afterward.

My ISP is notoriously cagey about the details of its (slow and
incremental) IPv6 rollout. I suppose I can try customer service, but in
the meantime, I just don't know whether I have a misconfiguration on the
router, or whether the ISP is not actually providing working IPv6.

--
Celejar

[Tim Woodall]

On Mon, 23 Jan 2023, Celejar wrote:

> Thank you - the plot thickens. I tried radvdump, and I was indeed
> receiving IPv6 advertisements. I inspected my router (an OpenWrt box)
> more carefully, and lo and behold, the router *thinks* that it has IPv6
> connectivity: it reports that it has configured its WAN interface via
> DHCPv6, that it has an IPv6 prefix delegation (a /56), and that it has
> accordingly handed out an IPv6 address via DHCPv6 to my Debian system
> (which indeed shows having its network interface configured with that
> address). The problem is that IPv6 is not actually working at all:
> ping6 to IPv6 hosts, even from the router itself, get no response, and
> traceroute to IPv6 hosts show replies from only the first two hops, and
> just asterisks afterward.
>
> My ISP is notoriously cagey about the details of its (slow and
> incremental) IPv6 rollout. I suppose I can try customer service, but in
> the meantime, I just don't know whether I have a misconfiguration on the
> router, or whether the ISP is not actually providing working IPv6.
>

Does ping/traceroute work in ipv4?

It's depressingly common to find people who block icmp for 'security
reasons'. While you can mostly get away with that in ipv4, ipv6 breaks
catastrophically without at least some working icmp.

Also try tcptraceroute - although if some router is configured to drop
all icmp then I don't think ipv6 can work at all

[to...@tuxteam.de]

On Tue, Jan 24, 2023 at 06:41:56PM +0000, Tim Woodall wrote:
> On Mon, 23 Jan 2023, Celejar wrote:
>

> > Thank you - the plot thickens [...]

> It's depressingly common to find people who block icmp for 'security
> reasons'. While you can mostly get away with that in ipv4, ipv6 breaks
> catastrophically without at least some working icmp.

IPv4, too, does funny things when dropping ICMP. Path MTU discovery,
breaks, for one.

Cheers
--
t

[Celejar]

On Tue, 24 Jan 2023 18:41:56 +0000 (GMT)
Tim Woodall <debia...@woodall.me.uk> wrote:

> On Mon, 23 Jan 2023, Celejar wrote:
>
> > Thank you - the plot thickens. I tried radvdump, and I was indeed
> > receiving IPv6 advertisements. I inspected my router (an OpenWrt box)
> > more carefully, and lo and behold, the router *thinks* that it has IPv6
> > connectivity: it reports that it has configured its WAN interface via
> > DHCPv6, that it has an IPv6 prefix delegation (a /56), and that it has
> > accordingly handed out an IPv6 address via DHCPv6 to my Debian system
> > (which indeed shows having its network interface configured with that
> > address). The problem is that IPv6 is not actually working at all:
> > ping6 to IPv6 hosts, even from the router itself, get no response, and
> > traceroute to IPv6 hosts show replies from only the first two hops, and
> > just asterisks afterward.
> >
> > My ISP is notoriously cagey about the details of its (slow and
> > incremental) IPv6 rollout. I suppose I can try customer service, but in
> > the meantime, I just don't know whether I have a misconfiguration on the
> > router, or whether the ISP is not actually providing working IPv6.
> >
>
> Does ping/traceroute work in ipv4?

Yes - pinging the same hosts that don't work using IPv6 works with ping
-4, and traceroute works with IPv4 addresses as well (I get some
asterisks, but many / most of the hops, including the destinations,
respond).

> It's depressingly common to find people who block icmp for 'security
> reasons'. While you can mostly get away with that in ipv4, ipv6 breaks
> catastrophically without at least some working icmp.
>
> Also try tcptraceroute - although if some router is configured to drop
> all icmp then I don't think ipv6 can work at all

tcptraceroute work fine. tcptraceroute6 gives the same problem as
regular traceroute6 - all asterisks after the first couple of hops.

--
Celejar

[Tim Woodall]

Indeed. Hence the --clamp-mss-to-pmtu magic in iptables.

[to...@tuxteam.de]

On Tue, Jan 24, 2023 at 09:24:25PM +0000, Tim Woodall wrote:
> On Tue, 24 Jan 2023, to...@tuxteam.de wrote:

[...]

> > IPv4, too, does funny things when dropping ICMP. Path MTU discovery,
> > breaks, for one.
> >
> Indeed. Hence the --clamp-mss-to-pmtu magic in iptables.

More magic :)

Cheers & thanks for the hint
--
t

[Andy Smith]

Hi,

On Sun, Jan 22, 2023 at 11:42:18PM -0500, Celejar wrote:
> Shouldn't this be included somewhere prominently in the Debian
> documentation, in the form of a Big Fat Warning that the standard
> dual-stack condiguration used by Debian can cause serious breakage if
> one's ISP doesn't support IPv6?

Not having IPv6 is not the problem. As you discovered later in this
thread, your ISP is providing just enough IPv6 for your router to
think it has a v6 block to hand out, but not enough that it actually
works. It's a misconfiguration by your ISP and not something that
any operating system's documentation needs to point out. They could
also let you connect to them but make IPv4 not work, it's just that
you'd notice that very quickly!

If you don't have a global scope IPv6 address on any interface then
at the point where your system is trying to decide what source
address to use for some packets, it will not that any IPv6 source
address to use. If there are IPv4 addresses to talk to then it will
source from one of your IPv4 addresses, or else give an error.

The change suggested here to gai.conf says to try IPv4 first, which
is why it fixes things for you.

I'm not sure but I think you may still have a bad time in the event
that a DNS response included ONLY IPv6 addresses, because the need
for a v6 source address would bypass gai.conf settings and then
instead of getting an instant failure you'd again get the long
timeout while your non-working IPv6 was used. But that is obviously
an edge case here.

Anyway, if the ISP can't fix the IPv6 and can't be convinced to stop
advertising it even when it doesn't work, I think you'd be best
off trying to disable radvd on the router. Failing that, disabling
IPv6 on all your clients (check your phones too, as they will try
IPv6 these days if they think it's available).

Cheers,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting

[Celejar]

On Sat, 28 Jan 2023 20:55:18 +0000
Andy Smith <an...@strugglers.net> wrote:

> Hi,
>
> On Sun, Jan 22, 2023 at 11:42:18PM -0500, Celejar wrote:
> > Shouldn't this be included somewhere prominently in the Debian
> > documentation, in the form of a Big Fat Warning that the standard
> > dual-stack condiguration used by Debian can cause serious breakage if
> > one's ISP doesn't support IPv6?
>
> Not having IPv6 is not the problem. As you discovered later in this
> thread, your ISP is providing just enough IPv6 for your router to
> think it has a v6 block to hand out, but not enough that it actually
> works. It's a misconfiguration by your ISP and not something that
> any operating system's documentation needs to point out. They could
> also let you connect to them but make IPv4 not work, it's just that
> you'd notice that very quickly!

Understood. There's apparently even a quasi-official name for this:
"IPv6 brokenness":

https://en.wikipedia.org/wiki/IPv6_brokenness_and_DNS_whitelisting

> In the field of IPv6 deployment, IPv6 brokenness was bad behavior seen
> in early tunneled or dual stack IPv6 deployments where unreliable or
> bogus IPv6 connectivity is chosen in preference to working IPv4
> connectivity. This often resulted in long delays in web page loading,
> where the user had to wait for each attempted IPv6 connection to time
> out before the IPv4 connection was tried. These timeouts ranged from
> being near-instantaneous in the best cases, to taking anywhere between
> four seconds to three minutes.

The article proceeds to assure the reader that:

> IPv6 brokenness is now generally regarded as a solved problem for
> almost all practical purposes, following improvements at both the
> transport and application layers.

But I guess I've managed to encounter a "practical" instance of
current IPv6 brokenness :|

...

> Anyway, if the ISP can't fix the IPv6 and can't be convinced to stop
> advertising it even when it doesn't work, I think you'd be best

Makes sense - I think I've seen other users of my ISP stating in online
forums that that's what they had to do at some point.

> off trying to disable radvd on the router. Failing that, disabling
> IPv6 on all your clients (check your phones too, as they will try

Thank you.

--
Celejar