Greater Anglia on-train Wifi and fragmented IP packets

Ian Jackson

unread,

Apr 9, 2014, 7:28:00 PM4/9/14

to

My VPN client didn't work properly when I was on a train. I suspected
a problem with IP fragmentation and today I had time to investigate
properly.

I connected to the wifi with networkmanager on Debian wheezy, and used
a vanilla iceweasel (Debian's firefox) browser to placate the captive
portal. After that I could ping my colo.

But, large pings would vanish. I did
ping -i3 -s2000 212.13.197.229
from my netbook. 212.13.197.229 is my colo. I ran tcpdump on both
ends, and obtained the packet traces you see below.

As you can see, my laptop is fragmenting the 2000-byte packet into two
pieces: 1348 and 700 bytes (the increase is due to the fragment
needing its own header). (The GA wifi has an MTU of 1350 - more about
that in a moment.)

Something in the GA wifi system is hopelessly mangling the fragmented
packets. The initial fragment has had the More Fragments flag
cleared, and the second fragment has had the fragment offset cleared.
This is bizarre and results in the packets being completely corrupt.
As a result, the ICMP checksum no longer matches. (Both packets have
had the Don't Fragment flag set, too.)

"Normal people" don't notice this problem because ordinary web
browsing will mostly work to most sites. This is because GA have set
the local wifi MTU to the unusually low value of 1350. As a result,
the passenger's device's TCP implementation will know to send smaller
packets, and offer a smaller MSS to the far end, and fragmentation
will not result.

I'm pretty sure that this low MTU has been set deliberately as a
workaround; without it, the system would probably fail even some of
the most basic tests (such as "can a random person with an ordinary
Android smartphone actually see web pages").

The workaround is effective so long as the actual path MTU is at least
1350 - if the other end of the connection (or, rarely, some
intervening bit of network) is on some kind of small-MTU link, things
will break. And it will only work for TCP and only when the TCP
connection is made directly by the machine connected to the wifi.
Other protocols will fail - for example many vpns, most voip, many
games, use of enhanced-services dns resolvers, probably mosh (the
remote text terminal client), etc. They may work unreliably, as
"small enough" packets are unaffected.

What should I do about this ?

Is it worth complaining to Greater Anglia ? Is there any way to get
this complaint to anyone who will understand it ?

This bug is almost certainly a firmware defect in the wifi box on
board the train. It reveals serious incompetence by the middlebox
vendor and the best way to fix it would be to fire the middlebox
vendor and get a different middlebox by someone more competent.

(Of course I'm going to put a workaround in my VPN software, since one
of the functions of VPN software is to provide
sane-IP-over-braindamage.)

reading from file greateranglia-wifi-bug/zealot.pcap, link-type EN10MB (Ethernet)
21:11:48.098925 IP (tos 0x0, ttl 64, id 43585, offset 0, flags [+], proto ICMP (1), length 1348)
10.101.1.159 > 212.13.197.229: ICMP echo request, id 11892, seq 1, length 1328
21:11:48.099011 IP (tos 0x0, ttl 64, id 43585, offset 1328, flags [none], proto ICMP (1), length 700)
10.101.1.159 > 212.13.197.229: ip-proto-1
21:11:51.101030 IP (tos 0x0, ttl 64, id 43586, offset 0, flags [+], proto ICMP (1), length 1348)
10.101.1.159 > 212.13.197.229: ICMP echo request, id 11892, seq 2, length 1328
21:11:51.101154 IP (tos 0x0, ttl 64, id 43586, offset 1328, flags [none], proto ICMP (1), length 700)
10.101.1.159 > 212.13.197.229: ip-proto-1
21:11:54.101255 IP (tos 0x0, ttl 64, id 43587, offset 0, flags [+], proto ICMP (1), length 1348)
10.101.1.159 > 212.13.197.229: ICMP echo request, id 11892, seq 3, length 1328
21:11:54.101324 IP (tos 0x0, ttl 64, id 43587, offset 1328, flags [none], proto ICMP (1), length 700)
10.101.1.159 > 212.13.197.229: ip-proto-1

reading from file greateranglia-wifi-bug/chiark.pcap, link-type EN10MB (Ethernet)
21:11:46.936255 IP (tos 0x0, ttl 50, id 43585, offset 0, flags [DF], proto ICMP (1), length 1348)
93.158.79.70 > 212.13.197.229: ICMP echo request, id 59721, seq 1, length 1328 (wrong icmp cksum e41d (->e975)!)
21:11:46.955171 IP (tos 0x0, ttl 50, id 43585, offset 0, flags [DF], proto ICMP (1), length 700)
93.158.79.70 > 212.13.197.229: ICMP type-#40, length 680 (wrong icmp cksum 2a2b (->24d3)!)
IP3 bad-hlen 0
21:11:49.906494 IP (tos 0x0, ttl 50, id 43586, offset 0, flags [DF], proto ICMP (1), length 1348)
93.158.79.70 > 212.13.197.229: ICMP echo request, id 59721, seq 2, length 1328 (wrong icmp cksum 9a14 (->9f6c)!)
21:11:49.916264 IP (tos 0x0, ttl 50, id 43586, offset 0, flags [DF], proto ICMP (1), length 700)
93.158.79.70 > 212.13.197.229: ICMP type-#40, length 680 (wrong icmp cksum 2a2b (->24d3)!)
IP3 bad-hlen 0
21:11:52.904151 IP (tos 0x0, ttl 50, id 43587, offset 0, flags [DF], proto ICMP (1), length 1348)
93.158.79.70 > 212.13.197.229: ICMP echo request, id 59721, seq 3, length 1328 (wrong icmp cksum be12 (->c36a)!)
21:11:52.913669 IP (tos 0x0, ttl 50, id 43587, offset 0, flags [DF], proto ICMP (1), length 700)
93.158.79.70 > 212.13.197.229: ICMP type-#40, length 680 (wrong icmp cksum 2a2b (->24d3)!)
IP3 bad-hlen 0

--
Ian Jackson personal email: <ijac...@chiark.greenend.org.uk>
These opinions are my own. http://www.chiark.greenend.org.uk/~ijackson/
PGP2 key 1024R/0x23f5addb, fingerprint 5906F687 BD03ACAD 0D8E602E FCF37657

Message has been deleted

Rupert Moss-Eccardt

unread,

Apr 10, 2014, 5:15:03 AM4/10/14

to

Ian Jackson wrote:
> My VPN client didn't work properly when I was on a train. I suspected
> a problem with IP fragmentation and today I had time to investigate
> properly.
>

> [snip]
I have emailed someone I know in Abellio. We'll see if they can point us
in the right direction.

Ian Jackson

unread,

Apr 10, 2014, 6:58:39 AM4/10/14

to

In article <li5bpi$jsi$1...@speranza.aioe.org>,
Hils <hi...@saynotospam.net> wrote:

>On 2014-04-10 00:28, Ian Jackson wrote:
>>Is it worth complaining to Greater Anglia ? Is there any way to get
>>this complaint to anyone who will understand it ?
>

>"The User acknowledges:- (a) that it is technically impossible to
>provide the Hotspot entirely free of faults [...]"
>http://www.abelliogreateranglia.co.uk/travel-information/your-journey/wifi/wi-fi-terms-and-conditions

Litigious though I am, it is surely obvious that sueing Abellio would
be a morally questionable and in any case entirely impractical
response to this difficulty. I was kind of hoping that someone with
the right connections (or suggestions) would be reading.

Ian Jackson

unread,

Apr 10, 2014, 6:58:59 AM4/10/14

to

In article <bqn5nk...@mid.individual.net>,

Rupert Moss-Eccardt <r.moss-...@computer.org> wrote:
>Ian Jackson wrote:
>> My VPN client didn't work properly when I was on a train. I suspected
>> a problem with IP fragmentation and today I had time to investigate
>> properly.

...

>I have emailed someone I know in Abellio. We'll see if they can point us
>in the right direction.

Great, thanks.

Message has been deleted

tony sayer

unread,

Apr 13, 2014, 6:17:27 AM4/13/14

to

In article <li6p7o$a49$1...@speranza.aioe.org>, Hils <hi...@saynotospam.net>
scribeth thus

>On 2014-04-10 11:58, Ian Jackson wrote:
>> In article <li5bpi$jsi$1...@speranza.aioe.org>,
>> Hils <hi...@saynotospam.net> wrote:
>>> On 2014-04-10 00:28, Ian Jackson wrote:
>>>> Is it worth complaining to Greater Anglia ? Is there any way to get
>>>> this complaint to anyone who will understand it ?
>>>
>>> "The User acknowledges:- (a) that it is technically impossible to
>>> provide the Hotspot entirely free of faults [...]"
>>> http://www.abelliogreateranglia.co.uk/travel-information/your-
>journey/wifi/wi-fi-terms-and-conditions
>>
>> Litigious though I am, it is surely obvious that sueing Abellio would
>> be a morally questionable and in any case entirely impractical
>> response to this difficulty. I was kind of hoping that someone with
>> the right connections (or suggestions) would be reading.
>

>If it's not resolved, Ofcom or the ASA may be interested. They may think
>differently, but ISTM that if a paid service is so crippled, the excuses
>shouldn't be buried in the T&Cs.

Doubt Ofcom would be the slightest bit interested in that sort of
misapplication of a service...
--
Tony Sayer