Google DNS resolving Google.com to...London?!

[chris...@gmail.com]

I've been testing using Google DNS (8.8.8.8/8.8.4.4) vs. Hurricane Electric's DNS (74.82.42.42). My ISP (in Missouri) uses HE in Kansas City as their primary transit uplink with a secondary to Cogent in St. Louis.

I've settled on Google DNS over HE's offering, even though HE's is closer and faster (9ms cached, 63ms uncached vs 23ms cached, 69ms uncached, per GRC DNS Benchmark), because Google DNS supports EDNS and DNSSEC, and even though I hit the HE anycast server in KC (6 hops/9ms away), I get the weirdest CDN results from it and get Netflix streaming from Miami, Akamai from Dallas (via Denver), Facebook from NYC, etc., whereas with Google DNS (and its EDNS support), pretty much everything I get is served to me from Chicago (~20ms away).

The only odd issue I get with Google DNS is that when I access Google's own services, I seem to reach halfway across the world to get them! Take this traceroute:

Tracing route to www.google.com [216.58.204.36]
over a maximum of 30 hops:


  1    <1 ms    <1 ms    <1 ms  router [192.168.56.1]
  2     2 ms    <1 ms    <1 ms  100.64.103.1
  3    <1 ms    <1 ms    <1 ms  38.131.218.xxx
  4     3 ms     3 ms     3 ms  38.65.114.217
  5    17 ms     9 ms    12 ms  v320.core1.mci3.he.net [184.105.41.97]
  6    13 ms    13 ms    14 ms  100ge9-2.core1.oma1.he.net [184.105.65.166]
  7    60 ms    58 ms    57 ms  206.126.235.14
  8    23 ms    23 ms    24 ms  108.170.244.2
  9    23 ms    23 ms    32 ms  216.239.59.151
 10    38 ms    37 ms    37 ms  209.85.252.39
 11   110 ms   110 ms   110 ms  72.14.236.8
 12   109 ms   109 ms   123 ms  216.239.58.128
 13     *        *        *     Request timed out.
 14   110 ms   110 ms   110 ms  108.170.238.119
 15   111 ms   110 ms   110 ms  lhr25s12-in-f36.1e100.net [216.58.204.36]


Trace complete.

Any ideas why on earth I'm getting Google served to me from London (LHR) instead of stopping somewhere a lot closer...like, perhaps, their datacenter in Omaha/Council Bluffs, which this path goes right past (and which my traceroute to 8.8.8.8 itself goes to)?

Tracing route to google-public-dns-a.google.com [8.8.8.8]
over a maximum of 30 hops:


  1    <1 ms     3 ms    <1 ms  router [192.168.56.1]
  2    <1 ms    <1 ms    <1 ms  100.64.103.1
  3    <1 ms    <1 ms    <1 ms  38.131.218.241
  4     3 ms     3 ms     3 ms  38.65.114.217
  5     9 ms    11 ms     9 ms  v320.core1.mci3.he.net [184.105.41.97]
  6    13 ms    13 ms    13 ms  100ge9-2.core1.oma1.he.net [184.105.65.166]
  7    62 ms    63 ms    66 ms  206.126.235.14
  8    23 ms    22 ms    24 ms  108.170.244.1
  9    23 ms    23 ms    23 ms  209.85.251.135
 10    23 ms    23 ms    23 ms  google-public-dns-a.google.com [8.8.8.8]


Trace complete.

Digging whoami.akamai.com gives me 74.125.72.16, and http://www.whatsmydnsserver.com/ gives me 173.194.103.137, both of which also appear to resolve to Council Bluffs, so with or without EDNS, I should at least get something local to Omaha/KC, if not closer.

Interestingly, if I use the HE nameserver or even run a local resolver, I get Google served out of DFW, so why is using Google's own DNS service with a resolver in Council Bluffs resolving not to Council Bluffs but across an ocean and a third of a way around the world?? This is repeatable several days in a row, too, so it's not a hiccup.

Traceroutes to other services like www.youtube.com also end up in London, though thankfully, it seems I'm getting actual YouTube content served to me out of Council Bluffs and Google Cloud Files content out of DFW, so at least streaming/download rates are decent.

Any ideas?

[Alex Dupuy]

What do you see when you make the following dig queries?

dig TXT o-o.myaddr.test.l.google.com. # (with @8.8.8.8 and without, using default resolver configuration)

dig TXT edns-client-sub.net. # (with and without @8.8.8.8)

These would reveal what EDNS Client Subnet (ECS) data is being sent to authorities. Perhaps somehow your DNS requests are going out with ECS value of 100.64.103.1 (which is a Carrier Grade NAT address, no more meaningful than 192.168.56.1.

[Chris Luth]

Thanks for the reply.

It's Windows, so I'm using nslookup, but with and without 8.8.8.8, I get 

edns0-client-subnet 38.131.218.243/32

and

"{'ecs_payload':{'family':'1','optcode':'0x08','cc':'US','ip':'38.131.218.0','mask':'24','scope':'0'},'ecs':'True','ts':'1515088007.77','recursive':{'cc':'US','srcip':'74.125.113.133','sport':'56313'}}"

Full results below. I'll not bother to obfuscate my ISP's CGNed public IP, as it's kinda pointless anyway. ;)

> set type=txt

> o-o.myaddr.test.l.google.com

Server:  router

Address:  192.168.56.1

Non-authoritative answer:

o-o.myaddr.test.l.google.com    text =

        "74.125.113.145"

o-o.myaddr.test.l.google.com    text =

        "edns0-client-subnet 38.131.218.243/32"

> o-o.myaddr.test.l.google.com 8.8.8.8

Server:  [8.8.8.8]

Address:  8.8.8.8

Non-authoritative answer:

o-o.myaddr.test.l.google.com    text =

        "74.125.183.72"

o-o.myaddr.test.l.google.com    text =

        "edns0-client-subnet 38.131.218.243/32"

> edns-client-sub.net

Server:  router

Address:  192.168.56.1

Non-authoritative answer:

edns-client-sub.net     text =

        "{'ecs_payload':{'family':'1','optcode':'0x08','cc':'US','ip':'38.131.218.0','mask':'24','scope':'0'},'ecs':'True','ts':'1515087998.46','recursive':{'cc':'US','srcip':'173.194.94.129','sport':'50825'}}"

> edns-client-sub.net 8.8.8.8

Server:  [8.8.8.8]

Address:  8.8.8.8

Non-authoritative answer:

edns-client-sub.net     text =

        "{'ecs_payload':{'family':'1','optcode':'0x08','cc':'US','ip':'38.131.218.0','mask':'24','scope':'0'},'ecs':'True','ts':'1515088007.77','recursive':{'cc':'US','srcip':'74.125.113.133','sport':'56313'}}"

[Chris Luth]

Just wanted to bump this and say there's been no change over the past week since I posted -- the exact endpoint fluctuates (today, it's 216.58.198.100), but it still resolves to some permutation of *lhr*.1e100.net with pings around 110ms.

[Chris Luth]

OK, so it's not a Google DNS issue. I set up Unbound as a local resolver and I'm getting the same thing. It's especially frustrating today as YouTube is now being served to me out of the London endpoint, meaning that I'm lucky to get smooth playback at 360p resolution. HD is impossible without buffering.

So it's obviously an issue with Google's internal CDN set-up/nameservers/whatever, not Google Public DNS (which makes sense, since I'm getting the same results from Google Public DNS, which passes my EDNS client subnet, as if I just run a resolver locally) and so off-topic for this mailing list. Alex or anyone else here, do you have contact info for someone in the right department at Google, or can someone here pass this along to the right place?

Local IP is 38.131.218.243, and my ASN is 55140. As we speak, I'm receiving YouTube content from lhr25s10-in-f165.1e100.net (216.58.198.165).

Thanks.

[Chris Luth]

This is still happening constantly and it's driving me nuts. Gmail gets annoyingly unresponsive sometimes. YouTube can barely stream to me in SD.

Today's server is lhr35s05-in-f69.1e100.net.

I know it's not strictly on-topic here (though I do use Google Public DNS), but since there appears to be no other way to report this issue to Google, I'm reliant on Alex or someone else at Google monitoring this list to pass my issue along to someone in the right department.

Again, as noted below, my local IP is 38.131.218.243 on ASN 55140 and every site I check correctly geolocates that to Southwest Missouri, so there is no reason that I should be getting served from London, yet London-based Google servers are almost always at the top of my network activity lists.

[Chris Luth]

Alex, I really need your help. This is driving me nuts.

Today I'm connecting to lhr25s14-in-f5.1e100.net and lhr35s06-in-f5.1e100.net. Pretty soon I will probably have a complete list of every server in your London datacenter, ha.

The problem is the long transatlantic link and high latency makes using Gmail difficult. The interface is laggy, uploading attachments is often tediously slow, and sometimes it breaks completely (like right now, a long message I tried to send has been stuck stuck on "Sending..." in the Gmail web interface for several minutes). I've already mentioned the issues with YouTube.

There is a problem here that should not exist and I need your help resolving it.

[Alex Dupuy]

I just checked, and addresses in 38.131.218.0/24 should be getting directed to frontends in Texas, Colorado, or Chicago. It can take a while (days to weeks) for mapping to converge for new address ranges. Are you still at that IP address?

Local IP is 38.131.218.243, and my ASN is 55140. As we speak, I'm receiving YouTube content from lhr25s10-in-f165.1e100.net (216.58.198.165).

I have pushed through internal issues to fix mapping problems in the past, and they have always resolved themselves by the time I got anyone to look at it, I suspect this might be true here as well.

The only other thing I can think of is that your browser or other intermediate DNS resolver is still using or returning cached answers for these lookups.

Have you tried a browser or router restart?

[Chris Luth]

Thanks for the reply.

Yes, that is a static IP and won't change anytime soon.

My system DNS resolver (which I push to all devices via the DHCP server I control for my network) is 8.8.8.8 and 8.8.4.4, so I am using Google Public DNS for all IP address resolutions. I did experiment with running my own local resolver directly querying the root servers, and the behavior was the same, so it seems to be with the Google authoritative name servers' geographic routing, perhaps?

I am using Chrome. However, behavior does replicate when I try Edge or Firefox as well as from the command line:

C:\Users\me>tracert www.google.com

Tracing route to www.google.com [216.58.206.36]

over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  router [192.168.56.1]

  2     1 ms    <1 ms    <1 ms  100.64.103.1

  3    <1 ms    <1 ms    <1 ms  38.131.218.241

  4     3 ms     2 ms     3 ms  38.65.114.217
  5     9 ms    16 ms     9 ms  v320.core1.mci3.he.net [184.105.41.97]

  6    13 ms    13 ms    13 ms  100ge9-2.core1.oma1.he.net [184.105.65.166]

  7    24 ms    24 ms    25 ms  206.126.235.14
  8    25 ms    25 ms    25 ms  108.170.243.165
  9    25 ms    25 ms    26 ms  209.85.249.136
 10    38 ms    38 ms    38 ms  209.85.252.47
 11   113 ms   113 ms   113 ms  209.85.254.250
 12   113 ms   113 ms   113 ms  216.239.57.206
 13   111 ms   111 ms   111 ms  108.170.246.225
 14   112 ms   112 ms   112 ms  216.239.41.207
 15   111 ms   111 ms   111 ms  lhr35s10-in-f4.1e100.net [216.58.206.36]

Trace complete.

C:\Users\me>nslookup www.google.com

Server:  router
Address:  192.168.56.1

Non-authoritative answer:

Name:    www.google.com
Addresses:  2a00:1450:4009:80c::2004
          216.58.206.36

C:\Users\me>nslookup www.google.com 8.8.8.8
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Non-authoritative answer:
Name:    www.google.com
Addresses:  2a00:1450:4009:80c::2004

          216.58.206.132

I can also replicate it on other machines on my network. I have rebooted those machines as well as my router multiple times.

If I watch Resource Monitor on Windows or my Mikrotik firewall's connections info when I load Gmail (or upload a file to Gmail) or YouTube, most of the time, I will see an LHR endpoint (i.e. right now, one of the top connections sorted by total B/sec in Resource Monitor is to lhr25s02-in-f101.1e100.net)

Occasionally, I'll see a DFW endpoint surface, but most of the traffic still goes to London:

C:\Users\me>tracert mail.google.com

Tracing route to googlemail.l.google.com [172.217.6.133]

over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  router [192.168.56.1]

  2    <1 ms    <1 ms    <1 ms  100.64.103.1

  3    <1 ms    <1 ms    <1 ms  38.131.218.241

  4     3 ms     2 ms     2 ms  38.65.114.217
  5     9 ms     9 ms     9 ms  v320.core1.mci3.he.net [184.105.41.97]
  6    26 ms    21 ms    21 ms  100ge12-1.core1.den1.he.net [184.105.64.49]
  7     *        *        *     Request timed out.
  8    31 ms    31 ms    31 ms  108.170.252.210
  9    31 ms    31 ms    31 ms  209.85.143.215
 10    28 ms    28 ms    28 ms  72.14.237.134
 11    28 ms    28 ms    28 ms  108.170.228.100
 12     *        *        *     Request timed out.
 13    28 ms    28 ms    28 ms  72.14.234.61
 14    27 ms    28 ms    27 ms  dfw25s16-in-f5.1e100.net [172.217.6.133]

Trace complete.

C:\Users\me>tracert ytimg.l.google.com

Tracing route to ytimg.l.google.com [216.58.213.110]

over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  router [192.168.56.1]
  2     2 ms    <1 ms    <1 ms  100.64.103.1

  3    <1 ms    <1 ms    <1 ms  38.131.218.241
  4     3 ms     2 ms     2 ms  38.65.114.217
  5     9 ms    16 ms     9 ms  v320.core1.mci3.he.net [184.105.41.97]
  6    21 ms    22 ms    24 ms  100ge12-1.core1.den1.he.net [184.105.64.49]
  7     *        *        *     Request timed out.
  8    31 ms    31 ms    31 ms  108.170.252.195
  9    32 ms    32 ms    32 ms  74.125.37.127
 10    32 ms    31 ms    32 ms  216.239.43.227
 11    33 ms    32 ms    32 ms  72.14.234.8
 12    48 ms    65 ms    45 ms  209.85.250.9
 13   119 ms   118 ms   147 ms  209.85.254.244
 14   118 ms   120 ms   118 ms  216.239.57.236
 15     *        *        *     Request timed out.
 16   119 ms   119 ms   119 ms  216.239.57.121
 17   118 ms   117 ms   118 ms  lhr25s02-in-f14.1e100.net [216.58.213.110]

Trace complete.

BTW, interesting reading at https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35590.pdf. Perhaps some of the issue is that my upstream ASNs are Cogent and HE (6939 and 174) , which both have widespread networks with several peering points with Google, including London, and so the way Google does geographic routing (by ASN latency, not client latency) might be contributing to the issue?

--
--
========================================================
You received this message because you are subscribed to the Google
Groups "public-dns-discuss" group.
To post to this group, send email to public-dns-discuss@googlegroups.com
To unsubscribe from this group, send email to
public-dns-discuss+unsub...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/public-dns-discuss
For more information on Google Public DNS, please visit
http://developers.google.com/speed/public-dns
========================================================
---
You received this message because you are subscribed to a topic in the Google Groups "public-dns-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/public-dns-discuss/25JxfnGqe40/unsubscribe.
To unsubscribe from this group and all its topics, send an email to public-dns-discuss+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ᐧ

[Chris Luth]

One screenshot to add showing traffic to London:

ᐧ

On Wed, Mar 7, 2018 at 2:32 PM, 'Alex Dupuy' via public-dns-discuss <public-dns-discuss@googlegroups.com> wrote:

I just checked, and addresses in 38.131.218.0/24 should be getting directed to frontends in Texas, Colorado, or Chicago. It can take a while (days to weeks) for mapping to converge for new address ranges. Are you still at that IP address?

Local IP is 38.131.218.243, and my ASN is 55140. As we speak, I'm receiving YouTube content from lhr25s10-in-f165.1e100.net (216.58.198.165).

I have pushed through internal issues to fix mapping problems in the past, and they have always resolved themselves by the time I got anyone to look at it, I suspect this might be true here as well.

The only other thing I can think of is that your browser or other intermediate DNS resolver is still using or returning cached answers for these lookups.

Have you tried a browser or router restart?

--
--
========================================================
You received this message because you are subscribed to the Google
Groups "public-dns-discuss" group.
To post to this group, send email to public-dns-discuss@googlegroups.com
To unsubscribe from this group, send email to

public-dns-discuss+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/public-dns-discuss
For more information on Google Public DNS, please visit
http://developers.google.com/speed/public-dns
========================================================
---
You received this message because you are subscribed to a topic in the Google Groups "public-dns-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/public-dns-discuss/25JxfnGqe40/unsubscribe.

To unsubscribe from this group and all its topics, send an email to public-dns-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

ᐧ

[Chris Luth]

Something may have changed with your updates--I'm now hitting YYZ (Toronto) for the bulk of my traffic and traceroutes.

Still a headscratcher why I'm not hitting DFW, ORD, or DEN when all of those are much closer (~20ms, and the route from me to YYZ actually transits DEN), but I can live with the 43ms latency to YYZ--I actually haven't seen any buffering/standard definition on YouTube or glitches with Gmail having trouble loading in the last few days.

Actually, actual YouTube content seems to be coming from somewhere I'm guessing around Chicago (IPs in the 173.194.162.8 netblock, 24ms away). Much better.

On Wednesday, March 7, 2018 at 6:15:06 PM UTC-5, Chris Luth wrote:

One screenshot to add showing traffic to London:

ᐧ

public-dns-discuss+unsub...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/public-dns-discuss
For more information on Google Public DNS, please visit
http://developers.google.com/speed/public-dns
========================================================
---
You received this message because you are subscribed to a topic in the Google Groups "public-dns-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/public-dns-discuss/25JxfnGqe40/unsubscribe.

To unsubscribe from this group and all its topics, send an email to public-dns-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

ᐧ