Bad resolution for Google related domains (google.com, gmail.com, gstatic.com)

[brave...@gmail.com]

We are experiencing an issue with the resolution of any Google related domain (google.com, gmail.com, gstatic.com, etc) from a specific location.

This location is Mexico City, the resolution of these domains all end up in IPs similar "eze04s05-in-f4.1e100.net" that i assume is Argentina all under the prefix 216.58.192.0/19. This is happening either using 8.8.8.8, 8.8.4.4 or ns1.google.com (the same happens by going directly to the root servers)

The latencies from Mexico City to any of those domains are high, more than 200ms for ICMP.

If i use CloudFlare DNS (1.1.1.1) the domains resolve to IPs under the prefix 74.125.127.0/24 that correspond to a rDNS similar to "ov-in-f147.1e100.net" , i don't know where this is located but the ICMP latencies for this are much lower, around 38ms.

There is obviously a wrong resolution for Google related domains from Mexico City that is making using Google much slower than it should be.

As you can see in these ThousandEyes links, the traffic within the Google network is being delayed for domains going to 216.58.192.0/19, from hop 209.85.247.116 to hop 209.85.254.154. Between those hops there is a latency of 128ms:

High latency: https://lcrzs.share.thousandeyes.com

Low latency: https://yftvz.share.thousandeyes.com

The consecuence of this is a page load of more than 4 seconds for any Google related service.

As extra information, the Mexico City IP where i'm doing the tests from is 138.186.140.212

Some dig outputs:

- 1.1.1.1:

$ dig @1.1.1.1 google.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @1.1.1.1 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34487
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;google.com.            IN    A

;; ANSWER SECTION:
google.com.        286    IN    A    172.217.12.46

;; Query time: 2 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Jul 20 23:19:10 UTC 2018
;; MSG SIZE  rcvd: 55

- ns1.google.com

$ dig @ns1.google.com google.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @ns1.google.com google.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6668
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;google.com.            IN    A

;; ANSWER SECTION:
google.com.        300    IN    A    172.217.28.174

;; Query time: 52 msec
;; SERVER: 216.239.32.10#53(216.239.32.10)
;; WHEN: Fri Jul 20 23:19:30 UTC 2018
;; MSG SIZE  rcvd: 44

- 8.8.8.8:

$ dig @8.8.8.8 google.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27417
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.            IN    A

;; ANSWER SECTION:
google.com.        299    IN    A    172.217.28.174

;; Query time: 45 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jul 20 23:19:40 UTC 2018
;; MSG SIZE  rcvd: 55

- With trace:

$ dig +trace google.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> +trace google.com
;; global options: +cmd
.            518400    IN    NS    a.root-servers.net.
.            518400    IN    NS    b.root-servers.net.
.            518400    IN    NS    c.root-servers.net.
.            518400    IN    NS    d.root-servers.net.
.            518400    IN    NS    e.root-servers.net.
.            518400    IN    NS    f.root-servers.net.
.            518400    IN    NS    g.root-servers.net.
.            518400    IN    NS    h.root-servers.net.
.            518400    IN    NS    i.root-servers.net.
.            518400    IN    NS    j.root-servers.net.
.            518400    IN    NS    k.root-servers.net.
.            518400    IN    NS    l.root-servers.net.
.            518400    IN    NS    m.root-servers.net.
;; Received 239 bytes from 208.67.222.222#53(208.67.222.222) in 48 ms

com.            172800    IN    NS    a.gtld-servers.net.
com.            172800    IN    NS    b.gtld-servers.net.
com.            172800    IN    NS    c.gtld-servers.net.
com.            172800    IN    NS    d.gtld-servers.net.
com.            172800    IN    NS    e.gtld-servers.net.
com.            172800    IN    NS    f.gtld-servers.net.
com.            172800    IN    NS    g.gtld-servers.net.
com.            172800    IN    NS    h.gtld-servers.net.
com.            172800    IN    NS    i.gtld-servers.net.
com.            172800    IN    NS    j.gtld-servers.net.
com.            172800    IN    NS    k.gtld-servers.net.
com.            172800    IN    NS    l.gtld-servers.net.
com.            172800    IN    NS    m.gtld-servers.net.
com.            86400    IN    DS    30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.            86400    IN    RRSIG    DS 8 1 86400 20180802170000 20180720160000 41656 . gimEIP+Ks+lKl/1HbC5ZOYXo2Fxn9ZG5YzWXhb92xnZjIHq/JqRjsqxO gqPKh0+yW8eSldY9YByWxY6QuH3+Gf4T0IEtkVUroXKumCTpKYVOLBcn a/W7OVdL+gMwdL7YSHkOcuk0vbf9qtGju+rYQdxvAcfSbr4Zwk+vB5n2 gatavpGwOSR9OegNekBh3MxShb9yRUSJlnbtGuR5ceeHmAIlSwAy0Xub DpKLkdTZNCOBE2aXX/olRxrI2EDvGcRHQL6TVxihCNr/tQPUzWqLXX5f XYNx0zzfh8rRoC1UecwSv5A8zMp3sIw6oHB6JA+0qu4bla8R6RIWRl2f 0VjRrw==
;; Received 1170 bytes from 199.7.83.42#53(l.root-servers.net) in 63 ms

google.com.        172800    IN    NS    ns2.google.com.
google.com.        172800    IN    NS    ns1.google.com.
google.com.        172800    IN    NS    ns3.google.com.
google.com.        172800    IN    NS    ns4.google.com.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20180727044456 20180720033456 46475 com. bZcUamH7dLp0d/N7iFLFHGxlTSmArGCDM3ufxdSp82loGtQA313NTJjC X6KxacakbmdzVB0UKsbYbEOjTsSg/FnI61DeJUUva5bp+dGOiQOKShtQ oh1AtS10bcxIB/OmkB1phw5Jiy4YN6wX92mUC5vYQ9QvFNZDens4KAAK exk=
S849LHDDSVU9A9N2FIRO5NKMQB321BEP.com. 86400 IN NSEC3 1 1 0 - S84CEFMDU6ABFSN4V0L2VLLOASCD5IV2 NS DS RRSIG
S849LHDDSVU9A9N2FIRO5NKMQB321BEP.com. 86400 IN RRSIG NSEC3 8 2 86400 20180724050949 20180717035949 46475 com. G2rqahJJayY0Xh1UttpVnJJtrueZC7XrarUnYVrx8JwgPb99thlGDtUg ichVVDelPj5djylrwwXb1ZVlzSrysx0YgEZqzwuV9mv96PU7cY3gyt+7 wtUKJcEhGIe2v3zVHxpbV8zuCj4QXtUA3gnkxe+6yNNkcVsnxzD7FwGW zso=
;; Received 772 bytes from 192.26.92.30#53(c.gtld-servers.net) in 85 ms

google.com.        300    IN    A    172.217.28.174
;; Received 44 bytes from 216.239.32.10#53(ns1.google.com) in 52 ms

[brave...@gmail.com]

Adding more information.

The IP block for Mexico was assigned recently, so it seems that before that block was in Argentina.

Google uses "EDNS Client Subnet (ECS)" to geolocate requests, this means Google needs to update it or geolocate again this IP block.