Why are DNS requests from Ireland resolving in Belgium & Netherlands?

[C]

Hi folks,

Long time user of Google Public DNS here but beyond typing 8.8.8.8 / 8.8.4.4 into every device I can find, I've never really spent much time delving into the system

Tonight while doing a bit of reading and testing, I inadvertently discovered that my lookups here in Ireland to Google DNS were being handled by Google servers in Belgium and the Netherlands. I found this strange as I had always assumed requests to Google DNS were handled within the country if Google had infrastructure in that country and Google have plenty of cloud infrastructure in Dublin so I always figured my requests were being resolved there. I checked different devices on both the Vodafone Ireland fixed-line network and the Three Ireland mobile network and the results are the same - lookups are being resolved by IP's such as 74.125.47.143, 74.125.73.66, etc...

I'm a little disappointed by this as it seems Google aren't resolving our lookups in Ireland or even in the UK, which is still closer than the EU mainland. It's makes the perceived performance of Google DNS a little misleading too as a quick ping to 8.8.8.8 has low RTT of 10ms but a ping to any of the aformentioned IP's which are actually resolving the lookups have a RTT of 30+ ms. Tracerts are the same... a trace to 8.8.8.8 completes in 5 or 6 hops but a trace to the actual resolver IP's can be as high as 18 hops. It would appear from this that Google have an "8.8.8.8 endpoint" in Ireland but this simply forwards the requests out to Belgium or the Netherlands. Granted, I do run Namebench tests from time to time and Google Public DNS still scores favourably but I'd prefer my lookups to be resolved closer to home. OpenDNS and even smaller players have endpoints in Ireland and the UK

Can anyone from Google confirm if there is DNS resolving infrastructure in Ireland or even in the UK?

Thanks

[Alex Dupuy]

Ping and traceroute can be helpful (the latter especially to understand the data path), but at the end of the day, you will get a more meaningful measure of DNS performance if you look at the ranges for round-trip time of DNS queries rather than ICMP packets. The latter can be useful to put lower bounds on network delay and therefore distance, but it isn't really reasonable to evaluate a DNS service based on its ICMP performance.

$ ping -qc 3 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---

3 packets transmitted, 3 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 13.459/14.912/16.573/1.280 ms

You can get DNS query timings with dig:

$ dig +noall +nocl +answer +stats www.google.com @8.8.8.8

www.google.com. 182 A 172.217.2.4

;; Query time: 22 msec

;; SERVER: 8.8.8.8#53(8.8.8.8)

;; WHEN: Wed Jan 18 02:49:11 2017

;; MSG SIZE  rcvd: 48

which shows the 22ms latency of a request answered from cache (you can tell this by the TTL of 182 rather than 300 or 299) and you can compare times for other resolvers in the same way. You can put an upper bound on the distance of the source of your answer by multiplying that by half the speed of light in fiber, but that is an upper bound, a more realistic estimate of distance would be 100km of distance for each millisecond of ping RTT, subtracting 8-40ms for DNS processing depending on whether you are getting cached data or your query is going all the way to the authoritative servers). For an example of the latter, you can query the SOA of dod.mil, which is only served from the US (and isn't likely to be in the cache of any resolver):

$ dig +noall +nocl +answer +stats SOA dod.mil @8.8.8.8

dod.mil. 43199 SOA dns-mas-a1-t.pentagon.mil. sysadmin.pentagon.mil. 2006066315 900 90 2592000 3600

;; Query time: 103 msec

;; SERVER: 8.8.8.8#53(8.8.8.8)

;; WHEN: Wed Jan 18 02:50:19 2017

;; MSG SIZE  rcvd: 92

You can even use tools like namebench 1.3.1 to benchmark the overall performance of the resolver you are using (there is also an unfinished namebench 2.0) if you are feeling adventurous.

While the resolver locations listed at developers.google.com/speed/public-dns/faq#locations do include Dublin, not all locations are actually used for Google Public DNS, as you noticed, so queries from Google Public DNS to authoritative servers will come only from a subset of the locations listed. However, bear in mind that Google Public DNS caches may be located closer to you than the IP addresses used to query authoritative servers, and this can be measured to some degree of accuracy by the query time difference between queries to Google Public DNS that are answered from cache, and queries directly to the authoritative name servers.

It's also important to note that with many websites now served by Content Deliver Networks (CDNs), the speed with which you get a DNS response may be much less important overall than the proximity of the address that is returned. To help improve the quality of responses by this important metric, Google Public DNS and other public resolvers, notably OpenDNS, developed the EDNS Client Subnet (ECS) extension that has been adopted by the IETF as RFC 7871 and by many authoritative servers (although not all, Microsoft and Facebook being the most notable current exceptions), allowing those that support ECS to return geographically accurate responses for clients in Ireland and India even when the resolvers sending them the query are located in Belgium and Singapore.

[C]

Hi Alex,

Thanks for the comprehensive response and apologies for not following up sooner

OK, I got my dig commands running here (I'm a Windows kid!) and I'm seeing as follows:

RTT to 8.8.8.8

hrping -L 84 -q -n 9 8.8.8.8

Pinging 8.8.8.8 [8.8.8.8] with 56 bytes data:

Packets: sent=9, rcvd=9, error=0, lost=0 (0.0% loss) in 4.024245 sec

RTTs in ms: min/avg/max/dev: 17.109 / 17.455 / 17.747 / 0.223

DIG to google.ie

dig +noall +nocl +answer +stats www.google.ie @8.8.8.8

www.google.ie.          299     A       216.58.198.67

;; Query time: 46 msec

;; SERVER: 8.8.8.8#53(8.8.8.8)

;; WHEN: Thu Feb 16 22:44:14 GMT Standard Time 2017

;; MSG SIZE  rcvd: 58

Repeat DIG to google.ie

dig +noall +nocl +answer +stats www.google.ie @8.8.8.8

www.google.ie.          288     A       216.58.198.67

;; Query time: 15 msec

;; SERVER: 8.8.8.8#53(8.8.8.8)

;; WHEN: Thu Feb 16 22:44:25 GMT Standard Time 2017

;; MSG SIZE  rcvd: 58

Based on your notes above and my DIG's, would it be a correct observation to say that the first DIG to google.ie may have been resolved in Belgium / Netherlands and then the follow-up request was served from a possibe DNS cache in Dublin?

I didn't make it clear either in the first post but my realisation that lookups from Ireland were possibly being handled in mainland Europe came from the online service ipleak.net. When I load this site, it correctly identifies I'm using Google's DNS infrastructure (I use at router level) but it then loads a list of 61 Google servers across Belgium and the Netherlands which are likely the resolvers. I've fired off an email to IP Leak to asking how this list of servers is ascertained and if it's possible their own infrastructure is listing their nearest Google DNS resolvers

All said... it's not a show stopper. If Google can provide faster and more reliable DNS than the Irish ISP's, I'll stick with them - even if some requests are being handled abroad

[Alex Dupuy]

Google Public DNS uses a multi-level cache structure, and tools like ipleak.net can only identify the locations where we originate queries to (external) authoritative name servers (and which are documented in the FAQ as I wrote previously). They don't have any visibility into our internal cache locations, which are located not only in our data centers but also our Edge PoPs, as I wrote in this recent forum post: https://groups.google.com/forum/#!topic/public-dns-discuss/oiyfQnXfpqk.

Your reduced resolution time of 15 ms for recently queried data certainly indicates that you are getting an answer from an Edge PoP in Ireland or the UK, although resolution latency is not entirely determined by distance × speed of light, the number of levels of caching your query goes through adds some time as well.

[C]

Thanks again Alex

I think the real issue here is my blatant lack of knowledge on the workings of DNS!! That said, it's been interesting reading up on it, understanding current workings and reading through different proposals. I've an Ubuntu box near ready here which I'll be test-running a DNS server on to get a bit more familiar with it

Looking again at where the requests are apparently being resolved and cross-checking with the list of DNS locations (https://developers.google.com/speed/public-dns/faq#locations), the IP's being identified as the Netherlands by the various tools are now in Brussels according to Google's list and the actual Netherland ranges from the list (GRQ -173.194.169.x & .170.x) don't appear to be in use on any of the tools. Is it possible that at some point, Google done a bit of consolidation and shifted the bulk of the resolving work to Brussels? My reasoning for this assumption being that why else would IP ranges for the Netherlands and Dublin appear on Google's location list if they weren't originating queries (stealing your phrase) in these locations at some point. Granted, they're likely caching in these locations but if caches aren't visible, then I doubt the IP ranges on the list for GRQ and DUB are for caches

Again, this is all just observation and it's not impacting my DNS experience that I can tell of

If Google did infact move to originating queries from Brussels, I'm struggling to grasp is the local / edge caching concept. Looking at my DIG's to google.ie in the last post, surely in Ireland the IP of google.ie would always be hot in the Irish cache but there seems to be a pattern that if I run DIG's on the same domain a few times in succession, I experience low RTT's (served from Dublin cache perhaps) but then I experience one longer RTT with a higher TTL (served from where?) then back to low RTT again. What exactly is happening when this higher RTT appears? Is the request going up to the originating servers for something that was in the cache just seconds before?

[Alex Dupuy]

C wrote:

My reasoning for this assumption being that why else would IP ranges for the Netherlands and Dublin appear on Google's location list if they weren't originating queries (stealing your phrase) in these locations at some point. Granted, they're likely caching in these locations but if caches aren't visible, then I doubt the IP ranges on the list for GRQ and DUB are for caches

The locations list at https://developers.google.com/speed/public-dns/faq#locations is for outbound queries to authoritative servers. It includes all possible locations where queries could egress from Google, but many of them are not (yet) used for Google Public DNS service. Besides the Google Public DNS resolvers in almost all the core data centers (see https://peering.google.com/#/infrastructure), there are caching resolvers in many of the Edge PoPs, but these are not listed in the locations list (unless the location happens to also contain a core data center or Cloud region, see below).

Currently, all outbound queries to authoritative servers originate from our core data center locations , the caching resolvers in the Edge PoPs forward all uncached queries through the data centers. DUB is the one core data center that is not currently running Google Public DNS, so it is not used; as you saw, uncached queries handled through the Edge PoP in DUB are sent to BRU.

The locations list also includes a number of Google Cloud regions (https://cloud.google.com/about/locations/ – IAD, SYD, LHR, FRA, BOM, GRU, NRT being the current ones where there is no core data center) but outbound queries from those locations would only be made on behalf of Cloud systems and customer applications using default resolvers, and not for queries routed through the Google Public DNS anycast IP addresses 8.8.8.8 et al.

Looking at my DIG's to google.ie in the last post, surely in Ireland the IP of google.ie would always be hot in the Irish cache but there seems to be a pattern that if I run DIG's on the same domain a few times in succession, I experience low RTT's (served from Dublin cache perhaps) but then I experience one longer RTT with a higher TTL (served from where?) then back to low RTT again. What exactly is happening when this higher RTT appears? Is the request going up to the originating servers for something that was in the cache just seconds before?

Queries to Google Public DNS are handled by a large set of resolvers with their own independent caches. When you send a series of queries, it is quite common that each query will get answers from different resolvers at the nearest location (whether Edge PoP as in your case, or data center for users in Belgium). While google.ie is going to be commonly in cache in DUB, some resolvers may not have a cached entry with unexpired TTL, so the query will be forwarded to a data center (BRU). In the case where you get a larger TTL of 229, it would likely have been a cache miss at the data center as well, and would have been forwarded to the authoritative servers for google.ie: ns[1234].google.com.