Incorrect DNS Resolution for krk.kargo.com with 8.8.8.8

96 views
Skip to first unread message

jer...@kargo.com

unread,
Jan 15, 2019, 5:50:06 PM1/15/19
to public-dns-discuss
Hey guys,

When using Google's DNS Server (8.8.8.8), we periodically see DNS resolution issues, but none of the other DNS servers have the same issue. You can reproduce by running `dig @8.8.8.8 krk.kargo.com` a bunch of times in a row in comparison to `dig @1.1.1.1 krk.kargo.com`.

1.1.1.1 always returns the following answer section...
;; ANSWER SECTION:
krk
.kargo.com. 227 IN CNAME kraken.production.us-east-1.kops.kargo.com.
kraken
.production.us-east-1.kops.kargo.com. 8 IN A 52.72.14.87
kraken
.production.us-east-1.kops.kargo.com. 8 IN A 52.204.49.101
kraken
.production.us-east-1.kops.kargo.com. 8 IN A 204.236.242.253


Whereas 8.8.8.8 sometimes returns an authority section instead...
;; AUTHORITY SECTION:
krk
.kargo.com. 1080 IN SOA ns1.p24.dynect.net. email.kargo.com. 2019011111 3600 600 604800 1800


There is one error, but we don't think it applies. Is that an incorrect assumption? 
kargo.com/DNSKEY: The response had an invalid RCODE (SERVFAIL). (204.13.250.24, 204.13.251.24, 208.78.70.24, 208.78.71.24, 2001:500:90:1::24, 2001:500:94:1::24, UDP_0_EDNS0_32768_512, UDP_0_NOEDNS)



We talked to our DNS provider and were told that this is an issue within Google's DNS server. Any clue how to best handle this?

Thanks,
Jeremy

Alex Dupuy

unread,
Jan 15, 2019, 6:16:46 PM1/15/19
to public-dns-discuss
Your DNS provider (Dyn) is definitely at least a little bit broken since they should be returning a NODATA response (rather than SERVFAIL) for the attempt to lookup the DNSKEY.

$ dig +norec +dnssec DNSKEY kargo.com @ns1.p24.dynect.net

; <<>> DiG 9.11.2-P1-1-Debian <<>> +norec +dnssec DNSKEY kargo.com @ns1.p24.dynect.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 23011
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;kargo.com. IN DNSKEY

;; Query time: 11 msec
;; SERVER: 2001:500:90:1::24#53(2001:500:90:1::24)
;; WHEN: Tue Jan 15 17:57:48 EST 2019
;; MSG SIZE  rcvd: 38

The domain in the authority section you sometimes see is rather odd, since krk.kargo.com isn't a delegated zone (it should be kargo.com). Do you have both a CNAME and some other record here? Or is this perhaps some special kind of ALIAS record rather than a plain CNAME?

Jeremy Sadwith

unread,
Jan 16, 2019, 10:05:11 AM1/16/19
to public-dns-discuss
Hey Alex,

Thanks for the quick response. We use Dyn's Traffic Director to route traffic based on location. All traffic in the US/APAC is routed to the closest AWS Region, and we block Europe traffic due to GDPR. That's the reason why krk.kargo.com isn't a delegated zone. 

When running `dig +norec +dnssec DNSKEY krk.kargo.com @ns1.p24.dynect.net`, it looks like the response is NOERROR, and the response seems to always return the right values.

What do you think should be our next troubleshooting step?

Thanks,
Jeremy

Jeremy Sadwith

unread,
Jan 17, 2019, 4:52:15 PM1/17/19
to public-dns-discuss
Following up here with our solution in case anyone is experiencing the same issue.

Dyn thinks that this is due to: "Google Public DNS does some really weird routing, sometimes shipping requests from one POP to another for responses. If they are shipping a US-based request to a EU POP, this could explain this behavior.". Since we're blocking traffic in the EU, this could cause the weird resolution we're seeing in the US.

The solution for now is to use ECS: https://help.dyn.com/edns-client-subnet-faq-info/
Per Dyn, "Google will receive a DNS query, append the originating /24 prefix. in the edns-client-subnet (ECS) data, and send that information to our nameservers. If we see that ECS information in the query, we will take advantage of it in our traffic director response."

Hope this helps someone else someday.

Jeremy

jsad...@gmail.com

unread,
Jan 18, 2019, 7:16:41 PM1/18/19
to public-dns-discuss
One quick clarification based on an offline concern of Alex's - we use multiple levels of blocking (DNS/CDN/service) to prevent traffic from the EU.
Reply all
Reply to author
Forward
0 new messages