DNSSEC Wildcard CNAME Failing on Google DNS

[dnsforum...@gmail.com]

Hello,

I've noticed an issue when resolving a wildcard CNAME *.dodlive.mil when DNSSEC is enabled.  Google consistently returns SERVFAIL on 8.8.8.8 and 8.8.4.4.  You can see that AT&T resolves just fine.

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6 <<>> +additional anything.dodlive.mil. @8.8.4.4
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 58525
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;anything.dodlive.mil.          IN      A

;; Query time: 51 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Tue Mar 15 18:08:34 2016
;; MSG SIZE  rcvd: 38

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6 <<>> +additional anything.dodlive.mil. @165.87.13.129
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52756
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;anything.dodlive.mil.          IN      A

;; ANSWER SECTION:
anything.DODLIVE.mil.   3195    IN      CNAME   www.dodlive.mil.edgekey.net.
www.dodlive.mil.edgekey.net. 239 IN     CNAME   e5757.dscb.akamaiedge.net.
e5757.dscb.akamaiedge.net. 20   IN      A       23.7.177.136

;; Query time: 10 msec
;; SERVER: 165.87.13.129#53(165.87.13.129)
;; WHEN: Tue Mar 15 18:08:34 2016
;; MSG SIZE  rcvd: 148

Now if I specify CNAME I do get a response but the client behavior is that the DNS fails to resolve and the page is not displayed.

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6 <<>> CNAME +additional anything.dodlive.mil. @8.8.4.4
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3421
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;anything.dodlive.mil.          IN      CNAME

;; ANSWER SECTION:
anything.dodlive.mil.   3599    IN      CNAME   www.dodlive.mil.edgekey.net.

;; Query time: 91 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Tue Mar 15 18:10:33 2016
;; MSG SIZE  rcvd: 79

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6 <<>> CNAME +additional anything.dodlive.mil. @165.87.13.129
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41648
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;anything.dodlive.mil.          IN      CNAME

;; ANSWER SECTION:
anything.DODLIVE.mil.   3076    IN      CNAME   www.dodlive.mil.edgekey.net.

;; Query time: 9 msec
;; SERVER: 165.87.13.129#53(165.87.13.129)
;; WHEN: Tue Mar 15 18:10:33 2016
;; MSG SIZE  rcvd: 96

Would really appreciate if you could look into the root cause of this failure.

[Alexander Dupuy]

This is a problem with the DNSSEC implementation on your nameservers - interestingly it is one that neither DNSViz nor Verisign DNSSEC Analyzer properly detect.

http://dnsviz.net/d/*.dodlive.mil/dnssec/

http://dnssec-debugger.verisignlabs.com/w.i.l.d.dodlive.mil

There are special considerations for wildcards when DNSSEC is involved - in particular, since there is no signed record for the specific domain queried that matches the wildcard, an NSEC or NSEC3 record indicating a wildcard must be returned to validate the response, per https://tools.ietf.org/html/rfc7129#section-5.3.

The authoritative nameservers for dodlive.mil are returning an RRSIG (for the wildcard) but no NSEC or NSEC3 record:

$ dig +dnssec +noall +answer *.dodlive.mil @155.7.208.112 +aa

*.dodlive.mil. 3600 IN CNAME www.dodlive.mil.edgekey.net.

*.dodlive.mil. 3600 IN RRSIG CNAME 8 2 3600 20160322075526 20160312065526 32698 dodlive.mil. DKCaARZGr0Ui4BPpPi9NAXmnd308VswWekPgabugy3Hm0Ds2I9QicBcg eUwYHvc2WPTWqJOdxnX6+sITSyy4v90ME/T9JjAstEIVN6i2RJzgAEyG hhS7C+PQeWZ5t+XimtcwA7XWRSRCGcyK2XopiR5Fp2CpTndtz5NqItkV 51Q=

www.dodlive.mil.edgekey.net. 38 IN CNAME e5757.dscb.akamaiedge.net.

e5757.dscb.akamaiedge.net. 16 IN A 23.79.200.50

Note that the Verisign public resolver at 64.6.64.6 also returns SERVFAIL unless you pass +CD, same as Google Public DNS.

[dnsforum...@gmail.com]

I agree with you per the RFC that NSEC3 hash for the wildcard should be returned; however, the previous servers hosting this record also did not return the wildcard NSEC3 response when querying against a match for the wildcard CNAME.  Is it possible something within Google and Verisign changed over the last three weeks?  If not, I'm not certain this is the root cause of the issue.  I hadn't noticed that DNSVIZ returns a clean record for the wildcard itself; if you use a domain name like anything.dodlive.mil you should see the NSEC3 complaint as one of the errors.

Thanks for your response!

[Alexander Dupuy]

It would be unlikely for both Google Public DNS and Verisign's public resolver to have changed in that time, but as you probably never checked with them before, it's fairly likely that they always failed these wildcard CNAMEs.

There probably were no changes to Google Public DNS that affected this; more likely might be that the new servers have different TTLs for the NSEC3 records and/or the RRSIG records signing them. Since the behavior is out-of-spec for the RFC it probably doesn't matter so much about mitigating details that might have allowed Google Public DNS to successfully resolve the wildcard CNAMEs with the previous resolvers.

@alex

[Bryan Price]

On Wed, Mar 16, 2016 at 3:53 PM, 'Alexander Dupuy' via public-dns-discuss <public-dn...@googlegroups.com> wrote:

It would be unlikely for both Google Public DNS and Verisign's public resolver to have changed in that time, but as you probably never checked with them before, it's fairly likely that they always failed these wildcard CNAMEs.

There probably were no changes to Google Public DNS that affected this; more likely might be that the new servers have different TTLs for the NSEC3 records and/or the RRSIG records signing them. Since the behavior is out-of-spec for the RFC it probably doesn't matter so much about mitigating details that might have allowed Google Public DNS to successfully resolve the wildcard CNAMEs with the previous resolvers.

I've checked all three public DNS that check DNSSEC.

Nobody currently resolves it.

DNSSEC validated DNS only


C:\>nslookup anything.dodlive.mil 8.8.8.8
*** google-public-dns-a.google.com can't find anything.dodlive.mil: Server failed

Server:  google-public-dns-a.google.com
Address:  8.8.8.8


C:\>f:\bind\dig @8.8.8.8 anything.dodlive.mil

; <<>> DiG 9.10.1 <<>> @8.8.8.8 anything.dodlive.mil
; (1 server found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 50668
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512

;; QUESTION SECTION:
;anything.dodlive.mil.        IN    A

;; Query time: 186 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Mar 16 18:27:00 Eastern Daylight Time 2016
;; MSG SIZE  rcvd: 49


C:\>nslookup anything.dodlive.mil 75.75.75.75
*** cdns01.comcast.net can't find anything.dodlive.mil: Server failed

Server:  cdns01.comcast.net
Address:  75.75.75.75


C:\>f:\bind\dig @75.75.75.75 anything.dodlive.mil

; <<>> DiG 9.10.1 <<>> @75.75.75.75 anything.dodlive.mil
; (1 server found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 24134
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512

;; QUESTION SECTION:
;anything.dodlive.mil.        IN    A

;; Query time: 17 msec
;; SERVER: 75.75.75.75#53(75.75.75.75)
;; WHEN: Wed Mar 16 18:27:02 Eastern Daylight Time 2016
;; MSG SIZE  rcvd: 49


C:\>nslookup anything.dodlive.mil 64.6.64.6
*** recpubns1.nstld.net can't find anything.dodlive.mil: Server failed

Server:  recpubns1.nstld.net
Address:  64.6.64.6


C:\>f:\bind\dig @64.6.64.6 anything.dodlive.mil

; <<>> DiG 9.10.1 <<>> @64.6.64.6 anything.dodlive.mil
; (1 server found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 29957
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096

;; QUESTION SECTION:
;anything.dodlive.mil.        IN    A

;; Query time: 62 msec
;; SERVER: 64.6.64.6#53(64.6.64.6)
;; WHEN: Wed Mar 16 18:27:03 Eastern Daylight Time 2016
;; MSG SIZE  rcvd: 49


From Florida, USA ISP Comcast

[dnsforum...@gmail.com]

Hey Guys,

Appreciate the response and feedback.  With byte posting the additional failures on 64.6.64.6 I agree the problem is not limited to an issue with Google. (Not that it matters a ton but 75.75.75.75 worked for me without issue.)  Are you able to confirm that the RFC deviation when presenting the NSEC3 hash is definitely the reason the Goggle cluster is tossing the SERVFAIL?  That would be infinitely helpful in isolating the problem to a specific configuration.  Thanks again for all your help! 

[Bryan Price]

On Thu, Mar 17, 2016 at 7:04 AM, <dnsforum...@gmail.com> wrote:

Hey Guys,

Appreciate the response and feedback.  With byte posting the additional failures on 64.6.64.6 I agree the problem is not limited to an issue with Google. (Not that it matters a ton but 75.75.75.75 worked for me without issue.)  Are you able to confirm that the RFC deviation when presenting the NSEC3 hash is definitely the reason the Goggle cluster is tossing the SERVFAIL?  That would be infinitely helpful in isolating the problem to a specific configuration.  Thanks again for all your help! 

I've ran into the same thing on occasion.  Of course, I'm probably using something different than you are for the anything.  But yeah, I wish I could understand why Comcast doesn't always match up with the other two.

And this has quickly escalated out of my understanding.

[Alexander Dupuy]

(Not that it matters a ton but 75.75.75.75 worked for me without issue.)  Are you able to confirm that the RFC deviation when presenting the NSEC3 hash is definitely the reason the Goggle cluster is tossing the SERVFAIL?  That would be infinitely helpful in isolating the problem to a specific configuration.  Thanks again for all your help! 

I've ran into the same thing on occasion.  Of course, I'm probably using something different than you are for the anything.  But yeah, I wish I could understand why Comcast doesn't always match up with the other two.

It's odd that Comcast succeeded for the OP but failed for Bryan; I can't imagine how that would happen, unless it had cached data from the previous working configuration.

For the historical record and folks playing along at home, "dnsforumuser9281" escalated this to our issues tracker (this is OK, we do also collect statistics on issue reports, which we don't do for the forum, and issues are not publicly accessible, which allows people to be more forthcoming about their internal configurations, etcetera) and that extra information allowed me to provide a useful direction to follow.

Since issues are not searchable, and in the spirit of helping the next person who has this problem before they escalate to us, I am copying my response on the issue to the forum. I hope that "dnsforumuser9281" will share their final resolution on the forum as well.

@alex

"Windows" was the magic word you didn't offer on the forum.  With that, our mystical Google powers allow us to find this:

lmgtfy.com/?q=windows+dns+wildcard+dnssec+invalid&l=1

Reading the linked Microsoft notice it does seem quite relevant; although the particular problem you're having might or might not be fixed by this update. It would certainly make sense to start by applying this update, and if you continue to have problems, you should escalate with Microsoft support.  With all three major DNSSEC-validating resolvers failing your wildcard CNAMEs, it certainly seems like a bug they should fix (if their existing update doesn't already fix it).

As for [previous server] not failing even though it didn't return an NSEC record - it really depends on what the RRSIG was signing.  If a DNS server is generating the signature dynamically, using the queried domain name as the owner, the returned CNAME is not distinguishable from a non-wildcard response, and no NSEC/NSEC3 is required to validate it.  (Note that dynamic signature generation requires the private ZSK be accessible to the DNS server, and that it have support for dynamic signing.)  I don't know that either of those is the case with the RRSIG that your [previous] servers were (are) returning (the hex digits in the dig output are not particularly illuminating), but it is at least a plausible possibility.

The Windows-generated RRSIG for the CNAME is using the wildcard *.dodlive.mil as the owner, and in that case an NSEC/NSEC3 (with its own RRSIG) is required, but your Windows DNS server is not providing that.  Microsoft needs to fix that (and perhaps they already have).  Please let us know what ends up fixing the problem, so that we can add it to our FAQ for the next person who runs into this problem.

[dnsforum...@gmail.com]

Alex and the folks at Google DNS have been great, I apologize for not disclosing more details on the problem the private tracker was a great way to do that.  KB3133717 is applied on all servers.  It looks like AT&T and OpenDNS specifically discard DNSSEC responses so they aren't the best points of reference for my original post.  I'll circle back and let everyone know what the end conclusion is; but at this point I think we are confident that Google is not the fault point.  Thanks again for the time and effort on your end, really great to see how responsive Google's staff are!

[dnsforum...@gmail.com]

The MS DNS Service was the source of the problem.  KB 3185279 was released and fixes the issue where certain wildcard queries fail to produce the NSEC3 records.

https://support.microsoft.com/en-us/kb/3185279
https://support.microsoft.com/en-us/help/24717/windows-8-1-windows-server-2012-r2-update-history