BIND9 SERVFAIL Issue with Windows 2008 R2 DNS Server

[David Lam]

Hello all.

I was looking over a strange problem with BIND 9 when one of my
Windows 2008 R2 instances is pointed to it as a forwarder.
Specifically, when DNSSEC is turned on in BIND, some domain names fail
to resolve under specific circumstances. These problems resolve
spontaneously when switched to a public DNS server, like Google's
8.8.8.8.

Looking at this further, it appears when EDNS is turned on in the
Windows 2008 R2 DNS server (default, accepting DNSSEC responses),
resolution fails occasionally with a SERVFAIL when NODATA is returned
to BIND (i.e. 0 answers with a status code of NOERROR.)

For example, mx2.comcast.com type SRV will fail when looked up in the
Windows 2008 R2 DNS server pointed to BIND as a forwarder, but
bat.comcast.com type SRV works just fine.

Doing the query locally with dig, I get these results:

mx2.comcast.com SRV - Local BIND query

; <<>> DiG 9.9.2-P1 <<>> @127.0.0.1 mx2.comcast.com SRV +dnssec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42484
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;mx2.comcast.com. IN SRV

;; AUTHORITY SECTION:
mx2.comcast.com. 3600 IN RRSIG NSEC 5 3 3600
20130711200520 20130704170020 2643 comcast.com.
pmOHJX7dSNuFSRiFvxNIIuhQk/Sh6/9xSiZ2wj2I6RDKkrQlDScdFjDB
nSpeWt9068Wq+aQE36dbTsvyyCKgtrPcJIUxKVCtsXzTavXdx9XVGwG9
cKF6TrQx+MGPRwRwjPorDmPJxImveGMeE7X4Nl1mkGk/lRJwbvk1yFWV w1w=
mx2.comcast.com. 3600 IN NSEC mx3.comcast.com. A RRSIG NSEC
comcast.com. 3600 IN SOA dns101.comcast.net.
domregtech.comcastonline.com. 2009085823 7200 3600 1209600 3600
comcast.com. 3600 IN RRSIG SOA 5 2 3600
20130711200520 20130704170020 2643 comcast.com.
Te6jKcUXakWpPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni
QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x
DDQkdcJo2jWhNHGr6zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=

Same query, but made with Google's DNS server:

; <<>> DiG 9.9.2-P1 <<>> @8.8.8.8 mx2.comcast.com SRV +dnssec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3537
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;mx2.comcast.com. IN SRV

;; AUTHORITY SECTION:
comcast.com. 1800 IN SOA dns101.comcast.net.
domregtech.comcastonline.com. 2009085823 7200 3600 1209600 3600
comcast.com. 1800 IN RRSIG SOA 5 2 3600
20130711200520 20130704170020 2643 comcast.com.
Te6jKcUXakWpPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni
QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x
DDQkdcJo2jWhNHGr6zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=
mx2.comcast.com. 3600 IN NSEC mx3.comcast.com. A RRSIG NSEC
mx2.comcast.com. 3600 IN RRSIG NSEC 5 3 3600
20130711200520 20130704170020 2643 comcast.com.
pmOHJX7dSNuFSRiFvxNIIuhQk/Sh6/9xSiZ2wj2I6RDKkrQlDScdFjDB
nSpeWt9068Wq+aQE36dbTsvyyCKgtrPcJIUxKVCtsXzTavXdx9XVGwG9
cKF6TrQx+MGPRwRwjPorDmPJxImveGMeE7X4Nl1mkGk/lRJwbvk1yFWV w1w=

When using Windows with the BIND server as forwarder:

; <<>> DiG 9.9.3-P1 <<>> mx2.comcast.com SRV @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 57054
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;mx2.comcast.com. IN SRV

and with Google's DNS as forwarder:

; <<>> DiG 9.9.3-P1 <<>> mx2.comcast.com SRV @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56582
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;mx2.comcast.com. IN SRV

;; AUTHORITY SECTION:
comcast.com. 900 IN SOA dns101.comcast.net.
domregtech.comcastonline.com. 2009085823 7200 3600 1209600 3600

Now, trying this with bat.comcast.com:

; <<>> DiG 9.9.2-P1 <<>> @127.0.0.1 bat.comcast.com SRV +dnssec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2383
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;bat.comcast.com. IN SRV

;; AUTHORITY SECTION:
comcast.com. 1603 IN SOA dns101.comcast.net.
domregtech.comcastonline.com. 2009085823 7200 3600 1209600 3600
comcast.com. 1603 IN RRSIG SOA 5 2 3600
20130711200520 20130704170020 2643 comcast.com.
Te6jKcUXakWpPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni
QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x
DDQkdcJo2jWhNHGr6zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=
awrelaypool02.comcast.com. 1603 IN RRSIG NSEC 5 3 3600
20130711200520 20130704170020 2643 comcast.com.
U87nbvAj7j7pAk4kigqMyVy8XDeHqRP9756PTQsucrRTEchtScfBKWLl
Eo7cWJc4Vcsfept+ixg0IiAxpwHATqwNTmq/giAeglFfeFmMHlXrhdOl
Bl5myReo1gSXlpm0+bvinOFRek/MUlYGLvDAq17noJag2k1oXrvhaNBo qWo=
awrelaypool02.comcast.com. 1603 IN NSEC www.bat.comcast.com. A
RRSIG NSEC

and Google's resolver:

; <<>> DiG 9.9.2-P1 <<>> @8.8.8.8 bat.comcast.com SRV +dnssec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28253
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;bat.comcast.com. IN SRV

;; AUTHORITY SECTION:
comcast.com. 1800 IN SOA dns101.comcast.net.
domregtech.comcastonline.com. 2009085823 7200 3600 1209600 3600
comcast.com. 1800 IN RRSIG SOA 5 2 3600
20130711200520 20130704170020 2643 comcast.com.
Te6jKcUXakWpPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni
QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x
DDQkdcJo2jWhNHGr6zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=
awrelaypool02.comcast.com. 3600 IN NSEC www.bat.comcast.com. A
RRSIG NSEC
awrelaypool02.comcast.com. 3600 IN RRSIG NSEC 5 3 3600
20130711200520 20130704170020 2643 comcast.com.
U87nbvAj7j7pAk4kigqMyVy8XDeHqRP9756PTQsucrRTEchtScfBKWLl
Eo7cWJc4Vcsfept+ixg0IiAxpwHATqwNTmq/giAeglFfeFmMHlXrhdOl
Bl5myReo1gSXlpm0+bvinOFRek/MUlYGLvDAq17noJag2k1oXrvhaNBo qWo=

Once again with Windows (BIND Resolver):

; <<>> DiG 9.9.3-P1 <<>> bat.comcast.com SRV @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11140
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;bat.comcast.com. IN SRV

;; AUTHORITY SECTION:
comcast.com. 900 IN SOA dns101.comcast.net.
domregtech.comcastonline.com. 2009085823 7200 3600 1209600 3600

Once again with Windows (Google Resolver):

; <<>> DiG 9.9.3-P1 <<>> bat.comcast.com SRV @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22907
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;bat.comcast.com. IN SRV

;; AUTHORITY SECTION:
comcast.com. 900 IN SOA dns101.comcast.net.
domregtech.comcastonline.com. 2009085823 7200 3600 1209600 3600

Looking at these results, Windows resolution fails on mx2.comcast.com,
yet succeeds on bat.comcast.com, and from the error code that Windows
reported (SERVFAIL), it may seem initially that DNSSEC validation is
failing, although this was not the case since all of the query
responses had the 'ad' (authenticated) bit set on them. That said,
BIND appears to have an intriguing tendency to tamper with the order
in which the authority section RRs appear. Looking at the Google query
for mx2.comcast.com, we can see that the authority section appear in
this order (this is how the authoritative server respond too):

SOA
RRSIG - SOA
NSEC
RRSIG - NSEC

whereas BIND returns responses in this order:

RRSIG - NSEC
NSEC
SOA
RRSIG - SOA

For bat.comcast.com, Google responds in this order:

SOA
RRSIG - SOA
NSEC
RRSIG - NSEC

and BIND responds in this order:

SOA
RRSIG - SOA
RRSIG - NSEC
NSEC

Given that the first query failed in Windows yet the second works just
fine, it seems apparent that Windows 2008 R2 requires that the SOA
record appear first when there are no answers and return code =
NOERROR. (Do note that if the remote server returned a NXDOMAIN, then
the ordering of these RRs does not seem to matter, and Windows will
return NXDOMAIN accordingly back to the client).

Looking at the BIND documentation and see if there are any
configuration options that control the ordering of these RRs, but to
no avail. Here's what I have tried:

rfc2308-type1 yes;
minimal-responses yes;
rrset-order {order fixed;};

I have also tried upgrading the local BIND version to 9.9.3-P1 from
9.9.2-P1, but the behavior did not seem to have changed.

Lastly, I could theoretically disable EDNS support in Windows 2008 R2
as a workaround and have these queries work (since disabling EDNS will
also suppress the DO flag for DNSSEC, thus omitting the RRSIG and the
NSEC RRs in the response), although I would rather have leave EDNS
turned on for its efficiency over UDP.

Does anyone know anything that I am missing here, or ran into similar
situations?

Any comments would be greatly appreciated!

--
David Lam
Security Administrator
Information Educational Technology
dav...@ucdavis.edu
(530) 752-6971

[Spain, Dr. Jeffry A.]

> Looking at this further, it appears when EDNS is turned on in the Windows 2008 R2 DNS server (default, accepting DNSSEC responses), resolution fails occasionally with a SERVFAIL when NODATA is returned to BIND (i.e. 0 answers with a status code of NOERROR.)

I'm using Windows Server 2012 DNS with BIND 9.9.3 forwarders, and can't reproduce the issue. I tested "dig mx2.comcast.com srv +dnssec" and "dig bat.comcast.com srv +dnssec" against a Windows domain controller (simon) and its BIND 9.9.3 forwarder (nr1). All four queries, shown below, returned NOERROR. Perhaps this will provide you a useful basis for comparison in any event.

Jeffry A. Spain
Network Administrator
Cincinnati Country Day School

--------------------

Windows PowerShell
Copyright (C) 2012 Microsoft Corporation. All rights reserved.

PS C:\> dig '@simon' mx2.comcast.com srv +dnssec

; <<>> DiG 9.9.3 <<>> @simon mx2.comcast.com srv +dnssec
; (2 servers found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1927
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4000

;; QUESTION SECTION:
;mx2.comcast.com. IN SRV

;; AUTHORITY SECTION:

comcast.com. 899 IN SOA dns101.comcast.net. domregtech.comcastonline.com. 2009085823 7200 3600 1
209600 3600
comcast.com. 899 IN RRSIG SOA 5 2 3600 20130711200520 20130704170020 2643 comcast.com. Te6jKcUXakW

pPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x DDQkdcJo2jWhNHGr6
zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=

mx2.comcast.com. 899 IN NSEC mx3.comcast.com. A RRSIG NSEC

;; Query time: 31 msec
;; SERVER: 2001:4870:20ca:158:2c59:7bdf:ab15:4270#53(2001:4870:20ca:158:2c59:7bdf:ab15:4270)
;; WHEN: Sat Jul 06 21:12:35 Eastern Daylight Time 2013
;; MSG SIZE rcvd: 331

PS C:\> dig '@nr1' mx2.comcast.com srv +dnssec

; <<>> DiG 9.9.3 <<>> @nr1 mx2.comcast.com srv +dnssec
; (2 servers found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38367

;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;mx2.comcast.com. IN SRV

;; AUTHORITY SECTION:

mx2.comcast.com. 2173 IN RRSIG NSEC 5 3 3600 20130711200520 20130704170020 2643 comcast.com. pmOHJX7dSN

uFSRiFvxNIIuhQk/Sh6/9xSiZ2wj2I6RDKkrQlDScdFjDB nSpeWt9068Wq+aQE36dbTsvyyCKgtrPcJIUxKVCtsXzTavXdx9XVGwG9 cKF6TrQx+MGPRwRw
jPorDmPJxImveGMeE7X4Nl1mkGk/lRJwbvk1yFWV w1w=

mx2.comcast.com. 2173 IN NSEC mx3.comcast.com. A RRSIG NSEC
comcast.com. 2173 IN SOA dns101.comcast.net. domregtech.comcastonline.com. 2009085823 7200 3600 1
209600 3600
comcast.com. 2173 IN RRSIG SOA 5 2 3600 20130711200520 20130704170020 2643 comcast.com. Te6jKcUXakW

pPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x DDQkdcJo2jWhNHGr6
zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=

;; Query time: 46 msec
;; SERVER: 2001:4870:20ca:158:8c2f:b9ff:31f7:3836#53(2001:4870:20ca:158:8c2f:b9ff:31f7:3836)
;; WHEN: Sat Jul 06 21:12:46 Eastern Daylight Time 2013
;; MSG SIZE rcvd: 502

PS C:\> dig '@simon' bat.comcast.com srv +dnssec

; <<>> DiG 9.9.3 <<>> @simon bat.comcast.com srv +dnssec
; (2 servers found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26028
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4000

;; QUESTION SECTION:
;bat.comcast.com. IN SRV

;; AUTHORITY SECTION:
comcast.com. 900 IN SOA dns101.comcast.net. domregtech.comcastonline.com. 2009085823 7200 3600 1
209600 3600

comcast.com. 900 IN RRSIG SOA 5 2 3600 20130711200520 20130704170020 2643 comcast.com. Te6jKcUXakW

pPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x DDQkdcJo2jWhNHGr6
zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=

awrelaypool02.comcast.com. 900 IN NSEC www.bat.comcast.com. A RRSIG NSEC

;; Query time: 62 msec
;; SERVER: 2001:4870:20ca:158:2c59:7bdf:ab15:4270#53(2001:4870:20ca:158:2c59:7bdf:ab15:4270)
;; WHEN: Sat Jul 06 21:13:18 Eastern Daylight Time 2013
;; MSG SIZE rcvd: 349

PS C:\> dig '@nr1' bat.comcast.com srv +dnssec

; <<>> DiG 9.9.3 <<>> @nr1 bat.comcast.com srv +dnssec
; (2 servers found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60015

;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;bat.comcast.com. IN SRV

;; AUTHORITY SECTION:

comcast.com. 3583 IN SOA dns101.comcast.net. domregtech.comcastonline.com. 2009085823 7200 3600 1
209600 3600
comcast.com. 3583 IN RRSIG SOA 5 2 3600 20130711200520 20130704170020 2643 comcast.com. Te6jKcUXakW

pPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x DDQkdcJo2jWhNHGr6
zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=

awrelaypool02.comcast.com. 3583 IN RRSIG NSEC 5 3 3600 20130711200520 20130704170020 2643 comcast.com. U87nbvAj7j
7pAk4kigqMyVy8XDeHqRP9756PTQsucrRTEchtScfBKWLl Eo7cWJc4Vcsfept+ixg0IiAxpwHATqwNTmq/giAeglFfeFmMHlXrhdOl Bl5myReo1gSXlpm0
+bvinOFRek/MUlYGLvDAq17noJag2k1oXrvhaNBo qWo=
awrelaypool02.comcast.com. 3583 IN NSEC www.bat.comcast.com. A RRSIG NSEC

;; Query time: 46 msec
;; SERVER: 2001:4870:20ca:158:8c2f:b9ff:31f7:3836#53(2001:4870:20ca:158:8c2f:b9ff:31f7:3836)
;; WHEN: Sat Jul 06 21:13:36 Eastern Daylight Time 2013
;; MSG SIZE rcvd: 520

[David Lam]

Hi Jeff.
Thanks for the quick response.
I have tested this behavior on our test Windows 2012 Server instance,
and just like what you have found, the responses indeed return with a
NOERROR instead of a SERVFAIL. On the very same identical stock
configuration (except with forwarders set), Windows 2008 R2 fails with
a SERVFAIL as described in my email. Seemingly it looks like an oddity
with Windows 2008 R2 in terms of how the records are parsed, although
I still find it quite odd that BIND9 fiddles around with the ordering
of these RRs and get Windows confused in the first place.
Perhaps someone who has a Windows 2008 R2 domain can go ahead and
confirm this, but so far the only way I can see to mitigate this issue
is either:

1. Disable EDNS on Windows 2008 R2 (which essentially disables the
ability to accept DNSSEC based responses)
or
2. Disable DNSSEC support in BIND9 with dnssec-enable no; (setting
dnssec-validation no; has no effect)

Anyone here has any thoughts?
Thanks!
David

[Spain, Dr. Jeffry A.]

> Perhaps someone who has a Windows 2008 R2 domain can go ahead and confirm this, but so far the only way I can see to mitigate this issue is either:

> 1. Disable EDNS on Windows 2008 R2 (which essentially disables the ability to accept DNSSEC based responses) or 2. Disable DNSSEC support in BIND9 with dnssec-enable no; (setting dnssec-validation no; has no effect)

One additional comment on this: When I first set up a DNSSEC-enabled BIND 9 resolver and used it as a forwarder from Windows Server 2008 R2 DNS, I found that Windows DNS would return answers for known-bad queries, thus defeating the entire purpose of using a DNSSEC-enabled forwarder. DNSSEC-Tools maintains a test zone with various problematic records. See http://dnssec-tools.org/testzone/index.html. "dig badsign-A.test.dnssec-tools.org" issued to the BIND9 resolver returns a SERVFAIL response, as you would expect with an invalid RRSIG. The same query, however, issued to the domain controller returned the answer 75.119.216.33. This turned out to be happening because Windows DNS was actually sending its query as "dig badsign-A.test.dnssec-tools.org +dnssec +cdflag", in other words telling BIND not to perform DNSSEC validation. Based on a Microsoft tech support case that I opened, the only way to fix this was to turn off EDNS ("dnscmd /config /EnableEDnsProbes 0"). It turned out not to be possible to enable DNSSEC validation on the Windows domain controller itself, because the mechanism for entering the DNS root trust anchor was also broken.

What response do you get from your domain controller with "dig badsign-A.test.dnssec-tools.org"?

This also seems to have been fixed in Windows Server 2012.

Thanks. Jeff.

[David Lam]

> This turned out to be happening because Windows DNS was actually sending its query as "dig badsign-A.test.dnssec-tools.org +dnssec +cdflag", in other words telling BIND not to perform DNSSEC validation.

Agreed. Looking at a Wireshark capture, it does look like this was the
case with the Windows 2008 R2 query:

Flags: 0x0110 Standard query
0... .... .... .... = Response: Message is a query
.000 0... .... .... = Opcode: Standard query (0)
.... ..0. .... .... = Truncated: Message is not truncated
.... ...1 .... .... = Recursion desired: Do query recursively
.... .... .0.. .... = Z: reserved (0)
.... .... ...1 .... = Non-authenticated data: Acceptable <--
CD flag set here

with:

Additional records
<Root>: type OPT
Name: <Root>
Type: OPT (EDNS0 option)
UDP payload size: 4000
Higher bits in extended RCODE: 0x0
EDNS0 version: 0
Z: 0x8000
Bit 0 (DO bit): 1 (Accepts DNSSEC security RRs) <-- DO
flag set here
Bits 1-15: 0x0 (reserved)
Data length: 0

returning this as a result (non-validated, even with DNSSEC validation
turned on at the BIND side):

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18664
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;badsign-A.test.dnssec-tools.org. IN A

;; ANSWER SECTION:
badsign-A.test.dnssec-tools.org. 85968 IN A 75.119.216.33

whereas Windows 2012 does have the CD flag turned off by default:

Flags: 0x0100 Standard query
0... .... .... .... = Response: Message is a query
.000 0... .... .... = Opcode: Standard query (0)
.... ..0. .... .... = Truncated: Message is not truncated
.... ...1 .... .... = Recursion desired: Do query recursively
.... .... .0.. .... = Z: reserved (0)
.... .... ...0 .... = Non-authenticated data: Unacceptable <--
CD flag is off

Additional records
<Root>: type OPT
Name: <Root>
Type: OPT (EDNS0 option)
UDP payload size: 4000
Higher bits in extended RCODE: 0x0
EDNS0 version: 0
Z: 0x8000
Bit 0 (DO bit): 1 (Accepts DNSSEC security RRs) <-- DO
flag is on
Bits 1-15: 0x0 (reserved)
Data length: 0

and SERVFAIL returned:

; <<>> DiG 9.9.3-P1 <<>> @127.0.0.1 badsign-A.test.dnssec-tools.org
; (1 server found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 48614
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

> It turned out not to be possible to enable DNSSEC validation on the Windows domain controller itself, because the mechanism for entering the DNS root trust anchor was also broken.

Looks that way to me as well. From what I can see in R2, the only
trust anchor algorithm that is supported is RSA/SHA-1, not compatible
with the root's algorithm 8 (RSA/SHA-256). Looking at the algorithm
list in 2012 though, it seems like many new algorithms are now
supported.

> Based on a Microsoft tech support case that I opened, the only way to fix this was to turn off EDNS ("dnscmd /config /EnableEDnsProbes 0").

> This also seems to have been fixed in Windows Server 2012.

What a bummer, this essentially stops anyone from using DNSSEC
validation correctly on R2. And while DNSSEC validation is a useful
utility, what concerns me more is the inability for other
organizations / entities to be able to look up our DNSSEC signed
zones, especially with the fact that IPv6 is enabled by default on R2,
causing unanticipated service failures for these organizations /
entities.

Taking comcast.com as an example again, if Exchange was to send mail
to a @comcast.com address, it will do the following lookups:

; <<>> DiG 9.9.3-P1 <<>> @127.0.0.1 comcast.com MX
; (1 server found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19571
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 5

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;comcast.com. IN MX

;; ANSWER SECTION:
comcast.com. 600 IN MX 5 mx1.comcast.com.
comcast.com. 600 IN MX 5 mx4.comcast.com.
comcast.com. 600 IN MX 5 mx2.comcast.com.
comcast.com. 600 IN MX 5 mx3.comcast.com.

;; ADDITIONAL SECTION:
mx1.comcast.com. 3600 IN A 76.96.32.247
mx4.comcast.com. 7200 IN A 69.241.43.118
mx2.comcast.com. 7200 IN A 76.96.32.252
mx3.comcast.com. 7200 IN A 69.241.43.117

----

; <<>> DiG 9.9.3-P1 <<>> @127.0.0.1 mx1.comcast.com AAAA
; (1 server found)

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 13421
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;mx1.comcast.com. IN AAAA

----

And mail bounces in Exchange because DNS lookup for the mail server
supposedly "failed." (it never gets to try the A records themselves).
Unless the system administrator knows about this issue, I really doubt
they will go try disabling EDNS or IPv6. Instead I think our DNS
servers would be to blame for this issue (which I half agree, due to
the tampering of the order of the RRs by BIND9 returned).
Now just hoping that there is a directive that we can use to maintain
the authority section RRs' order.

[Spain, Dr. Jeffry A.]

>> Based on a Microsoft tech support case that I opened, the only way to fix this was to turn off EDNS ("dnscmd /config /EnableEDnsProbes 0").
>> This also seems to have been fixed in Windows Server 2012.

> What a bummer, this essentially stops anyone from using DNSSEC validation correctly on R2. And while DNSSEC validation is a useful utility, what concerns me more is the inability for other organizations / entities to be able to look up our DNSSEC signed zones, especially with the fact that IPv6 is enabled by default on R2, causing unanticipated service failures for these organizations / entities.

I think the best bet with Windows Server 2008 R2 DNS is to disable recursion, turn off EDNS ("dnscmd /config /EnableEDnsProbes 0"), and continue to use one or more DNSSEC-enabled BIND 9 recursive resolvers as a forwarders ("options { dnssec-validation auto; allow-query { domain-controllers; }; allow-recursion { domain-controllers; }; };"). If you do this, querying the domain controller with "dig badsign-A.test.dnssec-tools.org" does return a proper SERVFAIL response. DNSSEC-validation is being performed by the BIND resolver, but this is transparent to the Windows environment.

I have continued to do things this way with my Windows Server 2012 domain controllers, although as you pointed out, it hasn't been necessary to disable EDNS since the CD flag in queries from the domain controller to the forwarders is cleared by default in this version.

Back to your original question, I have a Windows Server 2008 R2 test VM available and so built a domain controller and attempted to confirm your findings with dig, shown below. All four dig queries returned NOERROR. The query "dig mx2.comcast.com srv +dnssec" caused the domain controller to query the forwarder, which returned the Authority records in the order shown below. This was confirmed by Wireshark, and is the same order as shown in your queries posted earlier. If I understand you correctly, this contradicts your hypothesis that Windows Server 2008 R2 DNS requires that the SOA record be returned first in the Authority section to avoid a SERVFAIL response.

Regards, Jeff.

--------------------

Windows PowerShell
Copyright (C) 2009 Microsoft Corporation. All rights reserved.

PS C:\> dig mx2.comcast.com srv +dnssec

; <<>> DiG 9.9.3 <<>> mx2.comcast.com srv +dnssec

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32036
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 1

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags: do; udp: 4000
;; QUESTION SECTION:
;mx2.comcast.com. IN SRV

;; AUTHORITY SECTION:

mx2.comcast.com. 60 IN NSEC mx3.comcast.com. A RRSIG NSEC
mx2.comcast.com. 3600 IN RRSIG NSEC 5 3 3600 20130711200520 20130704170020 2643 comcast.com. pmOHJX7dSN

uFSRiFvxNIIuhQk/Sh6/9xSiZ2wj2I6RDKkrQlDScdFjDB nSpeWt9068Wq+aQE36dbTsvyyCKgtrPcJIUxKVCtsXzTavXdx9XVGwG9 cKF6TrQx+MGPRwRw
jPorDmPJxImveGMeE7X4Nl1mkGk/lRJwbvk1yFWV w1w=

;; Query time: 124 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Jul 07 15:46:43 Eastern Daylight Time 2013
;; MSG SIZE rcvd: 252

PS C:\> dig '@2001:4870:20ca:158:8c2f:b9ff:31f7:3836' mx2.comcast.com srv +dnssec

; <<>> DiG 9.9.3 <<>> @2001:4870:20ca:158:8c2f:b9ff:31f7:3836 mx2.comcast.com srv +dnssec

; (1 server found)
;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48676
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;mx2.comcast.com. IN SRV

;; AUTHORITY SECTION:

mx2.comcast.com. 3600 IN RRSIG NSEC 5 3 3600 20130711200520 20130704170020 2643 comcast.com. pmOHJX7dSN

uFSRiFvxNIIuhQk/Sh6/9xSiZ2wj2I6RDKkrQlDScdFjDB nSpeWt9068Wq+aQE36dbTsvyyCKgtrPcJIUxKVCtsXzTavXdx9XVGwG9 cKF6TrQx+MGPRwRw
jPorDmPJxImveGMeE7X4Nl1mkGk/lRJwbvk1yFWV w1w=

mx2.comcast.com. 3600 IN NSEC mx3.comcast.com. A RRSIG NSEC
comcast.com. 3600 IN SOA dns101.comcast.net. domregtech.comcastonline.com. 2009085823 7200 3600 1
209600 3600
comcast.com. 3600 IN RRSIG SOA 5 2 3600 20130711200520 20130704170020 2643 comcast.com. Te6jKcUXakW

pPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x DDQkdcJo2jWhNHGr6
zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=

;; Query time: 78 msec

;; SERVER: 2001:4870:20ca:158:8c2f:b9ff:31f7:3836#53(2001:4870:20ca:158:8c2f:b9ff:31f7:3836)

;; WHEN: Sun Jul 07 15:48:32 Eastern Daylight Time 2013
;; MSG SIZE rcvd: 502

PS C:\> dig bat.comcast.com srv +dnssec

; <<>> DiG 9.9.3 <<>> bat.comcast.com srv +dnssec

;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49117
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 1

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags: do; udp: 4000
;; QUESTION SECTION:
;bat.comcast.com. IN SRV

;; AUTHORITY SECTION:
comcast.com. 900 IN SOA dns101.comcast.net. domregtech.comcastonline.com. 2009085823 7200 3600 1
209600 3600
comcast.com. 900 IN RRSIG SOA 5 2 3600 20130711200520 20130704170020 2643 comcast.com. Te6jKcUXakW
pPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x DDQkdcJo2jWhNHGr6
zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=
awrelaypool02.comcast.com. 900 IN NSEC www.bat.comcast.com. A RRSIG NSEC

;; Query time: 62 msec

;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Jul 07 15:48:49 Eastern Daylight Time 2013
;; MSG SIZE rcvd: 349

PS C:\> dig '@2001:4870:20ca:158:8c2f:b9ff:31f7:3836' bat.comcast.com srv +dnssec

; <<>> DiG 9.9.3 <<>> @2001:4870:20ca:158:8c2f:b9ff:31f7:3836 bat.comcast.com srv +dnssec

; (1 server found)
;; global options: +cmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30832
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;bat.comcast.com. IN SRV

;; AUTHORITY SECTION:

comcast.com. 3600 IN SOA dns101.comcast.net. domregtech.comcastonline.com. 2009085823 7200 3600 1
209600 3600
comcast.com. 3600 IN RRSIG SOA 5 2 3600 20130711200520 20130704170020 2643 comcast.com. Te6jKcUXakW

pPGQYpZICPShPZYEHHEcCnfFoof6VfOLPhhQP5MlWMbni QSQTY1UZLLCqU0j2U5n48wAMrSLSXoye+9W+pFnHtSl00fCQoQJ2ts+x DDQkdcJo2jWhNHGr6
zsP6y9clhLUkFRW7ZVdqCV62KtTumU8Qe4UOjNK R3s=

awrelaypool02.comcast.com. 3600 IN RRSIG NSEC 5 3 3600 20130711200520 20130704170020 2643 comcast.com. U87nbvAj7j

7pAk4kigqMyVy8XDeHqRP9756PTQsucrRTEchtScfBKWLl Eo7cWJc4Vcsfept+ixg0IiAxpwHATqwNTmq/giAeglFfeFmMHlXrhdOl Bl5myReo1gSXlpm0
+bvinOFRek/MUlYGLvDAq17noJag2k1oXrvhaNBo qWo=

awrelaypool02.comcast.com. 3600 IN NSEC www.bat.comcast.com. A RRSIG NSEC

;; Query time: 78 msec

;; SERVER: 2001:4870:20ca:158:8c2f:b9ff:31f7:3836#53(2001:4870:20ca:158:8c2f:b9ff:31f7:3836)

;; WHEN: Sun Jul 07 15:49:05 Eastern Daylight Time 2013
;; MSG SIZE rcvd: 520