Event 5504 when using root hints on Server 2008 R2
Andrew Hodgson
Since we moved our domain controllers onto Windows Server 2008 R2 last
week I tried to use the DNS servers to resolve external names using
root hints instead of relying on an old Linux server to do this.
The first problem I ran into was EDNS0 blocking on the firewall, which
I got resolved by speaking nicely to the firewall team, and asking
them to allow DNS packets larger than 512 bytes (they currently set
this to 1280 bytes, as per the default settings in the MS DNS server).
All seems to work well, but we then had issues resolving a specific
DNS name - www.lloydstsb.co.uk. This resolves fine when resolving
through the Linux DNS forwarder, but when doing this using the root
hints, we get the following in the event log:
Event Type: Information
Event Source: DNS
Event Category: None
Event ID: 5504
Date: 09/12/2009
Time: 13:29:07
User: N/A
Computer: [...]
Description:
The DNS server encountered an invalid domain name in a packet from
193.34.230.74. The packet will be rejected. The event data contains
the DNS packet.
For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.
Data:
0000: cb a6 80 11 00 00 00 00 ˦€.....
0008: 00 00 00 00 ....
The users trying to access that site get a DNS server failure response
from Squid (our proxy).
I did a dig +trace on the Linux machine and found out that the server
that was causing the problem was at ns3.lloydstsb.co.uk, and I also
got events fired for ns2.lloydstsb.co.uk. However, all the DNS
responses were under the 512 bytes.
Anyone seen this before? Is the Windows Server DNS service no good
for this type of usage? (the Server 2008 standard one had issues as
well!)
Thanks.
Andrew.
Ace Fekay [MCT]
> 0008: 00 00 00 00 ....
>
> The users trying to access that site get a DNS server failure response
> from Squid (our proxy).
>
> I did a dig +trace on the Linux machine and found out that the server
> that was causing the problem was at ns3.lloydstsb.co.uk, and I also
> got events fired for ns2.lloydstsb.co.uk. However, all the DNS
> responses were under the 512 bytes.
>
> Anyone seen this before? Is the Windows Server DNS service no good
> for this type of usage? (the Server 2008 standard one had issues as
> well!)
>
> Thanks.
> Andrew.
Hi Andrew,
Take a look at the following:
http://eventid.net/display.asp?eventid=5504&eventno=642&source=DNS&phase=1
A 5504 is indicative of an illegal character in the FQDN and not an
EDNS0 issue. Apparently BIND (assuming you are running BIND on LInux)
is ignoring the illegal character. If I may suggest, to set a forwarder
to your Linux box from your internal DNS servers. This way you control
the focal point for outside resolution through that box.
--
Ace
This posting is provided "AS-IS" with no warranties or guarantees and
confers no rights.
Please reply back to the newsgroup or forum for collaboration benefit
among responding engineers, and to help others benefit from your
resolution.
Ace Fekay, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007, MCSE &
MCSA 2003/2000, MCSA Messaging 2003
Microsoft Certified Trainer
For urgent issues, please contact Microsoft PSS directly. Please check
http://support.microsoft.com for regional support phone numbers.
Andrew Hodgson
<ace...@mvps.RemoveThisPart.org> wrote:
>Take a look at the following:
>http://eventid.net/display.asp?eventid=5504&eventno=642&source=DNS&phase=1
Yes saw this.
>
>A 5504 is indicative of an illegal character in the FQDN and not an
>EDNS0 issue. Apparently BIND (assuming you are running BIND on LInux)
>is ignoring the illegal character.
Yes we do use Bind9 as the Linux resolver. I am checking the DNS path
to see what is illegal, because I tried several DNS resolvers, and
they are all able to resolve the names ok.
>If I may suggest, to set a forwarder
>to your Linux box from your internal DNS servers. This way you control
>the focal point for outside resolution through that box.
I wanted to get away from this since the box is old, and I wanted the
system to be redundant. It looks like I may just use ISP forwarders.
Thanks.
Andrew.
Ace Fekay [MCT]
news:j8uvh5183hgnsf1nu...@news.giganews.com...
Normally ISP forwarders is the "standard" recommendation, so to speak. For
some that have an independent resolver internally, as I assumed the BIND box
was intended for, then we suggest to forward to that guy, then from him to
the ISP.
Let us know what you find.
Ace
Andrew Hodgson
<ace...@mvps.RemoveThisPart.org> wrote:
Forwarding to the ISP works perfectly. I find it irritating though
that the MS DNS server always seems to have problems using root hints
and no forwarders, any other DNS server I use seems to manage this
without issue.
Andrew.
Ace Fekay [MCT]
I wouldn't blame Windows DNS if it's generating an error because of illegal
characters?
Anyway, there is a DNAME record that Windows DNS does not support, depending
on the Service Pack level and operating system version. If the record that
it was trying to resolve does have legit characters but is trying to resolve
a DNAME record, then it may be a hotfix to address it. Read the following
for more info.
Event 5504 is logged when a Windows Server 2003-based DNS server ...Note
Event ID 5504 may also be logged because of other DNS packets. If event ID
5504 is related to a DNAME record, it can only be viewed in a network trace.
...
http://support.microsoft.com/kb/920162
InformIT: BIND 9's New Resource Records > DNAME, Domain AliasRFC2672
specifies the DNAME record. The RFC is titled "Non-Terminal DNS Name
Redirection," which means that DNAME is like CNAME, but it does not alias a
...
http://www.informit.com/articles/article.aspx?p=19798
Ace
J de Boyne Pollard
also, is what characters are in the domain name. Ironically, the
information for diagnosing the issue was in plain sight, in the event
log message itself. Well done, therefore, for following the standard
problem reporting litany and giving the error messages that you see.
Here's the important part:
AH> Data:
AH> 0000: cb a6 80 11 00 00 00 00  Ë¦€.....
AH> 0008: 00 00 00 00 Â Â Â Â Â Â Â ....
What's happening here is a combination of the fact that Lloyds TSB
uses content DNS servers that are not EDNS0 capable, and the fact that
you are using resolving proxy DNS servers that are EDNS0 capable. A
quick query to 193.34.230.74 reveals that if it receives an EDNS0
query it responds with a "format error" response, which is what the
above DNS/UDP datagram actually decodes to. So, first of all, if
Lloyds TSB were using better content DNS servers you wouldn't be
experiencing this problem. Exacerbating the problem is the fact that
Lloyds TSB's response doesn't repeat the question in the query. It is
this that is throwing off Microsoft's DNS server. It's expecting to
have the question that it asked echoed back to it in the response.
But as you can see from the final four words in the aforegiven data,
there are no resource records *at all* in the response that Lloyds
TSB's content DNS servers are returning. The question isn't being
echoed. So when Microsoft's DNS server comes to decode the question
section, to check that the question returned is the same as the
question asked, it finds no question at all, and complains.
The reasons that you don't see this with other resolving proxy DNS
servers are twofold: First, older resolving proxy DNS servers won't
be sending EDNS0 queries, so won't be triggering this response from
Lloyds TSB's content DNS servers. Second, some other resolving proxy
DNS server softwares are more liberal in their handling of the DNS
protocol than Micorosoft's DNS server is being here in this instance.
They won't even look at the question section of a "format error"
response. (The downside of this liberty is that such resolving proxy
DNS servers are slightly more vulnerable to spoof responses from blind
attackers, albeit only if the attacker wants to spoof "format error"
responses.)
AH> The users trying to access that site get a DNS server failure
AH> response from Squid (our proxy).
If you wish to have Microsoft's DNS server perform query resolution,
you really have a choice of three courses of action, probably none of
which you will find appealing, here:
* You could ask Lloyds TSB to get better content DNS servers, that
support EDNS0.
* You could ask Microsoft to make the resolving proxy part of its DNS
server more liberal when it comes to "format error" responses.
* You could tell your users that they aren't missing out on much by
not being able to use Lloyds TSB's WWW site.
The alternative is to have some other resolving proxy DNS server
software perform query resolution for that domain name and its
subdomains. This is where forwarding proxy DNS service, for
"lloydstsb.co.uk." and its subdomains, comes into play, as you've
already discussed. That's a local fix. But a service fix would be
better.
Ace Fekay [MCT]
Good find, Jonathan. What query command did you use with Dig to
determine this, or did you use nslookup?
Ace
Andrew Hodgson
wrote:
> DNAME records aren't the issue. Â That's a complete red herring. Â So,
> also, is what characters are in the domain name. Â Ironically, the
> information for diagnosing the issue was in plain sight, in the event
> log message itself. Â Well done, therefore, for following the standard
> problem reporting litany and giving the error messages that you see.
> Here's the important part:
>
> AH> Data:
> AH> 0000: cb a6 80 11 00 00 00 00  Ë¦€.....
> AH> 0008: 00 00 00 00 Â Â Â Â Â Â Â ....
This was why I posted the full log and not just the error code,
because I could see the DNS data in there, but didn't have sufficient
tuits to be able to decode it.
> What's happening here is a combination of the fact that Lloyds TSB
> uses content DNS servers that are not EDNS0 capable, and the fact that
> you are using resolving proxy DNS servers that are EDNS0 capable. Â A
> quick query to 193.34.230.74 reveals that if it receives an EDNS0
> query it responds with a "format error" response, which is what the
> above DNS/UDP datagram actually decodes to. Â So, first of all, if
> Lloyds TSB were using better content DNS servers you wouldn't be
> experiencing this problem. Â Exacerbating the problem is the fact that
> Lloyds TSB's response doesn't repeat the question in the query. Â It is
> this that is throwing off Microsoft's DNS server. Â It's expecting to
> have the question that it asked echoed back to it in the response.
> But as you can see from the final four words in the aforegiven data,
> there are no resource records *at all* in the response that Lloyds
> TSB's content DNS servers are returning. Â The question isn't being
> echoed. Â So when Microsoft's DNS server comes to decode the question
> section, to check that the question returned is the same as the
> question asked, it finds no question at all, and complains.
Thanks.
>
[snip useful info]
>
> AH> The users trying to access that site get a DNS server failure
> AH> response from Squid (our proxy).
>
> If you wish to have Microsoft's DNS server perform query resolution,
> you really have a choice of three courses of action, probably none of
> which you will find appealing, here:
> * You could ask Lloyds TSB to get better content DNS servers, that
> support EDNS0.
> * You could ask Microsoft to make the resolving proxy part of its DNS
> server more liberal when it comes to "format error" responses.
> * You could tell your users that they aren't missing out on much by
> not being able to use Lloyds TSB's WWW site.
>
> The alternative is to have some other resolving proxy DNS server
> software perform query resolution for that domain name and its
> subdomains. Â This is where forwarding proxy DNS service, for
> "lloydstsb.co.uk." and its subdomains, comes into play, as you've
> already discussed. Â That's a local fix. Â But a service fix would be
> better.
Yes, we could add a "conditional forwarder" for that domain, and still
use the root hints. I would be happy to do this, but would face
questions about whether this would happen in the future.
Could an alternative to be to switch off EDNS0 support in the DNS
server itself?
Thanks.
Andrew.
Ace Fekay [MCT]
I would suggest to simply use a Conditional Forwarder for the domain,
but a general forwarder to the ISP works fine. If that is the case, I
don't expect you to see the error pop up again, so there would be no
need to create a Conditional Fowarder or disabling EDNS0.
If "lloydstsb.co.uk" DNS servers are out of date, why should you back
pedal to accomodate them or any other entity that can't keep up with
industry standards, such as EDNS0, which has been around for at least 8
years now.
Your Forwarder to the ISP works. As I mentioned, that is normally the
receommended 'best practice' to configure. I wouldn't disable EDNS0 if
this is working.
Ace
J de Boyne Pollard
Merci.
AF> What query command did you use with Dig to
AF> determine this, or did you use nslookup?
Neither. I used my own tool. I just sent an "A" query for
"www.lloydstsb.co.uk." to 193.34.230.74 with EDNS0 large datagram
support enabled. I don't remember, off the top of my head, what the
option for enabling that in "dig" is, but that's all that one needs to
do. With my own tool, it's the /LARGEUDP option, which yielded the
following:
[c:\]dnsqry /serverip:193.34.230.74 /largeudp a www.lloydstsb.co.uk.
[0.0.0.0:0000] -> [193.34.230.74:0035] 48
Header: 0000 1+0+0+1, Q, , query, no_error
Question: www.lloydstsb.co.uk. IN A
Additional: . 0x7fff OPT 0
[193.34.230.74:0035] -> [0.0.0.0:0000] 12
Header: 0000 0+0+0+0, R, , query, bad_format
[c:\]
That's the same 12-octet response (except for message ID and some
reserved flag bits) as is given in the reported event log data. As
you can see, there's no question section in the response. The first
thing that I actually did, that I then double-checked with the above
manual query, was simply ask a non-Microsoft resolving proxy DNS
server to look up the "A" resource record set for
"www.lloydstsb.co.uk.", and read its log of back-end queries sent and
responses received. Here's the relevant portion of that log (which
Google Groups will probably mangle a bit -- every @ timestamp is a new
line):
@4b20580e 15d9 rx 7f000009 07e1 query: www.lloydstsb.co.uk. IN A
@4b20580e ed99 tx c707422c 0035 (uk.) 1+0+0+1 48 www.lloydstsb.co.uk.
IN A
@4b20580e ed99 rx c707422c 0035 (uk.) 1+0+2+3 116 www.lloydstsb.co.uk.
IN A
@4b20580e ed99 rx c707422c 0035 cached: lloydstsb.co.uk. IN NS 172800
ns3.lloydstsb.co.uk.
@4b20580e ed99 rx c707422c 0035 cached: lloydstsb.co.uk. IN NS 172800
ns2.lloydstsb.co.uk.
@4b20580e ed99 rx c707422c 0035 cached: ns3.lloydstsb.co.uk. IN A
172800 193.34.230.74
@4b20580e ed99 rx c707422c 0035 cached: ns2.lloydstsb.co.uk. IN A
172800 193.34.230.73
@4b20580e ed99 tx c122e649 0035 (lloydstsb.co.uk.) 1+0+0+1 48
www.lloydstsb.co.uk. IN A
@4b20580f ed99 rx c122e649 0035 server indicated a bad format (1/2
lame or failed)
@4b20580f ed99 tx c122e649 0035 (lloydstsb.co.uk.) 1+0+0+0 37
www.lloydstsb.co.uk. IN A
@4b20580f ed99 rx c122e649 0035 (lloydstsb.co.uk.) 1+1+2+2 136
www.lloydstsb.co.uk. IN A
@4b20580f ed99 rx c122e649 0035 cached: www.lloydstsb.co.uk. IN A 900
141.92.130.226
@4b20580f ed99 rx c122e649 0035 cached: lloydstsb.co.uk. IN NS 900
ns2.lloydstsb.co.uk.
@4b20580f ed99 rx c122e649 0035 cached: lloydstsb.co.uk. IN NS 900
ns3.lloydstsb.co.uk.
@4b20580f ed99 rx c122e649 0035 cached: ns2.lloydstsb.co.uk. IN A
86400 193.34.230.73
@4b20580f ed99 rx c122e649 0035 cached: ns3.lloydstsb.co.uk. IN A
86400 193.34.230.74
@4b20580f 15d9 tx 7f000009 07e1 1+1+0+0 53 ok: www.lloydstsb.co.uk.
900
The "bad format" response to the initial EDNS0 query is, as you can
see, received at 4b20580f. This is the point where Microsoft's DNS
server fails to find an echoed question in the response and aborts,
logging the event that you see. The non-Microsoft resolving proxy DNS
server doesn't mind empty question sections in "format error"
responses to EDNS0 queries, and so just proceeds to switch off EDNS0
and re-try the query.
Ace Fekay [MCT]
"J de Boyne Pollard" <J.deBoyn...@Tesco.NET> wrote in message
news:4810a244-11b2-43ae...@c3g2000yqd.googlegroups.com...
Thank you for the explanation. Yes, I see exactly what you mean where
Microsoft DNS drops the query request.
I am archiving this for if an when it may arise again in the future.
Thank you, once again!
Ace
Ace Fekay [MCT]
news:eD6xXobe...@TK2MSFTNGP04.phx.gbl...
Oh, just to add, when trying to determine if a server responds with EDNS0 or
not, I usually use nslookup with the "set vc" switch. I was curious how you
determined that, and it was interesting. Once again, I thank you.
Ace
J de Boyne Pollard
AH> and still use the root hints. Â I would be happy to do this, but
AH> would face questions about whether this would happen in the
AH> future.
It's my preference of local fix.
AH> Could an alternative to be to switch off EDNS0 support in
AH> the DNS server itself?
That's another local fix, but one that I view as less preferable, for
the simple reason that large DNS/UDP datagram support *is* beneficial,
in my experience. So I'm, personally, reluctant to disable it.
But, as I said before, a service fix is better than any local fix,
here. Really, Microsoft's DNS server should cope better with content
DNS servers like Lloyds TSB's content DNS servers. Doing so *does*
trade off some small measure of security for interoperability.
There's no argument about that. But when it comes to the security of
the DNS protocol as a whole, it's somewhat of a drop in the ocean.
The DNS protocol is massively insecure, and badly designed. *This*
particular insecurity simply permits blind attackers to easily forge
negative, "format error", responses. Blind attackers are more
interested in forging *positive* responses. So this insecurity is on
the level of handing so-called "script kiddies" a mechanism for
preventing a victim from finding "whitehouse.gov." and its subdomains,
rather than on the level of actually being able to impersonate the
U.S. government, as other flaws in the DNS protocol do.
The service fix would address the issue that you touch upon. Lloyds
TSB is not the only organization with poor content DNS servers. (It's
more embarrassing for Lloyds TSB than you paint it to be, Ace, by the
way. RFC 2671 is dated August 1999. So it's over 10 years that
Lloyds TSB has had to get its content DNS servers capable of at least
*parsing* EDNS0 queries, even if they simply ignore the extensions.)
But the positive side is that it's one of a *few* such organizations
that has poor content DNS service. If they were many, the trouble
that you experience would be more widely reported over the past
decade.
I've been tempted, in recent months, to start a content DNS service
"Hall of Shame", listing content DNS services that don't get the DNS
protocol right, or that are woefully inadequate in their handling of
the DNS protocol, to the extent of causing interoperability problems
with widespread secure resolving proxy DNS servers that necessitate
variances from the protocol. Lloyds TSB not including a question
section in its responses to EDNS0 queries would be the third on such a
list, after Google (whose content DNS servers erroneously stop halfway
through constructing responses) and Amazon (whose content DNS servers
in combination put CNAME resource records on a delegation point). I
haven't done so, yet. But perhaps it would raise awareness of exactly
how much *bad protocol* softwares like Microsoft's DNS server have to
be coded to cope with, and the security tradeoffs that are forced as a
result; and how flawed the DNS protocol itself really is.
Ace Fekay [MCT]
> AH> Yes, we could add a "conditional forwarder" for that domain,
Jonathan,
I would welcome seeing such a list. It would provide Internet community
awareness. Also, I didn't look it up, but thank you for the RFC EDNS0
note. I thought it was longer than 8 years. :-)
Ace
J de Boyne Pollard
I'll give it some further consideration. For what it's worth, I had
this particular issue with content DNS servers that respond in this
particular way to EDNS0 documented some five years ago, in another,
related but not quite the same, Hall of Shame.
<URL:http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/dns-
superdomain-owner-hall-of-shame.html#Irony>
The situation isn't quite as bad now as it was when I wrote that.
Witness the aforegiven log excerpt. The "uk." content DNS server at
199.7.66.44 responded quite happily to an EDNS0 query, without the
need for any fallback.
Ace Fekay [MCT]
I guess as time has passed, many DNS owners have upgraded to support EDNS0
functionality, albeit the few that are left out there that haven't such as
Lloyd's.
If you put the list together, I will definitely reference it in future,
related posts.
Thank you,
Ace
Kerry Brown
This may be relevant to the discussion. If an EDNS query fails R2 doesn't
fall back to DNS. I believe BIND does.
http://weblogs.asp.net/owscott/archive/2009/09/15/windows-server-2008-r2-dns-issues.aspx
Hopefully at some point an update will fix this.
--
Kerry Brown
MS-MVP - Windows Desktop Experience: Systems Administration
http://www.vistahelp.ca/phpBB2/
J de Boyne Pollard
Only inasmuch as they're behind us. (-: They didn't figure out the
*reason* that the fallback doesn't happen in this particular case,
which can be deduced from what Microsoft's DNS server writes in the
event log.
It would be interesting to hear whether M. Hodgson ever had problems
with looking up "bing.com.". The "bing.com." content DNS servers also
return "format error", but in contrast to Lloyds TSB's content DNS
servers, they echo the question asked when they do:
[c:\]dnsqry /serverip:64.4.59.173 /largeudp a www.bing.com.
[0.0.0.0:0000] -> [64.4.59.173:0035] 41
Header: 0000 1+0+0+1, Q, , query, no_error
Question: www.bing.com. IN A
Additional: . 0x7fff OPT 0
[64.4.59.173:0035] -> [0.0.0.0:0000] 41
Header: 0000 1+0+0+1, R, , query, bad_format
Question: www.bing.com. IN A
Additional: . 0x7fff OPT 0
[c:\]
Andrew Hodgson
<kerry@kdbNOSPAMsys-tems.c*a*m> wrote:
>This may be relevant to the discussion. If an EDNS query fails R2 doesn't
>fall back to DNS. I believe BIND does.
Yes, enabling all the logging on our Bind box shows that we are
falling back to using non EDNS0 requests for that domain. I can see
the format error response when I do the lookup. The local fix I did
was to use forwarders on the DNS servers, until MS can fix this issue,
as it doesn't involve modifying the defaults, so we can just implement
it without too much documentation :).
>
>http://weblogs.asp.net/owscott/archive/2009/09/15/windows-server-2008-r2-dns-issues.aspx
>
>Hopefully at some point an update will fix this.
Well, in 2008 we had this issue:
http://support.microsoft.com/kb/968372
That held us from using root hints as well. They never fixed that;
not even with Service Pack 2. Although, it does seem to fall back
from using EDNS0 when resolving www.lloydstsb.co.uk, unlike its R2
counterpart.
Windows Server 2003 R2 was the most reliable DNS server I used with
root hints, it did use EDNS0, but seemed to fall back ok, and was not
affected by the 2008 issue.
Thanks.
Andrew.
Andrew Hodgson
<J.deBoyn...@Tesco.NET> wrote:
>It would be interesting to hear whether M. Hodgson ever had problems
>with looking up "bing.com.". The "bing.com." content DNS servers also
>return "format error", but in contrast to Lloyds TSB's content DNS
>servers, they echo the question asked when they do:
I can test this on Monday in the office on my test machine.
This system was up for over a day without this issue, and I would be
surprised if we didn't have users that used Bing throughout some part
of the day.
Andrew.
Ace Fekay [MCT]
I haven't seen this issue yet, with 2008. I am aware of the MaxCacheTTL.
Adding the entry and a value (articles suggests 2 days), keeps items in the
cache for the time you set and ignores the TTL on the record. Normally DNS
deletes the items in cache when TTL expires. However, I don't see forcing
this entry to help resolve some TLDs, unless there's something else this
key does with EDNS0 that's not stated in the following article.
MaxCacheTtl
http://technet.microsoft.com/en-us/library/cc959926.aspx
Also funny thing - What happens if DNS can't resolve a record? Then there
will be no record to be placed in cache.
Ace
Andrew Hodgson
wrote:
> It would be interesting to hear whether M. Hodgson ever had problems
> with looking up "bing.com.". Â The "bing.com." content DNS servers also
> return "format error", but in contrast to Lloyds TSB's content DNS
> servers, they echo the question asked when they do:
I tried this this morning by setting up one of our resolving
nameservers out of the way of the clients path to use root hints, then
cleared the cache on the server. I looked up www.lloydstsb.co.uk. to
start with, and got the same error (and event log message), but www.bing.com.
resolved fine, with no event log errors:
andrewh@tws-lilac:~$ dig www.lloydstsb.co.uk @10.4.2.7
; <<>> DiG 9.4.2-P2.1 <<>> www.lloydstsb.co.uk @10.4.2.7
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 59329
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.lloydstsb.co.uk. IN A
;; Query time: 3195 msec
;; SERVER: 10.4.2.7#53(10.4.2.7)
;; WHEN: Mon Dec 14 08:18:35 2009
;; MSG SIZE rcvd: 37
andrewh@tws-lilac:~$ dig www.bing.com @10.4.2.7
; <<>> DiG 9.4.2-P2.1 <<>> www.bing.com @10.4.2.7
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34664
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.bing.com. IN A
;; ANSWER SECTION:
www.bing.com. 3599 IN CNAME
search.ms.com.edgesuite.net.
search.ms.com.edgesuite.net. 21595 IN CNAME a134.g.akamai.net.
a134.g.akamai.net. 15 IN A 92.122.126.232
a134.g.akamai.net. 15 IN A 92.122.127.25
;; Query time: 3610 msec
;; SERVER: 10.4.2.7#53(10.4.2.7)
;; WHEN: Mon Dec 14 08:19:06 2009
;; MSG SIZE rcvd: 131
So it appears that we don't have any issues with resolving Bing, which
makes the above blog posting really strange :).
Let me know if you want me to test any other domains, since it isn't
too difficult to get this set up again.
Thanks.
Andrew.
J de Boyne Pollard
AF> I haven't seen this issue yet, with 2008. [...] I don't see
forcing
AF> this entry to help resolve some TLDs, Â unless there's something
AF> else this key does with EDNS0 that's not stated in the following
AF> article.
It's a bodge. It has all of the hallmarks of one. Stopping things
from being cached for more than 2 days prevents something as a side-
effect. It's not obvious what, though. My educated guess would be a
problem with caching the "glue" of the affected delegation points, but
a very quick review of "br." doesn't turn up anything that would
obviously cause a problem.
That's really a quite poor KB article when it comes to explaining what
the problem being bodged around is.
AF> Also funny thing - What happens if DNS can't resolve a record?
AF> Then there will be no record to be placed in cache.
Define "can't resolve". There's a different between the resolution
algorithm having to be aborted and the receipt of a negative answer.
The latter is most definitely cacheable. Remember: The database
schema that content and proxy DNS servers use isn't the schema used in
the DNS protocol. In the *true* DNS database schema, the one that DNS
server (and indeed DNS client) software writers actually have to
employ, resource records aren't considered individually, and empty
resource record sets and "no such name" indicators are concrete
things, with TTLs of their own.
J de Boyne Pollard
JdeBP> ever had problems with looking up "bing.com.". Â The
JdeBP> "bing.com." content DNS servers also return
JdeBP> "format error", but in contrast to Lloyds TSB's content
JdeBP> DNS servers, they echo the question asked when they do:
AH> [...] www.bing.com. resolved fine, with no event log errors:
AH> [...]
AH> So it appears that we don't have any issues with resolving
AH> Bing, which makes the above blog posting really strange :).
It makes it wrong. (-:
But it's independent confirmation of what I said. The problem here is
that (the resolving proxy part of) Microsoft's DNS server cannot
handle the missing question sections in responses from Lloyds TSB's
content DNS servers (and others), causing it to throw away the
response before it gets to the part of its algorithm that recognizes
an EDNS0 probe failure and the need for fallback. It doesn't have
this problem with the "bing.com." content DNS servers.
AH> Let me know if you want me to test any other domains,
AH> since it isn't too difficult to get this set up again.
I'll have a quick look around for some more content DNS servers like
Lloyds TSB's content DNS servers and like the "bing.com." content DNS
servers. But it's fairly conclusive with the results that we have.
Ace Fekay [MCT]
>
> AF> I haven't seen this issue yet, with 2008. [...] I don't see
> forcing
> AF> else this key does with EDNS0 that's not stated in the following
> AF> article.
>
> It's a bodge. It has all of the hallmarks of one. Stopping things
> from being cached for more than 2 days prevents something as a side-
> effect. It's not obvious what, though. My educated guess would be a
> problem with caching the "glue" of the affected delegation points, but
> a very quick review of "br." doesn't turn up anything that would
> obviously cause a problem.
Interesting. Problems with caching the glue may help with overriding
the TTL, but then again, that indicates it would need to be able to
resolve the glue, and perhaps if Lloyd's TTBS server won't respond to
that, then the chicken/egg theory comes into play.
>
> That's really a quite poor KB article when it comes to explaining what
> the problem being bodged around is.
I agree it is a bodge - hastily put together to address an issue
without explaining why it works or why it was suggested as a
workaround, and in this case, not necessarily a 'fix,' otherwise we
would have seen a 'dns.exe' hotfix.
>
> AF> Also funny thing - What happens if DNS can't resolve a record?
> AF> Then there will be no record to be placed in cache.
>
> Define "can't resolve". There's a different between the resolution
> algorithm having to be aborted and the receipt of a negative answer.
> The latter is most definitely cacheable.
It was more of a general statement such as if the Lloyd's A record is
not resolvable due to Lloyd's TTBS servers, e.g. not able to get a
response, then there would be no "A" record to cache, however, good
point that the negative response is cached.
> Remember: The database
> schema that content and proxy DNS servers use isn't the schema used in
> the DNS protocol. In the *true* DNS database schema, the one that DNS
> server (and indeed DNS client) software writers actually have to
> employ, resource records aren't considered individually, and empty
> resource record sets and "no such name" indicators are concrete
> things, with TTLs of their own.
Understood. As stated, I was making more of a general statement.
Ace
J de Boyne Pollard
AH> since it isn't too difficult to get this set up again.
JdeBP> I'll have a quick look around for some more content
JdeBP> DNS servers like Lloyds TSB's content DNS servers
JdeBP> and like the "bing.com." content DNS servers. Â But
JdeBP> it's fairly conclusive with the results that we have.
I've just searched my last 24 hours' logs for EDNS0 fallbacks caused
by "bad format" responses.
The "mii.instacontent.net." content DNS servers also reply "bad
format" to EDNS0 errors. But since they, too, echo the question in
their responses, you shouldn't have any trouble with, say, an "A"
lookup for "ant.mii.instacontent.net.".
J de Boyne Pollard
AF> I haven't seen this issue yet, with 2008. [...] I don't see
forcing
AF> this entry to help resolve some TLDs, unless there's something
AF> else this key does with EDNS0 that's not stated in the following
AF> article.
JdeBP> [...] Â It's not obvious what, though. Â My educated guess
JdeBP> would be a problem with caching the "glue" of the affected
JdeBP> delegation points, but a very quick review of "br." doesn't
JdeBP> turn up anything that would obviously cause a problem.
AF> Interesting. Problems with caching the glue may help with
AF> overriding the TTL, but then again, that indicates it would
AF> need to be able to resolve the glue, and perhaps if Lloyd's
AF> TSB's server won't respond to that, then the chicken/egg
AF> theory comes into play.
M. Hodgson said that he had the KB968372 problem last year. He didn't
say what domain names were affected, but the article gives "br." as
one example. It's not related to the Lloyds TSB problem, as far as I
can see. Some further research after I wrote the above indicates that
the cache not handling "glue" properly is indeed the KB968372
problem. (That one Microsoft employee suggested tweaking a
"LameDelegationTtl" property as a local fix was one clue. It's still
not clear what the actual problem is, though.) It's not a response
parsing issue, as the current Lloyds TSB problem is.
M. Hodgson's point, I believe, was that since we haven't yet seen a
service fix (or even a temporary fix) for the KB968372 problem,
whatever it is, we probably won't see a service fix for *this* problem
in the near future.
Ace Fekay [MCT]
>
> AF> I haven't seen this issue yet, with 2008. [...] I don't see
> forcing
> AF> this entry to help resolve some TLDs, unless there's something
> AF> else this key does with EDNS0 that's not stated in the following
> AF> article.
>
> JdeBP> would be a problem with caching the "glue" of the affected
> JdeBP> delegation points, but a very quick review of "br." doesn't
> JdeBP> turn up anything that would obviously cause a problem.
>
> AF> Interesting. Problems with caching the glue may help with
> AF> overriding the TTL, but then again, that indicates it would
> AF> need to be able to resolve the glue, and perhaps if Lloyd's
> AF> TSB's server won't respond to that, then the chicken/egg
> AF> theory comes into play.
>
> M. Hodgson said that he had the KB968372 problem last year. He didn't
> say what domain names were affected, but the article gives "br." as
> one example. It's not related to the Lloyds TSB problem, as far as I
> can see. Some further research after I wrote the above indicates that
> the cache not handling "glue" properly is indeed the KB968372
> problem. (That one Microsoft employee suggested tweaking a
> "LameDelegationTtl" property as a local fix was one clue. It's still
> not clear what the actual problem is, though.) It's not a response
> parsing issue, as the current Lloyds TSB problem is.
>
> M. Hodgson's point, I believe, was that since we haven't yet seen a
> service fix (or even a temporary fix) for the KB968372 problem,
> whatever it is, we probably won't see a service fix for *this* problem
> in the near future.
I agree that I don't think we will ever see a service fix for this
assuming that the engineers believe the problem is on the out of date
DNS servers, and should have been updated long ago. Maybe someone
should email Lloyds' to let them know.
Thanks for the info on the LameDelegationTtl explanation. It makes
sense that it's truly a glue problem.
Ace
J de Boyne Pollard
If you enjoyed that, you'll like these:
<URL:http://msdn.microsoft.com/en-us/library/cc422472(PROT.10).aspx>
<URL:http://msdn.microsoft.com/en-us/library/cc422509(PROT.10).aspx>
Ace Fekay [MCT]
news:ce9ac157-a192-4c77...@o28g2000yqh.googlegroups.com...
Interesting, thank you. I'll have to take a few moments to read them,
especially the Product Behavior link.
Ace
Andrew Hodgson
That is correct. I had the issue with mainly .co.uk addresses, and
our users couldn't resolve them (giving a servfail response), yet
other domains would still resolve fine. It was a completely separate
issue to the lloydstsb.co.uk issue I was talking about before.
Since I never understood what the registry fix would actually achieve,
I couldn't recommend implementing it in the live environment, so we
used a Linux machine running Bind to work round the issue instead. I
was disappointed not to receive any hot fix for this, and I was even
more disappointed when this didn't get fixed in SP2.
>
>M. Hodgson's point, I believe, was that since we haven't yet seen a
>service fix (or even a temporary fix) for the KB968372 problem,
>whatever it is, we probably won't see a service fix for *this* problem
>in the near future.
Yes, that is my point (although this problem has so far only affected
one domain that I know of).
Thanks.
Andrew.
Epstein@discussions.microsoft.com Lyle Epstein
Here is another very well known ISP that has the same issue: Earthlink.net.
When we attempt to send mail to one of their hosted customers,
bronsoncos.com which uses Earthlink.net as it's ISP, you get the 5504 event
ID on Windows 2008 R2.
Perhaps you can add them to your list of companies that don't comply!
Ace Fekay [MVP-DS, MCT]
in message news:IU.20100115....@J.de.Boyne.Pollard.localhost...
> I've been tempted, in recent months, to start a content DNS service
> "Hall of Shame", listing content DNS services that don't get the DNS
> protocol right, or that are woefully inadequate in their handling of the
> DNS protocol, to the extent of causing interoperability problems with
> widespread secure resolving proxy DNS servers that necessitate variances
> from the protocol. Lloyds TSB not including a question section in its
> responses to EDNS0 queries would be the third on such a list, after Google
> (whose content DNS servers erroneously stop halfway through constructing
> responses) and Amazon (whose content DNS servers in combination put CNAME
> resource records on a delegation point). I haven't done so, yet. But
> the security tradeoffs that are forced as a result; and how flawed the DNS
> protocol itself really is.
>
>
> Earthlink.net. When we attempt to send mail to one of their hosted
> customers, bronsoncos.com which uses Earthlink.net as it's ISP, you get
> the 5504 event ID on Windows 2008 R2.
>
> logs this problem thrice for the DNS lookups involved in sending mail.�
> This is because of the intermediate domain names:
>
>
> [C:\]dnsqry mx bronsoncos.com.|grep /b/u Answer:
> Answer: bronsoncos.com. IN MX 86066 10 store-forward.mspring.net.
> Answer: bronsoncos.com. IN MX 86066 5 mail.bronsoncos.com.
> Looking up both of the intermediate domain names mail.bronsoncos.com. and
> store-forward.mspring.net. results in A queries to the self-same content
> DNS servers that received the MX query.� All three queries will receive
> "bad format" responses with zero question section resource records.
>
>
> Perhaps you can add them to your list of companies that don't comply!
>
>
>
I thought you already did?
Ace
Ben Uhlig
Give that a shot if you have this issue, it might save you some time.
Ben
> 0000: cb a6 80 11 00 00 00 00 =CB=A6=80.....
> 0008: 00 00 00 00 ....
>
> The users trying to access that site get a DNS server failure response
> from Squid (our proxy).
>
> I did a dig +trace on the Linux machine and found out that the server
> that was causing the problem was at ns3.lloydstsb.co.uk, and I also
> got events fired for ns2.lloydstsb.co.uk. However, all the DNS
> responses were under the 512 bytes.
>
> Anyone seen this before? Is the Windows Server DNS service no good
> for this type of usage? (the Server 2008 standard one had issues as
> well!)
>
> Thanks.
> Andrew.
>> On Wednesday, December 09, 2009 11:21 AM Ace Fekay [MCT] wrote:
>> Hi Andrew,
>>
>> Take a look at the following:
>> http://eventid.net/display.asp?eventid=5504&eventno=642&source=DNS&phase=1
>>
>> A 5504 is indicative of an illegal character in the FQDN and not an
>> EDNS0 issue. Apparently BIND (assuming you are running BIND on LInux)
>> is ignoring the illegal character. If I may suggest, to set a forwarder
>> to your Linux box from your internal DNS servers. This way you control
>> the focal point for outside resolution through that box.
>>
>> --
>> Ace
>>
>> This posting is provided "AS-IS" with no warranties or guarantees and
>> confers no rights.
>>
>> Please reply back to the newsgroup or forum for collaboration benefit
>> among responding engineers, and to help others benefit from your
>> resolution.
>>
>> Ace Fekay, MCT, MCITP EA, MCTS Windows 2008 & Exchange 2007, MCSE &
>> MCSA 2003/2000, MCSA Messaging 2003
>> Microsoft Certified Trainer
>>
>> For urgent issues, please contact Microsoft PSS directly. Please check
>> http://support.microsoft.com for regional support phone numbers.
>>> On Wednesday, December 09, 2009 2:25 PM Andrew Hodgson wrote:
>>> Yes saw this.
>>>
>>> Yes we do use Bind9 as the Linux resolver. I am checking the DNS path
>>> to see what is illegal, because I tried several DNS resolvers, and
>>> they are all able to resolve the names ok.
>>>
>>>
>>> I wanted to get away from this since the box is old, and I wanted the
>>> system to be redundant. It looks like I may just use ISP forwarders.
>>>
>>> Thanks.
>>> Andrew.
>>>> On Wednesday, December 09, 2009 3:45 PM Ace Fekay [MCT] wrote:
>>>> Normally ISP forwarders is the "standard" recommendation, so to speak. For
>>>> some that have an independent resolver internally, as I assumed the BIND box
>>>> was intended for, then we suggest to forward to that guy, then from him to
>>>> the ISP.
>>>>
>>>> Let us know what you find.
>>>>
>>>> Ace
>>>>> On Wednesday, December 09, 2009 6:02 PM Andrew Hodgson wrote:
>>>>> Forwarding to the ISP works perfectly. I find it irritating though
>>>>> that the MS DNS server always seems to have problems using root hints
>>>>> and no forwarders, any other DNS server I use seems to manage this
>>>>> without issue.
>>>>>
>>>>> Andrew.
>>>>>> On Wednesday, December 09, 2009 7:47 PM Ace Fekay [MCT] wrote:
>>>>>> I would not blame Windows DNS if it is generating an error because of illegal
>>>>>> characters?
>>>>>>
>>>>>> Anyway, there is a DNAME record that Windows DNS does not support, depending
>>>>>> on the Service Pack level and operating system version. If the record that
>>>>>> it was trying to resolve does have legit characters but is trying to resolve
>>>>>> a DNAME record, then it may be a hotfix to address it. Read the following
>>>>>> for more info.
>>>>>>
>>>>>> Event 5504 is logged when a Windows Server 2003-based DNS server ...Note
>>>>>> Event ID 5504 may also be logged because of other DNS packets. If event ID
>>>>>> 5504 is related to a DNAME record, it can only be viewed in a network trace.
>>>>>> ...
>>>>>> http://support.microsoft.com/kb/920162
>>>>>>
>>>>>> InformIT: BIND 9's New Resource Records > DNAME, Domain AliasRFC2672
>>>>>> specifies the DNAME record. The RFC is titled "Non-Terminal DNS Name
>>>>>> Redirection," which means that DNAME is like CNAME, but it does not alias a
>>>>>> ...
>>>>>> http://www.informit.com/articles/article.aspx?p=19798
>>>>>>
>>>>>> Ace
>>>>>>> On Wednesday, December 09, 2009 10:20 PM Ace Fekay [MCT] wrote:
>>>>>>> Good find, Jonathan. What query command did you use with Dig to
>>>>>>> determine this, or did you use nslookup?
>>>>>>>
>>>>>>> Ace
>>>>>>>> On Thursday, December 10, 2009 3:03 AM J de Boyne Pollard wrote:
>>>>>>>> DNAME records are not the issue. That's a complete red herring. So,
>>>>>>>> also, is what characters are in the domain name. Ironically, the
>>>>>>>> information for diagnosing the issue was in plain sight, in the event
>>>>>>>> log message itself. Well done, therefore, for following the standard
>>>>>>>> problem reporting litany and giving the error messages that you see.
>>>>>>>> Here is the important part:
>>>>>>>>
>>>>>>>> AH> Data:
>>>>>>>> AH> 0000: cb a6 80 11 00 00 00 00 =A0 =CB=A6=80.....
>>>>>>>> AH> 0008: 00 00 00 00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ....
>>>>>>>>
>>>>>>>> What's happening here is a combination of the fact that Lloyds TSB
>>>>>>>> uses content DNS servers that are not EDNS0 capable, and the fact that
>>>>>>>> you are using resolving proxy DNS servers that are EDNS0 capable. A
>>>>>>>> quick query to 193.34.230.74 reveals that if it receives an EDNS0
>>>>>>>> query it responds with a "format error" response, which is what the
>>>>>>>> above DNS/UDP datagram actually decodes to. So, first of all, if
>>>>>>>> Lloyds TSB were using better content DNS servers you would not be
>>>>>>>> experiencing this problem. Exacerbating the problem is the fact that
>>>>>>>> Lloyds TSB's response does not repeat the question in the query. It is
>>>>>>>> this that is throwing off Microsoft's DNS server. it is expecting to
>>>>>>>> have the question that it asked echoed back to it in the response.
>>>>>>>> But as you can see from the final four words in the aforegiven data,
>>>>>>>> there are no resource records *at all* in the response that Lloyds
>>>>>>>> TSB's content DNS servers are returning. The question is not being
>>>>>>>> echoed. So when Microsoft's DNS server comes to decode the question
>>>>>>>> section, to check that the question returned is the same as the
>>>>>>>> question asked, it finds no question at all, and complains.
>>>>>>>>
>>>>>>>> The reasons that you do not see this with other resolving proxy DNS
>>>>>>>> servers are twofold: First, older resolving proxy DNS servers will not
>>>>>>>> be sending EDNS0 queries, so will not be triggering this response from
>>>>>>>> Lloyds TSB's content DNS servers. Second, some other resolving proxy
>>>>>>>> DNS server softwares are more liberal in their handling of the DNS
>>>>>>>> protocol than Micorosoft's DNS server is being here in this instance.
>>>>>>>> They will not even look at the question section of a "format error"
>>>>>>>> response. (The downside of this liberty is that such resolving proxy
>>>>>>>> DNS servers are slightly more vulnerable to spoof responses from blind
>>>>>>>> attackers, albeit only if the attacker wants to spoof "format error"
>>>>>>>> responses.)
>>>>>>>>
>>>>>>>> AH> The users trying to access that site get a DNS server failure
>>>>>>>> AH> response from Squid (our proxy).
>>>>>>>>
>>>>>>>> If you wish to have Microsoft's DNS server perform query resolution,
>>>>>>>> you really have a choice of three courses of action, probably none of
>>>>>>>> which you will find appealing, here:
>>>>>>>> * You could ask Lloyds TSB to get better content DNS servers, that
>>>>>>>> support EDNS0.
>>>>>>>> * You could ask Microsoft to make the resolving proxy part of its DNS
>>>>>>>> server more liberal when it comes to "format error" responses.
>>>>>>>> * You could tell your users that they are not missing out on much by
>>>>>>>> not being able to use Lloyds TSB's WWW site.
>>>>>>>>
>>>>>>>> The alternative is to have some other resolving proxy DNS server
>>>>>>>> software perform query resolution for that domain name and its
>>>>>>>> subdomains. This is where forwarding proxy DNS service, for
>>>>>>>> "lloydstsb.co.uk." and its subdomains, comes into play, as you have
>>>>>>>> already discussed. That's a local fix. But a service fix would be
>>>>>>>> better.
>>>>>>>>> On Thursday, December 10, 2009 9:53 AM Andrew Hodgson wrote:
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> This was why I posted the full log and not just the error code,
>>>>>>>>> because I could see the DNS data in there, but did not have sufficient
>>>>>>>>> tuits to be able to decode it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>> [snip useful info]
>>>>>>>>>
>>>>>>>>> Yes, we could add a "conditional forwarder" for that domain, and still
>>>>>>>>> use the root hints. I would be happy to do this, but would face
>>>>>>>>> questions about whether this would happen in the future.
>>>>>>>>>
>>>>>>>>> Could an alternative to be to switch off EDNS0 support in the DNS
>>>>>>>>> server itself?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>> Andrew.
>>>>>>>>>> On Thursday, December 10, 2009 11:07 AM Ace Fekay [MCT] wrote:
>>>>>>>>>> I would suggest to simply use a Conditional Forwarder for the domain,
>>>>>>>>>> but a general forwarder to the ISP works fine. If that is the case, I
>>>>>>>>>> do not expect you to see the error pop up again, so there would be no
>>>>>>>>>> need to create a Conditional Fowarder or disabling EDNS0.
>>>>>>>>>>
>>>>>>>>>> If "lloydstsb.co.uk" DNS servers are out of date, why should you back
>>>>>>>>>> pedal to accomodate them or any other entity that cannot keep up with
>>>>>>>>>> industry standards, such as EDNS0, which has been around for at least 8
>>>>>>>>>> years now.
>>>>>>>>>>
>>>>>>>>>> Your Forwarder to the ISP works. As I mentioned, that is normally the
>>>>>>>>>> receommended 'best practice' to configure. I would not disable EDNS0 if
>>>>>>>>>> this is working.
>>>>>>>>>>
>>>>>>>>>> Ace
>>>>>>>>>>> On Thursday, December 10, 2009 11:59 AM Ace Fekay [MCT] wrote:
>>>>>>>>>>> Thank you for the explanation. Yes, I see exactly what you mean where
>>>>>>>>>>> Microsoft DNS drops the query request.
>>>>>>>>>>>
>>>>>>>>>>> I am archiving this for if an when it may arise again in the future.
>>>>>>>>>>>
>>>>>>>>>>> Thank you, once again!
>>>>>>>>>>>
>>>>>>>>>>> Ace
>>>>>>>>>>>> On Thursday, December 10, 2009 12:04 PM Ace Fekay [MCT] wrote:
>>>>>>>>>>>> Oh, just to add, when trying to determine if a server responds with EDNS0 or
>>>>>>>>>>>> not, I usually use nslookup with the "set vc" switch. I was curious how you
>>>>>>>>>>>> determined that, and it was interesting. Once again, I thank you.
>>>>>>>>>>>>
>>>>>>>>>>>> Ace
>>>>>>>>>>>>> On Thursday, December 10, 2009 1:55 PM Ace Fekay [MCT] wrote:
>>>>>>>>>>>>> Jonathan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would welcome seeing such a list. It would provide Internet community
>>>>>>>>>>>>> awareness. Also, I did not look it up, but thank you for the RFC EDNS0
>>>>>>>>>>>>> note. I thought it was longer than 8 years. :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ace
>>>>>>>>>>>>>> On Thursday, December 10, 2009 4:43 PM J de Boyne Pollard wrote:
>>>>>>>>>>>>>> AF> Good find, Jonathan.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Merci.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> AF> What query command did you use with Dig to
>>>>>>>>>>>>>> AF> determine this, or did you use nslookup?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Neither. I used my own tool. I just sent an "A" query for
>>>>>>>>>>>>>> "www.lloydstsb.co.uk." to 193.34.230.74 with EDNS0 large datagram
>>>>>>>>>>>>>> support enabled. I do not remember, off the top of my head, what the
>>>>>>>>>>>>>> option for enabling that in "dig" is, but that is all that one needs to
>>>>>>>>>>>>>> do. With my own tool, it is the /LARGEUDP option, which yielded the
>>>>>>>>>>>>>> following:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [c:\]dnsqry /serverip:193.34.230.74 /largeudp a www.lloydstsb.co.uk.
>>>>>>>>>>>>>> [0.0.0.0:0000] -> [193.34.230.74:0035] 48
>>>>>>>>>>>>>> Header: 0000 1+0+0+1, Q, , query, no_error
>>>>>>>>>>>>>> Question: www.lloydstsb.co.uk. IN A
>>>>>>>>>>>>>> Additional: . 0x7fff OPT 0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [193.34.230.74:0035] -> [0.0.0.0:0000] 12
>>>>>>>>>>>>>> Header: 0000 0+0+0+0, R, , query, bad_format
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [c:\]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's the same 12-octet response (except for message ID and some
>>>>>>>>>>>>>> reserved flag bits) as is given in the reported event log data. As
>>>>>>>>>>>>>> you can see, there is no question section in the response. The first
>>>>>>>>>>>>>> thing that I actually did, that I then double-checked with the above
>>>>>>>>>>>>>> manual query, was simply ask a non-Microsoft resolving proxy DNS
>>>>>>>>>>>>>> server to look up the "A" resource record set for
>>>>>>>>>>>>>> "www.lloydstsb.co.uk.", and read its log of back-end queries sent and
>>>>>>>>>>>>>> responses received. Here is the relevant portion of that log (which
>>>>>>>>>>>>>> server does not mind empty question sections in "format error"
>>>>>>>>>>>>>> responses to EDNS0 queries, and so just proceeds to switch off EDNS0
>>>>>>>>>>>>>> and re-try the query.
>>>>>>>>>>>>>>> On Thursday, December 10, 2009 4:43 PM J de Boyne Pollard wrote:
>>>>>>>>>>>>>>> AH> Yes, we could add a "conditional forwarder" for that domain,
>>>>>>>>>>>>>>> AH> and still use the root hints. =A0I would be happy to do this, but
>>>>>>>>>>>>>>> AH> would face questions about whether this would happen in the
>>>>>>>>>>>>>>> AH> future.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> it is my preference of local fix.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> AH> Could an alternative to be to switch off EDNS0 support in
>>>>>>>>>>>>>>> AH> the DNS server itself?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That's another local fix, but one that I view as less preferable, for
>>>>>>>>>>>>>>> the simple reason that large DNS/UDP datagram support *is* beneficial,
>>>>>>>>>>>>>>> in my experience. So I am, personally, reluctant to disable it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But, as I said before, a service fix is better than any local fix,
>>>>>>>>>>>>>>> here. Really, Microsoft's DNS server should cope better with content
>>>>>>>>>>>>>>> DNS servers like Lloyds TSB's content DNS servers. Doing so *does*
>>>>>>>>>>>>>>> trade off some small measure of security for interoperability.
>>>>>>>>>>>>>>> There is no argument about that. But when it comes to the security of
>>>>>>>>>>>>>>> the DNS protocol as a whole, it is somewhat of a drop in the ocean.
>>>>>>>>>>>>>>> The DNS protocol is massively insecure, and badly designed. *This*
>>>>>>>>>>>>>>> particular insecurity simply permits blind attackers to easily forge
>>>>>>>>>>>>>>> negative, "format error", responses. Blind attackers are more
>>>>>>>>>>>>>>> interested in forging *positive* responses. So this insecurity is on
>>>>>>>>>>>>>>> the level of handing so-called "script kiddies" a mechanism for
>>>>>>>>>>>>>>> preventing a victim from finding "whitehouse.gov." and its subdomains,
>>>>>>>>>>>>>>> rather than on the level of actually being able to impersonate the
>>>>>>>>>>>>>>> U.S. government, as other flaws in the DNS protocol do.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The service fix would address the issue that you touch upon. Lloyds
>>>>>>>>>>>>>>> TSB is not the only organization with poor content DNS servers. (it is
>>>>>>>>>>>>>>> more embarrassing for Lloyds TSB than you paint it to be, Ace, by the
>>>>>>>>>>>>>>> way. RFC 2671 is dated August 1999. So it is over 10 years that
>>>>>>>>>>>>>>> Lloyds TSB has had to get its content DNS servers capable of at least
>>>>>>>>>>>>>>> *parsing* EDNS0 queries, even if they simply ignore the extensions.)
>>>>>>>>>>>>>>> But the positive side is that it is one of a *few* such organizations
>>>>>>>>>>>>>>> that has poor content DNS service. If they were many, the trouble
>>>>>>>>>>>>>>> that you experience would be more widely reported over the past
>>>>>>>>>>>>>>> decade.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have been tempted, in recent months, to start a content DNS service
>>>>>>>>>>>>>>> "Hall of Shame", listing content DNS services that do not get the DNS
>>>>>>>>>>>>>>> protocol right, or that are woefully inadequate in their handling of
>>>>>>>>>>>>>>> the DNS protocol, to the extent of causing interoperability problems
>>>>>>>>>>>>>>> with widespread secure resolving proxy DNS servers that necessitate
>>>>>>>>>>>>>>> variances from the protocol. Lloyds TSB not including a question
>>>>>>>>>>>>>>> section in its responses to EDNS0 queries would be the third on such a
>>>>>>>>>>>>>>> list, after Google (whose content DNS servers erroneously stop halfway
>>>>>>>>>>>>>>> through constructing responses) and Amazon (whose content DNS servers
>>>>>>>>>>>>>>> in combination put CNAME resource records on a delegation point). I
>>>>>>>>>>>>>>> have not done so, yet. But perhaps it would raise awareness of exactly
>>>>>>>>>>>>>>> how much *bad protocol* softwares like Microsoft's DNS server have to
>>>>>>>>>>>>>>> be coded to cope with, and the security tradeoffs that are forced as a
>>>>>>>>>>>>>>> result; and how flawed the DNS protocol itself really is.
>>>>>>>>>>>>>>>> On Thursday, December 10, 2009 11:23 PM J de Boyne Pollard wrote:
>>>>>>>>>>>>>>>> AF> I would welcome seeing such a list.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will give it some further consideration. For what it is worth, I had
>>>>>>>>>>>>>>>> this particular issue with content DNS servers that respond in this
>>>>>>>>>>>>>>>> particular way to EDNS0 documented some five years ago, in another,
>>>>>>>>>>>>>>>> related but not quite the same, Hall of Shame.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <URL:http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/dns-
>>>>>>>>>>>>>>>> superdomain-owner-hall-of-shame.html#Irony>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The situation is not quite as bad now as it was when I wrote that.
>>>>>>>>>>>>>>>> Witness the aforegiven log excerpt. The "uk." content DNS server at
>>>>>>>>>>>>>>>> 199.7.66.44 responded quite happily to an EDNS0 query, without the
>>>>>>>>>>>>>>>> need for any fallback.
>>>>>>>>>>>>>>>>> On Friday, December 11, 2009 7:53 AM Ace Fekay [MCT] wrote:
>>>>>>>>>>>>>>>>> I guess as time has passed, many DNS owners have upgraded to support EDNS0
>>>>>>>>>>>>>>>>> functionality, albeit the few that are left out there that have not such as
>>>>>>>>>>>>>>>>> Lloyd's.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If you put the list together, I will definitely reference it in future,
>>>>>>>>>>>>>>>>> related posts.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>> Ace
>>>>>>>>>>>>>>>>>> On Friday, December 11, 2009 9:36 AM Kerry Brown wrote:
>>>>>>>>>>>>>>>>>> This may be relevant to the discussion. If an EDNS query fails R2 does not
>>>>>>>>>>>>>>>>>> fall back to DNS. I believe BIND does.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> http://weblogs.asp.net/owscott/archive/2009/09/15/windows-server-2008-r2-dns-issues.aspx
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hopefully at some point an update will fix this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Kerry Brown
>>>>>>>>>>>>>>>>>> MS-MVP - Windows Desktop Experience: Systems Administration
>>>>>>>>>>>>>>>>>> http://www.vistahelp.ca/phpBB2/
>>>>>>>>>>>>>>>>>>> On Friday, December 11, 2009 12:48 PM J de Boyne Pollard wrote:
>>>>>>>>>>>>>>>>>>> KB> This may be relevant to the discussion.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Only inasmuch as they are behind us. (-: They did not figure out the
>>>>>>>>>>>>>>>>>>> *reason* that the fallback does not happen in this particular case,
>>>>>>>>>>>>>>>>>>> which can be deduced from what Microsoft's DNS server writes in the
>>>>>>>>>>>>>>>>>>> event log.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It would be interesting to hear whether M. Hodgson ever had problems
>>>>>>>>>>>>>>>>>>> with looking up "bing.com.". The "bing.com." content DNS servers also
>>>>>>>>>>>>>>>>>>> return "format error", but in contrast to Lloyds TSB's content DNS
>>>>>>>>>>>>>>>>>>> servers, they echo the question asked when they do:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [c:\]dnsqry /serverip:64.4.59.173 /largeudp a www.bing.com.
>>>>>>>>>>>>>>>>>>> [0.0.0.0:0000] -> [64.4.59.173:0035] 41
>>>>>>>>>>>>>>>>>>> Header: 0000 1+0+0+1, Q, , query, no_error
>>>>>>>>>>>>>>>>>>> Question: www.bing.com. IN A
>>>>>>>>>>>>>>>>>>> Additional: . 0x7fff OPT 0
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [64.4.59.173:0035] -> [0.0.0.0:0000] 41
>>>>>>>>>>>>>>>>>>> Header: 0000 1+0+0+1, R, , query, bad_format
>>>>>>>>>>>>>>>>>>> Question: www.bing.com. IN A
>>>>>>>>>>>>>>>>>>> Additional: . 0x7fff OPT 0
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [c:\]
>>>>>>>>>>>>>>>>>>>> On Friday, December 11, 2009 3:43 PM Andrew Hodgson wrote:
>>>>>>>>>>>>>>>>>>>> Yes, enabling all the logging on our Bind box shows that we are
>>>>>>>>>>>>>>>>>>>> falling back to using non EDNS0 requests for that domain. I can see
>>>>>>>>>>>>>>>>>>>> the format error response when I do the lookup. The local fix I did
>>>>>>>>>>>>>>>>>>>> was to use forwarders on the DNS servers, until MS can fix this issue,
>>>>>>>>>>>>>>>>>>>> as it does not involve modifying the defaults, so we can just implement
>>>>>>>>>>>>>>>>>>>> it without too much documentation :).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Well, in 2008 we had this issue:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> http://support.microsoft.com/kb/968372
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> That held us from using root hints as well. They never fixed that;
>>>>>>>>>>>>>>>>>>>> not even with Service Pack 2. Although, it does seem to fall back
>>>>>>>>>>>>>>>>>>>> from using EDNS0 when resolving www.lloydstsb.co.uk, unlike its R2
>>>>>>>>>>>>>>>>>>>> counterpart.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Windows Server 2003 R2 was the most reliable DNS server I used with
>>>>>>>>>>>>>>>>>>>> root hints, it did use EDNS0, but seemed to fall back ok, and was not
>>>>>>>>>>>>>>>>>>>> affected by the 2008 issue.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>>>>>>>> On Friday, December 11, 2009 3:52 PM Andrew Hodgson wrote:
>>>>>>>>>>>>>>>>>>>>> I can test this on Monday in the office on my test machine.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This system was up for over a day without this issue, and I would be
>>>>>>>>>>>>>>>>>>>>> surprised if we did not have users that used Bing throughout some part
>>>>>>>>>>>>>>>>>>>>> of the day.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>>>>>>>>> On Saturday, December 12, 2009 1:03 AM Ace Fekay [MCT] wrote:
>>>>>>>>>>>>>>>>>>>>>> I have not seen this issue yet, with 2008. I am aware of the MaxCacheTTL.
>>>>>>>>>>>>>>>>>>>>>> Adding the entry and a value (articles suggests 2 days), keeps items in the
>>>>>>>>>>>>>>>>>>>>>> cache for the time you set and ignores the TTL on the record. Normally DNS
>>>>>>>>>>>>>>>>>>>>>> deletes the items in cache when TTL expires. However, I do not see forcing
>>>>>>>>>>>>>>>>>>>>>> this entry to help resolve some TLDs, unless there is something else this
>>>>>>>>>>>>>>>>>>>>>> key does with EDNS0 that is not stated in the following article.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> MaxCacheTtl
>>>>>>>>>>>>>>>>>>>>>> http://technet.microsoft.com/en-us/library/cc959926.aspx
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Also funny thing - What happens if DNS cannot resolve a record? Then there
>>>>>>>>>>>>>>>>>>>>>> will be no record to be placed in cache.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Ace
>>>>>>>>>>>>>>>>>>>>>>> On Monday, December 14, 2009 3:25 PM Andrew Hodgson wrote:
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I tried this this morning by setting up one of our resolving
>>>>>>>>>>>>>>>>>>>>>>> nameservers out of the way of the clients path to use root hints, then
>>>>>>>>>>>>>>>>>>>>>>> cleared the cache on the server. I looked up www.lloydstsb.co.uk. to
>>>>>>>>>>>>>>>>>>>>>>> start with, and got the same error (and event log message), but www.bing.co=
>>>>>>>>>>>>>>>>>>>>>>> m.
>>>>>>>>>>>>>>>>>>>>>>> resolved fine, with no event log errors:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> So it appears that we do not have any issues with resolving Bing, which
>>>>>>>>>>>>>>>>>>>>>>> makes the above blog posting really strange :).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Let me know if you want me to test any other domains, since it is not
>>>>>>>>>>>>>>>>>>>>>>> too difficult to get this set up again.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>>>>>>>>>>> On Tuesday, December 15, 2009 10:17 AM Ace Fekay [MCT] wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Interesting. Problems with caching the glue may help with overriding
>>>>>>>>>>>>>>>>>>>>>>>> the TTL, but then again, that indicates it would need to be able to
>>>>>>>>>>>>>>>>>>>>>>>> resolve the glue, and perhaps if Lloyd's TTBS server will not respond to
>>>>>>>>>>>>>>>>>>>>>>>> that, then the chicken/egg theory comes into play.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I agree it is a bodge - hastily put together to address an issue
>>>>>>>>>>>>>>>>>>>>>>>> without explaining why it works or why it was suggested as a
>>>>>>>>>>>>>>>>>>>>>>>> workaround, and in this case, not necessarily a 'fix,' otherwise we
>>>>>>>>>>>>>>>>>>>>>>>> would have seen a 'dns.exe' hotfix.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> It was more of a general statement such as if the Lloyd's A record is
>>>>>>>>>>>>>>>>>>>>>>>> not resolvable due to Lloyd's TTBS servers, e.g. not able to get a
>>>>>>>>>>>>>>>>>>>>>>>> response, then there would be no "A" record to cache, however, good
>>>>>>>>>>>>>>>>>>>>>>>> point that the negative response is cached.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Understood. As stated, I was making more of a general statement.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Ace
>>>>>>>>>>>>>>>>>>>>>>>>> On Tuesday, December 15, 2009 10:58 AM J de Boyne Pollard wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> AH> http://support.microsoft.com/kb/968372
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> AF> I have not seen this issue yet, with 2008. [...] I do not see
>>>>>>>>>>>>>>>>>>>>>>>>> forcing
>>>>>>>>>>>>>>>>>>>>>>>>> AF> this entry to help resolve some TLDs, =A0unless there is something
>>>>>>>>>>>>>>>>>>>>>>>>> AF> else this key does with EDNS0 that is not stated in the following
>>>>>>>>>>>>>>>>>>>>>>>>> AF> article.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> it is a bodge. It has all of the hallmarks of one. Stopping things
>>>>>>>>>>>>>>>>>>>>>>>>> from being cached for more than 2 days prevents something as a side-
>>>>>>>>>>>>>>>>>>>>>>>>> effect. it is not obvious what, though. My educated guess would be a
>>>>>>>>>>>>>>>>>>>>>>>>> problem with caching the "glue" of the affected delegation points, but
>>>>>>>>>>>>>>>>>>>>>>>>> a very quick review of "br." does not turn up anything that would
>>>>>>>>>>>>>>>>>>>>>>>>> obviously cause a problem.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> That's really a quite poor KB article when it comes to explaining what
>>>>>>>>>>>>>>>>>>>>>>>>> the problem being bodged around is.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> AF> Also funny thing - What happens if DNS cannot resolve a record?
>>>>>>>>>>>>>>>>>>>>>>>>> AF> Then there will be no record to be placed in cache.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Define "cannot resolve". There is a different between the resolution
>>>>>>>>>>>>>>>>>>>>>>>>> algorithm having to be aborted and the receipt of a negative answer.
>>>>>>>>>>>>>>>>>>>>>>>>> The latter is most definitely cacheable. Remember: The database
>>>>>>>>>>>>>>>>>>>>>>>>> schema that content and proxy DNS servers use is not the schema used in
>>>>>>>>>>>>>>>>>>>>>>>>> the DNS protocol. In the *true* DNS database schema, the one that DNS
>>>>>>>>>>>>>>>>>>>>>>>>> server (and indeed DNS client) software writers actually have to
>>>>>>>>>>>>>>>>>>>>>>>>> employ, resource records are not considered individually, and empty
>>>>>>>>>>>>>>>>>>>>>>>>> resource record sets and "no such name" indicators are concrete
>>>>>>>>>>>>>>>>>>>>>>>>> things, with TTLs of their own.
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tuesday, December 15, 2009 10:58 AM J de Boyne Pollard wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> It would be interesting to hear whether M. Hodgson
>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> ever had problems with looking up "bing.com.". =A0The
>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> "bing.com." content DNS servers also return
>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> "format error", but in contrast to Lloyds TSB's content
>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> DNS servers, they echo the question asked when they do:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> AH> [...] www.bing.com. resolved fine, with no event log errors:
>>>>>>>>>>>>>>>>>>>>>>>>>> AH> [...]
>>>>>>>>>>>>>>>>>>>>>>>>>> AH> So it appears that we do not have any issues with resolving
>>>>>>>>>>>>>>>>>>>>>>>>>> AH> Bing, which makes the above blog posting really strange :).
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> It makes it wrong. (-:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> But it is independent confirmation of what I said. The problem here is
>>>>>>>>>>>>>>>>>>>>>>>>>> that (the resolving proxy part of) Microsoft's DNS server cannot
>>>>>>>>>>>>>>>>>>>>>>>>>> handle the missing question sections in responses from Lloyds TSB's
>>>>>>>>>>>>>>>>>>>>>>>>>> content DNS servers (and others), causing it to throw away the
>>>>>>>>>>>>>>>>>>>>>>>>>> response before it gets to the part of its algorithm that recognizes
>>>>>>>>>>>>>>>>>>>>>>>>>> an EDNS0 probe failure and the need for fallback. It does not have
>>>>>>>>>>>>>>>>>>>>>>>>>> this problem with the "bing.com." content DNS servers.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> AH> Let me know if you want me to test any other domains,
>>>>>>>>>>>>>>>>>>>>>>>>>> AH> since it is not too difficult to get this set up again.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I will have a quick look around for some more content DNS servers like
>>>>>>>>>>>>>>>>>>>>>>>>>> Lloyds TSB's content DNS servers and like the "bing.com." content DNS
>>>>>>>>>>>>>>>>>>>>>>>>>> servers. But it is fairly conclusive with the results that we have.
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tuesday, December 15, 2009 1:56 PM Ace Fekay [MCT] wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree that I do not think we will ever see a service fix for this
>>>>>>>>>>>>>>>>>>>>>>>>>>> assuming that the engineers believe the problem is on the out of date
>>>>>>>>>>>>>>>>>>>>>>>>>>> DNS servers, and should have been updated long ago. Maybe someone
>>>>>>>>>>>>>>>>>>>>>>>>>>> should email Lloyds' to let them know.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the info on the LameDelegationTtl explanation. It makes
>>>>>>>>>>>>>>>>>>>>>>>>>>> sense that it is truly a glue problem.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Ace
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tuesday, December 15, 2009 5:25 PM J de Boyne Pollard wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AH> Let me know if you want me to test any other domains,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AH> since it is not too difficult to get this set up again.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> I will have a quick look around for some more content
>>>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> DNS servers like Lloyds TSB's content DNS servers
>>>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> and like the "bing.com." content DNS servers. =A0But
>>>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> it is fairly conclusive with the results that we have.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have just searched my last 24 hours' logs for EDNS0 fallbacks caused
>>>>>>>>>>>>>>>>>>>>>>>>>>>> by "bad format" responses.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The "mii.instacontent.net." content DNS servers also reply "bad
>>>>>>>>>>>>>>>>>>>>>>>>>>>> format" to EDNS0 errors. But since they, too, echo the question in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> their responses, you should not have any trouble with, say, an "A"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup for "ant.mii.instacontent.net.".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tuesday, December 15, 2009 5:25 PM J de Boyne Pollard wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AH>http://support.microsoft.com/kb/968372
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> I have not seen this issue yet, with 2008. [...] I do not see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> forcing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> this entry to help resolve some TLDs, unless there is something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> else this key does with EDNS0 that is not stated in the following
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> article.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> [...] =A0it is not obvious what, though. =A0My educated guess
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> would be a problem with caching the "glue" of the affected
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> delegation points, but a very quick review of "br." does not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> JdeBP> turn up anything that would obviously cause a problem.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> Interesting. Problems with caching the glue may help with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> overriding the TTL, but then again, that indicates it would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> need to be able to resolve the glue, and perhaps if Lloyd's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> TSB's server will not respond to that, then the chicken/egg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> theory comes into play.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> M. Hodgson said that he had the KB968372 problem last year. He did not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> say what domain names were affected, but the article gives "br." as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one example. it is not related to the Lloyds TSB problem, as far as I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can see. Some further research after I wrote the above indicates that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cache not handling "glue" properly is indeed the KB968372
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> problem. (That one Microsoft employee suggested tweaking a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "LameDelegationTtl" property as a local fix was one clue. it is still
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not clear what the actual problem is, though.) it is not a response
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parsing issue, as the current Lloyds TSB problem is.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> M. Hodgson's point, I believe, was that since we have not yet seen a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> service fix (or even a temporary fix) for the KB968372 problem,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whatever it is, we probably will not see a service fix for *this* problem
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the near future.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Friday, December 18, 2009 2:00 AM Ace Fekay [MCT] wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Interesting, thank you. I will have to take a few moments to read them,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> especially the Product Behavior link.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ace
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Friday, December 18, 2009 5:45 AM Andrew Hodgson wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That is correct. I had the issue with mainly .co.uk addresses, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> our users could not resolve them (giving a servfail response), yet
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other domains would still resolve fine. It was a completely separate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue to the lloydstsb.co.uk issue I was talking about before.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since I never understood what the registry fix would actually achieve,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I could not recommend implementing it in the live environment, so we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used a Linux machine running Bind to work round the issue instead. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was disappointed not to receive any hot fix for this, and I was even
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more disappointed when this did not get fixed in SP2.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, that is my point (although this problem has so far only affected
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one domain that I know of).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Andrew.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Saturday, December 19, 2009 1:17 AM J de Boyne Pollard wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AF> Thanks for the info on the LameDelegationTtl explanation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If you enjoyed that, you will like these:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <URL:http://msdn.microsoft.com/en-us/library/cc422472(PROT.10).aspx>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <URL:http://msdn.microsoft.com/en-us/library/cc422509(PROT.10).aspx>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wednesday, January 06, 2010 6:01 PM Lyle Epstein wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is another very well known ISP that has the same issue: Earthlink.net.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When we attempt to send mail to one of their hosted customers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bronsoncos.com which uses Earthlink.net as it is ISP, you get the 5504 event
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ID on Windows 2008 R2.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Perhaps you can add them to your list of companies that do not comply!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Saturday, January 09, 2010 8:30 AM Jonathan de Boyne Pollard wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <html>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <head>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <title></title>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </head>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <body bgcolor="#ffffff" text="#000000">
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <blockquote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cite="mid:BEB12ED8-6CC0-441D...@microsoft.com"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type="cite">
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <blockquote type="cite">
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <p wrap="">I have been tempted, in recent months, to start a content
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DNS service "Hall of Shame", listing content DNS services that do not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get the DNS protocol right, or that are woefully inadequate in their
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handling of the DNS protocol, to the extent of causing interoperability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> problems with widespread secure resolving proxy DNS servers that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessitate variances from the protocol. Lloyds TSB not including a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> question section in its responses to EDNS0 queries would be the third
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on such a list, after Google (whose content DNS servers erroneously
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stop halfway through constructing responses) and Amazon (whose content
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DNS servers in combination put CNAME resource records on a delegation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point). I have not done so, yet. But perhaps it would raise awareness of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exactly how much <em>bad
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> protocol</em> softwares like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Microsoft's DNS server have to be coded to cope with, and the security
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tradeoffs that are forced as a result; and how flawed the DNS protocol
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> itself really is.<br>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </p>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </blockquote>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <p>Here is another very well known ISP that has the same issue:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Earthlink.net. When we attempt to send mail to one of their hosted
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> customers, <code>bronsoncos.com</code> which uses Earthlink.net as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is ISP, you get the 5504 event ID on Windows 2008 R2. </p>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </blockquote>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <p>The resolving proxy DNS server that I used to test this before
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually logs this problem thrice for the DNS lookups involved in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sending mail.?? This is because of the intermediate domain names:<br>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </p>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <blockquote>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pre wrap="">[C:\]dnsqry mx bronsoncos.com.|grep /b/u Answer:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Answer: bronsoncos.com. IN MX 86066 10 store-forward.mspring.net.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Answer: bronsoncos.com. IN MX 86066 5 mail.bronsoncos.com.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </pre>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </blockquote>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <p>Looking up both of the intermediate domain names <code>mail.bronsoncos.com.</code>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and <code>store-forward.mspring.net.</code> results in <code>A</code>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> queries to the self-same content DNS servers that received the <code>MX</code>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> query.?? All three queries will receive "bad format" responses with zero
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> question section resource records.<br>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </p>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <blockquote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cite="mid:BEB12ED8-6CC0-441D...@microsoft.com"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type="cite">
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <p wrap="">Perhaps you can add them to your list of companies that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do not comply!<br>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </p>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </blockquote>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You, too, want me to start that list, eh??? (-:<br>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </body>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </html>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tuesday, January 19, 2010 11:15 PM Ace Fekay [MVP-DS, MCT] wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Jonathan de Boyne Pollard" <J.deBoynePoll...@NTLWorld.COM> wrote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I thought you already did?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ace
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Submitted via EggHeadCafe - Software Developer Portal of Choice
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Changing WCF Service Implementation at Runtime
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://www.eggheadcafe.com/tutorials/aspnet/d9263dcc-f7ed-42f3-bc96-321461be3306/changing-wcf-service-implementation-at-runtime.aspx