Caching and upper case issue with BIND 9.9.7-P3

[cypher Nix]

After upgrading BIND from BIND 9.9.7-P2 to BIND 9.9.7-P3 on about a dozen
authoritative + recursive servers, we noticed a strange caching issue on one of the servers.

The server is authoritative for our main domain (let's assume example.com).
There are multiple subdomains under example.com that have been NS delegated to
other servers. Whenever the DNS server would respond to a recursive "A" record query from its
cache, the "Answers" part of the request would always be in upper case - such
as foo.bar.EXAMPLE.COM.
More details below
I noted the following behavior using a packet capture

-A client requests for "foo.bar.example.com" "A" record.

-Our server then does a lookup against bar.example.com authoritative server for "A" record "foo.bar.exampe.com"

-Our server gets a response from bar.example.com authoritative server:
; ANSWER SECTION:
foo.bar.example.com. 10 IN A 192.168.0.1

-Our server responds to the clients request for foo.bar.example.com and stores
the response for 10 seconds (the TTL of the record). At this point the
answer section is still in all lower case - the clients gets the following:
; ANSWER SECTION:
foo.bar.example.com. 10 IN A 192.168.0.1

-The next time the client queries for foo.bar.exmaple.com, our server responds
from the cache and changes the case from example.com to EXAMPLE.COM. It
continues to serve EXAMPLE.COM in upper case as part of the answer while the TTL is still valid.
; ANSWER SECTION:
foo.bar.EXAMPLE.COM. 9 IN A 192.168.0.1

-This behavior was observed for "A" record responses for queries under any
subdomain of example.com. The case was only change to upper case on the
answer section. Example.com also appeared under the question, authority, and
additional sections but only in the answer section was the case changed.

We eventually restarted BIND and the issue went
away. After restarting BIND all responses served from cache are now lower case, as expected.

Has anyone seen this behavior before ? Is this a bug ?
This caused issues for certain applications on our network that did string
comparison and expected the answer section to be in lower case.

[Tony Finch]

cypher Nix <ciph...@gmail.com> wrote:
>
> Has anyone seen this behavior before ? Is this a bug ?

See the CHANGES file:

4079. [func] Preserve the case of the owner name of records to
the RRset level. [RT #37442]

> This caused issues for certain applications on our network that did string
> comparison and expected the answer section to be in lower case.

That's a bug in those applications.

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
Viking, North Utsire: Easterly 4 or 5, increasing 6 at times. Slight or
moderate, but rough in southwest Viking. Showers later. Good, occasionally
poor later.

[Evan Hunt]

On Wed, Sep 23, 2015 at 08:18:45AM -0700, cypher Nix wrote:
> After upgrading BIND from BIND 9.9.7-P2 to BIND 9.9.7-P3 on about a dozen

... that's odd, the new case compression behavior was introduced a fair
bit earlier than that; are you sure you weren't upgrading a few servers
older than 9.9.7-P2?

> Has anyone seen this behavior before ? Is this a bug ?

> [...]

> This caused issues for certain applications on our network that did string
> comparison and expected the answer section to be in lower case.

Those applications should be changed to use strcasecmp() not strcmp().
However, if that's not practical, you can configure named to work the way
it used to for selected clients, or for all clients.

The case of the question is preserved in responses, but the case of the
answer can change. BIND (and most other DNS implementations) used to
compress names in responses in a way that forced the the answer to
conform to the case of the question. But the specification actually
says to preserve the original owner's case whenever possible, so we
changed it.

If you need the old behavior back for a particular client, add
"no-case-compress { <address> ; };" to your options. If you
want it back for all clients, then use "no-case-compress { any; };".

--
Evan Hunt -- ea...@isc.org
Internet Systems Consortium, Inc.

[cypher Nix]

On Wednesday, September 23, 2015 at 12:01:15 PM UTC-4, Tony Finch wrote:
>
> That's a bug in those applications.

I agree that this a bug on their application code. We've asked the application owners to fix their code.

[cypher Nix]

On Wednesday, September 23, 2015 at 12:18:37 PM UTC-4, Evan Hunt wrote:
> ... that's odd, the new case compression behavior was introduced a fair
> bit earlier than that; are you sure you weren't upgrading a few servers
> older than 9.9.7-P2?
>

We've ran 9.9.4, 9.9.6, 9.9.7, and 9.9.7-P2. We never had an issue until upgrading to 9.9.7-p3. Only one of the dozen severs we updated to experience this case issue. The issue cleared up once BIND was stopped and then restarted.

> Those applications should be changed to use strcasecmp() not strcmp().

Application owners have been asked to update their code. However, restarting BIND seem to have corrected the issue we were seeing.

> However, if that's not practical, you can configure named to work the way
> it used to for selected clients, or for all clients.
>
> The case of the question is preserved in responses, but the case of the
> answer can change. BIND (and most other DNS implementations) used to
> compress names in responses in a way that forced the the answer to
> conform to the case of the question. But the specification actually
> says to preserve the original owner's case whenever possible, so we
> changed it.

Thanks for clarifying. It seems like our server in question was actually changing the case to upper case for just one domain. This behavior stopped once we bounced the server. However, I feel that the application should account for different case in DNS names.

[Mark Andrews]

In message <alpine.LSU.2.00.1...@hermes-2.csi.cam.ac.uk>, Tony Fi
nch writes:

> cypher Nix <ciph...@gmail.com> wrote:
> >
> > Has anyone seen this behavior before ? Is this a bug ?
>

> See the CHANGES file:
>
> 4079. [func] Preserve the case of the owner name of records to
> the RRset level. [RT #37442]

BIND 9.9 only has

3645. [protocol] Use case sensitive compression when responding to
queries. [RT #34737]

which returns the case of the node when it was added to the tree.
4079 adds additional data at the node that records the case of the
entire name allowing it to be returned.

For foo.bar.EXAMPLE.COM there was likely a query for "XXX.EXAMPLE.COM"
which created the nodes "EXAMPLE" and "COM". Later on there was a
query for "foo.bar.example.com" which created the nodes "foo" and
"bar". Combining these you get "foo.bar.EXAMPLE.COM" being returned.

Even with 4079 you will see inconsistencies over time until the
authoritative servers are all upgraded to fully preserve the case
of owner names. Until that is done you will see the case of the
initial query being reflected in later responses, however you will
see less mixed case responses as most queries are either all uppercase
or all lowercase.

Mark

> > This caused issues for certain applications on our network that did string
> > comparison and expected the answer section to be in lower case.
>

> That's a bug in those applications.
>

> Tony.
> --
> f.anthony.n.finch <d...@dotat.at> http://dotat.at/
> Viking, North Utsire: Easterly 4 or 5, increasing 6 at times. Slight or
> moderate, but rough in southwest Viking. Showers later. Good, occasionally
> poor later.

> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
> from this list
>
> bind-users mailing list
> bind-...@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

[cypher Nix]

Thank you all for your help. I was able to reproduce this behavior in the lab using older versions of BIND (9.9.7 P2 and 9.9.6 P1). I setup two servers in the lab: 1. A recursive server that's also authoritative for example.com and 2. an non recursive server that's authoritative for foo.example.com. The zone foo.example.com was delegated to server number 2. I added multiple sample records to both zones including a.foo.EXAMPLE.COM. I first started BIND on server number 2. Upon starting BIND on server number 1, the first query I looked up was a.foo.EXAMPLE.COM. All other responses served from cache had EXAMPLE and COM in upper case.

I will use your responses along with my lab findings to convince the application owners that they must update their code in order to prevent issues in the future.

[/dev/rob0]

On Wed, Sep 23, 2015 at 08:18:45AM -0700, cypher Nix wrote:

> We eventually restarted BIND and the issue went away. After
> restarting BIND all responses served from cache are now lower
> case, as expected.

Restarting is a very painful way to fix cache issues. Consider as
better choices:
rndc flush
rndc flushtree example.com
--
http://rob0.nodns4.us/
Offlist GMX mail is seen only if "/dev/rob0" is in the Subject: