rndc flushname not working

[Frank Bulk]

Our ISP operations are running a mixture of 9.7.3 and 9.8.4 on several
Debian servers and we've noticed that rndc flushname doesn't work many
times.

This weekend we had a local institution whose own authoritative DNS servers
[all of them] were offline for 48+ hours and so there were several
negatively cached entries in our resolvers' caches from customers who were
querying for their institutions website and MX record. After the
institution restored connectivity to their DNS servers this afternoon I
checked all our resolvers and a few returned NXDOMAIN for the www and MX
record. Issuing rndc flushname <domain> and rndc flushname <www.domain>
didn't clear out the NXDOMAIN. I had to use "rndc flush" to resolve the
issue.

Is this expected behavior? The next time I see what, what troubleshoot
steps should I take diagnose the issue? Dump the DB?

Frank

[Mark Andrews]

Nameservers being down does not result in NXDOMAIN responses. I
suspect that some of the auth servers were producing NXDOMAIN
incorrectly. Flushing the name won't help in those cases.

> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
> from this list
>
> bind-users mailing list
> bind-...@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

[Frank Bulk]

Perhaps it wasn't NXDOMAIN -- I didn't capture the output. But there
definitely was not answer. The institution only has two authoritative
nameserver entries, both pointing to the same IP, so all it was all down.

In any case, why doesn't flushing the name work?

Frank

[Matus UHLAR - fantomas]

On 09.12.14 21:36, Frank Bulk wrote:
>Perhaps it wasn't NXDOMAIN -- I didn't capture the output. But there
>definitely was not answer. The institution only has two authoritative
>nameserver entries, both pointing to the same IP, so all it was all down.
>
>In any case, why doesn't flushing the name work?

I'm afraid we can't tell you without precise information.
However, the institution should get at least one backup DNS server...
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Microsoft dick is soft to do no harm

[Bob Harold]

On Wed, Dec 10, 2014 at 3:36 AM, Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:

On 09.12.14 21:36, Frank Bulk wrote:

Perhaps it wasn't NXDOMAIN -- I didn't capture the output.  But there
definitely was not answer.  The institution only has two authoritative
nameserver entries, both pointing to the same IP, so all it was all down.

In any case, why doesn't flushing the name work?

I'm afraid we can't tell you without precise information.
However, the institution should get at least one backup DNS server...
--
Matus UHLAR - fantomas,

If a DNS server does not respond, I believe that BIND caches that fact, but it is tied to the DNS server IP (or name?) and not to the domain, so flushing the domain name would not affect it.

 

[Matus UHLAR - fantomas]

>> On 09.12.14 21:36, Frank Bulk wrote:
>>> Perhaps it wasn't NXDOMAIN -- I didn't capture the output. But there
>>> definitely was not answer. The institution only has two authoritative
>>> nameserver entries, both pointing to the same IP, so all it was all down.
>>>
>>> In any case, why doesn't flushing the name work?

>On Wed, Dec 10, 2014 at 3:36 AM, Matus UHLAR - fantomas <uh...@fantomas.sk>
>wrote:
>> I'm afraid we can't tell you without precise information.
>> However, the institution should get at least one backup DNS server...

On 11.12.14 09:35, Bob Harold wrote:
>If a DNS server does not respond, I believe that BIND caches that fact, but
>it is tied to the DNS server IP (or name?) and not to the domain, so
>flushing the domain name would not affect it.

...and this cache is very short, 5 minutes IIRC.
it's really hard to tell without more info...

--

Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.

Boost your system's speed by 500% - DEL C:\WINDOWS\*.*

[Frank Bulk]

Next time I'll dump the db.

Frank

-----Original Message-----
From: bind-user...@lists.isc.org
[mailto:bind-user...@lists.isc.org] On Behalf Of Matus UHLAR -
fantomas
Sent: Thursday, December 11, 2014 10:32 AM
To: bind-...@lists.isc.org
Subject: Re: rndc flushname not working

[Frank Even]

Is there any place I can look to get a definitive answer in what cases
"flushname" will and will not work? I've been digging around in lists
and docs and can't seem to find any definitive answers. I've been
having odd troubles clearing a name from a cache and after even
clearing the name and the name that the name servers was attached to,
still had to flush the entire cache to get resolution working properly
on that domain again.

Thanks,
Frank

On Tue, Dec 9, 2014 at 8:31 PM, Mark Andrews <ma...@isc.org> wrote:
>
> Nameservers being down does not result in NXDOMAIN responses. I
> suspect that some of the auth servers were producing NXDOMAIN
> incorrectly. Flushing the name won't help in those cases.
>
> In message <001001d01429$1c857f70$55907e50$@iname.com>, "Frank Bulk" writes:
>> Our ISP operations are running a mixture of 9.7.3 and 9.8.4 on several
>> Debian servers and we've noticed that rndc flushname doesn't work many
>> times.
>>
>> This weekend we had a local institution whose own authoritative DNS servers
>> [all of them] were offline for 48+ hours and so there were several
>> negatively cached entries in our resolvers' caches from customers who were
>> querying for their institutions website and MX record. After the
>> institution restored connectivity to their DNS servers this afternoon I
>> checked all our resolvers and a few returned NXDOMAIN for the www and MX
>> record. Issuing rndc flushname <domain> and rndc flushname <www.domain>
>> didn't clear out the NXDOMAIN. I had to use "rndc flush" to resolve the
>> issue.
>>
>> Is this expected behavior? The next time I see what, what troubleshoot
>> steps should I take diagnose the issue? Dump the DB?
>>
>> Frank
>>

>> _______________________________________________
>> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
>> from this list
>>
>> bind-users mailing list
>> bind-...@lists.isc.org
>> https://lists.isc.org/mailman/listinfo/bind-users

> --
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

[Matus UHLAR - fantomas]

On 09.04.15 13:25, Frank Even wrote:
>Is there any place I can look to get a definitive answer in what cases
>"flushname" will and will not work?

it will work if you have old entries in the cache.
that will NOT help you if any of the servers that are supposed to be
authoritative for a domain will return invalid answers for the domain.

> I've been digging around in lists
>and docs and can't seem to find any definitive answers. I've been
>having odd troubles clearing a name from a cache and after even
>clearing the name and the name that the name servers was attached to,
>still had to flush the entire cache to get resolution working properly
>on that domain again.

this indicates that any of NS records the domain points to returns NXDOMAIN
for the domain.

hard to tell without more info, but some web DNS checkers are able to trace
this kind of issues...

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.

Windows found: (R)emove, (E)rase, (D)elete

[Frank Even]

On Thu, Apr 9, 2015 at 1:48 PM, Matus UHLAR - fantomas
<uh...@fantomas.sk> wrote:
> On 09.04.15 13:25, Frank Even wrote:
>>
>> Is there any place I can look to get a definitive answer in what cases
>> "flushname" will and will not work?
>
>
> it will work if you have old entries in the cache.
> that will NOT help you if any of the servers that are supposed to be
> authoritative for a domain will return invalid answers for the domain.
>
>> I've been digging around in lists
>> and docs and can't seem to find any definitive answers. I've been
>> having odd troubles clearing a name from a cache and after even
>> clearing the name and the name that the name servers was attached to,
>> still had to flush the entire cache to get resolution working properly
>> on that domain again.
>
>
> this indicates that any of NS records the domain points to returns NXDOMAIN
> for the domain.
>
> hard to tell without more info, but some web DNS checkers are able to trace
> this kind of issues...
> --

So flushname does not address NXDOMAIN responses? That's the point
I'm getting at, there is no good documentation on this that I can
find. All the responses in the lists seem to be around "well it
depends on your situation, need more data, etc." I'd like to just be
able to find documentation on the specific behaviors of these options
so I can understand properly what they do to maintain my environment
properly without properly understanding what command will do or not do
for me. The closest I have found is this -
https://kb.isc.org/article/AA-01002/0/How-do-I-flush-or-delete-incorrect-records-from-my-recursive-server-cache.html
- but it does not tgo on to tell what is or is not stored in the ADB
(or give a link to figure that out) to properly understand what I can
and cannot get dumped from cache by executing an "rndc flushname"
command.

I no longer have the data regardless, a full flush "fixed" the issue.
We have some automation around running a "flushname" on the servers
though and that addresses a large number of issues with cache
weirdness, so when I got pulled in for something where it wasn't
working I was curious as to why. It seems this is a recurring
question on the lists, but I can't find where there are any definitive
answers anywhere. If there is something that I'm missing I would be
highly appreciative of being pointed towards that information.

Thanks,
Frank

[Matus UHLAR - fantomas]

On 09.04.15 13:57, Frank Even wrote:
>So flushname does not address NXDOMAIN responses?

oh yes, it does. But it seems that after you flush the name and resolve
again, someone is providing you with further NXDOMAIN.
Flushname can not help you with broken server or broken delegation.

>I'm getting at, there is no good documentation on this that I can
>find. All the responses in the lists seem to be around "well it
>depends on your situation, need more data, etc."

yes, more data would help - if someone was able to look at the real domain
you have problems with, they could find what the issue is.

If you ask us generic question, we can only give you generic response.

>I no longer have the data regardless, a full flush "fixed" the issue.
>We have some automation around running a "flushname" on the servers
>though and that addresses a large number of issues with cache
>weirdness

Flushname should be only needed if something breaks. It should not be needed
to run that automatically. If you need flush more often than just
ocationally, there is apparently broken nameserver or too big TTL set and
fixing the server or lowering the TTL should fix the issue.

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.

Silvester Stallone: Father of the RISC concept.

[Chris Buxton]

On Apr 9, 2015, at 4:25 PM, Frank Even <lists+...@elitists.org> wrote:
>
> Is there any place I can look to get a definitive answer in what cases

> "flushname" will and will not work? I've been digging around in lists

> and docs and can't seem to find any definitive answers. I've been
> having odd troubles clearing a name from a cache and after even
> clearing the name and the name that the name servers was attached to,
> still had to flush the entire cache to get resolution working properly
> on that domain again.

'rndc flushname' will clear a single name (all records, all types) out of the normal recursion cache. However, as noted by Mark last December, if an authoritative server for the domain is returning bad answers, the bad answer might come back on the next query for that rrset or that name.

If you want to clear an entire domain out of cache (example.com and all of its child names), use 'rndc flushtree' instead.

Regards,
Chris

[Frank Even]

On Fri, Apr 10, 2015 at 2:19 AM, Matus UHLAR - fantomas
<uh...@fantomas.sk> wrote:
> On 09.04.15 13:57, Frank Even wrote:
>>
>> So flushname does not address NXDOMAIN responses?
>
>
> oh yes, it does. But it seems that after you flush the name and resolve
> again, someone is providing you with further NXDOMAIN.
> Flushname can not help you with broken server or broken delegation.

Some of the things I have read seem to have conflicting information on
it and there is no good documentation on exactly what flushname can
and cannot clear (which is the problem and why this question seems to
keep coming up in the lists).

>> I'm getting at, there is no good documentation on this that I can
>> find. All the responses in the lists seem to be around "well it
>> depends on your situation, need more data, etc."

What exactly does flushname clear is not necessarily a generic
question. If it was better documented you may avoid answering these
questions over and over.

>> I no longer have the data regardless, a full flush "fixed" the issue.
>> We have some automation around running a "flushname" on the servers
>> though and that addresses a large number of issues with cache
>> weirdness
>
>
> Flushname should be only needed if something breaks. It should not be needed
> to run that automatically. If you need flush more often than just
> ocationally, there is apparently broken nameserver or too big TTL set and
> fixing the server or lowering the TTL should fix the issue.

I suspect what happened is someone hit these servers and cached a
record prior to a domain being fully setup. BUT, the fact that
flushname did not work on the domain itself, or on dumping the domain
of the NS records (a different domain), yet flushing the cache got the
domain working again makes me wonder exactly what flushname does and
does not address (and nowhere can I find where that is documented).
The existing documentation does say that flushname will not clear some
things, but does not follow up on any way to get a complete idea of
what those things are.

[Frank Even]

flushtree does not work in the current distributed version of bind in
EL6 packages. In this particular case, when the issue came to me, the
name servers for the domain were able to return the result with no
problems, other caching servers throughout the company had no issues,
but this group of servers that apparently had tests run against the
domain prior to it being fully setup could not resolve it and nothing
short of a full flush of the cache would fix the issue. That leads me
to believe there is some data that gets held in cache not cleared by a
flushname that is relevant but not documented.

[John Wobus]

> In this particular case, when the issue came to me, the
> name servers for the domain were able to return the result with no
> problems, other caching servers throughout the company had no issues,
> but this group of servers that apparently had tests run against the
> domain prior to it being fully setup could not resolve it and nothing
> short of a full flush of the cache would fix the issue. That leads me
> to believe there is some data that gets held in cache not cleared by a
> flushname that is relevant but not documented.

It could be a nameserver in the delegation chain cached
wrongly. If that nameserver isn't in the domain to which
flushtree was directed, another flush aimed at that nameserver's name
would be needed. I recall the "fun" of hunting down some of those.
CNAMEs can also create fun.

John Wobus
Cornell University IT

[Tony Finch]

There was a bug in 9.9 and earlier that rndc flushtree only flushed the main cache, not adb or bad cache. This was fixed in 9.10 - see item 3606 in the CHANGES file.

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at

[Frank Even]

On Sat, Apr 11, 2015 at 6:49 AM, Tony Finch <d...@dotat.at> wrote:
> There was a bug in 9.9 and earlier that rndc flushtree only flushed the main cache, not adb or bad cache. This was fixed in 9.10 - see item 3606 in the CHANGES file.

...and where could I find info on what is stored in ADB and any other
particular items that flushname might not deal with? That's where my
frustration largely is, that I can't find clear documentation on this
point.

[Evan Hunt]

On Mon, Apr 13, 2015 at 11:05:05AM -0700, Frank Even wrote:
> ...and where could I find info on what is stored in ADB and any other
> particular items that flushname might not deal with? That's where my
> frustration largely is, that I can't find clear documentation on this
> point.

I believe "rndc dumpdb -all" will give you that.

Note, "rndc flushname" always took care of the ADB and the bad cache for a
given name. "rndc flushtree" is the one that used to be a problem -- it
would recursively clear all the names below the specified one from the
DNS cache, but it wasn't touching the ADB or the bad cache.

--
Evan Hunt -- ea...@isc.org
Internet Systems Consortium, Inc.

[Frank Even]

Ah...it does appear I was confusing flushtree and flushname in the
most specific doc I could find -
https://kb.isc.org/article/AA-01002/0/How-do-I-flush-or-delete-incorrect-records-from-my-recursive-server-cache.html

I have to apologize for that. I'd still definitely be curious to know
what info is stored in the ADB though since according to the docs ADB
was never intended to be flushed with a "flushtree" (although that has
now apparently been addressed in a new version as stated earlier in
the thread).

Thanks.

[Evan Hunt]

On Mon, Apr 13, 2015 at 11:17:41AM -0700, Frank Even wrote:
> I have to apologize for that. I'd still definitely be curious to know
> what info is stored in the ADB though since according to the docs ADB
> was never intended to be flushed with a "flushtree" (although that has
> now apparently been addressed in a new version as stated earlier in
> the thread).

The ADB is the "address database"; it holds the addresses of previously
encountered authoritative name servers and information about them such as
whether they support EDNS and the largest packet size they're known to be
able to accommodate.

The "bad cache" is a separate collection of name server addresses that have
been found to be irretrievably broken within the last several minutes. It
prevents named from wasting too much of its time trying to get answers from
broken servers.

(The reason flushtree didn't originally clear those, if your curiousity
extends this far, is that they're implemented as hash tables instead of
trees like the DNS cache, so deleting all records below a given name is
inefficient - it requires a linear search through the entire hash table.
It turned out to be a useful thing to do, though, so we eventually
decided to go ahead and put up with the inefficiency.)