All Bind servers crashed

[Magnus Schmidt]

Hello,

all three of our bind-Servers crashed tonight at the same time (in a 30
seconds time window), all of them are recursors. Following has been logged:

Log-File
Nov 16 05:30:41 xxx named[1326]: critical: query.c:1781: INSIST(!
dns_rdataset_isassociated(sigrdataset)) failed, back trace
Nov 16 05:30:41 xxx named[1326]: critical: #0 0xb76dafa0 in ??
Nov 16 05:30:41 xxx named[1326]: critical: #1 0xb72f9093 in ??
Nov 16 05:30:41 xxx named[1326]: critical: #2 0xb76e7ddf in ??
Nov 16 05:30:41 xxx named[1326]: critical: #3 0xb7599e38 in ??
Nov 16 05:30:41 xxx named[1326]: critical: #4 0xb75ccff8 in ??
Nov 16 05:30:41 xxx named[1326]: critical: #5 0xb76df7ff in ??
Nov 16 05:30:41 xxx named[1326]: critical: #6 0xb76e563a in ??
Nov 16 05:30:41 xxx named[1326]: critical: #7 0xb76e6bb1 in ??
Nov 16 05:30:41 xxx named[1326]: critical: #8 0xb731bebb in ??
Nov 16 05:30:41 xxx named[1326]: critical: #9 0xb711d955 in ??
Nov 16 05:30:41 xxx named[1326]: critical: #10 0xb6f73e7e in ??
Nov 16 05:30:41 xxx named[1326]: critical: exiting (due to assertion
failure)

Time is GMT+1. All are running under Debian Squeeze with version
1:9.7.3.dfsg-1~squeeze3.

Magnus Schmidt

--
Bisping & Bisping GmbH & Co. KG Dipl.Inf. (FH) Magnus Schmidt
*Internet & Network* m...@bisping.net
Spitalstrasse 21-24-26 phone +49-9123-9740-630
D-91207 Lauf a. d. Pegnitz fax +49-9123-9740-97
http://www.bisping.net * Service und Support: sup...@bisping.net

Bisping & Bisping GmbH & Co. KG
Sitz Lauf a.d. Peg. * Handelsregister Nürnberg HRA Nr. 10845
UStID: DE 132809745 * Steuernummer: 221/152/55405

Persönlich haftende Gesellschafterin: Bisping Media Group GmbH
Sitz Lauf a.d. Peg. * Handelsregister Nürnberg HRB Nr. 19061
Geschäftsführer: Johannes Bisping, Matthias Bisping

[Stephane Bortzmeyer]

On Wed, Nov 16, 2011 at 09:47:48AM +0100,
Magnus Schmidt <m...@bisping.de> wrote

a message of 49 lines which said:

> Nov 16 05:30:41 xxx named[1326]: critical: query.c:1781: INSIST(!
> dns_rdataset_isassociated(sigrdataset)) failed, back trace

It looks like CVE-2010-3613
<http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-3613> <http://www.isc.org/software/bind/advisories/cve-2010-3613> but:

> All are running under Debian Squeeze with version
> 1:9.7.3.dfsg-1~squeeze3.

Strange because it should not be vulnerable. May be a similar
unreported bug? Samer Khattab and nicku just reported a very similar bug
(messages "BIND 9.7.3-P3 crash on multiple cashing servers" and
"bind-9.8.1: INSIST(! dns_rdataset _isassociated(sigrdataset))
failed").

[Bill Owens]

On Wed, Nov 16, 2011 at 09:57:18AM +0100, Stephane Bortzmeyer wrote:
> On Wed, Nov 16, 2011 at 09:47:48AM +0100,
> Magnus Schmidt <m...@bisping.de> wrote
> a message of 49 lines which said:
>
> > Nov 16 05:30:41 xxx named[1326]: critical: query.c:1781: INSIST(!
> > dns_rdataset_isassociated(sigrdataset)) failed, back trace

This behavior makes me bet that the trigger is a name in an incoming email message, being resolved by an anti-spam filter. That appeared to trigger a site-wide resolver crash back in May, when the oversigned .gov zone was mentioned on a list (this particular list, I think). That suggests looking in the inbound mail spool to see what might have been received at the time of the crash might be productive.

Regardless of how the query was started, if this theory of propagation is correct I'd suggest that posting the triggering name unobscured in an email message would be A Bad Thing, even if one is emailing it to ISC as they've suggested. Perhaps *especially* in that case, unless they've taken care to have one production recursor running Unbound ;)

Bill (who is downloading Unbound right now)

[bi...@namor.ca]

We had the same thing happen, across multiple, geographically-diverse
servers overnight, around the exact same time as the OP. That seems a
little odd to be an email, as it would have to cover a myriad of
destinations all at once.

While that's possible, I'm just finding it lacking as the sole reason for
the conclusion.

Using 9.7.3-P3 from ISC sources, here, too.

[Bill Owens]

On Wed, Nov 16, 2011 at 07:59:10AM -0600, bi...@namor.ca wrote:
> On Wed, 16 Nov 2011, Bill Owens wrote:
> >This behavior makes me bet that the trigger is a name in an incoming
> >email message, being resolved by an anti-spam filter.
>

> We had the same thing happen, across multiple, geographically-diverse
> servers overnight, around the exact same time as the OP. That seems a
> little odd to be an email, as it would have to cover a myriad of
> destinations all at once.
>
> While that's possible, I'm just finding it lacking as the sole reason for
> the conclusion.

Looks like I'll lose the bet - a NYSERNet member campus admin tells me that his campus servers were affected, but he runs individual copies of BIND on his mail servers specifically to handle the load anti-spam queries and they had no problems.

Bill.