Recovering from over enthusiastic key cleanup...

[Warren Kumari]

Hi all,

So, I decided to roll keys on a test zone (af7.org) -- of course, I decided to do this a: late at night and b: while juggling many other things.

So, I generated a new key and submitted my DS to my registrar, and deleted an older one - so far, all good, everything working fine. Problem solved, off to bed...

Oh! Hang on a sec, my keys directory is cluttered with old keys. I've just deleted the DS for one of them from the registrar, so it ain't being used, guess I'll remove the key files.
rm Kaf7.org.+005+27780.*
Yay, everything is still working, clean up some unrelated stuff, now really off to bed. Type 'rndc sign af7.org' for giggles, and.... well...

Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.578 general: info: received control channel command 'sign af7.org'
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.578 general: info: zone af7.org/IN/external: reconfiguring zone keys
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.579 general: warning: dns_dnssec_keylistfromrdataset: error reading private key file af7.org/RSASHA1/27780: file not found
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.580 general: warning: dns_dnssec_findzonekeys2: error reading private key file af7.org/RSASHA1/27780: file not found
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.581 general: notice: zone af7.org/IN/external: setting keywarntime to 01-Feb-2012 22:19:57.578
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.592 general: warning: zone af7.org/IN/external: Key af7.org/RSASHA1/27780 missing or inactive and has no replacement: retaining signatures.
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.603 general: info: zone af7.org/IN/external: next key event: 01-Feb-2012 23:19:57.603
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.604 general: warning: dns_dnssec_findzonekeys2: error reading private key file af7.org/RSASHA1/27780: file not found
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.699 xfer-out: info: client 75.102.1.178#59905: view external: transfer of 'af7.org/IN': IXFR started
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.699 xfer-out: info: client 75.102.1.178#59905: view external: transfer of 'af7.org/IN': IXFR ended
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: rdata.c:393: REQUIRE(((rdata)->data == ((void *)0) && (rdata)->length == 0 && (rdata)->rdclass == 0 && (rdata)->type == 0 &&\
(rdata)->flags == 0 && !((void *)(((rdata))->link.prev) != (void *)(-1)))) failed, back trace
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #0 0x413f2c in assertion_failed()+0x4c
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #1 0x57a97a in isc_assertion_failed()+0xa
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #2 0x4cc384 in dns_rdata_fromregion()+0x64
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #3 0x4ada8a in rdataset_current()+0x5a
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #4 0x540fb0 in del_sigs()+0x230
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #5 0x5515d7 in zone_sign()+0x7b7
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #6 0x555116 in zone_timer()+0x166
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #7 0x596ef9 in run()+0x1c9
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #8 0x7fac41b94971 in _fini()+0x7fac415e9699
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #9 0x7fac418f092d in _fini()+0x7fac41345655
Feb 1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: exiting (due to assertion failure)


Oh. Well, that is sad. Restart BIND. Boom, dies again...
Erm, restore keyfiles from backup (after trying to remember how the restore works, and what passphrase I used for this GPG key)...
Still no love.

Ended up removing the zone stanza from named.conf so I could start BIND and have a working nameserver, then running ldns-read-zone -s af7.org | grep -v DS | grep -v TYPE65 > af7.org, re-enabling the zone stanza, resigning with new keys, submitting new DS, etc...

So, is there:
A: an easy way to figure out what keyfiles are no longer being used / referenced?
B: a simpler way to recover from this when one *does* make a boo boo?

BIND apparently *tried* to continue running without being able to access the keyfile ("Key af7.org/RSASHA1/27780 missing or inactive and has no replacement: retaining signatures.") but then went "Boom".

W

[Spain, Dr. Jeffry A.]

> So, is there:
> A: an easy way to figure out what keyfiles are no longer being used / referenced?
> B: a simpler way to recover from this when one *does* make a boo boo?

What a fun evening. For the sake of interest, which version of bind is in use? With regard to item A, how about executing the following from your key directory:

for f in *.private; do echo; echo $f; dnssec-settime -p all "$f"; done

Any key file for which the Inactive time is in the past would not be needed for signing. Bind would publish it in the zone if the key file were present and the Delete time were in the future (and the Publish time in the past). Any key for which the Delete time is in the past would not need to be retained in the key directory, as it would not be needed for publication or signing.

With regard to B, I don't understand why restoring the deleted key files didn't fix the problem, and so will leave further comment to the experts.

Jeffry A. Spain
Network Administrator
Cincinnati Country Day School

[Warren Kumari]

On Feb 2, 2012, at 11:43 AM, Spain, Dr. Jeffry A. wrote:

>> So, is there:
>> A: an easy way to figure out what keyfiles are no longer being used / referenced?
>> B: a simpler way to recover from this when one *does* make a boo boo?
>
> What a fun evening. For the sake of interest, which version of bind is in use?

Doh. I always get annoyed with folk forget to include this... and then I did it :-P

BIND 9.8.1-P1 built with '--with-openssl=yes' '--with-randomdev=/dev/urandom' '--enable-threads'
using OpenSSL version: OpenSSL 0.9.8o 01 Jun 2010

> With regard to item A, how about executing the following from your key directory:
>
> for f in *.private; do echo; echo $f; dnssec-settime -p all "$f"; done
>
> Any key file for which the Inactive time is in the past would not be needed for signing. Bind would publish it in the zone if the key file were present and the Delete time were in the future (and the Publish time in the past). Any key for which the Delete time is in the past would not need to be retained in the key directory, as it would not be needed for publication or signing.

Hmmm. Yeah, that will work...

Thanks
W

[Mark Andrews]

Grab 9.8.2rc1, it will address the the require assertion.

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org