inline-signing: SOA serial out of sync

[Axel Rau]

Hi all,

occasionally named 9.11.3 fails to increment SOA serial like here:

file: 2018060605 dns: 2018060604

zone file was edited by script and a rndc reload given.
This usually works perfect, but here:

Only entry in log file:

notify: debug 3: zone lrau.net/IN (signed): sending notify to …

Config detail:

key-directory "master/signed/lrau.net/";
auto-dnssec maintain;
inline-signing yes;
dnssec-secure-to-insecure no;

Manual fixing requires another cycle with zone file editing:

——-——-
[hermes:master/signed/lrau.net] root# service named stop
Stopping named.
Waiting for PIDS: 37110.
[hermes:master/signed/lrau.net] root# ls -l *.jbk *.jnl *.signed
-rw-r--r-- 1 bind pki_op 512 Jan 11 13:12 lrau.net.zone.jbk
-rw-r--r-- 1 bind pki_op 16409 Jun 6 21:05 lrau.net.zone.jnl
-rw-r--r-- 1 bind pki_op 50263 Jun 6 21:19 lrau.net.zone.signed
-rw-r--r-- 1 bind pki_op 682052 Jun 6 21:05 lrau.net.zone.signed.jnl
[hermes:master/signed/lrau.net] root# rm *.jbk *.jnl *.signed
[hermes:master/signed/lrau.net] root# service named start
Starting named.
[hermes:master/signed/lrau.net] root# ls -l *.jbk *.jnl *.signed
-rw-r--r-- 1 bind pki_op 512 Jun 7 12:37 lrau.net.zone.jbk
-rw-r--r-- 1 bind pki_op 8222 Jun 7 12:37 lrau.net.zone.signed
-rw-r--r-- 1 bind pki_op 57521 Jun 7 12:37 lrau.net.zone.signed.jnl
[hermes:master/signed/lrau.net] root# dig SOA lrau.net @localhost

; <<>> DiG 9.11.3 <<>> SOA lrau.net @localhost
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36163
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 7

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 9abf10cb4372b10e0eae26085b190b0d3486a4bef440b95c (good)
;; QUESTION SECTION:
;lrau.net. IN SOA

;; ANSWER SECTION:
lrau.net. 86400 IN SOA ns4.lrau.net. hostmaster.lrau.net. 2018060632 86400 7200 604800 3600
. . .
[hermes:local/etc/namedb] root# named-checkzone lrau.net master/signed/lrau.net/lrau.net.zone
zone lrau.net/IN: loaded serial 2018060606 <<<<<< still not in sync
OK

# edited zone file manually (serial set to 2018060640):

[hermes:master/signed/lrau.net] root# rndc reload
server reload successful
[hermes:local/etc/namedb] root# named-checkzone lrau.net master/signed/lrau.net/lrau.net.zone
zone lrau.net/IN: loaded serial 2018060640
OK
[hermes:master/signed/lrau.net] root# dig SOA lrau.net. @localhost
. . .
;; ANSWER SECTION:
lrau.net. 86400 IN SOA ns4.lrau.net. hostmaster.lrau.net. 2018060640 86400 7200 604800 3600
——————


What is going wrong here?
What can I do to get this fixed?

Thanks, Axel
---
PGP-Key:29E99DD6 ☀ computing @ chaos claudius

[Tony Finch]

Axel Rau <Axel...@Chaos1.DE> wrote:
>
> occasionally named 9.11.3 fails to increment SOA serial like here:
>
> file: 2018060605 dns: 2018060604

With inline signing the signed and unsigned zones have separate serial
numbers, so this is normal. If I understand inline-signing correctly, when
you only modify the unsigned zone's serial number, that is not a big
enough change to require an update to the signed version of the zone.

You can use `rndc zonestatus` to see the server's view of both serial
numbers.

You can use `rndc signing -serial` to set the serial number of the signed
zone.

You might want to set `serial-update-method` if you want something more
meaningful than an increment for each update (e.g. `date`).

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
champion the freedom, dignity, and well-being of individuals

[Matthew Pounsett]

On 7 June 2018 at 07:36, Axel Rau <Axel...@chaos1.de> wrote:

Hi all,

occasionally named 9.11.3 fails to increment SOA serial like here:

        file: 2018060605 dns: 2018060604

zone file was edited by script and a rndc reload given.

[...] 

Manual fixing requires another cycle with zone file editing:

 

You don't say this clearly, but it sounds like you're reporting more than just the serial not updating.  Is that correct?  Are there actual updates to the zone that are not being picked up?  As Tony says, the serial number can differ from the file to what's served by the name server when the name server is doing automatic signing.

Can you clarify which it is?

[Axel Rau]

Hi Tony,

sorry for the late replay.

Am 07.06.2018 um 14:20 schrieb Tony Finch <d...@dotat.at>:

Axel Rau <Axel...@Chaos1.DE> wrote:

occasionally named 9.11.3 fails to increment SOA serial like here:

file: 2018060605 dns: 2018060604

With inline signing the signed and unsigned zones have separate serial
numbers, so this is normal. If I understand inline-signing correctly, when
you only modify the unsigned zone's serial number, that is not a big
enough change to require an update to the signed version of the zone.

I changed a RR and the serial. Immediately after such a change, both

serials are usually equal, which my script checks.

If thea are different, this usually indicates some error with signing.

You can use `rndc zonestatus` to see the server's view of both serial
numbers.

I see.

You can use `rndc signing -serial` to set the serial number of the signed
zone.

You might want to set `serial-update-method` if you want something more
meaningful than an increment for each update (e.g. `date`).

OK.

Thanks for responding,

[Axel Rau]

Hi Matthew,

sorry for my late answer.

> Am 07.06.2018 um 15:31 schrieb Matthew Pounsett <ma...@conundrum.com>:
>
>
>
> On 7 June 2018 at 07:36, Axel Rau <Axel...@chaos1.de> wrote:
> Hi all,
>

> occasionally named 9.11.3 fails to increment SOA serial like here:
>
> file: 2018060605 dns: 2018060604
>

> zone file was edited by script and a rndc reload given.
> [...]
> Manual fixing requires another cycle with zone file editing:
>
>
> You don't say this clearly, but it sounds like you're reporting more than just the serial not updating. Is that correct?

Yes.

> Are there actual updates to the zone that are not being picked up?

Yes, that’s the point. If the problem happens, the signing machinery is blocked until resolved manually.
I don’t know the reason. named-checkzone reported no errors, but in case of syntax-errors, named behaves similar.

> As Tony says, the serial number can differ from the file to what's served by the name server when the name server is doing automatic signing.
>
> Can you clarify which it is?

I hope, I did (-:

There is nothing special with this zone file:

- - -
[hermes:~] root# rndc zonestatus lrau.net
name: lrau.net
type: master
files: master/signed/lrau.net/lrau.net.zone, master/signed/lrau.net/caldav.lrau.net.tlsa, master/signed/lrau.net/git3.lrau.net.tlsa, master/signed/lrau.net/git4.lrau.net.tlsa, master/signed/lrau.net/lists3.lrau.net.tlsa, master/signed/lrau.net/lists4.lrau.net.tlsa, master/signed/lrau.net/mailout3.lrau.net.tlsa, master/signed/lrau.net/mailout4.lrau.net.tlsa, master/signed/lrau.net/mx3.lrau.net.tlsa, master/signed/lrau.net/mx4.lrau.net.tlsa, master/signed/lrau.net/timap3.lrau.net.tlsa, master/signed/lrau.net/tmx3.lrau.net.tlsa, master/signed/lrau.net/acme_challenges.inc
serial: 2018060805
signed serial: 2018060805
nodes: 88
last loaded: Thu, 07 Jun 2018 10:37:34 GMT
secure: yes
inline signing: yes
key maintenance: automatic
next key event: Sat, 09 Jun 2018 13:08:21 GMT
next resign node: gw2.m6d2.lrau.net/NSEC
next resign time: Fri, 29 Jun 2018 21:38:07 GMT
dynamic: no
reconfigurable via modzone: no

[hermes:local/etc/namedb] root# named-checkzone lrau.net /usr/local/etc/namedb/master/signed/lrau.net/lrau.net.zone
zone lrau.net/IN: loaded serial 2018060805
OK
- - -

Thanks, Axel

[Axel Rau]

Am 07.06.2018 um 13:36 schrieb Axel Rau <Axel...@chaos1.de>:

occasionally named 9.11.3 fails to increment SOA serial like here:

file: 2018060605 dns: 2018060604

It just happened again. An included zone file has been changed from 2 TLSA RRs to one:

- - -

_443._tcp.git.nussberg.de. 3600 IN TLSA 3 0 1 DAE0AC343A6694DEAF0BAB42FC8A6B1F82E42799654BD667B458DC91655C6AB4

- - -

After reload no TLSAs are picked up by the server:

- - -

[hermes:local/etc/rc.d] root# dig AXFR nussberg.de. @localhost | grep TLSA

[hermes:local/etc/rc.d] root#

- - -

Zone status:

- - -

[hermes:local/etc/rc.d] root# rndc zonestatus nussberg.de

name: nussberg.de

type: master

files: master/signed/nussberg.de/nussberg.de.zone, master/signed/nussberg.de/git.nussberg.de.tlsa, master/signed/nussberg.de/acme_challenges.inc

serial: 2018061301

signed serial: 2018060702

nodes: 12

last loaded: Tue, 05 Jun 2018 07:08:59 GMT

secure: yes

inline signing: yes

key maintenance: automatic

next key event: Thu, 14 Jun 2018 10:05:11 GMT

next resign node: email1._domainkey.nussberg.de/TXT

next resign time: Sun, 17 Jun 2018 19:29:37 GMT

dynamic: no

reconfigurable via modzone: no

- - -

What else can I collect to help fixing this?

Thanks, Axel

PS: Why does

dig TLSA _443._tcp.git.nussberg.de. @localhost

not work at all?

[Matthew Pounsett]

On 14 June 2018 at 06:27, Axel Rau <Axel...@chaos1.de> wrote:

Am 07.06.2018 um 13:36 schrieb Axel Rau <Axel...@chaos1.de>:

occasionally named 9.11.3 fails to increment SOA serial like here:

file: 2018060605 dns: 2018060604

It just happened again. An included zone file has been changed from 2 TLSA RRs to one:

- - -

_443._tcp.git.nussberg.de. 3600 IN TLSA 3 0 1 DAE0AC343A6694DEAF0BAB42FC8A6B1F82E42799654BD667B458DC91655C6AB4

- - -

After reload no TLSAs are picked up by the server:

- - -

[hermes:local/etc/rc.d] root# dig AXFR nussberg.de. @localhost | grep TLSA

[hermes:local/etc/rc.d] root#

This now sounds very different from the original report.  Are you saying that the zone started with two TLSA records, you changed it to have only one, reloaded the zone, but then none were present?

That's a very different problem from just not picking up a zone update.

Have you checked the logs for errors during zone loading?  

[Alan Clegg]

On 6/14/18 9:44 AM, Matthew Pounsett wrote:

> It just happened again. An included zone file has been changed from
> 2 TLSA RRs to one:

[...]

> This now sounds very different from the original report.  Are you saying
> that the zone started with two TLSA records, you changed it to have only
> one, reloaded the zone, but then none were present?
>
> That's a very different problem from just not picking up a zone update.
>
> Have you checked the logs for errors during zone loading? 

Additionally, I read this as "the records changed are in an included
file" -- is the serial number in the "including" zone being incremented?

AlanC

[Axel Rau]

Am 14.06.2018 um 16:12 schrieb Alan Clegg <al...@clegg.com>:

Additionally, I read this as "the records changed are in an included
file" -- is the serial number in the "including" zone being incremented?

Yes.

Axel

[Axel Rau]

Am 14.06.2018 um 15:44 schrieb Matthew Pounsett <ma...@conundrum.com>:

This now sounds very different from the original report.  Are you saying that the zone started with two TLSA records, you changed it to have only one, reloaded the zone, but then none were present?

Yes.

That's a very different problem from just not picking up a zone update.

Have you checked the logs for errors during zone loading?  

 zone nussberg.de/IN (signed): reconfiguring zone keys

 zone nussberg.de/IN (signed): next key event: 13-Jun-2018 21:05:10.419

 zone nussberg.de/IN (unsigned): loaded serial 2018061301

 zone nussberg.de/IN (signed): receive_secure_serial: not exact

 zone nussberg.de/IN (signed): reconfiguring zone keys

 zone nussberg.de/IN (signed): next key event: 13-Jun-2018 22:05:10.428

[Matthew Pounsett]

On 14 June 2018 at 10:16, Axel Rau <Axel...@chaos1.de> wrote:

Am 14.06.2018 um 16:12 schrieb Alan Clegg <al...@clegg.com>:

Additionally, I read this as "the records changed are in an included
file" -- is the serial number in the "including" zone being incremented?

Yes.

I think at this point you're going to need to provide a lot more detail about your configuration, and what's special about these zones.  It sounds like this is not a straightforward configuration, and it's hard to figure out from the bits and pieces you've provided what could be going wrong.

 

[Axel Rau]

> Am 14.06.2018 um 18:30 schrieb Axel Rau <Axel...@chaos1.de>:
>
> I include the zone file with the 2 included files, a AXFR dump of it and the options and zone statement (which is not in a view) of the server config in a zip archiv.

I saw no comments on the provided data, so I assume, nobody has a clue on this.

To summarise:

Occasionally it happens after rndc reload that the serial in the zone file is bigger than that in served SOA.
In this case, named begins serving stale data.
Probability for this to happen increases with size of the journal file.

I’m using auto-dnssec maintain; inline-signing yes; serial-update-method increment;
The ARM does not state clearly, which is the base of the increment if the zone file changes (file or internal).
In my case, the served serial is based on the zone file and usually bigger than that.

Question: Could the problem arise by incrementing the serial without changing the zone data?
(Occasionally this could happen with my script)

I have upgraded to bind 9.12.1P2 and look forward. . .

Thanks, Axel

[Stefan Förster]

Hi,

* Axel Rau <Axel...@Chaos1.DE>:

>Occasionally it happens after rndc reload that the serial in the zone file is bigger than that in served SOA.
>In this case, named begins serving stale data.
>Probability for this to happen increases with size of the journal file.

I used to see something similar (although views were involved), where
BIND was not picking up changes to a zone when only included files were
changed:

https://lists.isc.org/pipermail/bind-users/2017-September/099145.html

The setup then was something like:

<master zone>
[data]
$INCLUDE "ns.include"

<ns.include>
@ IN NS ...
@ IN NS ...

If needed, I can provide the real zone data. I didn't encounter this
problem again after I quit using views.


Cheers,
Stefan

[Axel Rau]

> Am 21.06.2018 um 08:18 schrieb Stefan Förster via bind-users <bind-...@lists.isc.org>:
>
>
> I used to see something similar (although views were involved), where BIND was not picking up changes to a zone when only included files were changed:
>
> https://lists.isc.org/pipermail/bind-users/2017-September/099145.html

Thanks for pointing this out.

My problem happened again with bind 9.12.1.
As you can see below, „next key event" is in the past!

Axel

PS:
—-
[hermes:master/signed/lrau.net] root# rndc zonestatus lrau.net

name: lrau.net
type: master
files: master/signed/lrau.net/lrau.net.zone, master/signed/lrau.net/caldav.lrau.net.tlsa, master/signed/lrau.net/git3.lrau.net.tlsa, master/signed/lrau.net/git4.lrau.net.tlsa, master/signed/lrau.net/lists3.lrau.net.tlsa, master/signed/lrau.net/lists4.lrau.net.tlsa, master/signed/lrau.net/mailout3.lrau.net.tlsa, master/signed/lrau.net/mailout4.lrau.net.tlsa, master/signed/lrau.net/mx3.lrau.net.tlsa, master/signed/lrau.net/mx4.lrau.net.tlsa, master/signed/lrau.net/timap3.lrau.net.tlsa, master/signed/lrau.net/tmx3.lrau.net.tlsa, master/signed/lrau.net/acme_challenges.inc

serial: 2018062201
signed serial: 2018062201
nodes: 88
last loaded: Fri, 15 Jun 2018 15:17:08 GMT

secure: yes
inline signing: yes
key maintenance: automatic

next key event: Fri, 22 Jun 2018 21:07:51 GMT
next resign node: _25._tcp.lists4.lrau.net/NSEC
next resign time: Fri, 29 Jun 2018 21:38:07 GMT

dynamic: no
reconfigurable via modzone: no

[hermes:master/signed/lrau.net] root# date
Sat Jun 23 11:08:53 CEST 2018
[hermes:master/signed/lrau.net] root# grep Serial lrau.net.zone
2018062202 ; Serial number