slave zone files unreadable

[Manuel Ramirez Montero]

Hi,

since i have upgraded to 9.9.5-P1 is not possible to read slave zone files.

I have read an article about this :

https://kb.isc.org/article/AA-00608/0/Converting-Zone-Files-Between-Text-and-Raw-Formats.html

convert raw zone file "example.net.raw", containing data for zone example.net, to text-format zone file "example.net.text"
named-compilezone -f raw -F text -o example.net.text example.net example.net.raw

*************************************

This is the configuration for one of the slaves zone i´m having problems in the named.conf :

zone "movilmap.es" in {
                type slave;
                file "/var/named/slaves/movilmap.es.hosts";
                masters { 10.1.29.179; };
                allow-update-forwarding { dns; };
                };

What should be the correct syntax for the named-compilezone in my case?

Thanks

[Reindl Harald]

Am 09.07.2014 10:29, schrieb Manuel Ramirez Montero:
> since i have upgraded to 9.9.5-P1 is not possible to read slave zone files.
> I have read an article about this :
>
> https://kb.isc.org/article/AA-00608/0/Converting-Zone-Files-Between-Text-and-Raw-Formats.html
>

> convert raw zone file "example.net.raw", containing data for zone example.net <http://example.net>, to text-format
> zone file "example.net.text"
> named-compilezone -f raw -F text -o example.net.text example.net <http://example.net> example.net.raw

>
> *************************************
> This is the configuration for one of the slaves zone i´m having problems in the named.conf :
>

> zone "movilmap.es <http://movilmap.es>" in {

> type slave;
> file "/var/named/slaves/movilmap.es.hosts";
> masters { 10.1.29.179; };
> allow-update-forwarding { dns; };
> };
>
> What should be the correct syntax for the named-compilezone in my case?

dunno, but i perfer text-format anyways

* masterfile-format text;
* delete the zone file on the slave
* restart the slave

P.S.: please don't post HT;L on mailing-lists, you see above what happens with quotes

[Anand Buddhdev]

On 09/07/2014 13:21, Reindl Harald wrote:

> dunno, but i perfer text-format anyways
>
> * masterfile-format text; * delete the zone file on the slave *
> restart the slave

Plain text zone files are fine if you have a small number of zones, or
small zones. But for servers with large numbers of zones, or large
zones, reading back from text files when starting the server can take
a long time. The "raw" format helps by speeding up load time. The
newer "map" format in BIND 9.10 is even better, and can improve zone
load time over the "raw" format by a factor of 3.

As an example, on one of my BIND slaves with over 5000 zones, reading
from plain text zone files takes about 90 seconds, from raw files, it
takes 45 seconds, and from map files, just 15 seconds.

Regards,

Anand

[Reindl Harald]

agreed, but in case of debugging temporary switch to text
makes thins easier to watch including sniff the traffic
of zone-transfers - made it as example possible for me
to *really* find out that the master was sending expected
zone content and the cisco router between mangeled it

[Reindl Harald]

however, i wonder what takes 90 seconds to load 5000 zones

our master has 533 zones, zone files generated with self written
scripts as plaintext and loads them due a hard restart within
0.3 seconds and all notifies to the slave happen in the same second

the records-sql table has 3000 entries for all zones (backend
generates the zone-files in another table as text-field and
that content is fetched in case of changes from cronobs on the
nameservers and written down to zonefiles in case of changes)

[root@ns2:~]$ cat named.log | grep "loaded serial" | wc -l
533

09-Jul-2014 15:06:30.734 general: exiting
09-Jul-2014 15:06:30.827 general: managed-keys-zone: loaded serial 0
09-Jul-2014 15:06:30.948 general: all zones loaded
09-Jul-2014 15:06:30.968 notify: zone ***/IN: sending notifies (serial 2014070201)

last message on the slave - all happens at 15:06
09-Jul-2014 15:06:57.961 general: zone ***/IN: notify from 10.0.0.16#15394: zone is up to date

[Phil Mayers]

On 09/07/14 14:16, Reindl Harald wrote:

> however, i wonder what takes 90 seconds to load 5000 zones

Depends how big they are.

> the records-sql table has 3000 entries for all zones (backend

That is not very big. We've got zones with nearly 1M records in them,
including NSEC/RRSIG.

[Evan Hunt]

On Wed, Jul 09, 2014 at 03:16:04PM +0200, Reindl Harald wrote:
> however, i wonder what takes 90 seconds to load 5000 zones

It scales with both the number of zones and the number of records per
zone. Some of those zones are probably quite large.

When you're reading text, it takes time to do the lexical analysis and
parsing. When reading text *or* raw, it takes time to examine each node in
the zone file, determine its name, walk down through the nodes of a growing
red-black tree, allocate memory, and add the data.

A map file is a memory image of a fully-formed red-black tree; it can
be zapped into memory in one go, then we walk through the tree validating
checksums and updating the pointers, which is obviously much quicker.

(In fact it could be almost instant if we did the checksums-and-pointers
bit lazily, as the data was accessed rather than immediately at load time.
That introduces a lot of complexity, though; if a zone file is corrupt,
BIND expects to discover the fact right away, not at some random time
later on.)

--
Evan Hunt -- ea...@isc.org
Internet Systems Consortium, Inc.

[John Wobus]

In cases analogous to this, software often saves both
text and binary, and when initializing, uses mtime to
decide whether it can safely use the binary. Some resources
are spent storing the extra file and admins have yet
another way to screw things up, but the strategy
does have benefits.

John Wobus
Cornell University IT

> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to
> unsubscribe from this list
>
> bind-users mailing list
> bind-...@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users

[Evan Hunt]

On Fri, Jul 11, 2014 at 12:12:22PM -0400, John Wobus wrote:
> In cases analogous to this, software often saves both
> text and binary, and when initializing, uses mtime to
> decide whether it can safely use the binary. Some resources
> are spent storing the extra file and admins have yet
> another way to screw things up, but the strategy
> does have benefits.

You can't always rely on mtime for that. The disk may have been hit by
cosmic rays since last time you read the file, and mtime hasn't changed but
the file is different anyway.

If named had been designed from the beginning with the supposition that it
might, at any time, discover that its database was corrupt, and simply
stopped serving that particular zone until corrective measures had been
taken, then that would be okay. But named is designed to verify its
data thoroughly at load time, and then throw an assertion failure if
anything looks wrong later. The assumption that the database is reliable,
and if it isn't we should crash, runs all through the code; it would be a
huge job of work to change it.

So with map files, we load the database image with mmap(), then walk down
through the tree and verify the checksums. That means the whole database
gets paged into physical memory at load time, and that's why map files
don't load instantly. Still 3-4 times faster than raw, though.

(The idea of keeping both text and map versions of master files around,
though -- loading the zone from the map, or else loading from text and
creating a new map, depending on mtime -- is definitely under consideration
and may turn up in 9.11 if time permits.)

[Barry Margolin]

In article <mailman.559.1405100...@lists.isc.org>,

Evan Hunt <ea...@isc.org> wrote:

> On Fri, Jul 11, 2014 at 12:12:22PM -0400, John Wobus wrote:
> > In cases analogous to this, software often saves both
> > text and binary, and when initializing, uses mtime to
> > decide whether it can safely use the binary. Some resources
> > are spent storing the extra file and admins have yet
> > another way to screw things up, but the strategy
> > does have benefits.
>
> You can't always rely on mtime for that. The disk may have been hit by
> cosmic rays since last time you read the file, and mtime hasn't changed but
> the file is different anyway.

BIND already assumes mtime is reliable -- if you do "ndc reload", it
only reloads zone files whose mtimes are newer than when the zone was
previously loaded.

If you're worried about cosmic rays, you can't rely on ANYTHING on the
disk. Why would you consider the file contents to be reasonable if they
could change spontaneously.

Anyway, don't all modern disks have ECC codes in them? These will detect
and correct bit flips in the mtime just as well as the contents. If you
want extra safety, use RAID.

--
Barry Margolin
Arlington, MA

[Evan Hunt]

On Fri, Jul 11, 2014 at 08:23:34PM -0400, Barry Margolin wrote:
> BIND already assumes mtime is reliable -- if you do "ndc reload", it
> only reloads zone files whose mtimes are newer than when the zone was
> previously loaded.

Of course, but it *checks them for validity* when it loads them.
If the file's corrupt, it logs an error and carries on serving the
previous version.

With map files, we had a choice:

a) check validity during load (which takes a bit time),
b) mmap() into memory without validating, and hope really hard that
the file isn't corrupt (which gives you near-instant server startup,
but could cause assertion failures or serve bad data if there was a
problem), or
c) mmap() the file into memory without validating, and rewrite named
to cope robustly with zone database corruption if it's detected
later on (which turns out to be Hard).

We decided to spend the time and validate map files before serving
data from them.

[Mark Andrews]

In message <barmar-79D854....@news.eternal-september.org>, Barry Margolin writes:
> In article <mailman.559.1405100...@lists.isc.org>,
> Evan Hunt <ea...@isc.org> wrote:
>

> > On Fri, Jul 11, 2014 at 12:12:22PM -0400, John Wobus wrote:
> > > In cases analogous to this, software often saves both
> > > text and binary, and when initializing, uses mtime to
> > > decide whether it can safely use the binary. Some resources
> > > are spent storing the extra file and admins have yet
> > > another way to screw things up, but the strategy
> > > does have benefits.
> >
> > You can't always rely on mtime for that. The disk may have been hit by
> > cosmic rays since last time you read the file, and mtime hasn't changed but
> > the file is different anyway.
>

> BIND already assumes mtime is reliable -- if you do "ndc reload", it
> only reloads zone files whose mtimes are newer than when the zone was
> previously loaded.
>

> If you're worried about cosmic rays, you can't rely on ANYTHING on the
> disk. Why would you consider the file contents to be reasonable if they
> could change spontaneously.
>
> Anyway, don't all modern disks have ECC codes in them? These will detect
> and correct bit flips in the mtime just as well as the contents. If you
> want extra safety, use RAID.

The real problem is humans. They like to tinker with files (hence
the subject line). There really shouldn't be a reason for anyone
to need to read slave database files. They are there so named can
have the zone content when it starts up rather than having to
re-transfer the content at startup. If you need the contents of the
zone axfr them from the server. That way you actually get up to
date content not 15 minute old content.

If we could get people away from wanting to use a editor on master
files directly we would. The practice is highly error prone even
for experts.

> --
> Barry Margolin
> Arlington, MA

> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
>
> bind-users mailing list
> bind-...@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

[Reindl Harald]

Am 12.07.2014 03:08, schrieb Mark Andrews:
> If we could get people away from wanting to use a editor on master
> files directly we would. The practice is highly error prone even
> for experts.

uhm people wrote interfaces to generate them :-)

i am one of that people because no other software
is flexible enough or comes with dependency hell

please don't consider future versions making that impossible
__________________________________________________________

there are translations for NAT/WAN as example to have 500
zones only once maintained, generated from sql tables into
text fields and a replace table maintains a copy for the
internal LAN , customized for the environment and deeply
integrated in other admin backends (epp->dns->mail/web server)

modify them manually is not always bad :-)

[Alan Clegg]

On 7/11/14, 9:41 PM, Reindl Harald wrote:

> i am one of that people because no other software
> is flexible enough or comes with dependency hell

nsupdate

If BIND is installed, no dependencies and about as flexible as you can get.

AlanC

[Noel Butler]

On 12/07/2014 11:08, Mark Andrews wrote:
>
> The real problem is humans. They like to tinker with files (hence
> the subject line). There really shouldn't be a reason for anyone
> to need to read slave database files. They are there so named can
> have the zone content when it starts up rather than having to
> re-transfer the content at startup. If you need the contents of the
> zone axfr them from the server. That way you actually get up to
> date content not 15 minute old content.
>

> If we could get people away from wanting to use a editor on master
> files directly we would. The practice is highly error prone even
> for experts.
>

Most management systems in hosting comps typically open file < blah EOF
and stuff, so maybe 99.999999999999% of the internet :D (of course
these, and those of us who know how to write them by hand have no
trouble - because we all learnt the hard way at some time)


Also, I may be having a blonde moment (got a nasty case of te flu at
present) but whatever happened to the once discussed advantages of
having bind load zone files in the same way Apache httpd does using
(Include/IncludeOptional sompath_under_"directory"/* ), if the zone is
there it loads it, if not, it doesnt/ignores it - not just bail out
completely, that removes the dangers of a corrupted named.conf with tens
of thousands of zones. Testing showed with 11.5K hosts, the load time
was only 3 or so seconds longer IIRC (maybe less), not bad for peace of
mind ('n yes I know in DNS 3 seconds is a long time, but WTF takes pri
and sec's offline at same time (ok I guess the clowns who run them both
on hte same cheap over subscribed VPS but thats another rant for another
day)

[Reindl Harald]

i talk about web-interfaces generating complete zonefiles from
scratch out of a records table, orchestrating 4 remote nameservers
with the data which must be also maintainable by normal operators
and last but not least perfectly integrated in already self
developed admin backends for other services - well, and since the
nameservers are pulling ready-to-use zone-content via cronscript
additional nameservers could be added without touch anything but
sql permissions on the internal infrastructure

"This allows resource records to be added or removed from a
zone without manually editing the zone file" don't create you
a zone from scratch based on database records nor adds it the
zone from "named.conf" or would remove it based on a webbackend

such command line tools are nice but not useable to do the same
as a database backed webinterface maintaining 4 nameservers with
different IP addresses for two of them in case of otherwise
mirrored records for each zone - how do you integrate this in
cronjobs? in our case the cronjob pulls the complete zone-content
out from a database and writes it to disk, the single records are
to re-create the two textfields with the zone internal and WAN

additionally it's completly error-prone implement that way triggers
like "oh that domain got recently a mail-address in DBMail so now
we add a MX record if not already there as well as autoconfig and
autodiscover in the database via the DNS-API class" which does
codewise the same as if you would add that 3 records in the webUI

[Gary Wallis]

DNS experts,

What are the drawbacks, if any, of running only master name servers for
the set of authoritative NSs?

For example given:

[root@rc37 unxsVZ]# dig latimes.com NS +short
dns1.tribune.com.
dns2.tribune.com.
dns4.tribune.com.
dns3.tribune.com.

Where all 4 dnsN servers are in fact masters (this is just a
hypothetical, the NS above are most likely secondary servers)

Thank you!

[Reindl Harald]

practically none if all is going fine

if you are making a config mistake preventing named to work
it makes a difference because the master goes down and the
slaves have no chance to pull the mistake

been there done that for ISP breaking zone-transfer reasons
______________________________________________________________

example:

* subdomain1.example.com -> CNAME to whatever
* later a mailsub-domain get addeded
* you add MX subdomain1.example.com
* named won't load that zone because CNAME and others are not allowed
* the slave has no chance to pull such breakage

well, that mistake happened years ago and needed to be fixed
in our dns-backend to not allow, however at that time the
secondary nameserver was a slave and nothing happened

if both would have been configured as master and get the
same input the zone would have gone offline

[Alan Clegg]

On 7/12/14, 5:33 AM, Reindl Harald wrote:

> Am 12.07.2014 04:48, schrieb Alan Clegg:

>> nsupdate
>>
>> If BIND is installed, no dependencies and about as flexible as you can get
>
> i talk about web-interfaces generating complete zonefiles from
> scratch out of a records table, orchestrating 4 remote nameservers
> with the data which must be also maintainable by normal operators

> [...]

Believe it or not, I've done setups like what you ask for using database
backends, web front-ends and multiple "remote" servers with the DNS
infrastructure being maintained nearly completely using nsupdate.

I wasn't providing a full implementation, that costs money.

AlanC

[Doug Barton]

Please don't reply to a message on the list and change the subject line.
Doing so causes your new topic to show "under" the previous one for
those using mail readers that thread properly, and may cause your
message to be missed altogether if someone has blocked that thread.

Instead, save the list address and start a completely new message.

hope this helps,

Doug

[Gary Wallis]

Hello,

What are the drawbacks, if any, of running only master name servers for
the set of authoritative NSs?

For example given:

[root@rc37 unxsVZ]# dig latimes.com NS +short
dns1.tribune.com.
dns2.tribune.com.
dns4.tribune.com.
dns3.tribune.com.

Where all 4 dnsN servers are in fact masters (this is just a
hypothetical, the NS above are most likely secondary servers)

Thank you!

[Mike Hoskins (michoski)]

I'm not aware of any drawbacks from a strict DNS perspective. There could
be administrative overhead depending how you set it up, but we have hidden
masters which allow central control of our "public masters" which in turn
serve a few zones + act as origins for services like Akamai. It's worked
well for us over the past decade.

[Reindl Harald]

Am 12.07.2014 16:11, schrieb Gary Wallis:
> DNS experts,
>

> What are the drawbacks, if any, of running only master name servers for the set of authoritative NSs?
>
> For example given:
>
> [root@rc37 unxsVZ]# dig latimes.com NS +short
> dns1.tribune.com.
> dns2.tribune.com.
> dns4.tribune.com.
> dns3.tribune.com.
>
> Where all 4 dnsN servers are in fact masters (this is just a hypothetical, the NS above are most likely secondary
> servers)

why do you start the same thread again?

-------- Original-Nachricht --------
Betreff: Re: Public facing authoritative NS all masters
Datum: Sat, 12 Jul 2014 16:23:15 +0200
Von: Reindl Harald <h.re...@thelounge.net>
An: bind-...@lists.isc.org

Am 12.07.2014 16:11, schrieb Gary Wallis:
> DNS experts,
>

> What are the drawbacks, if any, of running only master name servers for the set of authoritative NSs?
>
> For example given:
>
> [root@rc37 unxsVZ]# dig latimes.com NS +short
> dns1.tribune.com.
> dns2.tribune.com.
> dns4.tribune.com.
> dns3.tribune.com.
>
> Where all 4 dnsN servers are in fact masters (this is just a hypothetical, the NS above are most likely secondary
> servers)

[Joseph S D Yao]

On 2014-07-13 12:11, Gary Wallis wrote:
> Hello,

>
> What are the drawbacks, if any, of running only master name servers
> for the set of authoritative NSs?
>
> For example given:
>
> [root@rc37 unxsVZ]# dig latimes.com NS +short
> dns1.tribune.com.
> dns2.tribune.com.
> dns4.tribune.com.
> dns3.tribune.com.
>
> Where all 4 dnsN servers are in fact masters (this is just a
> hypothetical, the NS above are most likely secondary servers)

...

If you think about it, it is not the servers themselves that are master
or slave. For each zone, it is the copy of that zone that is considered
a master copy on that server, or a copy slaved to the copy on another
server. And this can be different for each zone served on those
servers.

There should in fact be only one master copy that you change, for each
zone, and the other name servers somehow get an identical copy of that
master copy.

Whether all four of the visible authoritative name servers are
configured to have "master" in their internal configurations, saying to
use the copy of the zone found on disk, or "slave", saying to use the
DNS standard method of slaving the server's copy to a copy of that zone
found on another server, is in fact invisible to users and irrelevant to
them. What is important is that all servers declared as authoritative
for that one zone DO serve that zone, and respond with the same
information. All four could be configured as having a "master" copy of
a zone and get it by other means from a hidden master copy, or all four
could be configured as having "slave" copies slaved to, again, a hidden
master copy; or some combination (e.g., 4's copy is slaved to 3's copy
which is slaved to 2's copy, and both 1 and 2 are declared as having
master copies and get them via SCP from a hidden master, just to get
complicated).

Now, what was the reason you asked this question? Is there an
underlying question behind that?

Joseph Yao

[Tony Finch]

Gary Wallis <wgg...@gmail.com> wrote:
>
> What are the drawbacks, if any, of running only master name servers for the
> set of authoritative NSs?

That depends entirely on how you are replicating the zone data.

The DNS's own replication (AXFR, IXFR, NOTIFY, TSIG) is pretty hard to
beat: it is fast, secure, and copes gracefully with outages.

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
Shannon: Southwest veering west 5 to 7, decreasing 4 later. Moderate or rough.
Rain at first. Good, occasionally poor at first.

[Gary Wallis]

Thank you Tony and Joseph,

I think you have explained this well, and most importantly, exposed the
underlying issues.

Best regards,
Gary