[Samba] DomainDnsZone Replication Shows 200,000 Objects
lp101
that need to be replicated. This takes several hours to complete if at
all. I don't believe this to be correct. I'm currently running Samba
4.1.0 on several DC's across a couple sites. Tried to join a new DC
using Samba4.1.0 as well but it failed with an error code similar to the
one found here
https://lists.samba.org/archive/samba/2013-October/176237.html.
Reverted back to a 4.0.9 build and it completed the join process
without this error. I would like to join another DC but it takes an
excessive amount of time to replicate the DomainDnsZone partition. I
can't fathom this containing 200,000+ objects. My domain consist of
approximately 125 users and 150 machines. Thanks for any help.
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba
Andrew Bartlett
> During the samba domain join process I see over 200,000+ objects
> that need to be replicated. This takes several hours to complete if at
> all. I don't believe this to be correct. I'm currently running Samba
> 4.1.0 on several DC's across a couple sites. Tried to join a new DC
> using Samba4.1.0 as well but it failed with an error code similar to the
> one found here
>
> https://lists.samba.org/archive/samba/2013-October/176237.html.
>
> Reverted back to a 4.0.9 build and it completed the join process
> without this error. I would like to join another DC but it takes an
> excessive amount of time to replicate the DomainDnsZone partition. I
> can't fathom this containing 200,000+ objects. My domain consist of
> approximately 125 users and 150 machines. Thanks for any help.
few months ago, purporting to fix issues with clients not being able to
update their DNS records.
The fix caused the create of a new deleted record for every DNS
transaction, even one that should have had no impact on the database
(same IP).
The only workaround to avoid creating more is to change from the
internal DNS server to the BIND9 DLZ module, but this won't fix the
issue with having a database that is drowning in deleted records. We
don't have a tool to purge these at this time, and by default they will
be kept for 6 months.
We do realise we are going to have to come up with a better fix, but
sadly nobody has yet proposed a patch to do this properly. (We should
probably at least revert the one that was put in).
Sorry,
Andrew Bartlett
--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
lp101
for 6 months. Any possibilty of a parameter being introduced to modify this?
-James
Achim Gottinger
tombstone lifetime.
https://blogs.technet.com/b/networking/archive/2008/03/19/don-t-be-afraid-of-dns-scavenging-just-be-patient.aspx?Redirected=true.
On my first samba4 setup I did a few modifications to an dhcp
updatescript to suse samba-tool dns to delete and recreate dns entries
because nsupdate did not work. After a month i had ~80000 deleted
entries in the dns database. and
/var/lib/samba/priveate/sam.ldb.d/DC=DomainDnsZones,DC=Domain,DC=local.ldb
had growen to ~300MB with just ~15 clients in the network.
I noticed that the mentioned database file does not shrink by itself. To
lower it's size I use tdbbackup to make an backup of that ldb database
with samba stopped and copy the resulting file back to the ldb file.
After an few month it is now down at ~70MB again. Still alot for such an
small network.
Had tried to delete the deleted records from the ldb Database with
ldbedit but afterwards dns was broken, for example i could not create
new records.
achim~
Günter Kukkukk
Hi Andrew,
why do you throw such info
"We should probably at least revert the one that was put in"
knowing that the former version without that patch "was NOT working at ALL"!
In case you missed those long discussions about that failure those days,
please just DO NOT COMMENT!
I was _not_ aware about the (somewhat hidden) "tombstone" problem those days.
(neither was Kai .... or others)
And btw - I'm not completely sure whether the DLZ dns implementation is
working right in this area ...
Achim Gottinger <ac...@ag-web.biz> brought this to my/our attention:
https://blogs.technet.com/b/networking/archive/2008/03/19/don-t-be-afraid-of-dns-scavenging-just-be-patient.aspx?Redirected=true
I think - worth reading...
At least samba should behave "as friendly as possible" regarding deleted dns entries.
Looking at that (...o_o ..) code for days now.
Cheers, Günter
Achim Gottinger
>>>
>>> Achim Gottinger <ac...@ag-web.biz> brought this to my/our attention:
>>> https://blogs.technet.com/b/networking/archive/2008/03/19/don-t-be-afraid-of-dns-scavenging-just-be-patient.aspx?Redirected=true
>>>
>>> I think - worth reading...
>>>
>>> At least samba should behave "as friendly as possible" regarding deleted dns entries.
>>>
>>> Looking at that (...o_o ..) code for days now.
>>>
>>> Cheers, Günter
>>>
While the article is interesting indeed it did not cover how tombstones
are treated.
Here an retention time of 7 days before the deleted dns records
dissapear from ad is mentioned.
Günter Kukkukk
> Am 25.12.2013 06:15, schrieb Günter Kukkukk:
>>>>
>>>> Achim Gottinger <ac...@ag-web.biz> brought this to my/our attention:
>>>> https://blogs.technet.com/b/networking/archive/2008/03/19/don-t-be-afraid-of-dns-scavenging-just-be-patient.aspx?Redirected=true
>>>>
>>>> I think - worth reading...
>>>>
>>>> At least samba should behave "as friendly as possible" regarding deleted dns entries.
>>>>
>>>> Looking at that (...o_o ..) code for days now.
>>>>
>>>> Cheers, Günter
>>>>
>
> While the article is interesting indeed it did not cover how tombstones are treated.
>
> Here an retention time of 7 days before the deleted dns records dissapear from ad is mentioned.
>
> https://blogs.technet.com/b/isrpfeplat/archive/2010/09/23/dns-scavenging-internals-or-what-is-the-dnstombstoned-attribute-for-ad-integrated-zones-dstombstoneinterval-dnstombstoned.aspx?Redirected=true
>
>
>
Hi Achim,
thanks, great new link again! :-)
Cheers, Günter
--
lp101
Here are a couple articles that explain how these objects work that may
prove helpful.
http://blogs.technet.com/b/networking/archive/2011/08/17/tracking-dns-record-deletion.aspx
http://technet.microsoft.com/en-us/library/cc759204%28WS.10%29.aspx
Achim Gottinger
>
> Here are a couple articles that explain how these objects work that
> may prove helpful.
>
> http://blogs.technet.com/b/networking/archive/2011/08/17/tracking-dns-record-deletion.aspx
>
>
> http://technet.microsoft.com/en-us/library/cc759204%28WS.10%29.aspx
So if an DNS record ist dnsTombstoned and older than seven days it gets
AD tombstoned and lives in the database for an default of 180 days. I
thought it would be removed completely from the ad database in that case.
Since samba does not use dnsTombstone's any update to an dns record via
sama-tool dns, nsupdate or an windows client results in an directly ad
tombstoned record. So the only way to reduce the number of deleted dns
records is at the moment to lower the ad tombstone lifetime atribute
(CN=Directory Service,CN=Windows
NT,CN=Services,CN=Configuration,CN={GUID}). I reduced it to 30 days for
now. One has to be careful with that attribute if an ad domain server is
down for an longer period it can cause problems with replication
afterwards and the server must be rejoined.
achim~
lp101
unable to join a new server as the machine runs out of memory during the
replication process and kicks back to the command line. This is with 6GB
of memory. It failed with roughly 10,000 objects left to replicate. Can
you provide me with the correct syntax to view these objects in sam.ldb
and I assume you used ADSI to edit the tombstone attribute? Thanks.
Achim Gottinger
ldbsearch -H
/var/lib/samba/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=DOMAIN,DC=LOCAL.ldb '(&(name=HOSTNAME)(isDeleted=TRUE))'
dn
to search for specific hosts or
ldbsearch -H
/var/lib/samba/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=DOMAIN,DC=LOCAL.ldb 'isDeleted=TRUE'
dn
to get an list of all deleted dn's.
To change the default lifetime of tombstones with ADSI navigate to
CN=Directory Service,CN=Windows
NT,CN=Services,CN=Configuration,DC=DOMAIN,DC=LOCAL open the properties
of that folder and change the attribute called tombstoneLifetime to the
desired value. I restarted samba afterwards and it took a few minutes
till samba had deleted all the outdated objects.
lp101
records and it returned 391131 entries. Changed tombstone attribute and
restarted Samba. Records are not being deleted and replication according
to showrepl has failed. This was in log.samba
[2014/01/10 12:21:48.842660, 0]
../source4/dns_server/dns_utils.c:282(dns_replace_records)
Deleting record failed; 50
[2014/01/10 12:41:55.254616, 0]
../source4/dns_server/dns_utils.c:282(dns_replace_records)
Deleting record failed; 50
[2014/01/10 12:42:02.278754, 0]
../source4/dns_server/dns_utils.c:282(dns_replace_records)
Deleting record failed; 50
[2014/01/10 12:42:07.973631, 0]
../source4/dsdb/dns/dns_update.c:294(dnsupdate_nameupdate_done)
../source4/dsdb/dns/dns_update.c:294: Failed DNS update -
NT_STATUS_IO_TIMEOUT
[2014/01/10 12:43:46.925354, 0]
../source4/rpc_server/common/forward.c:51(dcesrv_irpc_forward_callback)
IRPC callback failed for DsExecuteKCC - NT_STATUS_IO_TIMEOUT
Now it appears replication is working because I can create users
and see them replicated on other DC's. If I switch to bind will this
delete these entries and allow me to join a new DC with the deleted
entries gone? As of now I'm unable to join any new DC's as the server
runs out of memory and exits to a command prompt at around 350,000
entries being replicated. I know see that updates are turned off.
schema_fsmo_init: we are master[yes] updates allowed[no]
Replication appears to fail when checking samba-tool with
rpc fault: WERR_EPT_S_CANT_PERFORM_OP
and I see this when using
On 1/2/2014 10:36 PM, Achim Gottinger wrote:
> ldbsearch -H
> /var/lib/samba/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=DOMAIN,DC=LOCAL.ldb
> 'isDeleted=TRUE' dn
Achim Gottinger
my office which has two ad DC's both running on an debian wheezy vm's,
one runs sernet 4.1.3 the other one an backported debian samba package
version 4.0.10. The server i modified the attribute on was the one with
sernet 4.1.3 and this one also has alle the fsmo roles. Here it did not
take long till the deleted objects started decreasing after i restarted
that server. Just checked both servers and they habe no replication
errors and both show the same number of ~390 deleted records. Before one
of my windows 7 clients alone had around 800 deleted records.
Are you shure you changed tombstoneLifetime to an small enoght value to
cache some of your deleted records? I'd also verify that the
tomstoneLiftime attribute replicated successfull to all your dc's.
> schema_fsmo_init: we are master[yes] updates allowed[no]
unrelated to Configuration changes or DNS updates.
lp101
replicated to the other DC's. It did. I changed the value to 30 from
180. All 3 DC's show different values for deleted records so I know
replication is broken.
lp101
sprang back to life. This was on the DC that held all the FSMO roles.
While things are working again I'm still back to square one with all the
deleted domain entries.
On 1/10/2014 3:56 PM, Achim Gottinger wrote:
Achim Gottinger
> Just an FYI. I reverted the tombstone back to 180 and replication
> sprang back to life. This was on the DC that held all the FSMO roles.
> While things are working again I'm still back to square one with all
> the deleted domain entries.
domain whom will be connected via 1-2MBit SDSL lines, looking at the
time it took your server to replicate the dns database during join makes
me curious how long it will take on my side.
You said your servers had different amounts of deleted records, is that
still the case after you got replication working? If not did they diminish?
My test setup was pretty simple two servers connected via an 2GBit VM
interface. So the changes i made to the tomstoneLifetime attribute
should have been replicated almost instantly.
On an bigger setup it may be better to wait till the change got
replicated to all dc's. The purging of outdated deleted object should
also happen on a daily basis without an restart of the samba services. I
think the active directory docs mentioned somewhere that ad objects do
not get deleted if there are replication errors. I'd change the
attribute more modest to for example 160 days and wait till samba-tool
drs shorrepl shows an successfull replication after the modification.
lp101
Will give it some time and check again. I assume my internal DNS must be
working 100% to have these entries deleted? I'm still getting this in my
log.samba:
[2014/01/10 22:17:32.665570, 0]
../source4/dsdb/dns/dns_update.c:294: Failed DNS update -
NT_STATUS_IO_TIMEOUT
lp101
hours. This was after changing the attribute to 30 days. Do you know how
to force replication for the Domain DNS Deleted Objects? Replicating the
DominDnsZones using Samba-tool drs replicate doesn't appear to replicate
these objects.
I've attempted to join a DC again over a 1.5Mbit Wan link using
Samba 4.1.4 on Ubuntu 12.04. At this moment I'm over 19hrs in with
312355/385196 replicated. I joined using "--domain-critical-only"
thinking it may exclude these items but I was wrong.
On 1/10/2014 8:33 PM, Achim Gottinger wrote:
Achim Gottinger
> It looks like 15,000 records have been deleted over a period of 8
> hours. This was after changing the attribute to 30 days. Do you know
> how to force replication for the Domain DNS Deleted Objects?
> Replicating the DominDnsZones using Samba-tool drs replicate doesn't
> appear to replicate these objects.
> I've attempted to join a DC again over a 1.5Mbit Wan link using
> Samba 4.1.4 on Ubuntu 12.04. At this moment I'm over 19hrs in with
> 312355/385196 replicated. I joined using "--domain-critical-only"
> thinking it may exclude these items but I was wrong.
directly connected? This does slow down replication. Hope it works for
you this time, but didn't it fail at ~350000 objects last time?
lp101
deleted objects during join.
Günter Kukkukk
> Am 13.01.2014 18:39, schrieb lp101:
>> It looks like 15,000 records have been deleted over a period of 8 hours. This was after changing the attribute to 30 days. Do you know how to
>> force replication for the Domain DNS Deleted Objects? Replicating the DominDnsZones using Samba-tool drs replicate doesn't appear to replicate these
>> objects.
>> I've attempted to join a DC again over a 1.5Mbit Wan link using Samba 4.1.4 on Ubuntu 12.04. At this moment I'm over 19hrs in with 312355/385196
>> replicated. I joined using "--domain-critical-only" thinking it may exclude these items but I was wrong.
> Thank you fro the update. Can it be you have an few sites whom are not directly connected? This does slow down replication. Hope it works for you this
> time, but didn't it fail at ~350000 objects last time?
>
>
When all child DNS entry are gone, it _leaves_ the directory storage as:
(sambatool dns query .... output)
Name=mytest, Records=0, Children=0
So the record is _not_ deleted - more or less "left as an unused entry".
Those entries can be re-used later, but can also accumulate when not
being re-used.
As i've seen with a windows7 client during normal operation, it deletes
its A and AAAA records and then registers one/both again in some interval
of about 5 to 10 minutes! (Could be due i was running the MS MMC DNS plugin).
This behavior is atm handled fine with the DLZ driver - but is somewhat FATAL
for the internal DNS server: It creates LOTS of deleted dns entries!
So i've reverted the patch
8b24c43b382740106474e26dec59e1419ba77306
which was deleting the whole dns entry.
After this revert the internal dns server behaves the same as the DLZ driver and
leaves those
Name=mytest, Records=0, Children=0
records around - BUT THEN the current implementation is NOT able to add
new incoming records!
https://bugzilla.samba.org/show_bug.cgi?id=9559
Atm i did a very first simple patch to the internal dns, which allows
to add new entries in that
Name=mytest, Records=0, Children=0
formerly failing state.
Now the internal dns _seems_ to behave similar to the DLZ driver, but
more investigation is needed because dns entries can be "static" or
"time stamped" ....
So i'm still looking at all related infos ....
Btw - has someone seen "strange" behavior in this area when the
DLZ driver is used?
Cheers, Günter
--
Günter Kukkukk
Atm i'm trying to collect as much info as possible.
Can someone comment on this article/patch ?
http://support.microsoft.com/kb/2548145/en-us
lp101
http://technet.microsoft.com/en-us/magazine/2007.09.tombstones.aspx
The hot fix link you posted sounds like my issue.
--
Achim Gottinger
moment? If in git i read it as moving to DLZ should fix the issue meanwhile.
The link you posted also suggests an workaround. "To work around the
issue, change the value of *Security updates* to *None* or to *Nonsecure
and secure* in the DNS server zones.". Does this work with samba also?
Guess it belongs to the issue described here
https://blogs.technet.com/b/isrpfeplat/archive/2010/09/23/dns-scavenging-internals-or-what-is-the-dnstombstoned-attribute-for-ad-integrated-zones-dstombstoneinterval-dnstombstoned.aspx?Redirected=true
*Note: The default behavior for Windows 2008 R2 was modified and will be
acting as if the SID of the machine has changed regardless of whether
it's true or not. Meaning when a record update is sent to a DNS server
hosted on a Windows 2008 R2 Domain Controller and the record's
dnsTombstone=True the record is deleted regardless of the permissions
issue described earlier. This was to fix the issue described in
**http://support.microsoft.com/kb/952087**.*
*In certain cases (with many updates and low No-Refresh interval) this
can cause issues. The resolution can be found in
**http://support.microsoft.com/kb/2548145/en-us**.*
Günter Kukkukk
some additional notes.
When the samba DLZ dns driver is creating dyn. dns entries, a time stamp
is set on that record! This is the expected behavior for dynamic entries.
This is _NOT_ the case when the internal dns used! :-(
Because the time stamp is zero, such a dns entry is treated as being "static"!
"static" dns entries are _never_ used by scavenging (aging)!
(beside when "dnscmd /ageallrecords ...." has been issued at some time)
See samba dns dlz vs internal differences here:
http://picpaste.com/samba-dns-6UEpowtv.png
--------------------
One can also use the MS "dnscmd" command against the internal dns entries:
C:\Users\administrator>dnscmd linux4771 /enumrecords intranet01.hom mytest /additional /detail
Zurückgegebene Datensätze:
RPC-Knoten:
ptr = 0000000000141BE0
wLength = 16
wRecordCount = 1
dwChildCount = 0
dwFlags = 00000000 Knotenname = @
A Datensatzinformationen:
ptr = 0000000000141BF0
wType = A (1)
wDataLength = 4
dwFlags = f0
rank = f0
dwSerial = 0000006E
dwTtlSeconds = 3600
dwTimeStamp = 0 ([ 0: 0: 0] [ 1/ 1/1601]) A 192.168.200.202
Der Befehl wurde erfolgreich ausgeführt.
----------------------
dwTimeStamp is zero ===> STATIC dns entry!
This adds some additional infos to all our former posts about scavenging ...
Will keep you informed.
Cheers, Günter
Günter Kukkukk
Internal samba DNS server:
The missing time stamp issue should be solved like this:
diff --git a/source4/dns_server/dns_update.c b/source4/dns_server/dns_update.c
index 9edc40b..b2c2bd7 100644
--- a/source4/dns_server/dns_update.c
+++ b/source4/dns_server/dns_update.c
@@ -292,6 +292,7 @@ static WERROR dns_rr_to_dnsp(TALLOC_CTX *mem_ctx,
char *tmp;
char *txt_record_txt;
char *saveptr = NULL;
+ NTTIME t;
if (rrec->rr_type == DNS_QTYPE_ALL) {
return DNS_ERR(FORMAT_ERROR);
@@ -305,6 +306,12 @@ static WERROR dns_rr_to_dnsp(TALLOC_CTX *mem_ctx,
/* TODO: Autogenerate this somehow */
r->dwSerial = 110;
+ /* We need to set a time stamp on this record */
+ unix_to_nt_time (&t, time(NULL));
+ t /= 10*1000*1000; /* convert to seconds */
+ t /= 3600; /* convert to hours */
+ r->dwTimeStamp = (uint32_t)t;
With my other revert and (internal) patch to allow dyn. dns updates also in case of
Name=mytest, Records=0, Children=0
all seems to be working fine now! :-)
Cheers, Günter
Hmmm - working at least similar to the DLZ driver ...
--
steve
> >
>
> some additional notes.
>
> When the samba DLZ dns driver is creating dyn. dns entries, a time stamp
> is set on that record! This is the expected behavior for dynamic entries.
>
> This is _NOT_ the case when the internal dns used! :-(
>
> Because the time stamp is zero, such a dns entry is treated as being "static"!
Hi
Would this explain why our dns update requests using nsupdate fail
against the internal dns? The DNS record is created once when a Linux
client joins the domain, but is never subsequently updated by requests
from (in our case sssd which uses. . .) nsupdate. Converting to bind9
dlz, works fine.
Just to add a bit more confusion, windows boxes update fine with either
server.
Günter: did you say you have a patch to correct time-stamp on the
internal server? My boss has been pushing me to internal dns and so I'd
like to be able to test. Of course, will share back.
Thanks,
Steve