Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

[Samba] Samba Join as DC failed

1,378 views
Skip to first unread message

Donaldson Jeff

unread,
Oct 17, 2013, 9:10:02 AM10/17/13
to
Attempted to join domain via

./bin/samba-tool domain join ncs.k12.de.us<http://ncs.k12.de.us> DC -Uadministrator --realm=ncs.k12.de.us<http://ncs.k12.de.us>

But this failed with

Committing SAM database
Failed to apply linked attribute change 'attribute 'isRecycled': invalid modify flags on 'CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us': 0x0'
dn: <GUID=4d560497-5f00-4d97-96a0-47ae1799ba92>;<SID=S-1-5-21-276688905-1455118844-2751846679-67110292>;CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us

Join failed - cleaning up
checking sAMAccountName
ERROR(ldb): uncaught exception - attribute 'isRecycled': invalid modify flags on 'CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us': 0x0
File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py", line 175, in _run
return self.run(*args, **kwargs)
File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/domain.py", line 552, in run
machinepass=machinepass, use_ntvfs=use_ntvfs, dns_backend=dns_backend)
File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1169, in join_DC
ctx.do_join()
File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1074, in do_join
ctx.join_replicate()
File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 848, in join_replicate
ctx.local_samdb.transaction_commit()

As suggestion found here https://irclog.samba.org/2013/09/20130908-Sun.log: is to use

ldbedit -H /usr/local/samba/private/sam.ldb --show-deleted '(isDeleted=*)'

to manually delete all the accounts with this attribute. When doing this I should stop samba on all DCs and then edit the local sam.ldb on each. Then restart samba on the DC and re-try joining the domain after deleting all files /usr/local/samba/private on the DC I am attempting to join to the domain as a DC?

Also saw on Samba list Nikos Mita had similar issue. It was suggested to try using samba-tool dbcheck -fix. Should I try this first? I'm just concerned whether this would complete or not. I have 94,443 records and this server only has 8GB of memory.

I want to make certain I get the sequence correct.

Also, before doing any of the above, I will make a copy of the private directories on the DC just in case ...

Any help is appreciated. Thanks!

Regards,
Jeff

Jeff Donaldson
Technology Director
Newark Charter School
jeff.do...@ncs.k12.de.us
(302) 369-2001 ext: 425
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

Andrew Bartlett

unread,
Oct 17, 2013, 11:00:02 PM10/17/13
to
On Thu, 2013-10-17 at 12:50 +0000, Donaldson Jeff wrote:
> Attempted to join domain via
>
> ./bin/samba-tool domain join ncs.k12.de.us<http://ncs.k12.de.us> DC -Uadministrator --realm=ncs.k12.de.us<http://ncs.k12.de.us>
>
> But this failed with
>
> Committing SAM database
> Failed to apply linked attribute change 'attribute 'isRecycled': invalid modify flags on 'CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us': 0x0'
> dn: <GUID=4d560497-5f00-4d97-96a0-47ae1799ba92>;<SID=S-1-5-21-276688905-1455118844-2751846679-67110292>;CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us
>
> Join failed - cleaning up
> checking sAMAccountName
> ERROR(ldb): uncaught exception - attribute 'isRecycled': invalid modify flags on 'CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us': 0x0
> File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py", line 175, in _run
> return self.run(*args, **kwargs)
> File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/domain.py", line 552, in run
> machinepass=machinepass, use_ntvfs=use_ntvfs, dns_backend=dns_backend)
> File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1169, in join_DC
> ctx.do_join()
> File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1074, in do_join
> ctx.join_replicate()
> File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 848, in join_replicate
> ctx.local_samdb.transaction_commit()
>
> As suggestion found here https://irclog.samba.org/2013/09/20130908-Sun.log: is to use
>
> ldbedit -H /usr/local/samba/private/sam.ldb --show-deleted
> '(isDeleted=*)'

This is not good advise for the general case. Deleting the objects
manually breaks replication (because the purpose of the deleted object
is to replicate the fact that it is deleted!), and should be a last
resort.

> to manually delete all the accounts with this attribute. When doing
> this I should stop samba on all DCs and then edit the local sam.ldb on
> each. Then restart samba on the DC and re-try joining the domain after
> deleting all files /usr/local/samba/private on the DC I am attempting
> to join to the domain as a DC?
>
> Also saw on Samba list Nikos Mita had similar issue. It was suggested
> to try using samba-tool dbcheck -fix. Should I try this first? I'm
> just concerned whether this would complete or not. I have 94,443
> records and this server only has 8GB of memory.
>
> I want to make certain I get the sequence correct.
>
> Also, before doing any of the above, I will make a copy of the private
> directories on the DC just in case ...
>
> Any help is appreciated. Thanks!

G'Day,

It seems to be the week for very, very large Samba installations!

I've looked at the code, and I know the line that fails, but don't I
know why this happens. Can you show me the failing object with
ldbsearch?

ldbsearch --show-deleted -H /usr/local/samba/private/sam.ldb -s base -b
'CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us'

The thing is, an object that has isRecycled set on it should not be able
to get to the line of code that fails, so I'm quite puzzled. I can fix
the 'error' simply (just need to create a new blank modification, rather
than re-using a search result), but I first want to know why it is
wrong.

Can you also let me know the full history of this domain? A user that
is deleted should have a name with "DEL" and a GUID in it.

The second part, once I have that is working out why our tests didn't
cover this code path, and working out how to make them do that.

But while you won't need to run dbcheck now, you will at some point in
the future. What we clearly do need is for a few of our very large
installations to club together and work on/isolate the remaining issues
at the scale you have.

Thank you so much for taking Samba to the extreme, and I will do what I
can to best assist you.

Andrew Bartlett

--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz

Donaldson Jeff

unread,
Oct 18, 2013, 12:50:02 PM10/18/13
to
Andrew,

The number of records indicated in the last email was based on these lines that were returned during the failed samba join. This is the last line of that sequence.

Partition[DC=DomainDnsZones,DC=ncs,DC=k12,DC=de,DC=us] objects[94443/94443] linked_values[0/0]

I think we're probably closer to 2,200 objects, so I apologize for any confusion. I also ran the ldbsearch you requested. Here's the output...

root@ncssamba1:~# ldbsearch --show-deleted -H /usr/local/samba/private/sam.ldb -s base -b 'CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us'
search error - No such Base DN: CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us

This account was producing errors in the log on this server. Since this was an old account no longer used, I used ADUC on a 2008 R2 server to delete the user account. I thought at the time this would eliminate the errors in the log. I believe it instead created an orphaned record and I'm not sure how to go about getting it removed cleanly.

As Dave mentioned in previous email, we installed the 4.2 Alpha using git accidentally. Here's the output from samba -V, if it helps.

root@ncssamba1:~# samba -V
Version 4.2.0pre1-GIT-b505111

We are now trying to get two servers running the 4.1.0 stable release up and running to eventually phase out the 4.2 servers. Here's the output from samba -V from server trying to join the domain unsuccessfully.

root@ncsauth2:~# samba -V
Version 4.1.0

Please let me know if there is anything else you need to help us troubleshoot the problem. We are truly grateful for your support!

Regards,
Jeff

Jeff Donaldson
Technology Director
Newark Charter School
jeff.do...@ncs.k12.de.us
(302) 369-2001 ext: 425

________________________________
From: David Hopkins <dahopk...@gmail.com>
Sent: Friday, October 18, 2013 10:58 AM
To: Andrew Bartlett
Cc: Donaldson Jeff; sa...@lists.samba.org; O'Neill James
Subject: Re: [Samba] Samba Join as DC failed

Jeff,

My response for the history of the domain:

Our prior authentication system was based on a custom Samba3+Openldap solution (originally developed with people from the K12LTSP list). This authentication system had been very stable for 10+ years. We installed the latest version of Samba using git (perhaps unfortunately, because we pulled 4.2) and upgraded to Samba4 to provide better support for Windows 7 and Server 2008/2012 systems. We then installed a second authentication server using the same process and joined that server to the domain as a second DC (also pulled the 4.2 version). Authentication was working very well until recently when the first server began to randomly stop responding to dns requests. We decided to install Samba 4.1 (in an effort to move back to the stable version). It was on trying to join the Samba 4.1 server to the domain as an AD DC that we got the above issue. We have two zones in DNS (10.179.0.0 and 10.186.0.0, subnet mask 255.255.224.0) The server with DNS issues is in the 10.179.0.0 z
one. The other server is working properly. Replication seems to be working properly.

As for the size of the domain, did I misread the screen? The number reported is the number that was returned during the join operation.


Sincerely,
Dave


On Thu, Oct 17, 2013 at 10:57 PM, Andrew Bartlett <abar...@samba.org<mailto:abar...@samba.org>> wrote:
On Thu, 2013-10-17 at 12:50 +0000, Donaldson Jeff wrote:
> Attempted to join domain via
>
> ./bin/samba-tool domain join ncs.k12.de.us<http://ncs.k12.de.us><http://ncs.k12.de.us> DC -Uadministrator --realm=ncs.k12.de.us<http://ncs.k12.de.us><http://ncs.k12.de.us>
>
> But this failed with
>
> Committing SAM database
> Failed to apply linked attribute change 'attribute 'isRecycled': invalid modify flags on 'CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us': 0x0'
> dn: <GUID=4d560497-5f00-4d97-96a0-47ae1799ba92>;<SID=S-1-5-21-276688905-1455118844-2751846679-67110292>;CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us
>
> Join failed - cleaning up
> checking sAMAccountName
> ERROR(ldb): uncaught exception - attribute 'isRecycled': invalid modify flags on 'CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us': 0x0
> File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py", line 175, in _run
> return self.run(*args, **kwargs)
> File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/domain.py", line 552, in run
> machinepass=machinepass, use_ntvfs=use_ntvfs, dns_backend=dns_backend)
> File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1169, in join_DC
> ctx.do_join()
> File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1074, in do_join
> ctx.join_replicate()
> File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 848, in join_replicate
> ctx.local_samdb.transaction_commit()
>
> As suggestion found here https://irclog.samba.org/2013/09/20130908-Sun.log: is to use
>
> ldbedit -H /usr/local/samba/private/sam.ldb --show-deleted
> '(isDeleted=*)'

This is not good advise for the general case. Deleting the objects
manually breaks replication (because the purpose of the deleted object
is to replicate the fact that it is deleted!), and should be a last
resort.

> to manually delete all the accounts with this attribute. When doing
> this I should stop samba on all DCs and then edit the local sam.ldb on
> each. Then restart samba on the DC and re-try joining the domain after
> deleting all files /usr/local/samba/private on the DC I am attempting
> to join to the domain as a DC?
>
> Also saw on Samba list Nikos Mita had similar issue. It was suggested
> to try using samba-tool dbcheck -fix. Should I try this first? I'm
> just concerned whether this would complete or not. I have 94,443
> records and this server only has 8GB of memory.
>
> I want to make certain I get the sequence correct.
>
> Also, before doing any of the above, I will make a copy of the private
> directories on the DC just in case ...
>
> Any help is appreciated. Thanks!

Donaldson Jeff

unread,
Oct 19, 2013, 11:20:02 AM10/19/13
to
Andrew,

As an add on to previous message...even though the Samba join as DC reported that it failed, the server shows up in ADUC as a DC (although shows unavailable, probably because we did not start samba service on the machine with the error) as well as in Sites and Services. It also shows replication to our other two DCs, but not from them. In addition, /usr/local/samba/private is populated but I'm not sure if it is complete.

How do we pull the new DC out of our environment cleanly or can we just attempt to re-add as DC after running dbcheck on PDC to have it pull remaining pieces for a complete join?

Regards,
Jeff

Jeff Donaldson
Technology Director
Newark Charter School
jeff.do...@ncs.k12.de.us
(302) 369-2001 ext: 425

________________________________
From: Donaldson Jeff
Sent: Friday, October 18, 2013 12:48 PM
To: David Hopkins; Andrew Bartlett
Cc: sa...@lists.samba.org; O'Neill James
Subject: RE: [Samba] Samba Join as DC failed

Donaldson Jeff

unread,
Oct 19, 2013, 12:40:02 PM10/19/13
to
Andrew,

Perhaps another hint...ran the following against the offending user account. Noticed that it shows up on a list of users with the --show-deleted flag. Also dbcheck without --fix flags this account on the PDC, but on the other DC it does not show up. We also saw that samba-tool drs showrepl indicates that the servers are properly replicating. The fact that dbcheck shows two different outputs is confusing as replication is working properly.

ncssamba1:~# ldbsearch -H /usr/local/samba/private/sam.ldb -s base -b 'CN=test_user,CN=Users,DC=ncs,DC=k12,DC=de,DC=us'
# returned 0 records
# 0 entries
# 0 referrals
ncssamba1:~# ldbsearch --show-deleted -H /usr/local/samba/private/sam.ldb -s base -b 'CN=test_user,CN=Users,DC=ncs,DC=k12,DC=de,DC=us'
# record 1
dn: CN=test_user,CN=Users,DC=ncs,DC=k12,DC=de,DC=us
cn: test_user
instanceType: 4
whenCreated: 20130726175012.0Z
uSNCreated: 13699
objectGUID: 4d560497-5f00-4d97-96a0-47ae1799ba92
badPwdCount: 0
badPasswordTime: 0
lastLogoff: 0
lastLogon: 0
objectSid: S-1-5-21-276688905-1455118844-2751846679-67110292
logonCount: 0
sAMAccountName: test_user
objectClass: top
objectClass: posixAccount
objectClass: person
objectClass: organizationalPerson
objectClass: user
manager: CN=jdonaldson,CN=Users,DC=ncs,DC=k12,DC=de,DC=us
memberOf: CN=Teachers,CN=Users,DC=ncs,DC=k12,DC=de,DC=us
userAccountControl: 66048
userParameters:: IA==
whenChanged: 20131011151907.0Z
isDeleted: TRUE
uSNChanged: 142163
name:: dGVzdF91c2VyCkRFTDo0ZDU2MDQ5Ny01ZjAwLTRkOTctOTZhMC00N2FlMTc5OWJhOTI=
lastKnownParent: CN=Users,DC=ncs,DC=k12,DC=de,DC=us
isRecycled: TRUE
distinguishedName: CN=test_user,CN=Users,DC=ncs,DC=k12,DC=de,DC=us

# returned 1 records
# 1 entries
# 0 referrals



Jeff Donaldson
Technology Director
Newark Charter School
jeff.do...@ncs.k12.de.us
(302) 369-2001 ext: 425

________________________________________
From: samba-...@lists.samba.org <samba-...@lists.samba.org> on behalf of Donaldson Jeff <Jeff.Do...@ncs.k12.de.us>
Sent: Saturday, October 19, 2013 11:16 AM
To: David Hopkins; Andrew Bartlett
Cc: sa...@lists.samba.org; O'Neill James

Andrew Bartlett

unread,
Oct 21, 2013, 6:00:01 AM10/21/13
to
On Fri, 2013-10-18 at 16:48 +0000, Donaldson Jeff wrote:
> Andrew,
>
> The number of records indicated in the last email was based on these lines that were returned during the failed samba join. This is the last line of that sequence.
>
> Partition[DC=DomainDnsZones,DC=ncs,DC=k12,DC=de,DC=us] objects[94443/94443] linked_values[0/0]

These are probably deleted DNS records. Are you using the internal DNS
server or bind9_dlz? Either way, find out if this is still growing, we
may have an issue we need to work on here.

> I think we're probably closer to 2,200 objects, so I apologize for any confusion. I also ran the ldbsearch you requested. Here's the output...
>
> root@ncssamba1:~# ldbsearch --show-deleted -H /usr/local/samba/private/sam.ldb -s base -b 'CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us'
> search error - No such Base DN: CN=test_user,CN=Deleted Objects,DC=ncs,DC=k12,DC=de,DC=us
>
> This account was producing errors in the log on this server. Since this was an old account no longer used, I used ADUC on a 2008 R2 server to delete the user account. I thought at the time this would eliminate the errors in the log. I believe it instead created an orphaned record and I'm not sure how to go about getting it removed cleanly.
>
> As Dave mentioned in previous email, we installed the 4.2 Alpha using git accidentally. Here's the output from samba -V, if it helps.
>
> root@ncssamba1:~# samba -V
> Version 4.2.0pre1-GIT-b505111
>
> We are now trying to get two servers running the 4.1.0 stable release up and running to eventually phase out the 4.2 servers. Here's the output from samba -V from server trying to join the domain unsuccessfully.
>
> root@ncsauth2:~# samba -V
> Version 4.1.0
>
> Please let me know if there is anything else you need to help us troubleshoot the problem. We are truly grateful for your support!

In terms of AD stuff, these two versions are pretty similar, but best to
use 4.1.0 rather than GIT in production.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org


Andrew Bartlett

unread,
Oct 21, 2013, 6:00:02 AM10/21/13
to
This is very, very odd. Clearly the user has been subject to faulty
conflict resolution prior to our fix to ensure deleted objects stay
deleted. I guess we will need to add logic to fix this into dbcheck.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org


daho...@comcast.net

unread,
Oct 21, 2013, 6:50:02 AM10/21/13
to
Should we run samba-tool dbcheck --fix at this point on both servers to try and correct this?

daho...@comcast.net

unread,
Oct 21, 2013, 7:00:02 AM10/21/13
to

>
> The number of records indicated in the last email was based on these lines that were returned during the failed samba join. This is the last line of that sequence.
>
> Partition[DC=DomainDnsZones,DC=ncs,DC=k12,DC=de,DC=us] objects[94443/94443] linked_values[0/0]

>These are probably deleted DNS records. Are you using the internal DNS
>server or bind9_dlz? Either way, find out if this is still growing, we
>may have an issue we need to work on here.

We are using the internal DNS server. We have two zones (10.179.0.0/19 and 10.186.0.0/19). After a period of time, DNS quits working one of the servers and at that point authentication (using nslcd/nscd from our linux systems, and we get RPC errors on our Windows domain members) using that server also seems to fail. How can we test if this is still growing?

Sincerely,
Dave Hopkins

Andrew Bartlett

unread,
Oct 21, 2013, 3:00:02 PM10/21/13
to
I'm not aware of any code in dbcheck that will fix this, so perhaps let
me know what dbcheck is proposing to do, or what it does on a backup (it
has a --verbose mode) when we --fix it.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org


Andrew Bartlett

unread,
Oct 21, 2013, 3:10:02 PM10/21/13
to
On Mon, 2013-10-21 at 10:49 +0000, daho...@comcast.net wrote:
> >
> > The number of records indicated in the last email was based on these lines that were returned during the failed samba join. This is the last line of that sequence.
> >
> > Partition[DC=DomainDnsZones,DC=ncs,DC=k12,DC=de,DC=us] objects[94443/94443] linked_values[0/0]
>
> >These are probably deleted DNS records. Are you using the internal DNS
> >server or bind9_dlz? Either way, find out if this is still growing, we
> >may have an issue we need to work on here.
>
> We are using the internal DNS server. We have two zones (10.179.0.0/19 and 10.186.0.0/19). After a period of time, DNS quits working one of the servers and at that point authentication (using nslcd/nscd from our linux systems, and we get RPC errors on our Windows domain members) using that server also seems to fail. How can we test if this is still growing?
>
> Sincerely,
> Dave Hopkins

Simply check the number of records in the database, say by ldbsearch
--show-deleted -s sub -b DC=DomainDnsZones,DC=ncs,DC=k12,DC=de,DC=us

This looks very much like what Amitay fixed for the BIND9_DLZ backend
in:

commit 169db333033b72b6f9ac1e7b23f0f2c151218c1f
Author: Amitay Isaacs <ami...@gmail.com>
Date: Thu Feb 9 10:17:02 2012 +1100

dlz_bind9: Do not remove LDB record in subrdataset and delrdataset

This fixes the problem of large number of deleted records in DNS
partitions due to frequent dynamic dns updates from windows
clients. The typical pattern for dynamic update get converted
into subrdataset() followed by addrdataset(). If there are no
dnsRecord attributes left as a result of sub/delrdataset(),
leave the LDB entry for dns name as is. The subsequent
addrdataset() would add the dnsRecord attribute without
re-creating the same entry.

Do you know if for your use case, the internal DNS server, did it only
start happening after this commit?

This code has logic that shouldn't delete an object when just changing
it's IP, but perhaps something else is wrong. I've CC'ed Kai, the
maintainer of the internal DNS server.

commit 673678474791d2f71ba7d8d0f73e20b2a974ae9a
Author: Kai Blin <k...@samba.org>
Date: Sat Jun 1 10:24:11 2013 +0200

dns: Delete dnsNode objects when they are empty

If an update leaves the dnsNode without any entries, the dnsNode
object
should be deleted. Thanks to Günter Kukkukk for his excellent
debugging
work on this one.

This should fix bug #9559

Signed-off-by: Kai Blin <k...@samba.org>

Reviewed-by: Andrew Bartlett <abar...@samba.org>
(cherry picked from commit 8b24c43b382740106474e26dec59e1419ba77306)

The last 3 patches address bug #9559 - Only initial signed DNS
update for a
works.

Autobuild-User(v4-0-test): Karolin Seeger <kse...@samba.org>
Autobuild-Date(v4-0-test): Mon Jun 3 14:16:16 CEST 2013 on
sn-devel-104

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

daho...@comcast.net

unread,
Oct 21, 2013, 7:20:01 PM10/21/13
to
Andrew,

Here is the last part of the output from the ldbsearch command. It appears that DNS is still growing rapidly and is being replicated across the servers.

---------------------------------------------------------------------------------------------------------------------

# record 117569
dn: DC=NCS-FINANCE\0ADEL:17f969f3-ef19-4c8a-9d27-fa802257678b,CN=Deleted Objects,DC=DomainDnsZones,DC=ncs,DC=k12,DC=de,DC=us
objectClass: top
objectClass: dnsNode
instanceType: 4
whenCreated: 20130831222333.0Z
uSNCreated: 25571
objectGUID: 17f969f3-ef19-4c8a-9d27-fa802257678b
isDeleted: TRUE
lastKnownParent: DC=ncs.k12.de.us,CN=MicrosoftDNS,DC=DomainDnsZones,DC=ncs,DC=
k12,DC=de,DC=us
isRecycled: TRUE
dc:: TkNTLUZJTkFOQ0UKREVMOjE3Zjk2OWYzLWVmMTktNGM4YS05ZDI3LWZhODAyMjU3Njc4Yg==
name:: TkNTLUZJTkFOQ0UKREVMOjE3Zjk2OWYzLWVmMTktNGM4YS05ZDI3LWZhODAyMjU3Njc4Yg=
=
whenChanged: 20130831232332.0Z
uSNChanged: 25584
distinguishedName: DC=NCS-FINANCE\0ADEL:17f969f3-ef19-4c8a-9d27-fa802257678b,C
N=Deleted Objects,DC=DomainDnsZones,DC=ncs,DC=k12,DC=de,DC=us

# returned 117569 records
# 117569 entries
# 0 referrals

So .. is there a way to clean up the DNS issues without wiping the servers? I did not get exactly the same results on both samba4 AD DC's. One server reported 117569 records, the other 117562. Could be a timing issue given how quickly the database is growing?

We didn't even build our samba4 domain until approximately Aug 24/2013 so definitely after the commit date.

Sincerely,
Dave Hopkins

Andrew Bartlett

unread,
Oct 21, 2013, 7:30:01 PM10/21/13
to
In the short term, can you try and use DLZ_BIND9 until we can sort this
out in the internal server? Then we can look at trying to expire these
tombstones.

Thanks,

Andrew Bartlett

daho...@comcast.net

unread,
Oct 21, 2013, 7:40:01 PM10/21/13
to
We can try. Are we sure that the conversion from internal to external DNS will be ok? Given the other issues we are facing, I'm extremely leery of completely losing everything. We'll backup the samba/private directories, and other key config files before the attempt. I'm assuming that if it goes badly, I could then just restore those directories and config files and restart samba.

We could also (time consuming but do-able) assign static IP addresses though not sure this would resolve anything.

Sincerely,
Dave Hopkins

daho...@comcast.net

unread,
Oct 21, 2013, 7:40:02 PM10/21/13
to
I ran dbcheck on both servers, they reported different issues. On one server, I also ran the --fix since all the issues were/are related to a server that we had tried to demote from its role as an AD DC. On the server that the above issue, I checked with the verbose option and the information was the same (for the errors) as shown below (ncsamba1 output).

Another issue (mentioned initiall) is that we built a 4.1 server that we tried to join as another AD DC but although it claimed that the join failed, that system is still listed in ADUC as a member server, and replication (samba-tool drs showrepl) shows that both current AD DC show that server as in inbound replication partner. In fact, the verbose output includes the line

Checking object CN=NCSAUTH2,OU=Domain Controllers,DC=ncs,DC=k12,DC=de,DC=us

So .. if the join says it failed, but samba/private seems to be populated on this new system and there is output from the dbcheck .. how do we either complete the join or clean up the system (e.g. just delete the samba/private data on ncsauth2) so that we can correctly join the domain once we have the test_user issue and the DNS issues resolved.

The following is the output requested.

--------------------------------------------
run on ncssamba3

/usr/local/samba/var# samba-tool dbcheck --fix
Checking 2163 objects
ERROR: target DN is deleted for masteredBy in object DC=ncs,DC=k12,DC=de,DC=us - <GUID=4ce872f9-90c4-4255-9d85-18903249f8a2>;CN=NTDS Settings,CN=NCSSAMBA2\0ADEL:831b85aa-87cf-40fc-9410-3574bc7456a4,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ncs,DC=k12,DC=de,DC=us
Target GUID points at deleted DN CN=NTDS Settings,CN=NCSSAMBA2\0ADEL:831b85aa-87cf-40fc-9410-3574bc7456a4,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ncs,DC=k12,DC=de,DC=us
Remove DN link? [y/N/all/none] y
Removed deleted DN on attribute masteredBy
ERROR: target DN is deleted for msDS-IsDomainFor in object DC=ncs,DC=k12,DC=de,DC=us - <GUID=4ce872f9-90c4-4255-9d85-18903249f8a2>;CN=NTDS Settings,CN=NCSSAMBA2\0ADEL:831b85aa-87cf-40fc-9410-3574bc7456a4,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ncs,DC=k12,DC=de,DC=us
Target GUID points at deleted DN CN=NTDS Settings,CN=NCSSAMBA2\0ADEL:831b85aa-87cf-40fc-9410-3574bc7456a4,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ncs,DC=k12,DC=de,DC=us
Remove DN link? [y/N/all/none] y
Removed deleted DN on attribute msDS-IsDomainFor
ERROR: target DN is deleted for msDs-masteredBy in object DC=ncs,DC=k12,DC=de,DC=us - <GUID=4ce872f9-90c4-4255-9d85-18903249f8a2>;CN=NTDS Settings,CN=NCSSAMBA2\0ADEL:831b85aa-87cf-40fc-9410-3574bc7456a4,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ncs,DC=k12,DC=de,DC=us
Target GUID points at deleted DN CN=NTDS Settings,CN=NCSSAMBA2\0ADEL:831b85aa-87cf-40fc-9410-3574bc7456a4,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ncs,DC=k12,DC=de,DC=us
Remove DN link? [y/N/all/none] y
Removed deleted DN on attribute msDs-masteredBy
Checked 2163 objects (3 errors)
---------------------------------------------------------
run on ncssamba1
ncssamba1:~# samba-tool dbcheck
Checking 2163 objects
ERROR: incorrect RMD_FLAGS value 0 for attribute 'manager' in CN=test_user,CN=Users,DC=ncs,DC=k12,DC=de,DC=us for link <GUID=0ac90a0d-9275-4032-962d-447b2f564bd1>;<RMD_ADDTIME=130193455900000000>;<RMD_CHANGETIME=130193455900000000>;<RMD_FLAGS=0>;<RMD_INVOCID=83af4e4e-38f9-4ddf-b3e4-4c694e7b26dc>;<RMD_LOCAL_USN=13723>;<RMD_ORIGINATING_USN=13723>;<RMD_VERSION=0>;<SID=S-1-5-21-276688905-1455118844-2751846679-6922>;CN=jdonaldson,CN=Users,DC=ncs,DC=k12,DC=de,DC=us
Not fixing incorrect RMD_FLAGS 0
ERROR: target DN is deleted for member in object CN=Teachers,CN=Users,DC=ncs,DC=k12,DC=de,DC=us - <GUID=4d560497-5f00-4d97-96a0-47ae1799ba92>;<SID=S-1-5-21-276688905-1455118844-2751846679-67110292>;CN=test_user,CN=Users,DC=ncs,DC=k12,DC=de,DC=us
Target GUID points at deleted DN CN=test_user,CN=Users,DC=ncs,DC=k12,DC=de,DC=us
Not removing
ERROR: target DN is deleted for directReports in object CN=jdonaldson,CN=Users,DC=ncs,DC=k12,DC=de,DC=us - <GUID=4d560497-5f00-4d97-96a0-47ae1799ba92>;<SID=S-1-5-21-276688905-1455118844-2751846679-67110292>;CN=test_user,CN=Users,DC=ncs,DC=k12,DC=de,DC=us
Target GUID points at deleted DN CN=test_user,CN=Users,DC=ncs,DC=k12,DC=de,DC=us
Not removing
Please use --fix to fix these errors
Checked 2163 objects (2 errors)
----------------------------------------------------------

Sincerely,
Dave Hopkins

David Hopkins

unread,
Oct 27, 2013, 11:30:02 PM10/27/13
to
Jeff,

My response for the history of the domain:

Our prior authentication system was based on a custom Samba3+Openldap
solution (originally developed with people from the K12LTSP list). This
authentication system had been very stable for 10+ years. We installed the
latest version of Samba using git (perhaps unfortunately, because we pulled
4.2) and upgraded to Samba4 to provide better support for Windows 7 and
Server 2008/2012 systems. We then installed a second authentication server
using the same process and joined that server to the domain as a second DC
(also pulled the 4.2 version). Authentication was working very well until
recently when the first server began to randomly stop responding to dns
requests. We decided to install Samba 4.1 (in an effort to move back to
the stable version). It was on trying to join the Samba 4.1 server to the
domain as an AD DC that we got the above issue. We have two zones in DNS
(10.179.0.0 and 10.186.0.0, subnet mask 255.255.224.0) The server with DNS
issues is in the 10.179.0.0 zone. The other server is working properly.
Replication seems to be working properly.

As for the size of the domain, did I misread the screen? The number
reported is the number that was returned during the join operation.


Sincerely,
Dave


Andrew Bartlett

unread,
Nov 27, 2013, 11:00:02 PM11/27/13
to
I've asked Garming (my intern here at Catalyst IT) to have a look into
this, but he can't reproduce this with nsupdate -g.

Can you get me a network trace of the traffic that causes the extra
entries so we can try and reproduce and fix it?

Thanks,

Andrew Bartlett

--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Garming Sam

unread,
Nov 27, 2013, 11:20:01 PM11/27/13
to
Hi there, sorry about that, I managed to reproduce the error. You don't
have to worry about sending in a network trace anymore. Now that I've
reproduced it, I'll work with Andrew to fix the issue in the next few days.



Cheers,

Garming Sam

Andrew Bartlett

unread,
Feb 28, 2014, 2:50:01 AM2/28/14
to
On Thu, 2013-10-17 at 12:50 +0000, Donaldson Jeff wrote:
Did you ever get to the bottom of this? I'm working on a patch for this
issue because I'm worried about a broader corruption that this may or
may not be related to. Did you ever run Samba from GIT or a 4.1
pre-release?

Thanks,

Andrew Bartlett
--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba


Daniel von Obernitz

unread,
Feb 28, 2014, 3:50:02 AM2/28/14
to
Hi Andrew,

I try to join a samba 4.1.4 (Debian Sernet package) as a DC into an
existing Windows 2008 RC2 AD as well and get a similar error message:



Committing SAM database
descriptor_sd_propagation_recursive:
DC=DomainDnsZones,DC=uni-greifswald,DC=de not found under
DC=uni-greifswald,DC=de
descriptor_sd_propagation_recursive:
DC=ForestDnsZones,DC=uni-greifswald,DC=de not found under
DC=uni-greifswald,DC=de
Failed to apply linked attribute change 'attribute 'isRecycled': invalid
modify flags on 'CN=NTDS
Settings\0ADEL:f0a27911-d9d3-4752-afd6-217c289b5baf,CN=SMB1\0ADEL:67680fef-790d-452c-ac87-95c546e54f81,CN=Servers,CN=Uni-Greifswald,CN=Sites,CN=Configuration,DC=uni-greifswald,DC=de':
0x0'
dn: <GUID=f0a27911-d9d3-4752-afd6-217c289b5baf>;CN=NTDS
Settings\0ADEL:f0a27911-d9d3-4752-afd6-217c289b5baf,CN=SMB1\0ADEL:67680fef-790d-452c-ac87-95c546e54f81,CN=Servers,CN=Uni-Greifswald,CN=Sites,CN=Configuration,DC=uni-greifswald,DC=de

changetype: modify
isRecycled: TRUE
-
replace: hasMasterNCs
hasMasterNCs:
<GUID=06b2fb6c-52f7-47a4-89bd-40750d030c93>;<RMD_ADDTIME=1303790

04690000000>;<RMD_CHANGETIME=130379004690000000>;<RMD_FLAGS=0>;<RMD_INVOCID=5


b9829f3-e5d3-41bf-bbe4-2c79f3877600>;<RMD_LOCAL_USN=36870>;<RMD_ORIGINATING_U

SN=79114298>;<RMD_VERSION=1>;CN=Configuration,DC=uni-greifswald,DC=de
-
replace: whenChanged
whenChanged: 20140228075037.0Z
-
replace: uSNChanged
uSNChanged: 36870
-



Join failed - cleaning up
checking sAMAccountName
Deleted CN=SMB1,OU=Domain Controllers,DC=uni-greifswald,DC=de
Deleted CN=dns-SMB1,CN=Users,DC=uni-greifswald,DC=de
Deleted CN=NTDS
Settings,CN=SMB1,CN=Servers,CN=UNI-GREIFSWALD,CN=Sites,CN=Configuration,DC=uni-greifswald,DC=de

Deleted
CN=SMB1,CN=Servers,CN=UNI-GREIFSWALD,CN=Sites,CN=Configuration,DC=uni-greifswald,DC=de

ERROR(ldb): uncaught exception - attribute 'isRecycled': invalid modify
flags on 'CN=NTDS
Settings\0ADEL:f0a27911-d9d3-4752-afd6-217c289b5baf,CN=SMB1\0ADEL:67680fef-790d-452c-ac87-95c546e54f81,CN=Servers,CN=Uni-Greifswald,CN=Sites,CN=Configuration,DC=uni-greifswald,DC=de':
0x0
File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
line 175, in _run
return self.run(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/samba/netcmd/domain.py", line
552, in run
machinepass=machinepass, use_ntvfs=use_ntvfs, dns_backend=dns_backend)
File "/usr/lib/python2.7/dist-packages/samba/join.py", line 1172, in
join_DC
ctx.do_join()
File "/usr/lib/python2.7/dist-packages/samba/join.py", line 1077, in
do_join
ctx.join_replicate()
File "/usr/lib/python2.7/dist-packages/samba/join.py", line 851, in
join_replicate
ctx.local_samdb.transaction_commit()


But in my case it seems doesn't seem to be a problem with a user but
with the host smb1. smb1 is the computer name of my samba server I want
to join.

Do you have any idea, what is causing this?
Thanks.
Daniel

Andrew Bartlett

unread,
Feb 28, 2014, 4:20:02 AM2/28/14
to
On Fri, 2014-02-28 at 09:39 +0100, Daniel von Obernitz wrote:
> Hi Andrew,
>
> I try to join a samba 4.1.4 (Debian Sernet package) as a DC into an
> existing Windows 2008 RC2 AD as well and get a similar error message:
>

>
> Do you have any idea, what is causing this?
> Thanks.
> Daniel

The attached patch should fix it. I don't understand how an object
becomes isRecycled but not isDeleted however, or how that doesn't cause
us to just skip out of this function.

Anyway, with the usual cautions about development patches, you could
give this a try if this is a test network (or you can reproduce against
such a test network). It is against GIT master.

Can you also get me a copy of the failing object searching directly
against your Windows 2008 R2 server? That will help me understand if
the isDeleted/isRecycled is a local thing caused during the join, or if
that is just how it is on Windows.

Thanks!
0001-dsdb-Do-not-reuse-search-message-for-modify-in-replm.patch

Daniel von Obernitz

unread,
Feb 28, 2014, 5:40:02 AM2/28/14
to
Hi Andrew,

thanks for the patch. Is it possible to add this to the sernet package?
All I can find is the already compiled .so-File. Or do I have to use the
samba sources for this test?

I do have a test network with an AD clone, so no worries
there...Unfortunately I can't find the failing object anywhere on the
Windows Server or I look wrong. Is it possible, that samba recieves all
data from the AD but fails when smb1 is not able to join itself to it?

Daniel

Damien Dye

unread,
Feb 28, 2014, 7:30:03 AM2/28/14
to
Daniel

you will need to test against the by getting the source from master git repo


then apply this patch

then compile it

--

Damien Dye
IT Manager
*Sondrel Ltd*
Sondrel House, Theale Lakes Business Park
Moulden Way, Sulhamstead, Berkshire, RG7 4GB, UK

Tel: +44(0)118 9838 550
www.sondrel.com

[image: Sondrel] <http://www.sondrel.com/>

This e-mail and any attachments may be confidential or legally privileged.
If you are not the intended recipient, you should destroy the e-mail
message and any attachments, and inform us of the erroneous delivery by
return e-mail. You are prohibited from retaining, distributing, disclosing
or using any information contained herein. Internet communications cannot
be guaranteed to be timely, secure, error or virus-free. Sondrel Ltd and
the sender do not accept liability for any errors or omissions, nor do we
accept liability for the content of this email, or for the consequences of
any actions taken on the basis of the information provided, unless that
information is consequently confirmed in writing under the personal
signature of a duly authorised officer of Sondrel Ltd.

This email is sent on behalf of Sondrel Ltd registered in England with
number 4491953, registered office Sondrel House, Theale Lakes Business
Park, Moulden Way, Sulhamstead, Berkshire, RG7 4GB, UK.


On 28 February 2014 10:33, Daniel von Obernitz <

Daniel von Obernitz

unread,
Mar 4, 2014, 3:30:03 AM3/4/14
to
Hi Andrew,

your patch worked! Joining took quite a while (at least 8 hours, but I
don't know the exact time), but samba started and I'm able to list users
and GPO.

I'll go on with the migration test now.

Thanks a lot!
Daniel

Andrew Bartlett

unread,
Mar 4, 2014, 4:50:04 PM3/4/14
to
On Tue, 2014-03-04 at 09:25 +0100, Daniel von Obernitz wrote:

> --------------ms040408040805090903060905
> Content-Type: text/plain; charset=UTF-8; format=flowed
> Content-Transfer-Encoding: quoted-printable
>
> Hi Andrew,
>
> your patch worked! Joining took quite a while (at least 8 hours, but I=20
> don't know the exact time), but samba started and I'm able to list users =
>
> and GPO.
>
> I'll go on with the migration test now.

8 hours is a very long time to replicate, is your domain large and your
machine under-powered, or are you doing this over a very slow link?

Andrew Bartlett

--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba




Daniel von Obernitz

unread,
Mar 5, 2014, 2:50:02 AM3/5/14
to
The domain has about 20000 users, don't know if that can be considered
as large.
The machine has 2 cores and 6GB ram and both machines (AD-clone and
samba) are on the same esx-host in an isolated net structure...

Daniel

Andrew Bartlett

unread,
Mar 6, 2014, 4:50:03 PM3/6/14
to
On Wed, 2014-03-05 at 08:44 +0100, Daniel von Obernitz wrote:

> --------------ms060807020701080105090506
> Content-Type: text/plain; charset=UTF-8; format=flowed
> Content-Transfer-Encoding: quoted-printable
>
> The domain has about 20000 users, don't know if that can be considered=20
> as large.
> The machine has 2 cores and 6GB ram and both machines (AD-clone and=20
> samba) are on the same esx-host in an isolated net structure...

That is pretty large for Samba's AD DC code, but folks have deployed
sites that large. Do you have any idea where it spent the time? Was it
CPU busy, or swapping, or something else?

We know we need to improve some issues at the large end, it would be
very interesting to run that join under 'perf record -g' on Linux (with
a very, very large disk and much more memory) to see what we are hitting
the worst, to see if we can improve it.

Thanks,

Daniel von Obernitz

unread,
Mar 7, 2014, 2:30:01 AM3/7/14
to
Hi Andrew,


Am 06.03.2014 22:47, schrieb Andrew Bartlett:
>>
>> The domain has about 20000 users, don't know if that can be considered=20
>> as large.
>> The machine has 2 cores and 6GB ram and both machines (AD-clone and=20
>> samba) are on the same esx-host in an isolated net structure...
>
> That is pretty large for Samba's AD DC code, but folks have deployed
> sites that large. Do you have any idea where it spent the time? Was it
> CPU busy, or swapping, or something else?

I just retested the join with "time", but it aborted because of not
enough disk space, which was correct. But until then it ran about
244minutes.. I will retest it again now to geht the full duration.

I didn't watch the process the whole time, but htop showed the process
using 1 CPU at 100% all the time.


> We know we need to improve some issues at the large end, it would be
> very interesting to run that join under 'perf record -g' on Linux (with
> a very, very large disk and much more memory) to see what we are hitting
> the worst, to see if we can improve it.
>

Would be glad to help, just give me some concrete values for very large
disk (and where do I have to mount it - never worked with perf before)
and how much memory and I'll see, what I can do here...


And I have another question: I joined the Samba using dns BIND9_DLZ .. I
have to use an already existing bind9-server and the generated
named-files are designed for a bind9 running on the same machine as
samba. Are there any manuals how to work with an existing external
bind9-DNS-Server? When I started samba with "samba -i" I got " Server
not found in Kerberos database" and a RID-Error which is connected to
that I think.

Thanks
Daniel

Daniel von Obernitz

unread,
Mar 7, 2014, 9:10:01 AM3/7/14
to
> That is pretty large for Samba's AD DC code, but folks have deployed
> sites that large. Do you have any idea where it spent the time? Was it
> CPU busy, or swapping, or something else?
>

time samba-tool domain join ....

real 291m42.305s
user 279m10.803s
sys 0m10.329s

All the time the process used one cpu on 100% / 20-40% Mem

Daniel

Daniel von Obernitz

unread,
Mar 19, 2014, 12:40:01 PM3/19/14
to
Hi Andrew,

>> We know we need to improve some issues at the large end, it would be
>> very interesting to run that join under 'perf record -g' on Linux (with
>> a very, very large disk and much more memory) to see what we are hitting
>> the worst, to see if we can improve it.
>>
>
> Would be glad to help, just give me some concrete values for very large
> disk (and where do I have to mount it - never worked with perf before)
> and how much memory and I'll see, what I can do here...
>
>

Still would be glad to help, but I need help with the values.


Finally I found the time to go on with my testing procedure using an
external BIND9_DLZ-Server.

I mountet the /usr/local/samba-directory via sshfs to my dns-server
"dns2" (bind9.8.4), so that the directory is also /usr/local/samba. Then
I included the named.conf and it worked so far.

Now I face a problem, when I edit the options-settings in the
named.conf.options, that I get the following error message when starting
bind:


default realm from krb5.conf (UNI-GREIFSWALD.DE) does not match
tkey-gssapi-credential (DNS/dns2.uni-greifswald.de)


krb5.conf
[libdefaults]
default_realm = UNI-GREIFSWALD.DE
dns_lookup_realm = false
dns_lookup_kdc = true


named.conf.options
...
options {
[...]
tkey-gssapi-credential "DNS/dns2.uni-greifswald.de"
tkey-domain "UNI-GREIFSWALD.DE"
tkey-gssapi-keytab "/usr/local/samba/private/dns.keytab";
[...]
};


If I don't use the first two lines in the named.conf.options, bind9 starts.


If I start samba via

samba -i

I get:

/usr/local/samba/sbin/samba_dnsupdate: tkey query failed: GSSAPI error:
Major = Unspecified GSS failure. Minor code may provide more information
, Minor = Server not found in Kerberos database.
/usr/local/samba/sbin/samba_dnsupdate: tkey query failed: GSSAPI error:
Major = Unspecified GSS failure. Minor code may provide more information
, Minor = Server not found in Kerberos database.
Calling samba_kcc script
../source4/dsdb/repl/drepl_ridalloc.c:43: RID Manager failed RID
allocation - WERR_BADFILE - extended_ret[0x0]


Any ideas?
Best regards

Daniel

Daniel von Obernitz

unread,
Apr 29, 2014, 5:50:01 AM4/29/14
to
Hi Andrew,

since I got rid of the dns-problem (had a configuration error) I now got
some more info about the join problem caused by the "isRecycled"-Flag:

I set up a complete new domain (Win Server 2008 R2), added just one more
user and then joined the samba-server (Sernet-Package and from the
sources) without problems. Replication also worked, tested with adding a
second user.

Then I activated the AD-internal recycle bin and tried the join again.
The sernet package failed, the sources including your patch worked, also
with replication.

The failing object is the samba-server itself, either samba doesn't
understand this flag correctly or windows can't set it right on the
object when joining.

Hope this helps a bit.

Best regards
Daniel
0 new messages