[Samba] Invalid data for index DN=@INDEX:OBJECTCLASS:DNSNODE

[ash-...@comtek.co.uk]

We have a Samba primary domain controller "empire", which seems to have
DNS update issues. We can seem to query all records on empire just fine,
and we can modify IPs for existing records, but it will not delete or
add new records. Attempting to delete via the AD tools shows "Local
security authority database contains an internal inconsistency". Adding
a record on the command line shows:

> samba-tool dns add empire chester-dc.example.com p-bats A 10.4.4.141
-U ash
> Password for [CHESTER-DC\ash]:
> ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
line 175, in _run
> return self.run(*args, **kwargs)
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/dns.py", line
1067, in run
> 0, server, zone, name, add_rec_buf, None)

We have two other DCs (hawaii and alaska), but we are reluctant to
switch to them, since they are located in another country, and have an
unreliable high latency link. The other two DCs accept DNS record
additions/deletions.

Our plan was to set up a 4th DC locally (v-ward), and ultimately make
that the primary server. Unfortunately, this results in:

>
> samba-tool domain join chester-dc.example.com DC -Uash
--realm=CHESTER-DC.EXAMPLE.COM
> Finding a writeable DC for domain 'chester-dc.example.com'
> Found DC empire.chester-dc.example.com
> Password for [CHESTER-DC\ash]:
> workgroup is CHESTER-DC
> realm is chester-dc.example.com
> checking sAMAccountName
> Adding CN=V-WARD,OU=Domain Controllers,DC=chester-dc,DC=example,DC=com
> Adding
CN=V-WARD,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=chester-dc,DC=example,DC=com
> Adding CN=NTDS
Settings,CN=V-WARD,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=chester-dc,DC=example,DC=com
> Adding SPNs to CN=V-WARD,OU=Domain
Controllers,DC=chester-dc,DC=example,DC=com
> Setting account password for V-WARD$
> Enabling account
> Calling bare provision
> No IPv6 address will be assigned
> Provision OK for domain DN DC=chester-dc,DC=example,DC=com
> Starting replication
> Schema-DN[CN=Schema,CN=Configuration,DC=chester-dc,DC=example,DC=com]
objects[402/1550] linked_values[0/0]
> Schema-DN[CN=Schema,CN=Configuration,DC=chester-dc,DC=example,DC=com]
objects[804/1550] linked_values[0/0]
> Schema-DN[CN=Schema,CN=Configuration,DC=chester-dc,DC=example,DC=com]
objects[1206/1550] linked_values[0/0]
> Schema-DN[CN=Schema,CN=Configuration,DC=chester-dc,DC=example,DC=com]
objects[1550/1550] linked_values[0/0]
> Analyze and apply schema objects
> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
objects[402/1634] linked_values[0/0]
> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
objects[804/1634] linked_values[0/0]
> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
objects[1206/1634] linked_values[0/0]
> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
objects[1608/1634] linked_values[0/0]
> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
objects[1634/1634] linked_values[53/0]
> Replicating critical objects from the base DN of the domain
> Partition[DC=chester-dc,DC=example,DC=com] objects[100/100]
linked_values[39/0]
> Partition[DC=chester-dc,DC=example,DC=com] objects[502/723]
linked_values[0/0]
> Partition[DC=chester-dc,DC=example,DC=com] objects[823/723]
linked_values[988/0]
> Done with always replicated NC (base, config, schema)
> Replicating DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[402/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[804/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[1206/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[1608/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[2010/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[2412/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[2814/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[3216/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[3618/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[4020/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[4422/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[4824/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[5226/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[5628/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[6030/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[6432/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[6834/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[7236/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[7638/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[8040/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[8442/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[8844/9093] linked_values[0/0]
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[9093/9093] linked_values[0/0]
> Replicating DC=ForestDnsZones,DC=chester-dc,DC=example,DC=com
> Partition[DC=ForestDnsZones,DC=chester-dc,DC=example,DC=com]
objects[27/27] linked_values[0/0]
> Partition[DC=ForestDnsZones,DC=chester-dc,DC=example,DC=com]
objects[54/27] linked_values[0/0]
> Committing SAM database
> descriptor_modify: Could not find SD for
DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com
>
> Join failed - cleaning up
> checking sAMAccountName
> Deleted CN=V-WARD,OU=Domain Controllers,DC=chester-dc,DC=example,DC=com
> Deleted CN=NTDS
Settings,CN=V-WARD,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=chester-dc,DC=example,DC=com
> Deleted
CN=V-WARD,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=chester-dc,DC=example,DC=com
> ERROR(ldb): uncaught exception - operations error at
../source4/dsdb/samdb/ldb_modules/descriptor.c:1147
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
line 175, in _run
> return self.run(*args, **kwargs)
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/domain.py",
line 555, in run
> machinepass=machinepass, use_ntvfs=use_ntvfs,
dns_backend=dns_backend)
> File "/usr/lib/python2.7/dist-packages/

I have noticed that the the DNS ldb file is rather large (300M):

> total 347988
> -rw------- 1 root root 10383360 May 13 14:13
CN%3DCONFIGURATION,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
> -rw------- 1 root root 10383360 May 13 14:13
CN%3DSCHEMA,CN%3DCONFIGURATION,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
> -rw------- 1 root root 17158144 May 13 14:13
DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
> -rw------- 1 root root 313745408 May 13 14:13
DC%3DDOMAINDNSZONES,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
> -rw------- 1 root root 4247552 May 13 14:13
DC%3DFORESTDNSZONES,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
> -rw-r----- 1 root root 421888 May 13 14:09 metadata.tdb

Investigating further:

> 0 root@empire:~[0] /usr/bin/samba-tool drs replicate
empire.chester-dc.example.com alaska.chester-dc.example.com
DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com --local
Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
objects[402/15688] linked_values[0/0]
> Invalid data for index DN=@INDEX:OBJECTCLASS:DNSNODE
>
> replmd_replicated_request rename
DC=DEELR013\0ADEL:08ae6b71-9b11-4003-9daf-f2e2ed3a58be,CN=Deleted
Objects,DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com =>
DC=DEELR013\0ACNF:08ae6b71-9b11-4003-9daf-f2e2ed3a58be\0ADEL:08ae6b71-9b11-4003-9daf-f2e2ed3a58be,CN=Deleted
Objects,DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com failed -
ldb_wait: > Operations error (1)
>
> Failed to apply records: ldb_wait: Operations error (1): Other
> Failed to commit objects: WERR_GENERAL_FAILURE
> ERROR(<type 'exceptions.TypeError'>): Error replicating DN
DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com - Failed to process
chunk: NT_STATUS_UNSUCCESSFUL
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/drs.py", line
269, in drs_local_replicate
> repl.replicate(NC, source_dsa_invocation_id, destination_dsa_guid)
> File "/usr/lib/python2.7/dist-packages/samba/drs_utils.py", line
256, in replicate
> schema=schema, req_level=req_level, req=req)

This pointed us at the DEELR013 record, so, I tried:

> 0 root@empire:~[0] ldbdel -H
/var/lib/samba/private/sam.ldb.d/DC%3DDOMAINDNSZONES,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
DC=DEELR013,DC=chester-dc.example.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com
> Invalid data for index DN=@INDEX:OBJECTCLASS:DNSNODE
>
> delete of
'DC=DEELR013,DC=chester-dc.example.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com'
failed - (Operations error) ldb_wait: Operations error (1)
>

Finally, stumbling around blindly I ran tdbbackup on the DOMAINDNSZONES
ldb file (which shrunk a few megabytes - no errors though), and I
managed to ldbedit and delete the file index, then it allowed me to
ldbdel. I Copied the newly modified file on top of the original one,
restarted Samba, and at that point I realised that the file was now over
700mb. Samba had hung and stopped accepting connections (I couldn't even
get a share list with smbclient). Unfortunately I can't give accurate
detail about this paragraph, because I rolled back to last night's LXC
snapshot.

Can anybody please give us advice on how to proceed from here?

> 0 root@empire:~[0] samba-tool -V
> 4.1.11-Debian
> 0 root@empire:~[0] dpkg -s samba |grep ^Ver
> Version: 2:4.1.11+dfsg-1
> 0 root@empire:~[0] uname -a
> Linux empire 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
x86_64 GNU/Linux


--
/---------------------------------------------------------------------\
|Ashley Griffiths Phone: +44 (0)1244 280 390 |
|IT manager Web:http://www.comtek.co.uk/ |
|Comtek Group |
\---------------------------------------------------------------------/




--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Rowland penny]

First things first, is there anyway you can update Samba ?
The 4.1.X series is now EOL and wasn't patched for badlock, depending on
what version of debian you are running, you should be able to upgrade
easily.

Please do not alter the ldb under sam.ldb.d directly, only modify the
sam.ldb file (this contains everything in sam.ldb.d)

With AD, there is no such thing as a primary domain controller, all DCs
are equal, the only difference is in which DC has the FSMO roles and
these do not need to be all on the same DC. I mention this because it
can get confusing when/if somebody asks a question about an NT-style PDC
problem.

Your domain zone growing in size is probably down to tombstone objects,
try searching on 'samba tombstone' for help on this.

Have you tried running 'samba-tool dbcheck' ??

Rowland

[ash-...@comtek.co.uk]

> First things first, is there anyway you can update Samba ?
> The 4.1.X series is now EOL and wasn't patched for badlock, depending
> on what version of debian you are running, you should be able to
> upgrade easily.

0 root@empire:/etc/apt[0] samba-tool -V
4.2.10-Debian
0 root@empire:/etc/apt[0] dpkg -s samba |grep ^Ver
Version: 2:4.2.10+dfsg-0+deb8u2
0 root@empire:/etc/apt[0] uname -a

Linux empire 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
x86_64 GNU/Linux

> Have you tried running 'samba-tool dbcheck' ??

0 root@empire:/etc/apt[0] samba-tool dbcheck
Checking 723 objects
Checked 723 objects (0 errors)

We have the same inability to add records.

We have the same error running: /usr/bin/samba-tool drs replicate
empire.chester-dc.example.com alaska.chester-dc.example.com
DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com --local

Thanks, but it doesn't seem like the upgrade has helped.

[Rowland penny]

On 13/05/16 17:19, ash-...@comtek.co.uk wrote:
>
>> First things first, is there anyway you can update Samba ?
>> The 4.1.X series is now EOL and wasn't patched for badlock, depending
>> on what version of debian you are running, you should be able to
>> upgrade easily.
> 0 root@empire:/etc/apt[0] samba-tool -V
> 4.2.10-Debian
> 0 root@empire:/etc/apt[0] dpkg -s samba |grep ^Ver
> Version: 2:4.2.10+dfsg-0+deb8u2
> 0 root@empire:/etc/apt[0] uname -a
> Linux empire 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
> x86_64 GNU/Linux
>> Have you tried running 'samba-tool dbcheck' ??
> 0 root@empire:/etc/apt[0] samba-tool dbcheck
> Checking 723 objects
> Checked 723 objects (0 errors)
>
> We have the same inability to add records.
>
> We have the same error running: /usr/bin/samba-tool drs replicate
> empire.chester-dc.example.com alaska.chester-dc.example.com
> DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com --local
>
> Thanks, but it doesn't seem like the upgrade has helped.

OK, could this just be a permissions problem i.e. user 'ash' doesn't
have the required rights to add a dns record, try again, but this time
use the 'Administrator' user.

Rowland

[ash-...@comtek.co.uk]

> OK, could this just be a permissions problem i.e. user 'ash' doesn't
> have the required rights to add a dns record, try again, but this time
> use the 'Administrator' user.

I've repeated the "samba-tool dns add", and the "samba-tool domain join"
commands with "-UAdministrator". I get the same errors with either user.

(the error for domain join is now the following)

> samba-tool domain join chester-dc.comtek.co.uk DC -Uash
--realm=CHESTER-DC.COMTEK.CO.UK
> Finding a writeable DC for domain 'chester-dc.comtek.co.uk'
> Found DC empire.chester-dc.comtek.co.uk

> Password for [CHESTER-DC\ash]:
> workgroup is CHESTER-DC

> realm is chester-dc.comtek.co.uk
> checking sAMAccountName
> Adding CN=V-WARD,OU=Domain
Controllers,DC=chester-dc,DC=comtek,DC=co,DC=uk

> Join failed - cleaning up
> checking sAMAccountName

> ERROR(ldb): uncaught exception - LDAP error 68
LDAP_ENTRY_ALREADY_EXISTS - <00002071: ../ldb_tdb/ldb_index.c:1216:
Failed to re-index objectSid in CN=V-WARD,OU=Domain
Controllers,DC=chester-dc,DC=comtek,DC=co,DC=uk -
../ldb_tdb/ldb_index.c:1148: unique index violation on objectSid in
CN=V-WARD,OU=Domain Controllers,DC=chester-dc,DC=comtek,DC=co,DC=uk> <>

> File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
line 175, in _run
> return self.run(*args, **kwargs)
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/domain.py",
line 555, in run
> machinepass=machinepass, use_ntvfs=use_ntvfs,
dns_backend=dns_backend)

> File "/usr/lib/python2.7/dist-packages/samba/join.py", line 1172,
in join_DC
> ctx.do_join()
> File "/usr/lib/python2.7/dist-packages/samba/join.py", line 1075,
in do_join
> ctx.join_add_objects()
> File "/usr/lib/python2.7/dist-packages/samba/join.py", line 515, in
join_add_objects
> ctx.samdb.add(rec)


Could permissions account for the " Invalid data for index
DN=@INDEX:OBJECTCLASS:DNSNODE", anyway?

Ash

--
/---------------------------------------------------------------------\
|Ashley Griffiths Phone: +44 (0)1244 280 390 |
|IT manager Web: http://www.comtek.co.uk/ |
|Comtek Group |
\---------------------------------------------------------------------/

[Rowland penny]

Possibly, if your user doesn't have the correct rights, then the command
could error, the full command that failed was this:

dns_conn.DnssrvUpdateRecord2(dnsserver.DNS_CLIENT_VERSION_LONGHORN, 0,

server, zone, name, add_rec_buf, None)

This relies on:

dns_conn = dns_connect(server, self.lp, self.creds)

The relevant part is this: 'self.creds'

This means the entire command would fail if the supplied user didn't
have the required rights

The above 'join' error seems to show that 'chester-dc' already exists in
AD (if only partially), you could try checking if this is possible. If
it does, you will need to find a way of removing it, but we will come to
that only if it does.

Rowland

[Andrew Bartlett]

On Fri, 2016-05-13 at 14:49 +0100, ash-...@comtek.co.uk wrote:
> We have a Samba primary domain controller "empire", which seems to
> have
> DNS update issues. We can seem to query all records on empire just
> fine,
> and we can modify IPs for existing records, but it will not delete or
> add new records. Attempting to delete via the AD tools shows "Local
> security authority database contains an internal inconsistency".
> Adding
> a record on the command line shows:

This certainly sounds stressful.

Another way to (on a backup, particularly given your history above) remove the index is with samba-tool dbcheck --reindex.

The missing ntSecurityDescriptor is a curious issue. Can you check if
it or the whole record is really missing? I'm guessing it is another
index issue, stopping us finding the record rather than the record not
being there. Look over an ldbdump of the backend DB in sam.ldb.d/ if
you have to, to confirm that.
Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

[ash-...@comtek.co.uk]

> Possibly, if your user doesn't have the correct rights, then the
> command could error, the full command that failed was this:
>
> dns_conn.DnssrvUpdateRecord2(dnsserver.DNS_CLIENT_VERSION_LONGHORN, 0,
> server, zone, name, add_rec_buf, None)
>
> This relies on:
>
> dns_conn = dns_connect(server, self.lp, self.creds)
>
> The relevant part is this: 'self.creds'
>
> This means the entire command would fail if the supplied user didn't
> have the required rights
>
> The above 'join' error seems to show that 'chester-dc' already exists
> in AD (if only partially), you could try checking if this is possible.
> If it does, you will need to find a way of removing it, but we will
> come to that only if it does.
>

Sorry for the delay in responding. It seems that upgrading empire caused
"drs replicate" to fail on the other two machines,
(LDAP_STRONG_AUTH_REQUIRED - <SASL:[GSS-SPNEGO]: Sign or Seal are
required), so we've had to find a way to quickly upgrade them.

We have noticed a new symptom since the 4.2 upgrade. We have a periodic
script which creates users. It now appears to be doing:

ERROR(ldb): Failed to add user 'john.smith': -

../ldb_tdb/ldb_index.c:1216: Failed to re-index objectSid in

CN=john.smith,CN=Users,DC=chester-dc,DC=example,DC=com -

../ldb_tdb/ldb_index.c:1148: unique index violation on objectSid in

CN=john.smith,CN=Users,DC=chester-dc,DC=example,DC=com

[ash-...@comtek.co.uk]

> This certainly sounds stressful.
Yes!

> Another way to (on a backup, particularly given your history above) remove the index is with samba-tool dbcheck --reindex.

Re-indexing...
completed re-index OK

0 root@empire:~[0] samba-tool dns add empire chester-dc.example.com
p-cats A 10.4.4.142 -U ash
Password for [CHESTER-DC\ash]:
Record added successfully

Thanks!

> The missing ntSecurityDescriptor is a curious issue. Can you check if
> it or the whole record is really missing? I'm guessing it is another
> index issue, stopping us finding the record rather than the record not
> being there. Look over an ldbdump of the backend DB in sam.ldb.d/ if
> you have to, to confirm that.
> Andrew Bartlett

I haven't actually got ldbdump on the machine, and I can't see it in the
Debian packages. That said, I do appear to be able to add DNS records
now, so I'm assuming it was the index. If you particularly want me to
find out then I'll try to get a dump, but as long as its working I'm
happy to leave it be!

Ash

[ash-...@comtek.co.uk]

>> Andrew Bartlett
> I haven't actually got ldbdump on the machine, and I can't see it in
> the Debian packages. That said, I do appear to be able to add DNS
> records now, so I'm assuming it was the index. If you particularly
> want me to find out then I'll try to get a dump, but as long as its
> working I'm happy to leave it be!
>
> Ash

Well, I will try to obtain that ldbdump

samba-tool dbcheck --reindex doesn't seem to have entirely worked. While
we can add DNS records we can't add users. For example:

> /usr/bin/samba-tool user add test.user --uid=test.user
--random-password --uid-number=10226 --surname=user --given-name=test
--job-title=Storekeeper --department=Repairs
--mail-address=test...@example.com --telephone-number=01244123456
--gid-number=513
> ERROR(ldb): Failed to add user 'test.user': -
../ldb_tdb/ldb_index.c:1216: Failed to re-index objectSid in CN=test
user,CN=Users,DC=chester-dc,DC=example,DC=com -

../ldb_tdb/ldb_index.c:1148: unique index violation on objectSid in

CN=test user,CN=Users,DC=chester-dc,DC=example,DC=com

We also can't add a DC:

> samba-tool domain join chester-dc.example.com DC -Uash
--realm=CHESTER-DC.EXAMPLE.COM
> Finding a writeable DC for domain 'chester-dc.example.com'
> Found DC empire.chester-dc.example.com
> Password for [CHESTER-DC\ash]:
> workgroup is CHESTER-DC
> realm is chester-dc.example.com
> checking sAMAccountName
> Adding CN=V-WARD,OU=Domain Controllers,DC=chester-dc,DC=example,DC=com

> Join failed - cleaning up
> checking sAMAccountName

> ERROR(ldb): uncaught exception - LDAP error 68

LDAP_ENTRY_ALREADY_EXISTS - <00002071: ../ldb_tdb/ldb_index.c:1216:
Failed to re-index objectSid in CN=V-WARD,OU=Domain
Controllers,DC=chester-dc,DC=example,DC=com -

../ldb_tdb/ldb_index.c:1148: unique index violation on objectSid in

CN=V-WARD,OU=Domain Controllers,DC=chester-dc,DC=example,DC=com> <>

> File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
line 175, in _run
> return self.run(*args, **kwargs)
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/domain.py",
line 555, in run
> machinepass=machinepass, use_ntvfs=use_ntvfs,
dns_backend=dns_backend)

> File "/usr/lib/python2.7/dist-packages/samba/join.py", line 1172,
in join_DC
> ctx.do_join()
> File "/usr/lib/python2.7/dist-packages/samba/join.py", line 1075,
in do_join
> ctx.join_add_objects()
> File "/usr/lib/python2.7/dist-packages/samba/join.py", line 515, in
join_add_objects
> ctx.samdb.add(rec)

Or add a member:

> root@p-bats:/etc/samba# net ads join -Uash
> Enter ash's password:
> Failed to join domain: failed to join domain 'CHESTER-DC.EXAMPLE.COM'
over rpc: None of the information to be translated has been translated.

[ash-...@comtek.co.uk]

On 16/05/16 16:41, ash-...@comtek.co.uk wrote:
>
>>> Andrew Bartlett
>> I haven't actually got ldbdump on the machine, and I can't see it in
>> the Debian packages. That said, I do appear to be able to add DNS
>> records now, so I'm assuming it was the index. If you particularly
>> want me to find out then I'll try to get a dump, but as long as its
>> working I'm happy to leave it be!
>>
>> Ash
>

Okay, I've managed to compile ldbdump, and doing ./ldbdump
/var/lib/samba/private/sam.ldb.d/DC%3DDOMAINDNSZONES,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
now only shows CN=Deleted Objects for the PC record which we considered
suspect. Each entry does have an ntSecurityDescriptor, but the original
object doesn't seem to be listed.

I believe that DNS is fine now, but the inability to add user records or
join machines to the domain is a bigger problem!

Ash

[Andrew Bartlett]

On Mon, 2016-05-16 at 16:41 +0100, ash-...@comtek.co.uk wrote:
> > > Andrew Bartlett
> > I haven't actually got ldbdump on the machine, and I can't see it
> > in
> > the Debian packages. That said, I do appear to be able to add DNS
> > records now, so I'm assuming it was the index. If you particularly
> > want me to find out then I'll try to get a dump, but as long as its
> > working I'm happy to leave it be!
> >
> > Ash
>
> Well, I will try to obtain that ldbdump
>
> samba-tool dbcheck --reindex doesn't seem to have entirely worked.
> While
> we can add DNS records we can't add users. For example:
>
> > /usr/bin/samba-tool user add test.user --uid=test.user
> --random-password --uid-number=10226 --surname=user --given-name=test
> --job-title=Storekeeper --department=Repairs
> --mail-address=test...@example.com --telephone-number=01244123456
> --gid-number=513
> > ERROR(ldb): Failed to add user 'test.user': -
> ../ldb_tdb/ldb_index.c:1216: Failed to re-index objectSid in CN=test
> user,CN=Users,DC=chester-dc,DC=example,DC=com -
> ../ldb_tdb/ldb_index.c:1148: unique index violation on objectSid in
> CN=test user,CN=Users,DC=chester-dc,DC=example,DC=com

G'Day,

This is a serious situation. What it means is that the nextRid value for that DC points at a user account that already exists, so when we go to create it, the create fails.

That, and the other issue, suggests you have had some serious DB corruption, and this may not be the only issues. Does a full dbcheck pass? (Not just the reindex).

Is there another DC that still works, that you can replicate from? (but you suggested other issues I think).

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

[ash-...@comtek.co.uk]

> G'Day,
>
> This is a serious situation. What it means is that the nextRid value for that DC points at a user account that already exists, so when we go to create it, the create fails.

I've just looked at the LDAP output, and nextRid is 1000 for both dn:
CN=Builtin,DC=chester-dc,etc and for dn: DC=chester-dc,etc

The most recent successful new user (that I'm aware of) is objectSid:
S-1-5-21-2702589905-558746101-3641499263-2825

I can't see any objectSid entries which end in 1000 though. The lowest
one we have is S-1-5-21-2702589905-558746101-3641499263-1101

> That, and the other issue, suggests you have had some serious DB corruption, and this may not be the only issues. Does a full dbcheck pass? (Not just the reindex).

dbcheck works on empire.

> Is there another DC that still works, that you can replicate from? (but you suggested other issues I think).

We can successfully "/usr/bin/samba-tool user add" with alaska (a
machine located on another continent, with a quite unreliable link!),
and that gives us an account with
S-1-5-21-2702589905-558746101-3641499263-7125 on -both- alaska and
empire, so there is clearly some amount of working replication.
Confusingly, after doing this nextRid is still 1000 on both machines.

Creating a new local DC (and decommissioning empire) would be a good
solution for us. I can add a new DC (v-ward) by specifying
--server=alaska.chester-dc, and I get no errors in the process. The
samba process on v-ward isn't working, though. I'm still trying to debug
this (currently it isn't even listening to port 389).

[Rowland penny]

On 17/05/16 12:11, ash-...@comtek.co.uk wrote:
>
>> G'Day,
>>
>> This is a serious situation. What it means is that the nextRid value
>> for that DC points at a user account that already exists, so when we
>> go to create it, the create fails.
> I've just looked at the LDAP output, and nextRid is 1000 for both dn:
> CN=Builtin,DC=chester-dc,etc and for dn: DC=chester-dc,etc

Same here.

>
> The most recent successful new user (that I'm aware of) is objectSid:
> S-1-5-21-2702589905-558746101-3641499263-2825
>
> I can't see any objectSid entries which end in 1000 though. The lowest
> one we have is S-1-5-21-2702589905-558746101-3641499263-1101
>> That, and the other issue, suggests you have had some serious DB
>> corruption, and this may not be the only issues. Does a full dbcheck
>> pass? (Not just the reindex).
> dbcheck works on empire.
>> Is there another DC that still works, that you can replicate from?
>> (but you suggested other issues I think).
>
> We can successfully "/usr/bin/samba-tool user add" with alaska (a
> machine located on another continent, with a quite unreliable link!),
> and that gives us an account with
> S-1-5-21-2702589905-558746101-3641499263-7125 on -both- alaska and
> empire, so there is clearly some amount of working replication.
> Confusingly, after doing this nextRid is still 1000 on both machines.

This could be because you are looking at the wrong attribute in the
wrong place.
Try looking at the object 'CN=RID Set,CN=ALASKA,OU=Domain
Controllers,DC=CHESTER-DC,DC=EXAMPLE,DC=COM' and the attribute
'rIDNextRID' it contains.

Rowland

[ash-...@comtek.co.uk]

>> We can successfully "/usr/bin/samba-tool user add" with alaska (a
>> machine located on another continent, with a quite unreliable link!),
>> and that gives us an account with
>> S-1-5-21-2702589905-558746101-3641499263-7125 on -both- alaska and
>> empire, so there is clearly some amount of working replication.
>> Confusingly, after doing this nextRid is still 1000 on both machines.
>
> This could be because you are looking at the wrong attribute in the
> wrong place.
> Try looking at the object 'CN=RID Set,CN=ALASKA,OU=Domain
> Controllers,DC=CHESTER-DC,DC=EXAMPLE,DC=COM' and the attribute
> 'rIDNextRID' it contains.

Interesting.

If, on Alaska, I do: ldbedit -H ldap://localhost -U ash

> # record 122
> dn: CN=RID Set,CN=ALASKA,OU=Domain
Controllers,DC=chester-dc,DC=example,DC=com
> objectClass: top
> objectClass: rIDSet
> cn: RID Set
> instanceType: 4
> whenCreated: 20141223180132.0Z
> whenChanged: 20141223180132.0Z
> uSNCreated: 12146
> uSNChanged: 12146
> showInAdvancedViewOnly: TRUE
> name: RID Set
> objectGUID: b2f1c43e-4bd7-46dd-bdd8-6cc31f259655
> rIDAllocationPool: 7100-7599
> rIDUsedPool: 0
> objectCategory:
CN=RID-Set,CN=Schema,CN=Configuration,DC=chester-dc,DC=example,
> DC=com
> rIDPreviousAllocationPool: 7100-7599
> rIDNextRID: 7126
> distinguishedName: CN=RID Set,CN=ALASKA,OU=Domain
Controllers,DC=chester-dc,DC
> =example,DC=com


on empire, the same command shows

> # record 122
> dn: CN=RID Set,CN=ALASKA,OU=Domain
Controllers,DC=chester-dc,DC=example,DC=com
> objectClass: top
> objectClass: rIDSet
> cn: RID Set
> instanceType: 4
> whenCreated: 20141223180132.0Z
> whenChanged: 20141223180132.0Z
> uSNCreated: 39967
> uSNChanged: 39967
> showInAdvancedViewOnly: TRUE
> name: RID Set
> objectGUID: b2f1c43e-4bd7-46dd-bdd8-6cc31f259655
> rIDAllocationPool: 7100-7599
> rIDPreviousAllocationPool: 0-0
> rIDUsedPool: 0
> rIDNextRID: 0
> objectCategory:
CN=RID-Set,CN=Schema,CN=Configuration,DC=chester-dc,DC=example,
> DC=com
> distinguishedName: CN=RID Set,CN=ALASKA,OU=Domain
Controllers,DC=chester-dc,DC
> =example,DC=com

The interesting thing is that alaska has got no other RID Set entries.
empire has a RID Set for each of empire, alaska, hawaii, v-ward (though
the value for rIDNextRID is 0 for each except for the empire entry
itself, which is 2828). Is this normal?

The rIDNextRID 2828 does collide with the SID entry for dn:
CN=DEEL059,CN=Computers,DC=chester-dc,DC=example,DC=com

[Rowland penny]

OK, I just checked on my test domain, DC1 has 'CN=RID Set' for both DCs,
but only shows 'rIDNextRID: 0' for DC2.
DC2 only has its own 'CN=RID Set' and shows rIDNextRID: 1605. It looks
like this part of your AD is correct.

A quick check reveals that 'rIDNextRID' is one of Microsofts famous
mis-named attributes, it should really have been 'rIDLastRIDused' and is
a non replicating attribute.

Rowland

[Andrew Bartlett]

On Tue, 2016-05-17 at 12:11 +0100, ash-...@comtek.co.uk wrote:
> > G'Day,
> >
> > This is a serious situation. What it means is that the nextRid
> > value for that DC points at a user account that already exists, so
> > when we go to create it, the create fails.
> I've just looked at the LDAP output, and nextRid is 1000 for both dn:
> CN=Builtin,DC=chester-dc,etc and for dn: DC=chester-dc,etc
>
> The most recent successful new user (that I'm aware of) is objectSid:
> S-1-5-21-2702589905-558746101-3641499263-2825
>
> I can't see any objectSid entries which end in 1000 though. The
> lowest
> one we have is S-1-5-21-2702589905-558746101-3641499263-1101
> > That, and the other issue, suggests you have had some serious DB
> > corruption, and this may not be the only issues. Does a full
> > dbcheck pass? (Not just the reindex).
> dbcheck works on empire.
> > Is there another DC that still works, that you can replicate from?
> > (but you suggested other issues I think).
>
> We can successfully "/usr/bin/samba-tool user add" with alaska (a
> machine located on another continent, with a quite unreliable link!),
> and that gives us an account with
> S-1-5-21-2702589905-558746101-3641499263-7125 on -both- alaska and
> empire, so there is clearly some amount of working replication.
> Confusingly, after doing this nextRid is still 1000 on both machines.

The value you need to look for is in the RID Set, not the domain, which
is a legacy figure we don't use. Sorry for the red herring.

> Creating a new local DC (and decommissioning empire) would be a good
> solution for us. I can add a new DC (v-ward) by specifying
> --server=alaska.chester-dc, and I get no errors in the process. The
> samba process on v-ward isn't working, though. I'm still trying to
> debug
> this (currently it isn't even listening to port 389).

OK. That seems serious.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba