[Samba] drs showrepl - Failed to bind to UUID - Undetermined error

[Jonathan Hunter via samba]

Hi Guys,

I have now updated to 4.5.0 - thank you to all the team for your efforts on
this :)

I was excited to read in the release notes that there were many replication
improvements, and I have run 'samba-tool dbcheck --cross-ncs --fix' on all
my DCs; there were many, many replPropertyMetaData and other errors which
have now been found and fixed - thanks!

However, I think something still isn't right in my domain; this is probably
not the fault of 4.5.0 but rather an inconsistency caused when one of my
DCs died and was rebuilt - however I'm now not sure where to look
(presumably with ADSIEdit / ldbsearch) to check which object I need to
remove / update.

The symptom I can see is that running 'samba-tool drs showrepl' fails on
one of my DCs, but works on the other two. On the failing DC I get the
message:

user@dc2:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
Failed to bind to uuid aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee for
ncacn_ip_tcp:1.2.3.4[1024,seal,target_hostname=dc2.mydomain.org.uk
,abstract_syntax=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee/0x00000004,

FUL
ERROR(<class 'samba.drs_utils.drsException'>): DRS connection to
dc2.mydomain.org.uk failed - drsException: DRS connection to
dc2.mydomain.org.uk failed: (-1073741823, 'Undetermined error')
File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/drs.py",
line 41, in drsuapi_connect
(ctx.drsuapi, ctx.drsuapi_handle, ctx.bind_supported_extensions) =
drs_utils.drsuapi_connect(ctx.server, ctx.lp, ctx.creds)
File "/usr/local/samba/lib/python2.7/site-packages/samba/drs_utils.py",
line 54, in drsuapi_connect
raise drsException("DRS connection to %s failed: %s" % (server, e))


Replication of objects between DCs does seem to work fine (at least,
changing the description on a test user object on any DC did propagate
between all 3 DCs) so I don't think the basic mechanism is broken.. but I
suspect whatever objects 'samba-tool drs showrepl' looks at aren't quite
right.

I don't understand why this only fails on one DC, though - all three are
built pretty much identically, so I would have expected this to work or not
equally across all three.

Where should I be looking in AD? The inter-site links seem to be defined OK
from what I can tell, but I don't know much about the internals of these
beyond looking in AD Sites & Services and things "look OK" there.

Any pointers would be much appreciated, I'll do some digging from there.

Thanks!

Jonathan



On an unrelated note, on DC3, 'samba-tool drs showrepl' does work, but
shows the following warnings/errors, before then working fine and showing
the usual output that I would expect to see. Should I file a bug for this -
can anyone else reproduce it?

user@dc3:~ $ sudo /usr/local/samba/bin/samba-tool drs showrepl
Failed to connect host 127.0.1.1 on port 135 - NT_STATUS_CONNECTION_REFUSED
Failed to connect host 127.0.1.1 (dc3.mydomain.org.uk) on port 135 -
NT_STATUS_CONNECTION_REFUSED.
Failed to connect host 127.0.1.1 on port 1024 - NT_STATUS_CONNECTION_REFUSED
Failed to connect host 127.0.1.1 (dc3.mydomain.org.uk) on port 1024 -
NT_STATUS_CONNECTION_REFUSED.
mysite\DC3
[...]


I think I have tracked this one down to the following smb.conf items that
were present on this machine (and which I have now removed):

bind interfaces only = yes
interfaces = eth0 lo
The 'lo' interface has the IP 127.0.0.1, but for some reason 'samba-tool
drs showrepl' is trying to connect to 127.0.1.1 - which then fails.

--
"If we knew what it was we were doing, it would not be called research,
would it?"
- Albert Einstein
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Rowland Penny via samba]

Is this in ubuntu ?
check /etc/hosts and remove any line starting with 127.0.1.1 (or
comment it out)
If networkmanager is running, stop it using dnsmasq.
This is what usually cause the ptoblem you are having.

Rowland

[L.P.H. van Belle via samba]

Quick, befor i get my beer here..

>> Failed to connect host 127.0.1.1

Check you hosts file for this one, you got 127.0.1.1 there from an install with dhcp.


Greetz,

Louis



> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-...@lists.samba.org] Namens Jonathan Hunter
> via samba
> Verzonden: vrijdag 9 september 2016 16:25
> Aan: samba
> Onderwerp: [Samba] drs showrepl - Failed to bind to UUID - Undetermined
> error

[Jonathan Hunter via samba]

Thank you Louis and Rowland!

I feel a little silly, having not checked the "obvious" place /etc/hosts -
but in my defence I have not used debian anything like as much as I have
used Slackware, RedHat and CentOS.. I'd never come across this behaviour at
all.

These DCs are running on Raspberry Pis which are Debian based. Clearly they
have static IPs now but apparently yes, some cruft from the originally
as-shipped DHCP config or whatever was still present.

I can confirm after I update /etc/hosts to 127.0.0.1 (from 127.0.1.1) that
that part works perfectly again on dc3. (I have still removed the lines
from smb.conf, as I don't need them - but it does now work with them in)

Now, I just need to find out why "samba-tool drs showrepl" doesn't work on
one of my other DCs, that's the real mystery :) Editing /etc/hosts on dc3
doesn't help dc2 to work, I still get the "Failed to bind to uuid" error
when running "samba-tool drs showrepl" on dc2.

On 9 September 2016 at 16:01, L.P.H. van Belle via samba <

[Andrew Bartlett via samba]

On Fri, 2016-09-09 at 15:24 +0100, Jonathan Hunter via samba wrote:
> Hi Guys,
>
> I have now updated to 4.5.0 - thank you to all the team for your
> efforts on
> this :)
>
> I was excited to read in the release notes that there were many
> replication
> improvements, and I have run 'samba-tool dbcheck --cross-ncs --fix'
> on all
> my DCs; there were many, many replPropertyMetaData and other errors
> which
> have now been found and fixed - thanks!
>
> However, I think something still isn't right in my domain; this is
> probably
> not the fault of 4.5.0 but rather an inconsistency caused when one of
> my
> DCs died and was rebuilt - however I'm now not sure where to look
> (presumably with ADSIEdit / ldbsearch) to check which object I need
> to
> remove / update.

It looks like others have solved your issue, but just checking on the
broader issue of removing servers.  Is the UUID for the removed server,
and if so how did you remove the DC that died?

We now have 'samba-tool domain demote --remove-other-dead-server' that
will do a more comprehensive job cleaning out the old DC.

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

[Jonathan Hunter via samba]

Thanks Andrew.

No - it was my fault for including an easily-solved side query in the same
email as the main query.. :) I haven't solved the original issue, which is
that 'samba-tool drs showrepl' runs on two of my DCs but not on the third.

I don't know if anything else also doesn't work, e.g. some aspect of
replication I haven't observed yet - but the only problem I can actually
see is that 'samba-tool drs showrepl' doesn't run on this one DC.

You ask a good question in terms of removing the DC that died. I think I
probably did not do this step correctly. I had two DCs die within a short
time of each other (disk issues) and I built new machines and simply joined
them to the domain 'over the top', using the same name and IP address as
previously. I now realise that this might not have been the best idea, as
they would now have new UUIDs and I have done nothing much to remove the
old UUIDs, apart from removing them from DNS/LDAP where I found them.
Perhaps I should have explicitly removed the DCs, before re-adding them? I
may well not have removed them fully myself.

Is there an easy place in AD where these UUIDs are stored - I'm happy to go
through and remove stale entries myself using ADSIEdit or similar? Or would
you recommend I temporarily remove each DC in turn using the demote tool,
then re-add? (Would the demote tool remove *all* UUIDs from the DCs, or
only the first one?)

Is there some form of AD-checker tool, perhaps (either MS or Samba) that
would check all the various LDAP entries, DNS entries (_msdcs, _sites,
_tcp, _kerberos etc.) and point out what I have wrong? :-)

At the moment I guess there might be multiple UUIDs somewhere in the
directory for this one DC, which might be why 'samba-tool drs showrepl'
chokes. There may well be multiple UUIDs for my other server that died,
too, but perhaps the first one that is returned from LDAP for that other
server is the current one, which is why 'samba-tool drs showrepl' works on
that?

Many thanks,

Jonathan

--
"If we knew what it was we were doing, it would not be called research,
would it?"
- Albert Einstein

[Andrew Bartlett via samba]

If you re-joined over the top, then it would have cleaned up most of
what it needed to.  It probably left some junk in DNS, but that is
mostly harmless.

Where the UUID may be is in the repsFrom and repsTo, but this should be
cleaned up by the new KCC.

There isn't currently a 'is this all correct' tool beyond dbcheck.

For the failure to connect, it is trying to connect to the local server
to perform the query, have you confirmed DNS is set resolving for the
name given and that the server is running, and listening?  Wireshark
may help, as might turning up the log level on the command eg -d10 and
the server.

Andrew Bartlett
-- 
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

[Jonathan Hunter via samba]

Thank you Andrew, you pointed me in exactly the right direction, and I now
have 'samba-tool drs showrepl' working fine on this DC. I unfortunately now
don't know if this worked before I upgraded to 4.5, or not - but it does
work again now.

I am once again ashamed to say that there was a config error on this
machine; I can only assume that when I was rebuilding it after the disk
problems, I rebuilt it in a rush. The /etc/hosts file contained the wrong
IP for the local machine (in fact, it had the IP of another DC where I had
perhaps copied the format of the file from!). I have no idea why this
hadn't caused any more issues than just this - DNS was still correct so
perhaps enough programs consulted DNS before /etc/hosts. I'm kind of
impressed that samba seemed to otherwise run just fine on this machine!

Sorry for the false alarm; I think that with the KCC changes in 4.5, I
decided now would be a good time to check 'samba-tool drs showrepl' and
that finally highlighted the typo I made a couple of months back when I
rebuilt the DC..

Thanks all :-)

J

> Samba Developer, Catalyst IT http://catalyst.net.nz/
> services/samba
>
>


--
"If we knew what it was we were doing, it would not be called research,
would it?"
- Albert Einstein

[Andrew Bartlett via samba]

On Sat, 2016-09-10 at 23:27 +0100, Jonathan Hunter via samba wrote:
> Thank you Andrew, you pointed me in exactly the right direction, and
> I now
> have 'samba-tool drs showrepl' working fine on this DC. I
> unfortunately now
> don't know if this worked before I upgraded to 4.5, or not - but it
> does
> work again now.

> Thanks all :-)

Great, glad you got it working.  A samba-tool domain validate that is
broader is scope than dbcheck would be a really nice feature sometime,
checking over all these kind of things.

Andrew Bartlett
-- 
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba