[Samba] "failed to lookup DC info for domain over rpc" when joining samba4 domain

[Richard Connon]

Hi,

I have an existing samba4 domain with 2 domain controllers on different
sites.

Both domain controllers are running samba 4.1.17

Until recently the domain operated exactly as expected. I recently tried
to join a new machine to the domain and received the error:

Failed to join domain: failed to lookup DC info for domain
'ADS.CONNON.ME.UK' over rpc

I'm not sure what has triggered this change in behaviour. Authentication
against the domain still works as normal.

I've attached a full debug log of the domain join process.

Anyone got any ideas where I should be looking?

Regards,
Richard

[Tim]

Hey Richard,

you should post your debug log because your attachment has been scrubbed.

Have you checked your DNS entries? I don't know why but I had an issue some days ago that the host a entry for my fsmo DC had disappeared.

Regards
Tim

>------------------------------------------------------------------------
>
>--
>To unsubscribe from this list go to the following URL and read the
>instructions: https://lists.samba.org/mailman/options/samba
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Richard Connon]

Hi again,

Seems you can't attach items to mails on this list so here's the debug log:
http://www.irconan.co.uk/net-ads-join.log

Regards,
Richard

[Richard Connon]

Hi Tim,

I re-posted with a link to the debug log.

I appear to have A and AAAA records present for both DCs. Not sure about
the relevant SRV records used in domain join. Got any pointers on how
they should look?

Regards,
Richard

[Rowland Penny]

On 09/03/15 21:59, Richard Connon wrote:
> On 09/03/2015 21:59, Rowland Penny wrote:
>> How did you try to join the machine to the domain ? I think I know,
>> but it would like you to confirm my suspicions.
>
> Hi Rowland,
>
> This output was generated with `net ads join -Uprovisioning%<password>
> -d10
>
> Regards,
> Richard

OK, well it isn't what I thought, moving on, what is in smb.conf (please
do not post any commented lines), /etc/resolv.conf, /etc/krb5.conf, what
OS etc

Rowland

[Richard Connon]

On 09/03/2015 21:59, Rowland Penny wrote:
> How did you try to join the machine to the domain ? I think I know,
> but it would like you to confirm my suspicions.

Hi Rowland,

This output was generated with `net ads join -Uprovisioning%<password> -d10

Regards,
Richard

[Rowland Penny]

How did you try to join the machine to the domain ? I think I know, but
it would like you to confirm my suspicions.

Rowland

[Richard Connon]

On 09/03/2015 22:07, Rowland Penny wrote:
> On 09/03/15 21:59, Richard Connon wrote:
>> On 09/03/2015 21:59, Rowland Penny wrote:
>>> How did you try to join the machine to the domain ? I think I know,
>>> but it would like you to confirm my suspicions.
>>
>> Hi Rowland,
>>
>> This output was generated with `net ads join
>> -Uprovisioning%<password> -d10
>>
>> Regards,
>> Richard
>
> OK, well it isn't what I thought, moving on, what is in smb.conf
> (please do not post any commented lines), /etc/resolv.conf,
> /etc/krb5.conf, what OS etc
>
> Rowland
>

Hi Rowland,

On all hosts of site CCPG-UK:
resolv.conf contains:
domain ads.connon.me.uk
nameserver 10.10.0.250
nameserver 10.10.0.252
nameserver 10.10.0.251

krb5.conf contains:
[libdefaults]
default_realm = ADS.CONNON.ME.UK
dns_lookup_realm = false
dns_lookup_kdc = true
rdns = false

The DC smb.conf contains:
[global]
netbios name = DC01
realm = ADS.CONNON.ME.UK
workgroup = CONNON
server role = active directory domain controller
server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc,
drepl, winbind, ntp_signd, kcc, dnsupdate
dedicated keytab file = /etc/krb5.keytab
kerberos method = dedicated keytab
dsdb:schema update allowed = Yes

[netlogin]
path = /var/lib/samba/sysvol/ads.connon.me.uk/scripts
read only = No
[sysvol]
path = /var/lib/samba/sysvol
read only = No

The client smb.conf contains:
[global]
security = ads
netbios name = SHELL01
realm = ADS.CONNON.ME.UK
workgroup = CONNON
dedicated keytab file = /etc/krb5.keytab
kerberos method = dedicated keytab

The OS for all machines is debian 7. The DC is using samba
4.1.17+dfsg-1~bpo70+1 from backports while the client is using
3.6.6-6+deb7u5.
I appreciate that samba 3.6 is now very old but I'd like to avoid
deviating from the standard install for clients. I'm reasonably sure
this should be fixable with a 3.6 client since it has worked so well in
the past.

It is possible that the DC has received a minor (4.1.x) upgrade since
domain join last worked.

Regards,
Richard

[Rowland Penny]

Hmm, everything looks ok and it shouldn't matter whether you use the
standard 3.6 from debian or 4.1.17 from backports except for the fact
that 3.6 isn't just old, it is EOL , so you may have to rely on debian
backporting any security updates themselves.

I take it that the three nameservers in the clients resolv.conf are all
DC's, if not, I suggest you remove any that aren't, could you also have
a look here:

https://wiki.samba.org/index.php/Setup_a_Samba_AD_Member_Server

Rowland

[Richard Connon]

On 09/03/2015 22:36, Rowland Penny wrote:
> Hmm, everything looks ok and it shouldn't matter whether you use the
> standard 3.6 from debian or 4.1.17 from backports except for the fact
> that 3.6 isn't just old, it is EOL , so you may have to rely on debian
> backporting any security updates themselves.
>
> I take it that the three nameservers in the clients resolv.conf are
> all DC's, if not, I suggest you remove any that aren't, could you also
> have a look here:
>
> https://wiki.samba.org/index.php/Setup_a_Samba_AD_Member_Server
>
> Rowland

Hi Rowland,

I'm aware of 3.6's security status. I'm planning to count on debian
backporting fixes for now and move to 4.1 (or 4.2) if and when required.
I have just tried, as an experiment, upgrading this failing client to
4.1.17 to no avail.

The nameservers in resolv.conf are just forwarders. They forward to my
DCs for anything under ads.connon.me.uk.
As an experiment I tried changing the resolv.conf on both the DC and the
client to contain just the DC for this site rather than my normal
recursive servers. Again, this didn't change the behaviour.

I'm not familiar with the RPC protocol very much. Are there some tools I
can use to perform some test queries against this DC?

Regards,
Richard

[Tim]

Hey Richard,

first of all I personally think it is better to post logfiles in plain text on the list so that it keeps readable for later users. Just my two cents :-)

What I first saw in your smb.conf is that the netlogon share is named netlogin.

Beside this, I will send you a list of DNS entries I have under _msdcs later. Perhaps it is worth to compare.

[Rowland Penny]

Your DC's must point to themselves for DNS and your domain clients must
point to the DC's, anything outside the domain the DC's will be obtain
from the forwarders set on them.

What I think is happening: your client is asking for the DC from your
forwarders, they do not know, so they ask the DC, who asks the
forwarder, who does not know and so on.

The resolv.conf on my DCs is simply this:

search example.com
nameserver 127.0.0.1

I use Bind and this is setup to forward to my router, so when a client
wants the DC, it contacts a DC (set in resolv.conf on client) which
knows all about the domain and replies with the correct info. You can do
this with the internal DC DNS server.

Rowland

[Tim]

Hey Richard,

here you are my _msdcs-dns-zone. I only have two dcs. I hope the text structure will be readable at your side.

Name Type Data
dc
+- _sites
+- Default-First-Site-Name
+- _tcp
_kerberos Service Identification (SRV) [0][100][88] DC1.example.samdom.com.
_kerberos Service Identification (SRV) [0][100][88] DC2.example.samdom.com.
_ldap Service Identification (SRV) [0][100][389] DC1.example.samdom.com.
_ldap Service Identification (SRV) [0][100][389] DC2.example.samdom.com.
+- _tcp
_kerberos Service Identification (SRV) [0][100][88] DC1.example.samdom.com.
_kerberos Service Identification (SRV) [0][100][88] DC2.example.samdom.com.
_ldap Service Identification (SRV) [0][100][389] DC1.example.samdom.com.
_ldap Service Identification (SRV) [0][100][389] DC2.example.samdom.com.
domains
+- <Domain-ID>
_tcp
+- _ldap
(identical with folder above) Service Identification (SRV) [0][100][389] DC1.example.samdom.com.
(identical with folder above) Service Identification (SRV) [0][100][389] DC2.example.samdom.com.
_ldap Service Identification (SRV) [0][100][389] DC1.example.samdom.com.
_ldap Service Identification (SRV) [0][100][389] DC2.example.samdom.com.

gc
+- _sites
+- Default-First-Site-Name
+- _tcp
_ldap Service Identification (SRV) [0][100][389] DC1.example.samdom.com.
_ldap Service Identification (SRV) [0][100][389] DC2.example.samdom.com.
+- _tcp
_ldap Service Identification (SRV) [0][100][389] DC1.example.samdom.com.
_ldap Service Identification (SRV) [0][100][389] DC2.example.samdom.com.
(identical with folder above) Host (A) <IP Adress DC1>
(identical with folder above) Host (A) <IP Adress DC2>

pdc
+- _tcp
_ldap Service Identification (SRV) [0][100][389] DC1.example.samdom.com.

<Unique ID of DC1> Alias (CNAME) DC1.example.samdom.com.
<Unique ID of DC2> Alias (CNAME) DC2.example.samdom.com.
(identical with folder above) Authority Source (SOA) [12], DC1.example.samdom.com., hostmaster.example.samdom.com.
(identical with folder above) Nameserver (NS) DC1.example.samdom.com.
(identical with folder above) Nameserver (NS) DC2.example.samdom.com.



Regards
Tim

[Richard Connon]

Hi Rowland,

Please see comments inline.

On 10/03/15 08:51, Rowland Penny wrote:
> Your DC's must point to themselves for DNS and your domain clients must
> point to the DC's, anything outside the domain the DC's will be obtain
> from the forwarders set on them.

This is contrary to what the wiki says.
https://wiki.samba.org/index.php/Setup_a_Samba_AD_Member_Server
This page indicates that as long as the client can resolve names in the
domain DNS zone (in my case ads.connon.me.uk) they should be fine.

> What I think is happening: your client is asking for the DC from your
> forwarders, they do not know, so they ask the DC, who asks the
> forwarder, who does not know and so on.

I can confirm this isn't happening since I can resolve (for example) the
SRV records on _ldap._tcp.ads.connon.me.uk through my forwarders, you
can even test this yourself with `dig -t SRV
_ldap._tcp.ads.connon.me.uk` or similar.

I'm currently looking into whether there are any records missing.

Regards,
Richard

[Rowland Penny]

On 10/03/15 14:11, Richard Connon wrote:
> Hi Rowland,
>
> Please see comments inline.
>
> On 10/03/15 08:51, Rowland Penny wrote:
>> Your DC's must point to themselves for DNS and your domain clients must
>> point to the DC's, anything outside the domain the DC's will be obtain
>> from the forwarders set on them.
>
> This is contrary to what the wiki says.
> https://wiki.samba.org/index.php/Setup_a_Samba_AD_Member_Server
> This page indicates that as long as the client can resolve names in
> the domain DNS zone (in my case ads.connon.me.uk) they should be fine.
>

I think that you are referring to this line:

Your DNS server(s) must be able to resolve the AD DNS zone, because
services, such as Kerberos, use it to locate other services in your
network.

Above that line in the wiki is this:


Configure your Member Servers /etc/resolv.conf to use the DNS server(s)
and search domain of your AD:

nameserver 192.168.1.1
search samdom.example.com

And if look further up 192.168.1.1 is the ip of a DC DNS server.

>> What I think is happening: your client is asking for the DC from your
>> forwarders, they do not know, so they ask the DC, who asks the
>> forwarder, who does not know and so on.
>
> I can confirm this isn't happening since I can resolve (for example)
> the SRV records on _ldap._tcp.ads.connon.me.uk through my forwarders,
> you can even test this yourself with `dig -t SRV
> _ldap._tcp.ads.connon.me.uk` or similar.
>

AGGHHHH, your Domain DCs are resolvable on the internet, *they shouldn't be*

rowland@ThinkPad ~ $ dig -t SRV _ldap._tcp.ads.connon.me.uk

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> -t SRV _ldap._tcp.ads.connon.me.uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42601
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 13, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_ldap._tcp.ads.connon.me.uk. IN SRV

;; ANSWER SECTION:
_ldap._tcp.ads.connon.me.uk. 899 IN SRV 0 100 389
dc02.ads.connon.me.uk.
_ldap._tcp.ads.connon.me.uk. 899 IN SRV 0 100 389
dc01.ads.connon.me.uk.

> I'm currently looking into whether there are any records missing.
>
> Regards,
> Richard
>

Probably not, it just seems to be set up incorrectly.

Your AD domain should be a sub domain of your registered domain (if you
have one) and should not be resolvable from the internet.

Rowland

[Richard Connon]

Hello again,

Rowland, thanks for the pointers regarding AD DNS best practice. I'll
look into blocking my ads.connon.me.uk zone from external networks.

The root cause of my issue, however, turned out to be something
unrelated. I concluded that the problem occurs when the join process
needs to connect to the IPC$ share on the DC. For some reason the shares
on the DC were not working due to a missing module:
/usr/lib/x86_64-linux-gnu/samba/vfs/acl_xattr.so

Installing the debian package samba-vfs-modules this has resolved the
issues with my join!

Regards,
Richard