Kerberos Auth - the specified credentials were rejected by the server

[mea...@iforium.com]

Hi guys,

On my Centos7 ansible control server I'm trying to connect to a windows host using a domain account. I've configured kerberos and installed everything (pip install pywinrm[kerberos]) and I have a ticket:

[meaton@MDE-001-IOM ansible-iom]# klist

Ticket cache: KEYRING:persistent:0:krb_ccache_g4NS8oC

Default principal: ans...@IOM.DOMAIN.COM

Valid starting     Expires            Service principal

29/03/17 17:16:35  30/03/17 03:16:35  krbtgt/IOM.DOM...@IOM.DOMAIN.COM

        renew until 05/04/17 17:16:32

However, trying to use these details it still says invalid credentials, am I missing something? 

Group Vars:

ansible_user =  ans...@IOM.DOMAIN.COM

ansible_password = password

ansible_port = 5986

ansible_connection = winrm

ansible_winrm_server_cert_validation = ignore

ansible_winrm_realm = IOM.DOMAIN.COM

ansible_winrm_service = HOST (I've tried with and without this, following various "solutions" - none of which have worked)

I used the powershell script to configure WinRM. I can run my playbook with no issues if I use a local account rather than a domain account. DNS is configured, realms etc all configured.

cat /etc/krb5.conf

[libdefaults]

 dns_lookup_realm = false

 ticket_lifetime = 24h

 renew_lifetime = 7d

 forwardable = true

 rdns = false

 default_realm = IOM.DOMAIN.COM

 default_ccache_name = KEYRING:persistent:%{uid}

[realms]

 IOM.DOMAIN.COM = {

  kdc = dc-01-iom.iom.domain.com

 }

[domain_realm]

 .iom.domain.com = IOM.DOMAIN.COM

# example.com = EXAMPLE.COM

Error from Playbook run:

TASK [Gathering Facts] ******************************************************************************************************************************************

Using module file /root/ansible/lib/ansible/modules/windows/setup.ps1

<appt-001-iom> ESTABLISH WINRM CONNECTION FOR USER: ans...@IOM.DOMAIN.COM on PORT 5986 TO appt-001-iom

<appt-001-iom> WINRM CONNECT: transport=ssl endpoint=https://appt-001-iom:5986/wsman

<appt-001-iom> WINRM CONNECTION ERROR: the specified credentials were rejected by the server

Traceback (most recent call last):

  File "/root/ansible/lib/ansible/plugins/connection/winrm.py", line 211, in _winrm_connect

    self.shell_id = protocol.open_shell(codepage=65001)  # UTF-8

  File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 132, in open_shell

    res = self.send_message(xmltodict.unparse(req))

  File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 207, in send_message

    return self.transport.send_message(message)

  File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 190, in send_message

    raise InvalidCredentialsError("the specified credentials were rejected by the server")

InvalidCredentialsError: the specified credentials were rejected by the server

fatal: [appt-001-iom]: UNREACHABLE! => {

    "changed": false,

    "msg": "ssl: the specified credentials were rejected by the server",

    "unreachable": true

}

Ansible Version:

ansible 2.4.0 (devel e084e8809e) last updated 2017/03/24 11:58:45 (GMT +100)

  config file =

  configured module search path = Default w/o overrides

  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

Thanks,

Michael

[Jordan Borean]

Are you able to set ansible_winrm_transport to Kerberos and see if that works out. I also believe in 2.4 there was a change made where ansible will get the Kerberos ticket for you removing the need for getting it manually beforehand. Another thing that would be good to know is the output of 'winrm get winrm/config' when running on your windows server.

[Michael Eaton]

Thanks,

That allowed me to get a bit further:


TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************************************************
Using module file /root/ansible/lib/ansible/modules/windows/setup.ps1
<appt-001-iom.IOM.DOMAIN.COM> ESTABLISH WINRM CONNECTION FOR USER: ans...@IOM.DOMAIN.COM on PORT 5986 TO appt-001-iom.IOM.DOMAIN.COM
creating Kerberos CC at /tmp/tmppm3JWz
calling kinit for principal ans...@IOM.DOMAIN.COM
kinit succeeded for principal ans...@IOM.DOMAIN.COM
<appt-001-iom.IOM.DOMAIN.COM> WINRM CONNECT: transport=kerberos endpoint=https://appt-001-iom.IOM.DOMAIN.COM:5986/wsman
<appt-001-iom.IOM.DOMAIN.COM> WINRM CONNECTION ERROR: authGSSClientStep() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))

Traceback (most recent call last):
File "/root/ansible/lib/ansible/plugins/connection/winrm.py", line 211, in _winrm_connect
self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 132, in open_shell
res = self.send_message(xmltodict.unparse(req))
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 207, in send_message
return self.transport.send_message(message)

File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 181, in send_message
prepared_request = self.session.prepare_request(request)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 407, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/lib/python2.7/site-packages/requests/models.py", line 306, in prepare
self.prepare_auth(auth, url)
File "/usr/lib/python2.7/site-packages/requests/models.py", line 543, in prepare_auth
r = auth(self)
File "/usr/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 308, in __call__
auth_header = self.generate_request_header(None, host, is_preemptive=True)
File "/usr/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 148, in generate_request_header
raise KerberosExchangeError("%s failed: %s" % (kerb_stage, str(error.args)))
KerberosExchangeError: authGSSClientStep() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))

fatal: [appt-001-iom.IOM.DOMAIN.COM]: UNREACHABLE! => {
"changed": false,
"msg": "kerberos: authGSSClientStep() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))",
"unreachable": true
}
to retry, use: --limit @/root/ansible-iom/windows.retry


As you can see the ticket request succeeds but I still get the error about the server not being found. DNS looks good - I can resolve both ways,, WinRM config...



Config
MaxEnvelopeSizekb = 500
MaxTimeoutms = 60000
MaxBatchItems = 32000
MaxProviderRequests = 4294967295
Client
NetworkDelayms = 5000
URLPrefix = wsman
AllowUnencrypted = false
Auth
Basic = true
Digest = true
Kerberos = true
Negotiate = true
Certificate = true
CredSSP = false
DefaultPorts
HTTP = 5985
HTTPS = 5986
TrustedHosts
Service
RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
MaxConcurrentOperations = 4294967295
MaxConcurrentOperationsPerUser = 1500
EnumerationTimeoutms = 240000
MaxConnections = 300
MaxPacketRetrievalTimeSeconds = 120
AllowUnencrypted = true
Auth
Basic = true
Kerberos = true
Negotiate = true
Certificate = false
CredSSP = false
CbtHardeningLevel = Relaxed
DefaultPorts
HTTP = 5985
HTTPS = 5986
IPv4Filter = *
IPv6Filter = *
EnableCompatibilityHttpListener = false
EnableCompatibilityHttpsListener = false
CertificateThumbprint
AllowRemoteAccess = true
Winrs
AllowRemoteShellAccess = true
IdleTimeout = 7200000
MaxConcurrentUsers = 2147483647
MaxShellRunTime = 2147483647
MaxProcessesPerShell = 2147483647
MaxMemoryPerShellMB = 2147483647
MaxShellsPerUser = 2147483647



Let me know if you need anything further - any ideas?

Best Regards,

Michael Eaton | DevOps Engineer



T: +44 (0) 203 4688271 | M: +44 (0) 7624 267 407
E: mea...@iforium.com
W: www.iforium.com







Twitter | Facebook | Linkedin

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/ORVozS2Nwqk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/3a2fe4e4-91ff-4080-b328-795a1b3cb53c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Davis]

Is the target host's HTTP SPN assigned to a user (instead of the computer account) in AD? Pywinrm isn't currently patching the service override through to the kerb layer (see https://github.com/diyan/pywinrm/pull/144), so if you're in that situation, you'll have to wait for the next pywinrm release that includes that bugfix.

To unsubscribe from this group and all its topics, send an email to ansible-project+unsubscribe@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.

[Michael Eaton]

Hey.

How do I check the spn? I've already applied that pull to pywinrm...

Thanks.

Michael 

-------- Original message --------

From: Matt Davis <mda...@ansible.com>

Date: 30/03/2017 18:08 (GMT+00:00)

To: Ansible Project <ansible...@googlegroups.com>

To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/144f3e78-ab94-407c-8e19-080f18ccd8ee%40googlegroups.com.

[Mike Fennemore]

Also might want to check reverse DNS entry for the host your connecting to. It needs to be resolvable from the Ansible node.

[Matt Davis]

Something like this will do it: https://social.technet.microsoft.com/wiki/contents/articles/18996.list-all-spns-used-in-your-active-directory.aspx

Oftentimes it's some random piece of software that reassigns a host's HTTP SPNs to do Kerberos in IIS with a custom user account. Normally the HTTP SPN is "implicit", so it shouldn't be assigned anywhere.

[Игорь Тиунов]

HI, try to check your search string on CentOS machine. It should be set to your domain name. And try to add short name of target server to hosts file. The 'server not found...' kerberos error is related to target server account name that passed to krb5 libraries. It should be a short name without any domain name.

[Noel Verhoeven]

I have struggled with this a lot. I have run into the fact that a new windows host, joined to a domain, sometimes needs to be rebooted again in order to have its domain name reflected correctly in its group policies. I have had GPresult /r say another domain name then what is displayed no the computer properties screen.

[J Hawkesworth]

If I recall I've had problems in the past where a machine has moved from one domain to another.  You can wind up with an orphaned computer account in active directory in the old domain (IIRC this affected older AD versions).

Fix was to remove the Computer account from the old domain.

That said, I am not sure now if this caused winrm problems or whether it affected something else that stopped us from working - thought I'd mention it anyway.

Jon