Kerberos in Ansible Tower, pulling my hair out.

[William McKenzie]

Here's my setup:

Ansible Tower 3.1.1, Basic License, using the vagrant box, with some post provisioning steps to setup krb5 and join the box to my domain.

Kerberos configuration is good. I can kinit, klist, etc. etc. from command line.

python winrm is good:

vagrant@ansible-tower ~]$ sudo su - awx

Last login: Sat Apr  1 23:12:18 JST 2017 on pts/1

Welcome to Ansible Tower!

Log into the web interface here: etc...

-bash-4.2$ cat test.py

import sys

from winrm.protocol import Protocol

HYPERV_SERVER = 'https://louis.home.cartewright.com:5986/wsman'

class RM():

    def __init__(self):

       self.win_connect = Protocol(endpoint=HYPERV_SERVER, transport='kerberos', server_cert_validation='ignore')

    def test(self):

       shell_id = self.win_connect.open_shell()

       cmd = "dir"

       command_id = self.win_connect.run_command(shell_id, cmd)

       output,error_value,exit_status = self.win_connect.get_command_output(shell_id, command_id)

       self.win_connect.cleanup_command(shell_id, command_id)

       self.win_connect.close_shell(shell_id)

       print output

def main():

   rm = RM()

   rm.test()

if __name__ == '__main__':

   main()

   sys.exit()

-bash-4.2$ python test.py

Volume in drive C is SAMSUNG 512GB SSD

Volume Serial Number is 2C8F-7BFA

Directory of C:\Users\ansible

03/31/2017  11:04 AM    <DIR>          .

03/31/2017  11:04 AM    <DIR>          ..

07/16/2016  06:47 AM    <DIR>          Desktop

03/31/2017  11:04 AM    <DIR>          Documents

07/16/2016  06:47 AM    <DIR>          Downloads

07/16/2016  06:47 AM    <DIR>          Favorites

07/16/2016  06:47 AM    <DIR>          Links

07/16/2016  06:47 AM    <DIR>          Music

07/16/2016  06:47 AM    <DIR>          Pictures

07/16/2016  06:47 AM    <DIR>          Saved Games

07/16/2016  06:47 AM    <DIR>          Videos

              0 File(s)              0 bytes

             11 Dir(s)  291,787,771,904 bytes free

-bash-4.2$

So now, I manually create some inventory in the default directories/files for ansible (not Tower). Works perfectly:

-bash-4.2$ cat /etc/ansible/group_vars/windows.yml 

ansible_connection: winrm

ansible_user: ans...@HOME.CARTEWRIGHT.COM

ansible_password: R1pflash

ansible_winrm_server_cert_validation: ignore

-bash-4.2$ 

-bash-4.2$ ansible windows -m win_ping -v

Using /etc/ansible/ansible.cfg as config file

louis.home.cartewright.com | SUCCESS => {

    "changed": false, 

    "ping": "pong"

}

Now, I create the exact same inventory in tower, exact same credentials (ans...@HOME.CARTEWRIGHT.COM) and now matter how I tweak it, always the same thing:

Using /etc/ansible/ansible.cfg as config file SSH password: Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/windows/win_ping.ps1 <louis.home.cartewright.com> ESTABLISH WINRM CONNECTION FOR USER: ans...@HOME.CARTEWRIGHT.COM on PORT 5986 TO louis.home.cartewright.com <louis.home.cartewright.com> WINRM CONNECT: transport=kerberos endpoint=https://louis.home.cartewright.com:5986/wsman <louis.home.cartewright.com> WINRM CONNECTION ERROR: authGSSClientInit() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ("Can't find client principal ans...@HOME.CARTEWRIGHT.COM in cache collection", -1765328243)) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 154, in _winrm_connect self.shell_id = protocol.open_shell(codepage=65001) # UTF-8 File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", line 132, in open_shell res = self.send_message(xmltodict.unparse(req)) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", line 207, in send_message return self.transport.send_message(message) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py", line 181, in send_message prepared_request = self.session.prepare_request(request) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/sessions.py", line 394, in prepare_request hooks=merge_hooks(request.hooks, self.hooks), File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/models.py", line 298, in prepare self.prepare_auth(auth, url) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/models.py", line 500, in prepare_auth r = auth(self) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 308, in __call__ auth_header = self.generate_request_header(None, host, is_preemptive=True) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 148, in generate_request_header raise KerberosExchangeError("%s failed: %s" % (kerb_stage, str(error.args))) KerberosExchangeError: authGSSClientInit() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ("Can't find client principal ans...@HOME.CARTEWRIGHT.COM in cache collection", -1765328243)) <louis.home.cartewright.com> WINRM CONNECT: transport=ssl endpoint=https://louis.home.cartewright.com:5986/wsman <louis.home.cartewright.com> WINRM CONNECTION ERROR: the specified credentials were rejected by the server Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 154, in _winrm_connect self.shell_id = protocol.open_shell(codepage=65001) # UTF-8 File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", line 132, in open_shell res = self.send_message(xmltodict.unparse(req)) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", line 207, in send_message return self.transport.send_message(message) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py", line 190, in send_message raise InvalidCredentialsError("the specified credentials were rejected by the server") InvalidCredentialsError: the specified credentials were rejected by the server louis.home.cartewright.com | UNREACHABLE! => { "changed": false, "msg": "kerberos: authGSSClientInit() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), (\"Can't find client principal ans...@HOME.CARTEWRIGHT.COM in cache collection\", -1765328243)), ssl: the specified credentials were rejected by the server", "unreachable": true }

Just completely unable to find the credentials in the cache, no matter how I do it. I've put the password in the tower credentials, in the inventory, using ASK, no matter what I do, it cannot find credentials.

What am I missing?

~Bill 

[Matt Davis]

Ansible doesn't manage the tickets for you until Ansible Core 2.3 (still in release candidate). Anything earlier, you'll have to do the kinit on the controller yourself (either via a cron job or as part of your playbook with a local action). 

[William McKenzie]

I think that's what I'm doing.

I've tried doing the kinit from the console, doing the kinit in a cron job, doing the kinit manually in a playbook before running the winrm play book, and doing it as a local_action in the winrm playbook itself.

In all cases (except the last one), the kinit succeeds; I can use klist to see the tickets (logged in as awx user). I can see the credential cache with the correct owner and attributes in the /tmp directory. I've also tried using the KEYRING instead of the FILE cache. For whatever reason, the winrm job is unable to see the credentials in the cache. 

When I run this playbook, it fails:

---
- name: WinPing
  hosts: all
   
  tasks:
  - name: knit
    local_action: command echo "xxxxxxx" | kinit -l 7d -r 7d -pf ansible@HOME.CARTEWRIGHT.COM
 
  - name: ping
    win_ping:

Produces this output. It seems to be running task setup before running my local action, and setup fails.

TASK [setup] *******************************************************************

07:39:32

8

Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/windows/setup.ps1

9

<louis.home.cartewright.com> ESTABLISH WINRM CONNECTION FOR USER: ans...@HOME.CARTEWRIGHT.COM on PORT 5986 TO louis.home.cartewright.com

10

<louis.home.cartewright.com> WINRM CONNECT: transport=kerberos endpoint=https://louis.home.cartewright.com:5986/wsman

11

<louis.home.cartewright.com> WINRM CONNECTION ERROR: authGSSClientInit() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ("Can't find client principal ans...@HOME.CARTEWRIGHT.COM in cache collection", -1765328243))

12

Traceback (most recent call last):

13

File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 154, in _winrm_connect

14

self.shell_id = protocol.open_shell(codepage=65001) # UTF-8

15

File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", line 132, in open_shell

16

res = self.send_message(xmltodict.unparse(req))

17

File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", line 207, in send_message

18

return self.transport.send_message(message)

19

File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py", line 181, in send_message

20

prepared_request = self.session.prepare_request(request)

21

File "/var/lib/awx/venv/ansible/lib/python2.7/site-packag…

22

[cupcake]

sanity check; is time in sync? windows AD/kerb wont auth if the skew is more than 5 or 10 minutes off. I also saw some weirdness like this recently and a reboot and then kinit again made it work but i think due to another config reason on my part.

[William McKenzie]

time is definitely good.

i run ntpdate in my vagrant provisioning script just to be sure.

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/-7e0do1u8dk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-project+unsubscribe@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/5ca67b34-4b01-4c10-bb3a-5956d24c24ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Davis]

Tower uses an isolation tech called proot that will often break shared ticket caches. If you can't wait for Ansible 2.3 (should be released within the next couple weeks), I'd suggest disabling proot (IIRC it's in settings.py, but my Tower-fu is getting rusty). 

[William McKenzie]

Bingo. I was suspecting some kind of isolation thing, because it was so clear that the environment was different when running a playbook than at the command line, even when sudo'd as awx. Many thanks. I'll never get that week back, but I love a mystery solved.

~Bill 

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/9df03222-6b17-40dc-980d-eb62311ad4e4%40googlegroups.com.

[Jarryd Took]

@william: 

The latest requests-ntlm in combination of Ansible 2.2.0 you can use NTLM against Windows machines. If this floats your boat as an alternative to kerberos tickets.

ansible_connection: winrm

ansible_winrm_transport: ntlm 

[William McKenzie]

Thanks, I have tried NTLM and basic, and they both work fine. However, I am evaluating this for a 20000+ node enterprise solution, and Kerberos is a must-have requirement.