Problem with WinRM Connections

1,587 views
Skip to first unread message

Michael Kennedy

unread,
Jun 11, 2022, 9:55:15 PM6/11/22
to Ansible Project
I am having a problem running WinRM connections with both basic and kerberos auth.  

My Ansible is deployed with RedHat AAP 4.2.0 on RHEL 9.  

I setup a test Windows 2019 machine and ran the ConfigureRemotingForAnsible.ps1 script against the host.  Rebooted the host for good measure.  Tested from the Ansible server that I can telnet to 5985 and 5986 (confirmed) but I cannot run a Windows test playbook against the host.

Skipping callback 'oneline', as we already have a stdout callback.
18
19
PLAYBOOK: test.yml *************************************************************
20
2 plays in windows/test.yml
21
22
PLAY [test raw module] *********************************************************18:47:21
23
24
TASK [Gathering Facts] *********************************************************18:47:21
25
task path: /runner/project/windows/test.yml:2
26
[WARNING]: The "winrm" connection plugin has an improperly configured remote
27
target value, forcing "inventory_hostname" templated value instead of the
28
string
29
redirecting (type: modules) ansible.builtin.setup to ansible.windows.setup
30
Using module file /usr/share/ansible/collections/ansible_collections/ansible/windows/plugins/modules/setup.ps1
31
Pipelining is enabled.
32
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: administrator on PORT 5986 TO inventory_hostname
33
fatal: [192.168.12.52]: UNREACHABLE! => {
34
"changed": false,
35
"msg": "ssl: HTTPSConnectionPool(host='inventory_hostname', port=5986): Max retries exceeded with url: /wsman (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa35f010f10>: Failed to establish a new connection: [Errno -2] Name or service not known'))",
36
"unreachable": true
37
}

Running a Windows test against a domain joined machine produces a different error that I also cannot resolve.  

PLAY [Ping] ********************************************************************18:53:19
3
4
TASK [Gathering Facts] *********************************************************18:53:19
5
[WARNING]: The "winrm" connection plugin has an improperly configured remote
6
target value, forcing "inventory_hostname" templated value instead of the
7
string
8
fatal: [srvrds04]: UNREACHABLE! => {"changed": false, "msg": "kerberos: authGSSClientStep() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))", "unreachable": true}

I am getting this error despite me confirming the SPN is fine.  Confirming I can connect to the host with WinRM from a different domain joined host.  Confirmed my Kerb ticket with kinit and list.  


Michael Kennedy

unread,
Jun 12, 2022, 11:17:51 AM6/12/22
to Ansible Project
Here is some more troubleshooting information.  Also as a side note.  I am running an partner NFR self-support only version of AAP and have zero support from RedHat on this.  It also doesn't seem to matter if I run the job through command line ansible or through AAP.  The error is the same.  

My Group Vars

---
ansible_connection: winrm
ansible_winrm_server_cert_validation: ignore

Host Inventory

---
ansible_hostname: mikes-wintest.DOMAIN.CA

my WinRM settings after running the ConfigureAnsibleRemoting.ps1 script

PS C:\Users\ubermike\Desktop> winrm configSDDL default
Service
    RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)(A;;GAGR;;;S-1-5-21-809043649-619790271-106372718-1977)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
    MaxConcurrentOperations = 4294967295
    MaxConcurrentOperationsPerUser = 1500
    EnumerationTimeoutms = 240000
    MaxConnections = 300
    MaxPacketRetrievalTimeSeconds = 120
    AllowUnencrypted = true
    Auth
        Basic = true
        Kerberos = true
        Negotiate = true
        Certificate = false
        CredSSP = true
        CbtHardeningLevel = Relaxed
    DefaultPorts
        HTTP = 5985
        HTTPS = 5986
    IPv4Filter = *
    IPv6Filter = *
    EnableCompatibilityHttpListener = false
    EnableCompatibilityHttpsListener = false
    CertificateThumbprint
    AllowRemoteAccess = true


My Ansible Host Kerb ticket

[uber...@domain.ca@ansible ~]$ klist -c
Ticket cache: KCM:1944601976:978
Default principal: uber...@DOMAIN.CA

Valid starting       Expires              Service principal
2022-06-12 08:10:56  2022-06-12 18:10:56  krbtgt/DOMA...@DOMAIN.CA
    renew until 2022-06-19 08:10:56


Latest Error Output from the job

{
  "unreachable": true,
  "msg": "kerberos: authGSSClientStep() failed: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377)), ssl: HTTPSConnectionPool(host='inventory_hostname', port=5986): Max retries exceeded with url: /wsman (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f3ca1fd2430>: Failed to establish a new connection: [Errno -2] Name or service not known'))",
  "changed": false
}

Urs Rau

unread,
Jun 13, 2022, 6:27:20 AM6/13/22
to Ansible Project
Hi Michael,

A stab in the dark winrm or Kerberos specifically depend on a fully working DNS.
Is your tower cluster properly looking up the windows domain DCs? 
I had tried a kludge myself with hard coded names in Hosts files but that gave me same error you are getting.
Oddly the kinit and list cmd works fine, but the actual ansible connections trying to use winrm or kerberos transport failed.
And as soon as I had added my local or internal dns zone to the cluster dns things started working for me.

HTH

-- 
Urs Rau
Message has been deleted
Message has been deleted

Michael Kennedy

unread,
Jun 13, 2022, 10:42:06 AM6/13/22
to Ansible Project
Hi Urs,

Yes, dealing with Active Directory, DNS was the first place I looked.  I have eliminated it being a problem with DNS.  My resolv.conf file is setup correctly.  DNS is correct, and rDNS is correct.  I have also eliminated other low hanging fruit such as, NTP, Firewalls, Windows Firewall, Ethernet Adapter zones.  

[root@ansible ~]# ping mikes-wintest
PING mikes-wintest.domain.ca (192.168.12.52) 56(84) bytes of data.
64 bytes from Mikes-WinTest.domain.ca (192.168.12.52): icmp_seq=1 ttl=123 time=14.1 ms
^C
--- mikes-wintest.domain.ca ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 14.127/14.127/14.127/0.000 ms
[root@ansible ~]# nslookup mikes-wintest
Server:        192.168.8.11
Address:    192.168.8.11#53

Name:    mikes-wintest.domain.ca
Address: 192.168.12.52

[root@ansible ~]# nslookup domain.ca
Server:        192.168.8.11
Address:    192.168.8.11#53

Name:    domain.ca
Address: 192.168.15.202
Name:    domain.ca
Address: 192.168.12.153
Name:    domain.ca
Address: 192.168.12.20
Name:    domain.ca
Address: 192.168.8.11
Name:    domain.ca
Address: 192.168.15.201
Name:    domain.ca
Address: 192.168.8.44
Name:    domain.ca
Address: 192.168.8.21
Name:    domain.ca
Address: 192.168.12.201
Name:    domain.ca
Address: 192.168.9.150

[root@ansible ~]#

Michael Kennedy

unread,
Jun 13, 2022, 11:06:13 AM6/13/22
to Ansible Project
I setup the inventory from CLI with a yml file and have run the win_ping test against the hosts.  One thing that stands out to me is this.  

<inventory_hostname> WINRM CONNECT: transport=kerberos endpoint=https://inventory_hostname:5986/wsman

Should it not be populating the real FQDN of the machine here?  

ansible [core 2.13.0]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.10 (main, Feb  9 2022, 00:00:00) [GCC 11.2.1 20220127 (Red Hat 11.2.1-9)]
  jinja version = 3.0.3
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /etc/ansible/projects/inventory/inventory.yml as it did not pass its verify_file() method
script declined parsing /etc/ansible/projects/inventory/inventory.yml as it did not pass its verify_file() method
Parsed /etc/ansible/projects/inventory/inventory.yml inventory source with ini plugin
Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python3.9/site-packages/ansible/plugins/callback/minimal.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.

Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.
META: ran handlers
[WARNING]: The "winrm" connection plugin has an improperly configured remote target value, forcing "inventory_hostname" templated value instead of the string
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Loading collection ansible.windows from /root/.ansible/collections/ansible_collections/ansible/windows
Using module file /root/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: uber...@DOMAIN.CA on PORT 5986 TO inventory_hostname
creating Kerberos CC at /tmp/tmpnx950wor
calling kinit with pexpect for principal uber...@DOMAIN.CA
[WARNING]: The "winrm" connection plugin has an improperly configured remote target value, forcing "inventory_hostname" templated value instead of the string
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Loading collection ansible.windows from /root/.ansible/collections/ansible_collections/ansible/windows
Using module file /root/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: uber...@DOMAIN.CA on PORT 5986 TO inventory_hostname
creating Kerberos CC at /tmp/tmpoh0zue5y
calling kinit with pexpect for principal uber...@DOMAIN.CA
[WARNING]: The "winrm" connection plugin has an improperly configured remote target value, forcing "inventory_hostname" templated value instead of the string
kinit succeeded for principal uber...@DOMAIN.CA
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Loading collection ansible.windows from /root/.ansible/collections/ansible_collections/ansible/windows
Using module file /root/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: uber...@DOMAIN.CA on PORT 5986 TO inventory_hostname
creating Kerberos CC at /tmp/tmp1d3m70sc
calling kinit with pexpect for principal uber...@DOMAIN.CA
kinit succeeded for principal uber...@DOMAIN.CA
<inventory_hostname> WINRM CONNECT: transport=kerberos endpoint=https://inventory_hostname:5986/wsman
kinit succeeded for principal uber...@DOMAIN.CA
<inventory_hostname> WINRM CONNECT: transport=kerberos endpoint=https://inventory_hostname:5986/wsman
<inventory_hostname> WINRM CONNECTION ERROR: authGSSClientStep() failed: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/winrm/vendor/requests_kerberos/kerberos_.py", line 245, in generate_request_header
    result = kerberos.authGSSClientStep(self.context[host],
kerberos.GSSError: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ansible/plugins/connection/winrm.py", line 448, in _winrm_connect
    self.shell_id = protocol.open_shell(codepage=65001)  # UTF-8
  File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 166, in open_shell
    res = self.send_message(xmltodict.unparse(req))
  File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 243, in send_message
    resp = self.transport.send_message(message)
  File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 320, in send_message
    prepared_request = self.session.prepare_request(request)
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 456, in prepare_request
    p.prepare(
  File "/usr/lib/python3.9/site-packages/requests/models.py", line 320, in prepare
    self.prepare_auth(auth, url)
  File "/usr/lib/python3.9/site-packages/requests/models.py", line 551, in prepare_auth
    r = auth(self)
  File "/usr/local/lib/python3.9/site-packages/winrm/vendor/requests_kerberos/kerberos_.py", line 453, in __call__
    auth_header = self.generate_request_header(None, host, is_preemptive=True)
  File "/usr/local/lib/python3.9/site-packages/winrm/vendor/requests_kerberos/kerberos_.py", line 260, in generate_request_header
    raise KerberosExchangeError("%s failed: %s" % (kerb_stage, str(error.args)))
winrm.vendor.requests_kerberos.exceptions.KerberosExchangeError: authGSSClientStep() failed: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))
Mikes-WinTest.domain.ca | UNREACHABLE! => {

    "changed": false,
    "msg": "kerberos: authGSSClientStep() failed: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))",
    "unreachable": true
}

Urs Rau

unread,
Jun 13, 2022, 1:11:56 PM6/13/22
to ansible...@googlegroups.com
Hi Michael,

OK, I should have said so earlier but as well as kinit and klist ping also worked, but ansible still would not connect … and gave me exactly the error message you are getting, I am not sure but I suspect the python kerberos transport is making another over the wire query off of the DC or another container is involved to open the kerberos winrm session.

I have messed with the resolvers.conf myself and cheated by adding my local ones but that alone did not resolve it. I think there is another container that tower uses .

I have posted today on this group what I did with getting the local dns info injected into the awx cluster. I am using minikube but maybe my way works on yours too?

Awx or tower uses a number of containers …

HTH

Urs 

On 13. Jun 2022, at 16:40, Michael Kennedy <indiem...@gmail.com> wrote:

Hi Urs,

Yes.  Dealing with Active Directory, DNS was the first thing I have eliminated as being a problem.  My resolv.conf file is setup correctly.  

[root@ansible ~]# ping mikes-wintest
PING mikes-wintest.sudden.ca (192.168.12.52) 56(84) bytes of data.


I have also eliminated other low-hanging fruit such as NTP, Firewalls, Windows Firewall, Ethernet adapter zone.  
On Monday, June 13, 2022 at 3:27:20 AM UTC-7 urs...@gmail.com wrote:

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/b064e330-b0cb-4929-9c4d-8c53a54eafccn%40googlegroups.com.

Michael Kennedy

unread,
Jun 13, 2022, 2:30:08 PM6/13/22
to Ansible Project
I could add the machines to my hosts file directly but I think I actually have a bug here.  The winRM plugin is actually trying to connect to https://inventory_hostname:5986/wsman instead of https://mikes-wintest.domain.ca:5986/wsman.  


WARNING]: The "winrm" connection plugin has an improperly configured remote target value, forcing "inventory_hostname" templated value instead of the string
kinit succeeded for principal uber...@DOMAIN.CA
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Loading collection ansible.windows from /root/.ansible/collections/ansible_collections/ansible/windows
Using module file /root/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: uber...@DOMAIN.CA on PORT 5986 TO inventory_hostname

creating Kerberos CC at /tmp/tmp1d3m70sc
calling kinit with pexpect for principal uber...@DOMAIN.CA
kinit succeeded for principal uber...@DOMAIN.CA
<inventory_hostname> WINRM CONNECT: transport=kerberos endpoint=https://inventory_hostname:5986/wsman
kinit succeeded for principal uber...@DOMAIN.CA

jbor...@gmail.com

unread,
Jun 13, 2022, 6:04:26 PM6/13/22
to Ansible Project
Have a look at the warning Ansible is giving you, it is telling you it was unable to use your configured remote target value and forced `inventory_hostname`. Solve that and you will probably solve this problem.

Matt Martz

unread,
Jun 13, 2022, 6:14:07 PM6/13/22
to ansible...@googlegroups.com
That bug was introduced in 2.13.0 and recently resolved:  https://github.com/ansible/ansible/pull/77894

It'll be included in 2.13.1 on June 20.



--
Matt Martz
@sivel
sivel.net

Michael Kennedy

unread,
Jun 13, 2022, 6:50:50 PM6/13/22
to Ansible Project
Matt,

Thank you for that.  Yes, it seems my hunch that it was a bug was correct if its already confirmed.  Guess I will need to wait and patch Ansible before I can run it against any windows hosts.  

Reply all
Reply to author
Forward
0 new messages