Problem with Rundeck v5.3 (WinrmPython - CredSSP)

[herve naga]

Hello,

I have a problem with Rundeck in using winrmpython for execute a command on remote windows 2012 R2 (with auth CredSSP)

I have to do some tests

On my old server Rundeck 4.7
py-winrm-plugin (2.1.2)


On my new serveur Rundeck 5.3
py-winrm-plugin (2.1.3)

On the both servers, node definition is the same (format xml)

<node name="xxxxxxx" description="Serveur Etudes Paris" tags="Prod"
    hostname="xxxxx.domain.xx" username="xxx...@xxxxx.domain.xx"
    osFamily="Windows" osName="Microsoft Windows Server 2012 R2" osArch="amd64"
    node-executor="WinRMPython" winrm-auth-type="CredSSP" winrm-protocol="http" winrm-spn-use-http="true"
    winrm-connection-timeout="28800000" winrm-timeout="PT28800.000S"
    winrm-cmd="CMD" winrm-kerberos-debug="false" winrm-domain="domain"
    winrm-password-storage-path="keys/Myaccount">
</node>


My test is a just the dir command (adhoc)

- With release v4.7 it's working.

- With release v5.3 i have the following error:

   [ERROR  ]  Execution finished with the following error (winrm-exec.py:378)[root]
   [ERROR  ]  Server did not response with a CredSSP token after step TLS Handshake - actual 'Negotiate, Kerberos, Basic realm="WSMAN", CredSSP' (winrm-exec.py:379)[root]

For information if i use a basic authenification the command working but i would like to use CredSSP (for double Hop)


The parameters on the project (Rundeck v5.3)

project.plugin.FileCopier.WinRMcpPython.authtype=basic
project.plugin.FileCopier.WinRMcpPython.interpreter=python3
project.plugin.FileCopier.WinRMcpPython.kinit=kinit
project.plugin.FileCopier.WinRMcpPython.krb5config=/etc/krb5.conf
project.plugin.FileCopier.WinRMcpPython.nossl=false
project.plugin.FileCopier.WinRMcpPython.retryconnection=1
project.plugin.FileCopier.WinRMcpPython.retryconnectiondelay=10
project.plugin.FileCopier.WinRMcpPython.winrmport=5985
project.plugin.FileCopier.WinRMcpPython.winrmtransport=http
project.plugin.NodeExecutor.WinRMPython.authtype=credssp
project.plugin.NodeExecutor.WinRMPython.exitbehaviour=console
project.plugin.NodeExecutor.WinRMPython.interpreter=python3
project.plugin.NodeExecutor.WinRMPython.kinit=kinit
project.plugin.NodeExecutor.WinRMPython.krb5config=/etc/krb5.conf
project.plugin.NodeExecutor.WinRMPython.krbdelegation=true
project.plugin.NodeExecutor.WinRMPython.nossl=false
project.plugin.NodeExecutor.WinRMPython.retryconnection=1
project.plugin.NodeExecutor.WinRMPython.retryconnectiondelay=10
project.plugin.NodeExecutor.WinRMPython.shell=powershell
project.plugin.NodeExecutor.WinRMPython.winrmport=5985
project.plugin.NodeExecutor.WinRMPython.winrmtransport=http
project.retry-counter=3
project.ssh-authentication=privateKey
project.use-sftp=true
resources.source.1.config.file=/var/lib/rundeck/projects/P_EDITION/resources.xml
resources.source.1.config.format=resourcexml
resources.source.1.config.generateFileAutomatically=true
resources.source.1.config.writeable=true
resources.source.1.type=file
service.FileCopier.default.provider=WinRMcpPython
service.NodeExecutor.default.provider=WinRMPython


The parameters on the project Rundeck v4.7

project.plugin.FileCopier.WinRMcpPython.authtype=ntlm
project.plugin.FileCopier.WinRMcpPython.interpreter=python3
project.plugin.FileCopier.WinRMcpPython.kinit=kinit
project.plugin.FileCopier.WinRMcpPython.krb5config=/etc/krb5.conf
project.plugin.FileCopier.WinRMcpPython.nossl=false
project.plugin.FileCopier.WinRMcpPython.winrmport=5985
project.plugin.FileCopier.WinRMcpPython.winrmtransport=http
project.plugin.NodeExecutor.WinRMPython.authtype=credssp
project.plugin.NodeExecutor.WinRMPython.exitbehaviour=console
project.plugin.NodeExecutor.WinRMPython.interpreter=python3
project.plugin.NodeExecutor.WinRMPython.kinit=kinit
project.plugin.NodeExecutor.WinRMPython.krb5config=/etc/krb5.conf
project.plugin.NodeExecutor.WinRMPython.nossl=false
project.plugin.NodeExecutor.WinRMPython.shell=powershell
project.plugin.NodeExecutor.WinRMPython.winrmport=5985
project.plugin.NodeExecutor.WinRMPython.winrmtransport=http
project.ssh-authentication=privateKey
project.winrm-auth-type=basic
project.winrm-cmd=CMD
project.winrm-connection-timeout=28800000
project.winrm-protocol=http
project.winrm-spn-use-http=true
project.winrm-timeout=PT28800.000S
project.winrm-user-option=rundeck
resources.source.1.config.file=/var/lib/rundeck/projects/PROD_INFRA_WINDOWS/etc/resources.xml
resources.source.1.config.generateFileAutomatically=true
resources.source.1.config.writeable=true
resources.source.1.type=file
resources.source.2.config.count=1
resources.source.2.config.delay=0
resources.source.2.config.prefix=node
resources.source.2.config.suffix=test
resources.source.2.config.tags=stub
resources.source.2.type=stub
service.FileCopier.default.provider=WinRMcpPython
service.NodeExecutor.default.provider=WinRMPython


Do you want more any informations ?

Thanks in advance for your help

[rac...@rundeck.com]

Hi,

Could you please try removing the 'project.plugin.NodeExecutor.WinRMPython.krbdelegation=true' node executor settings in 5.3 instance? is found in the 4.7 instance.

Also, are both Rundeck instances delivering jobs to the same Windows server?

Regards.

[herve naga]

Hello

Thanks for you answer.

I removed the krbdelegetation option  but the problem persist with the same error message

For the job it's the dir command executed (for simple test)

Thanks

[rac...@rundeck.com]

I see,

It appears that your configuration is fine, but there are some important elements there:

Regarding this message:

The server did not respond with a CredSSP token following the stage TLS Handshake.

and this config:

project.plugin.NodeExecutor.WinRMPython.nossl=false

Do you use SSL on the Windows server? Do you have the SSL certificate in the Rundeck 5.3 Java Cacert? Please double-check that the Rundeck server trusts the remote SSL service. This answer should help you.

Another configuration (related to the file copier) is the following line in Rundeck 5.3 Config.

project.plugin.FileCopier.WinRMcpPython.authtype=basic

In the working environment, you are using the NTLM authentication:

project.plugin.FileCopier.WinRMcpPython.authtype=ntlm

Also, try to set this in your Windows remote node (applied here).

Regards!

​

[herve naga]

Hello,

Thanks for your reply

I don't use SSL certifcat

I passed the plugin.NodeExecutor.WinRMPython.nossl parameter to true but the problem persist

and put the value ntlm for project.plugin.FileCopier.WinRMcpPython.authtype parameter

I executed the command winrm set winrm/config/service '@{CertificateThumbprint=""}'  but it wasn't defined on the remote Windows server

I didn't test the create thumbprint  on the remot Windows server (I don't know create it, i should be search )

Thanks yet

[herve naga]

Hello,

I'am doing some tests with script powershell

When i'am connecting on my PC windows and on another PC, my powershell script working

$username = "domain\myuser"

$password = "mypassword"
$passworddomain = ConvertTo-SecureString -String $password -AsPlainText -Force

$Cred = New-Object -Typename System.Management.Automation.PSCredential -Argumentlist $username, $passworddomain

Invoke-Command -ComputerName "myhostname" -credential $Cred -Authentication Credssp -ErrorAction Stop -ScriptBlock {Invoke-Expression -Command:"cmd.exe /c 'dir /B'"}

From Rundeck I have an error : reply: 'HTTP/1.1 500

$username = "domain\myuser"

$passworddomain = ConvertTo-SecureString -String @option.PWD@ -AsPlainText -Force  --> [PWD = it is an option who use the password key]

$Cred = New-Object -Typename System.Management.Automation.PSCredential -Argumentlist $username, $passworddomain

Invoke-Command -ComputerName "@node.hostname@" -credential $Cred -Authentication Credssp -ErrorAction Stop -ScriptBlock {Invoke-Expression -Command:"cmd.exe /c 'dir /B'"}

Did i forget anything on Rundeck?

Thanks

[herve naga]

Tests with script python

# On my Old Rundeck server (python3 -V Python 3.9.18)

 /usr/bin/python3 /var/lib/rundeck/libext/cache/py-winrm-plugin-2.1.2/winrm-check.py --hostname MyServer --username 'domain\user' --password xxxx' --authentication credssp --transport http --port 5985 --nossl true

b'\r\nConfiguration IP de Windows\r\n\r\n\r\nCarte Ethernet Ethernet1 :\r\n\r\n   Suffixe DNS propre \x85 la connexion. . . : \r\n   Adresse IPv6 de liaison locale. . . . .: fe80::9188:d63e:7f12:2259%13\r\n   Adresse IPv4. . . . . . . . . . . . . .: x.x.x.x\r\n   Masque de sous-r\x82seau. . . .\xff. . . . . : x.x.x.0\r\n   Passerelle par d\x82faut. . . .\xff. . . . . : 172.21.2.1\r\n\r\nCarte Tunnel isatap.{C9CBFA7F-4933-44ED-AAD0-EDAE1099B2EC} :\r\n\r\n   Statut du m\x82dia. . . . . . . . . . . . : M\x82dia d\x82connect\x82\r\n   Suffixe DNS propre \x85 la connexion. . . : \r\n'
Connection with host MyServer successfull


# On my New Rundeck (v5.3) server -->  (python3 -V Python 3.10.12)

/usr/bin/python3 /var/lib/rundeck/libext/cache/py-winrm-plugin-2.1.3/winrm-check.py --hostname MyServer --username 'domain\user' --password 'xxxx' --authentication credssp --transport http --port 5985 --nossl true

Traceback (most recent call last):
  File "/var/lib/rundeck/libext/cache/py-winrm-plugin-2.1.3/winrm-check.py", line 256, in <module>
    result = session.run_cmd(exec_command)
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/winrm/__init__.py", line 40, in run_cmd
    shell_id = self.protocol.open_shell()
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/winrm/protocol.py", line 166, in open_shell
    res = self.send_message(xmltodict.unparse(req))
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/winrm/protocol.py", line 243, in send_message
    resp = self.transport.send_message(message)
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/winrm/transport.py", line 309, in send_message
    self.build_session()
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/winrm/transport.py", line 292, in build_session
    self.setup_encryption()
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/winrm/transport.py", line 298, in setup_encryption
    self._send_message_request(prepared_request, '')
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/winrm/transport.py", line 327, in _send_message_request
    response = self.session.send(prepared_request, timeout=self.read_timeout_sec)
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/requests/sessions.py", line 710, in send
    r = dispatch_hook("response", hooks, r, **kwargs)
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/requests/hooks.py", line 30, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/requests_credssp/credssp.py", line 199, in response_hook
    response = self.handle_401(response, **kwargs)
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/requests_credssp/credssp.py", line 233, in handle_401
    in_token = self._get_credssp_token(response, credssp_regex, step_name)
  File "/var/lib/rundeck/.local/lib/python3.10/site-packages/requests_credssp/credssp.py", line 264, in _get_credssp_token
    raise AuthenticationException(error_msg)
requests_credssp.exceptions.AuthenticationException: Server did not response with a CredSSP token after step TLS Handshake - actual 'Negotiate, Kerberos, Basic realm="WSMAN", CredSSP'

Thanks

[rac...@rundeck.com]

Interesting. It seems to be the root reason. Could you try Rundeck 5.3 in your Python 3.9 environment? It appears to be a valid issue to post here. Also, could you compare the `pip3` modules on each server?

Regards.

[herve naga]

Hello

I have the same problem on new Rundeck server with python v3.9 however there is a little difference on the versions

Old Server : 3.9.18

New Server 3.9.19

Below the pip list of servers

## New server
Package             Version
------------------- -------------
Babel               2.8.0
bcrypt              3.2.0
blinker             1.4
certifi             2020.6.20
cffi                1.16.0
chardet             4.0.0
cryptography        42.0.8
dbus-python         1.2.18
distro              1.7.0
distro-info         1.1+ubuntu0.2
httplib2            0.20.2
idna                3.3
importlib-metadata  4.6.4
jeepney             0.7.1
Jinja2              3.0.3
jmespath            0.10.0
lazr.uri            1.0.6
MarkupSafe          2.0.1
more-itertools      8.10.0
netifaces           0.11.0
ntlm-auth           1.4.0
packaging           21.3
pip                 22.0.2
pycparser           2.22
PyGObject           3.42.1
PyJWT               2.3.0
pykerberos          1.1.14
PyNaCl              1.5.0
pyparsing           2.4.7
pyspnego            0.11.0
python-apt          2.4.0+ubuntu3
pytz                2022.1
pywinrm             0.4.3
PyYAML              5.4.1
requests            2.25.1
requests-credssp    2.0.0
requests_ntlm       1.3.0
resolvelib          0.8.1
setuptools          59.6.0
six                 1.16.0
ufw                 0.36.1
unattended-upgrades 0.1
urllib3             1.26.5
wadllib             1.3.6
wheel               0.37.1
xmltodict           0.12.0
zipp                1.0.0


## Old server
Package            Version
------------------ ----------
certifi            2023.11.17
cffi               1.16.0
charset-normalizer 3.3.2
cryptography       41.0.7
decorator          5.1.1
distlib            0.3.7
filelock           3.13.1
gssapi             1.8.3
idna               3.6
install            1.3.5
krb5               0.5.1
pexpect            4.9.0
pip                23.3.1
pipenv             2023.11.15
platformdirs       4.0.0
ptyprocess         0.7.0
pycparser          2.21
pyspnego           0.10.2
pywinrm            0.4.3
requests           2.31.0
requests-credssp   2.0.0
requests-kerberos  0.14.0
requests-ntlm      1.2.0
setuptools         69.0.2
six                1.16.0
urllib3            1.26.15
virtualenv         20.24.7
xmltodict          0.13.0

After i have installed the same version of modules requests on my new server, the problem persist

pip3 list |grep request
requests            2.31.0
requests-credssp    2.0.0
requests-kerberos   0.14.0
requests-ntlm       1.2.0

Thanks

[herve naga]

For information i tried with py-winrm-plugin-2.1.2 (as on may old server) and the problem is same.

On new server by default it's  py-winrm-plugin-2.1.3

Thanks

[herve naga]

Another information.

I tested (winrm-check.py script) in basic, ntlm and kerberos authentication method and it's working