Python WinRM Plugin - 500 Errors for Some Users

[Russ Robinson]

I've run into an odd issue with the python winrm plugin.  Most users are able to run jobs which connect just fine; but a few users receive a 500 error within the job.  I've looked on the target servers and they are in the same groups, permissions, etc.  Has anyone else ran into this oddity?

[DEBUG ] handle_other(): Handling: 500 (kerberos_.py:315)[winrm.vendor.requests_kerberos.kerberos_]

[ERROR ] handle_other(): Mutual authentication unavailable on 500 response (kerberos_.py:339)[winrm.vendor.requests_kerberos.kerberos_]

[DEBUG ] handle_response(): returning <Response [500]> (kerberos_.py:428)[winrm.vendor.requests_kerberos.kerberos_]

[ERROR ] Execution finished with the following exit code: -1 (winrm-exec.py:316)[root]

[ERROR ] -1 (winrm-exec.py:317)[root]

[ERROR ] The parameter is incorrect. (extended fault data: {'transport_message': 'Bad HTTP response returned from server. Code 500', 'http_status_code': 500, 'wsmanfault_code': '87', 'fault_code': 's:Sender', 'fault_subcode': 'w:InvalidParameter'}) (winrm-exec.py:318)[root]

[WinRMPython]: result code: 255, success: false

Failed: NonZeroResultCode: [WinRMPython] Result code: 255

[rac...@rundeck.com]

Hi Russ,

Could you try increasing the host timeout attribute? (winrm-readtimeout in your windows node definition, check the “connect/read times out” section here) It seems a weird issue in certain environments.

Regards!

​

[Russ Robinson]

I do have the following set at the project level:

project.plugin.NodeExecutor.WinRMPython.authtype=kerberos
project.plugin.NodeExecutor.WinRMPython.exitbehaviour=exitcode
project.plugin.NodeExecutor.WinRMPython.interpreter=/usr/bin/python3
project.plugin.NodeExecutor.WinRMPython.nossl=true
project.plugin.NodeExecutor.WinRMPython.operationtimeout=14400
project.plugin.NodeExecutor.WinRMPython.readtimeout=15400
project.plugin.NodeExecutor.WinRMPython.shell=cmd
project.plugin.NodeExecutor.WinRMPython.winrmport=5986
project.plugin.NodeExecutor.WinRMPython.winrmtransport=https

None of the winrm parameters are set at the node level.

[rac...@rundeck.com]

Hi Russ,

Testing the Windows node with those users you obtain the same error?

You can test it with the following python code:

import winrm 

s = winrm.Session('yourdomain ', 'transport='kerberos', auth=('user', 'password'), server_cert_validation='ignore')
r = s.run_cmd('ipconfig', ['/all'])

Greetings.

​

[Russ Robinson]

Python script works for the person (even running it from the same Linux server as Rundeck).  However, the job (targeting same windows server) fails with same error above.  Any further suggetsions?

[Konrad Tjaden]

Hi Russ,

Check that whatever code/payload you are passing to the windows node for execution does not contain a '%' character.
I have a job generating a random password including characters '{}[]!@#$%^&*' and it would appear every time I pass a '%' the python node executor fails with something like that.
Removing % from the script parameter i'm passing solved this for me.

Let me know if it's the same thing for you?

Cheers

Konrad

[Russ Robinson]

Sadly, the test job is just running "hostname" command on the target windows server.  Below is the output from debug mode:

[workflow] beginExecuteNodeStep(mytest.server.com_AdminServer): NodeDispatch: CommandItem{label='Issue Hostname Command', command=[1 words]}

[WinRMPython] execCommand started, command: hostname

[WinRMPython] executing: [[${config.interpreter}, -u, /var/lib/rundeck/libext/cache/py-winrm-plugin-2.0.10/winrm-exec.py, ${node.hostname}]]

[DEBUG ] Using option.username: akbartho (winrm-exec.py:165)[root]

[DEBUG ] Using option.winrmpassword (winrm-exec.py:180)[root]

[DEBUG ] ------------------------------------------ (winrm-exec.py:200)[root]

[DEBUG ] endpoint:https://mytest.server.com:5986 (winrm-exec.py:201)[root]

[DEBUG ] authentication:kerberos (winrm-exec.py:202)[root]

[DEBUG ] username:akbartho (winrm-exec.py:203)[root]

[DEBUG ] nossl:True (winrm-exec.py:204)[root]

[DEBUG ] diabletls12:False (winrm-exec.py:205)[root]

[DEBUG ] krb5config:/etc/krb5.conf (winrm-exec.py:206)[root]

[DEBUG ] kinit command:kinit (winrm-exec.py:207)[root]

[DEBUG ] kerberos delegation:False (winrm-exec.py:208)[root]

[DEBUG ] shell:cmd (winrm-exec.py:209)[root]

[DEBUG ] readtimeout:15400 (winrm-exec.py:210)[root]

[DEBUG ] operationtimeout:14400 (winrm-exec.py:211)[root]

[DEBUG ] exit Behaviour:exitcode (winrm-exec.py:212)[root]

[DEBUG ] ------------------------------------------ (winrm-exec.py:213)[root]

[DEBUG ] running kinit ['kinit'] (kerberosauth.py:28)[root]

[DEBUG ] Exist status: 0 (kerberosauth.py:47)[root]

[DEBUG ] kinit succeeded for akbartho (kerberosauth.py:56)[root]

[DEBUG ] HTTPKerberosAuth: Preemptive Authorization header: Negotiate

.....

.....

.....

(kerberos_.py:455)[winrm.vendor.requests_kerberos.kerberos_]

[DEBUG ] Starting new HTTPS connection (1): mytest.server.com:5986 (connectionpool.py:975)[urllib3.connectionpool]

[DEBUG ] handle_other(): Handling: 500 (kerberos_.py:315)[winrm.vendor.requests_kerberos.kerberos_]

[ERROR ] handle_other(): Mutual authentication unavailable on 500 response (kerberos_.py:339)[winrm.vendor.requests_kerberos.kerberos_]

[DEBUG ] handle_response(): returning <Response [500]> (kerberos_.py:428)[winrm.vendor.requests_kerberos.kerberos_]

[ERROR ] Execution finished with the following exit code: -1 (winrm-exec.py:316)[root]

[ERROR ] -1 (winrm-exec.py:317)[root]

[ERROR ] The parameter is incorrect. (extended fault data: {'transport_message': 'Bad HTTP response returned from server. Code 500', 'http_status_code': 500, 'wsmanfault_code': '87', 'fault_code': 's:Sender', 'fault_subcode': 'w:InvalidParameter'}) (winrm-exec.py:318)[root]

[WinRMPython]: result code: 255, success: false

Failed: NonZeroResultCode: [WinRMPython] Result code: 255

[workflow] finishExecuteNodeStep(mytest.server.com_AdminServer): NodeDispatch: NonZeroResultCode: [WinRMPython] Result code: 255

[rac...@rundeck.com]

Hi Russ,

Just to discard, are you using Python3 in your Rundeck Server? Also, check that the python3 interpreter path is well referenced at Project Settings > Edit Configuration > Default Node Executor > Python Interpreter (WinRM Node Executor Python). Check this and this.

Regards.

[Russ Robinson]

Project settings:

project.plugin.NodeExecutor.WinRMPython.interpreter=/usr/bin/python3

And on the Rundeck server itself:

-bash-4.2$ whoami
rundeck
-bash-4.2$ /usr/bin/python3 --version
Python 3.6.8

[Russ Robinson]

Thanks for the pointer Konrad.  The test script posted below worked for the user just fine.  However, the Rundeck jobs still did not work.  The problem ended up being the user had a % character in their password.  Once the user changed their password to one without the % character, all worked fine.  Since the test script worked with the % password, I believe this is a bug in the Rundeck python plugin not handling passwords with % correctly.

Thanks,

Russ

On Mar 18, 2021, at 12:45 PM, Russ Robinson <eagl...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/7944d511-0f51-4837-9402-c09d90e9c340n%40googlegroups.com.

[rac...@rundeck.com]

Hi Russ,

Please file this issue here.

Thanks!

[Russ Robinson]

Bug opened here.