WinRM Connection Failed

[Jeremy Vermillion]

I created a Rundeck instance with Azure Container Instance. I am using rundeck:5.7.0 with within a simple docker file that packages python3 and azure cli.

I have Rundeck running, Azure node plugin working and I am able to run jobs on localhost. I can't get winrm jobs to work. The node/VM I am trying to target is already working with  Rundeck installed on VM so it is configured correctly. 

I am able to successfully ping the node by IP and telnet the IP:PORT

I verified WinRM settings on the host I'm trying to run per Rundeck Documentation.

This is the job output of a simple Write-Output script


Rundeck logs

[2024-10-31T13:41:39,894] ERROR node.NodeStepPluginAdapter - Error executing node step. com.dtolabs.rundeck.core.execution.workflow.steps.node.NodeStepException: [WinRMcpPython]: external script failed with exit code: 1 at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.DefaultScriptFileNodeStepUtils.executeScriptFile(DefaultScriptFileNodeStepUtils.java:187) ~[rundeck-core-5.7.0-20241021.jar!/:?] at org.rundeck.plugin.scriptnodestep.ScriptFileNodeStepExecutor.executeScriptFile(ScriptFileNodeStepExecutor.groovy:89) ~[?:?] at org.rundeck.plugin.scriptnodestep.ScriptNodeStepPlugin.executeNodeStep(ScriptNodeStepPlugin.groovy:108) ~[?:?] at com.dtolabs.rundeck.core.execution.workflow.steps.node.NodeStepPluginAdapter.executeNodeStep(NodeStepPluginAdapter.java:180) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:207) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:130) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:61) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodesWith(ExecutionServiceImpl.java:263) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:234) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.workflow.steps.NodeDispatchStepExecutor.executeWorkflowStep(NodeDispatchStepExecutor.java:66) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:111) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWFItem(BaseWorkflowExecutor.java:285) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWorkflowStep(BaseWorkflowExecutor.java:681) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.workflow.engine.StepCallable.apply(StepCallable.java:71) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:76) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:32) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.rules.WorkflowEngineOperationsProcessor.lambda$beginOperation$1(WorkflowEngineOperationsProcessor.java:323) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[guava-32.0.1-jre.jar!/:?] at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75) ~[guava-32.0.1-jre.jar!/:?] at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[guava-32.0.1-jre.jar!/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: com.dtolabs.rundeck.core.execution.service.FileCopierException: [WinRMcpPython]: external script failed with exit code: 1 at com.dtolabs.rundeck.core.execution.service.ScriptPluginFileCopier.copyFile(ScriptPluginFileCopier.java:214) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.service.ScriptPluginFileCopier.copyFile(ScriptPluginFileCopier.java:96) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.fileCopyFile(ExecutionServiceImpl.java:320) ~[rundeck-core-5.7.0-20241021.jar!/:?] at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.DefaultScriptFileNodeStepUtils.executeScriptFile(DefaultScriptFileNodeStepUtils.java:172) ~[rundeck-core-5.7.0-20241021.jar!/:?] ... 22 more [2024-10-31T13:41:39,915] ERROR api.workflowstatus - Step 1 failed: script-inline [2024-10-31T13:41:40,217] ERROR services.ExecutionUtilService - Execution failed: 33 in project Test_Project: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [proc-00: NonZeroResultCode: [WinRMcpPython]: external script failed with exit code: 1 + {dataContext=MultiDataContextImpl(map={}, base=null)} ]}, Node failures: {proc-00=[NonZeroResultCode: [WinRMcpPython]: external script failed with exit code: 1 + {dataContext=MultiDataContextImpl(map={}, base=null)} ]}, status: failed] [2024-10-31T13:41:40,232] INFO execution.status admin finish [33:failed] Test_Project admin/- "-/Keep Rollin -"[cd6b467b-f128-4617-b0bb-2f28d31e755e] [2024-10-31T13:41:41,362] INFO web.requests "GET /project/Test_Project/execution/ajaxExecState/33" 10.102.5.8 http admin form 12 ? [application/json;charset=utf-8] (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36)

[rac...@rundeck.com]

Hi,

Did you configure the Default Node Executor / File Copier as WinRM Node Executor Python / WinRM Python File Copier? If so, did you set/test the right Python Interpreter in the "Python Interpreter" field? Can you share that config?

Greetings.

[Jeremy Vermillion]

#Mon Nov 04 13:52:21 GMT 2024
#edit below
project.description=
project.disable.executions=false
project.disable.schedule=false
project.execution.history.cleanup.batch=500
project.execution.history.cleanup.enabled=true
project.execution.history.cleanup.retention.days=60
project.execution.history.cleanup.retention.minimum=50
project.execution.history.cleanup.schedule=0 0 0 1/1 * ? *
project.jobs.gui.groupExpandLevel=1
project.keep-alive-max-alive-count=5
project.label=
project.later.executions.disable=false
project.later.executions.enable=false
project.later.schedule.disable=false
project.later.schedule.enable=false
project.name=Test_Project
project.nodeCache.enabled=true
project.nodeCache.firstLoadSynch=true
project.output.allowUnsanitized=false
project.plugin.FileCopier.WinRMcpPython.authtype=ntlm
project.plugin.FileCopier.WinRMcpPython.interpreter=python3
project.plugin.FileCopier.WinRMcpPython.kinit=kinit
project.plugin.FileCopier.WinRMcpPython.krb5config=/etc/krb5.conf
project.plugin.FileCopier.WinRMcpPython.nossl=false
project.plugin.FileCopier.WinRMcpPython.password_storage_path=keys/passwords/adminuser
project.plugin.FileCopier.WinRMcpPython.retryconnection=1
project.plugin.FileCopier.WinRMcpPython.retryconnectiondelay=10
project.plugin.FileCopier.WinRMcpPython.username=adminuser
project.plugin.FileCopier.WinRMcpPython.winrmport=5985
project.plugin.FileCopier.WinRMcpPython.winrmtransport=http
project.plugin.NodeExecutor.WinRMPython.authtype=ntlm
project.plugin.NodeExecutor.WinRMPython.exitbehaviour=console
project.plugin.NodeExecutor.WinRMPython.interpreter=python3
project.plugin.NodeExecutor.WinRMPython.kinit=kinit
project.plugin.NodeExecutor.WinRMPython.krb5config=/etc/krb5.conf
project.plugin.NodeExecutor.WinRMPython.nossl=false
project.plugin.NodeExecutor.WinRMPython.password_storage_path=keys/passwords/adminuser
project.plugin.NodeExecutor.WinRMPython.retryconnection=1
project.plugin.NodeExecutor.WinRMPython.retryconnectiondelay=10
project.plugin.NodeExecutor.WinRMPython.shell=powershell
project.plugin.NodeExecutor.WinRMPython.username=adminuser
project.plugin.NodeExecutor.WinRMPython.winrmport=5985
project.plugin.NodeExecutor.WinRMPython.winrmtransport=http
project.retry-counter=3
project.ssh-authentication=privateKey
resources.source.1.type=local
resources.source.2.config.client=<removed>
resources.source.2.config.extraMapping=tags.selector\=azure_status;hostname.selector\=name
resources.source.2.config.key=*****
resources.source.2.config.resourceGroup=DEV
resources.source.2.config.subscriptionId=<removed>
resources.source.2.config.tenant=<removed>
resources.source.2.type=azure-resource-model
service.FileCopier.default.provider=WinRMcpPython
service.NodeExecutor.default.provider=WinRMPython

[rac...@rundeck.com]

Thanks,

Can you test a simple command step? like: powershell Write-Host "hello world" Do you see something wrong in the log output/service.log?

Also, try this (check this too).

Regards.

​

[Jeremy Vermillion]

Ran the following with the same error

Logs from the job run

[2024-11-04T15:37:58,025] INFO  web.requests "GET /project/Test_Project/execution/ajaxExecState/43" 10.102.5.9 http admin form 11 ? [application/json;charset=utf-8] (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36)
[2024-11-04T15:37:58,120] ERROR node.NodeStepPluginAdapter - Error executing node step.

[2024-11-04T15:37:58,135] ERROR api.workflowstatus - Step 1 failed: script-inline
[2024-11-04T15:37:58,423] ERROR services.ExecutionUtilService - Execution failed: 43 in project Test_Project: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [proc-00: NonZeroResultCode: [WinRMcpPython]: external script failed with exit code: 1 + {dataContext=MultiDataContextImpl(map={}, base=null)} ]}, Node failures: {proc-00=[NonZeroResultCode: [WinRMcpPython]: external script failed with exit code: 1 + {dataContext=MultiDataContextImpl(map={}, base=null)} ]}, status: failed]
[2024-11-04T15:37:58,439] INFO  execution.status admin finish [43:failed] Test_Project admin/- "-/Keep Rollin  -"[cd6b467b-f128-4617-b0bb-2f28d31e755e]
[2024-11-04T15:37:59,712] INFO  web.requests "GET /project/Test_Project/execution/ajaxExecState/43" 10.102.5.9 http admin form 14 ? [application/json;charset=utf-8] (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36)

[Jeremy Vermillion]

I'm able successfully use pywinrm in the shell as well.

rundeck@SandboxHost-<guid>:~$ python3
Python 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import winrm
>>> ps_script = """$strComputer = $Host
... Clear
... $RAM = WmiObject Win32_ComputerSystem
... $MB = 1048576
...
... "Installed Memory: " + [int]($RAM.TotalPhysicalMemory /$MB) + " MB" """
>>> winrmsession = winrm.Session('10.0.3.0',auth=('adminuser','<CREDS>'),transport='ntlm')
>>> r = winrmsession.run_ps(ps_script)
>>> r.status_code
0
>>> r.std_out
b'Installed Memory: 4096 MB\r\n'

[rac...@rundeck.com]

Hello,

What about the command step? (not using a script-step, just dispatch a command like: powershell Write-Host "hello world"). Just to discard a global plugin issue (I think is related to the file copier config).

Greetings!

​

[Jeremy Vermillion]

No Luck :/

[ERROR ] Execution finished with the following error (winrm-exec.py:378)[root]

[ERROR ] Connection failed after 1 retries (winrm-exec.py:379)[root]

Failed: NonZeroResultCode: [WinRMPython] Result code: 1

Execution failed: 45 in project Test_Project: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [web-00: NonZeroResultCode: [WinRMPython] Result code: 1 + {dataContext=MultiDataContextImpl(map={ContextView(step:1, node:web-00)=BaseDataContext{{exec={exitCode=1}}}, ContextView(node:web-00)=BaseDataContext{{exec={exitCode=1}}}}, base=null)} ]}, Node failures: {web-00=[NonZeroResultCode: [WinRMPython] Result code: 1 + {dataContext=MultiDataContextImpl(map={ContextView(step:1, node:web-00)=BaseDataContext{{exec={exitCode=1}}}, ContextView(node:web-00)=BaseDataContext{{exec={exitCode=1}}}}, base=null)} ]}, status: failed]

[rac...@rundeck.com]

Hi,

Weird. I'm still curious. Could you test against a simple node defined in an XML/YAML file like this example? (instead of the Azure model source).

Thanks.

[Jeremy Vermillion]

That worked! That clued me in that I need to use the Azure Resource Model plugin mapping params and set this.
hostname.selector=netInterface:privateIp
Thanks for your help walking me through this!

[rac...@rundeck.com]

Happy to help! :)