Occasional failures in Rundeck
47 views
Skip to first unread message
Nandhu u
Feb 16, 2024, 2:03:37 PMFeb 16
to rundeck-discuss
Hi Team,
we are running process automation (Rundeck) clustered instances in Linux with Runner in Linux environment. And our target node is Windows server.
we are running process automation (Rundeck) clustered instances in Linux with Runner in Linux environment. And our target node is Windows server.
This is a cyclic job where it run every 2 mins. Occasionally we are seeing this below error. When we checked network and Windows logs we do not see any problem.
18:08:00 [admin@xxx _][VERBOSE] [workflow] Begin execution: node-first
18:08:00 [admin@xxx _][VERBOSE] preparing for sequential execution on 1 nodes
18:08:00 [admin@xxx _][VERBOSE] Executing command on node: hostname_1, NodeEntryImpl{tags=[IWF_AppServer], attributes={nodename=hostname_1, hostname=x.x.x.x, osFamily=windows, winrm-authtype=credssp, winrm-password-storage-path=keys/project/IWF/sp_ipmsscdctrm, description=IWF Prod App Server_ipmsscdctrm, node-executor=WinRMPython, file-copier=WinRMcpPython, username=xxx, tags=IWF_AppServer, winrm-user=xxx}, project='null'}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Start EngineWorkflowExecutor
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {before.step.1=true, after.step.1=false}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] start conditions for step [1]: []
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] skip conditions for step [1]: [(step.1.completed == 'true')]
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Create rule engine with rules: RuleEngine{ruleSet=[Rule: Conditions([(step.any.flowcontrol.halt == 'true')]) => DataState{state={workflow.done=true}}, Rule: Conditions([java.util.function.Predicate$$Lambda$801/0x0000000840916440@15cc348f]) => DataState{state={step.1.skip=true}}, Rule: Conditions([]) => DataState{state={step.1.start=true}}, Rule: Conditions([(workflow.keepgoing == 'false'), (step.any.state.failed == 'true')]) => DataState{state={workflow.done=true}}]}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Create workflow engine with state: StateLogger{state=DataState{state={workflow.id=b3c20e64-e843-4973-acb2-8136baa4a21d, workflow.keepgoing=false, before.step.1=true, after.step.1=false}}}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Begin: Workflow begin
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillProcessStateChange: state changes: init DataState{state={workflow.state=started}}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {workflow.state=started}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {step.1.start=true}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] DidProcessStateChange: applied state changes and rules (changed? true): init - StateLogger{state=DataState{state={workflow.id=b3c20e64-e843-4973-acb2-8136baa4a21d, workflow.keepgoing=false, before.step.1=true, step.1.start=true, after.step.1=false, workflow.state=started}}}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillRunOperation: operation starting: Step{stepNum=1, label='PIWF2DIWF_SCDsftp_IN'}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] LoopProgress: Pending(1) => run(1), skip(0), remain(0)
18:08:00 [admin@xxx 1][VERBOSE] [workflow] Begin step: 1,NodeDispatch
18:08:00 [admin@xxx 1][VERBOSE] 1: Workflow step executing: ScriptFileItem{label='PIWF2DIWF_SCDsftp_IN', script=[2743 chars]}
18:08:00 [admin@xxx 1][VERBOSE] preparing for sequential execution on 1 nodes
18:08:00 [admin@xxx 1][VERBOSE] Executing command on node: hostname_1, NodeEntryImpl{tags=[IWF_AppServer], attributes={nodename=hostname_1, hostname=x.x.x.x, osFamily=windows, winrm-authtype=credssp, winrm-password-storage-path=keys/project/IWF/sp_ipmsscdctrm, description=IWF Prod App Server_ipmsscdctrm, node-executor=WinRMPython, file-copier=WinRMcpPython, username=xxx, tags=IWF_AppServer, winrm-user=xxx}, project='null'}
18:08:00 [xxx@hostname_1 1][VERBOSE] [workflow] beginExecuteNodeStep(hostname_1): NodeDispatch: ScriptFileItem{label='PIWF2DIWF_SCDsftp_IN', script=[2743 chars]}
18:08:04 [xxx@hostname_1 1][VERBOSE] Operation state: Queued
18:08:04 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=1, log=[WinRMcpPython] executing: [${config.interpreter}, -u, /opt/apps/runner/plugin-cache/py-winrm-plugin-2.1.1/winrm-filecopier.py, ${node.hostname}, /opt/apps/runner/tmp/dispatch18117215979559193780.tmp, C:\WINDOWS\TEMP\14613-79992469-hostname_1-dispatch-script.tmp.ps1], level=VERBOSE, meta=null)
18:08:04 [xxx@hostname_1 1][VERBOSE] Operation state: Running
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=3, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=2, log=Traceback (most recent call last):, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=7, log= File "<string>", line 3, in raise_from, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=8, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=9, log= httplib_response = conn.getresponse(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=10, log= File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=11, log= response.begin(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=12, log= File "/usr/lib64/python3.9/http/client.py", line 320, in begin, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=13, log= version, status, reason = self._read_status(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=14, log= File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=15, log= line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1"), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=16, log= File "/usr/lib64/python3.9/socket.py", line 704, in readinto, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=5, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=4, log= httplib_response = self._make_request(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=6, log= six.raise_from(e, None), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=32, log= httplib_response = self._make_request(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=33, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=34, log= six.raise_from(e, None), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=35, log= File "<string>", line 3, in raise_from, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=37, log= httplib_response = conn.getresponse(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=36, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=38, log= File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=17, log= return self._sock.recv_into(b), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=18, log=ConnectionResetError: [Errno 104] Connection reset by peer, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=19, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=20, log=During handling of the above exception, another exception occurred:, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=21, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=22, log=Traceback (most recent call last):, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=23, log= File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=24, log= resp = conn.urlopen(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=25, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=26, log= retries = retries.increment(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=27, log= File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=28, log= raise six.reraise(type(error), error, _stacktrace), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=29, log= File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 708, in reraise, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=30, log= raise value.with_traceback(tb), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=31, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=51, log= File "/opt/apps/runner/plugin-cache/py-winrm-plugin-2.1.1/winrm-filecopier.py", line 385, in <module>, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=40, log= File "/usr/lib64/python3.9/http/client.py", line 320, in begin, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=41, log= version, status, reason = self._read_status(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=42, log= File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=43, log= line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1"), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=44, log= File "/usr/lib64/python3.9/socket.py", line 704, in readinto, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=45, log= return self._sock.recv_into(b), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=46, log=urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=47, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=48, log=During handling of the above exception, another exception occurred:, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=49, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=50, log=Traceback (most recent call last):, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=52, log= copy.winrm_upload(remote_path=destination,, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=53, log= File "/opt/apps/runner/plugin-cache/py-winrm-plugin-2.1.1/winrm-filecopier.py", line 187, in winrm_upload, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=54, log= self.session.run_ps('if (!(Test-Path {0})) {{ New-Item -ItemType directory -Path {0} }}'.format(remote_path)), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=55, log= File "/usr/local/lib/python3.9/site-packages/winrm/__init__.py", line 53, in run_ps, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=56, log= rs = self.run_cmd('powershell -encodedcommand {0}'.format(encoded_ps)), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=57, log= File "/usr/local/lib/python3.9/site-packages/winrm/__init__.py", line 40, in run_cmd, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=58, log= shell_id = self.protocol.open_shell(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=59, log= File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 166, in open_shell, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=60, log= res = self.send_message(xmltodict.unparse(req)), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=61, log= File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 243, in send_message, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=62, log= resp = self.transport.send_message(message), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=63, log= File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 309, in send_message, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=39, log= response.begin(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=81, log=requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=76, log= response = self.handle_401(response, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=77, log= File "/usr/local/lib/python3.9/site-packages/requests_credssp/credssp.py", line 228, in handle_401, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=78, log= response = response.connection.send(request, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=79, log= File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=80, log= raise ConnectionError(err, request=request), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=64, log= self.build_session(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=65, log= File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 292, in build_session, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=66, log= self.setup_encryption(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=67, log= File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 298, in setup_encryption, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=68, log= self._send_message_request(prepared_request, ''), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=69, log= File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 327, in _send_message_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=70, log= response = self.session.send(prepared_request, timeout=self.read_timeout_sec), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=71, log= File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 664, in send, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=72, log= r = dispatch_hook('response', hooks, r, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=73, log= File "/usr/lib/python3.9/site-packages/requests/hooks.py", line 31, in dispatch_hook, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=74, log= _hook_data = hook(hook_data, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=75, log= File "/usr/local/lib/python3.9/site-packages/requests_credssp/credssp.py", line 199, in response_hook, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=82, log=[WinRMcpPython]: result code: 1, level=VERBOSE, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Operation state: Error
18:08:10 [xxx@hostname_1 1][VERBOSE] [workflow] finishExecuteNodeStep(hostname_1): NodeDispatch: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
18:08:10 [admin@xxx 1][ERROR] Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
18:08:10 [admin@xxx 1][VERBOSE] Failed dispatching to node hostname_1: com.dtolabs.rundeck.core.execution.workflow.steps.node.NodeStepException: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.DefaultScriptFileNodeStepUtils.executeScriptFile(DefaultScriptFileNodeStepUtils.java:134)
at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.ScriptFileNodeStepExecutor.executeNodeStep(ScriptFileNodeStepExecutor.java:81)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:207)
at com.dtolabs.rundeck.core.execution.ExecutionService$executeNodeStep$12.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.executeNodeStep(RunnerExecutionService.groovy:109)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:130)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:61)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodesWith(ExecutionServiceImpl.java:263)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:234)
at com.dtolabs.rundeck.core.execution.ExecutionService$dispatchToNodes$9.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.dispatchToNodes(RunnerExecutionService.groovy:116)
at com.dtolabs.rundeck.core.execution.workflow.steps.NodeDispatchStepExecutor.executeWorkflowStep(NodeDispatchStepExecutor.java:66)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:111)
at com.dtolabs.rundeck.core.execution.ExecutionService$executeStep$8.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.executeStep(RunnerExecutionService.groovy:98)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWFItem(BaseWorkflowExecutor.java:285)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWorkflowStep(BaseWorkflowExecutor.java:681)
at com.dtolabs.rundeck.core.execution.workflow.engine.StepCallable.apply(StepCallable.java:71)
at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:76)
at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:32)
at com.dtolabs.rundeck.core.rules.WorkflowEngineOperationsProcessor.lambda$beginOperation$1(WorkflowEngineOperationsProcessor.java:323)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.dtolabs.rundeck.core.execution.service.FileCopierException: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
at jdk.internal.reflect.GeneratedConstructorAccessor1471.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:72)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:286)
at com.rundeck.runner.filecopier.RunnerFileCopier.copyFileStream(RunnerFileCopier.groovy:148)
at com.rundeck.runner.filecopier.RunnerFileCopier.copyFile(RunnerFileCopier.groovy:155)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.fileCopyFile(ExecutionServiceImpl.java:320)
at com.dtolabs.rundeck.core.execution.ExecutionService$fileCopyFile$15.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.fileCopyFile(RunnerExecutionService.groovy:147)
at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.DefaultScriptFileNodeStepUtils.executeScriptFile(DefaultScriptFileNodeStepUtils.java:119)
... 26 more
18:08:10 [admin@xxx 1][VERBOSE] 1: Workflow step finished, result: NodeDispatchFailure: Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
18:08:10 [admin@xxx 1][VERBOSE] [workflow] Finish step: 1,NodeDispatch
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] OperationFailed: operation completed, success? false: OperationCompleted(identity=[1]PIWF2DIWF_SCDsftp_IN, stepNum=1, newState=DataState{state={step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}, stepResultCapture=StepResultCapture{stepResult=NodeDispatchFailure: Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1, stepSuccess=false, statusString='null', controlBehavior=null, resultData=MultiDataContextImpl(map={}, base=null)}, success=false)
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillProcessStateChange: state changes: [1]PIWF2DIWF_SCDsftp_IN DataState{state={step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {workflow.done=true, step.1.start=true, step.1.skip=true}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] DidProcessStateChange: applied state changes and rules (changed? true): [1]PIWF2DIWF_SCDsftp_IN - StateLogger{state=DataState{state={workflow.id=b3c20e64-e843-4973-acb2-8136baa4a21d, workflow.keepgoing=false, workflow.done=true, step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.start=true, after.step.1=true, step.1.state=failure, workflow.state=started, step.1.skip=true}}}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WorkflowEndState: Workflow end state reached.
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillShutdown: Workflow engine shutting down (interrupted? false)
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Complete: Workflow complete: [Step{stepNum=1, label='PIWF2DIWF_SCDsftp_IN'}: OperationCompleted(identity=[1]PIWF2DIWF_SCDsftp_IN, stepNum=1, newState=DataState{state={step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}, stepResultCapture=StepResultCapture{stepResult=NodeDispatchFailure: Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1, stepSuccess=false, statusString='null', controlBehavior=null, resultData=MultiDataContextImpl(map={}, base=null)}, success=false)]
18:08:10 [admin@xxx _][VERBOSE] [workflow] Finish execution: node-first: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [hostname_1: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, Node failures: {hostname_1=[COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, status: failed]
18:08:10 [admin@xxx _][VERBOSE] [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [hostname_1: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, Node failures: {hostname_1=[COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, status: failed]
18:08:10 [admin@xxx _][ERROR] Execution failed: 79992469 in project IWF: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [hostname_1: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, Node failures: {hostname_1=[COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, status: failed]
18:08:00 [admin@xxx _][VERBOSE] preparing for sequential execution on 1 nodes
18:08:00 [admin@xxx _][VERBOSE] Executing command on node: hostname_1, NodeEntryImpl{tags=[IWF_AppServer], attributes={nodename=hostname_1, hostname=x.x.x.x, osFamily=windows, winrm-authtype=credssp, winrm-password-storage-path=keys/project/IWF/sp_ipmsscdctrm, description=IWF Prod App Server_ipmsscdctrm, node-executor=WinRMPython, file-copier=WinRMcpPython, username=xxx, tags=IWF_AppServer, winrm-user=xxx}, project='null'}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Start EngineWorkflowExecutor
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {before.step.1=true, after.step.1=false}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] start conditions for step [1]: []
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] skip conditions for step [1]: [(step.1.completed == 'true')]
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Create rule engine with rules: RuleEngine{ruleSet=[Rule: Conditions([(step.any.flowcontrol.halt == 'true')]) => DataState{state={workflow.done=true}}, Rule: Conditions([java.util.function.Predicate$$Lambda$801/0x0000000840916440@15cc348f]) => DataState{state={step.1.skip=true}}, Rule: Conditions([]) => DataState{state={step.1.start=true}}, Rule: Conditions([(workflow.keepgoing == 'false'), (step.any.state.failed == 'true')]) => DataState{state={workflow.done=true}}]}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Create workflow engine with state: StateLogger{state=DataState{state={workflow.id=b3c20e64-e843-4973-acb2-8136baa4a21d, workflow.keepgoing=false, before.step.1=true, after.step.1=false}}}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Begin: Workflow begin
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillProcessStateChange: state changes: init DataState{state={workflow.state=started}}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {workflow.state=started}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {step.1.start=true}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] DidProcessStateChange: applied state changes and rules (changed? true): init - StateLogger{state=DataState{state={workflow.id=b3c20e64-e843-4973-acb2-8136baa4a21d, workflow.keepgoing=false, before.step.1=true, step.1.start=true, after.step.1=false, workflow.state=started}}}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillRunOperation: operation starting: Step{stepNum=1, label='PIWF2DIWF_SCDsftp_IN'}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] LoopProgress: Pending(1) => run(1), skip(0), remain(0)
18:08:00 [admin@xxx 1][VERBOSE] [workflow] Begin step: 1,NodeDispatch
18:08:00 [admin@xxx 1][VERBOSE] 1: Workflow step executing: ScriptFileItem{label='PIWF2DIWF_SCDsftp_IN', script=[2743 chars]}
18:08:00 [admin@xxx 1][VERBOSE] preparing for sequential execution on 1 nodes
18:08:00 [admin@xxx 1][VERBOSE] Executing command on node: hostname_1, NodeEntryImpl{tags=[IWF_AppServer], attributes={nodename=hostname_1, hostname=x.x.x.x, osFamily=windows, winrm-authtype=credssp, winrm-password-storage-path=keys/project/IWF/sp_ipmsscdctrm, description=IWF Prod App Server_ipmsscdctrm, node-executor=WinRMPython, file-copier=WinRMcpPython, username=xxx, tags=IWF_AppServer, winrm-user=xxx}, project='null'}
18:08:00 [xxx@hostname_1 1][VERBOSE] [workflow] beginExecuteNodeStep(hostname_1): NodeDispatch: ScriptFileItem{label='PIWF2DIWF_SCDsftp_IN', script=[2743 chars]}
18:08:04 [xxx@hostname_1 1][VERBOSE] Operation state: Queued
18:08:04 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=1, log=[WinRMcpPython] executing: [${config.interpreter}, -u, /opt/apps/runner/plugin-cache/py-winrm-plugin-2.1.1/winrm-filecopier.py, ${node.hostname}, /opt/apps/runner/tmp/dispatch18117215979559193780.tmp, C:\WINDOWS\TEMP\14613-79992469-hostname_1-dispatch-script.tmp.ps1], level=VERBOSE, meta=null)
18:08:04 [xxx@hostname_1 1][VERBOSE] Operation state: Running
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=3, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=2, log=Traceback (most recent call last):, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=7, log= File "<string>", line 3, in raise_from, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=8, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=9, log= httplib_response = conn.getresponse(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=10, log= File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=11, log= response.begin(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=12, log= File "/usr/lib64/python3.9/http/client.py", line 320, in begin, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=13, log= version, status, reason = self._read_status(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=14, log= File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=15, log= line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1"), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=16, log= File "/usr/lib64/python3.9/socket.py", line 704, in readinto, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=5, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=4, log= httplib_response = self._make_request(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=6, log= six.raise_from(e, None), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=32, log= httplib_response = self._make_request(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=33, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=34, log= six.raise_from(e, None), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=35, log= File "<string>", line 3, in raise_from, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=37, log= httplib_response = conn.getresponse(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=36, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=38, log= File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=17, log= return self._sock.recv_into(b), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=18, log=ConnectionResetError: [Errno 104] Connection reset by peer, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=19, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=20, log=During handling of the above exception, another exception occurred:, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=21, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=22, log=Traceback (most recent call last):, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=23, log= File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=24, log= resp = conn.urlopen(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=25, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=26, log= retries = retries.increment(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=27, log= File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=28, log= raise six.reraise(type(error), error, _stacktrace), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=29, log= File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 708, in reraise, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=30, log= raise value.with_traceback(tb), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=31, log= File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=51, log= File "/opt/apps/runner/plugin-cache/py-winrm-plugin-2.1.1/winrm-filecopier.py", line 385, in <module>, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=40, log= File "/usr/lib64/python3.9/http/client.py", line 320, in begin, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=41, log= version, status, reason = self._read_status(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=42, log= File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=43, log= line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1"), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=44, log= File "/usr/lib64/python3.9/socket.py", line 704, in readinto, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=45, log= return self._sock.recv_into(b), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=46, log=urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=47, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=48, log=During handling of the above exception, another exception occurred:, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=49, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=50, log=Traceback (most recent call last):, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=52, log= copy.winrm_upload(remote_path=destination,, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=53, log= File "/opt/apps/runner/plugin-cache/py-winrm-plugin-2.1.1/winrm-filecopier.py", line 187, in winrm_upload, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=54, log= self.session.run_ps('if (!(Test-Path {0})) {{ New-Item -ItemType directory -Path {0} }}'.format(remote_path)), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=55, log= File "/usr/local/lib/python3.9/site-packages/winrm/__init__.py", line 53, in run_ps, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=56, log= rs = self.run_cmd('powershell -encodedcommand {0}'.format(encoded_ps)), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=57, log= File "/usr/local/lib/python3.9/site-packages/winrm/__init__.py", line 40, in run_cmd, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=58, log= shell_id = self.protocol.open_shell(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=59, log= File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 166, in open_shell, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=60, log= res = self.send_message(xmltodict.unparse(req)), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=61, log= File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 243, in send_message, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=62, log= resp = self.transport.send_message(message), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=63, log= File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 309, in send_message, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=39, log= response.begin(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=81, log=requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=76, log= response = self.handle_401(response, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=77, log= File "/usr/local/lib/python3.9/site-packages/requests_credssp/credssp.py", line 228, in handle_401, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=78, log= response = response.connection.send(request, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=79, log= File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=80, log= raise ConnectionError(err, request=request), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=64, log= self.build_session(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=65, log= File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 292, in build_session, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=66, log= self.setup_encryption(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=67, log= File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 298, in setup_encryption, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=68, log= self._send_message_request(prepared_request, ''), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=69, log= File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 327, in _send_message_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=70, log= response = self.session.send(prepared_request, timeout=self.read_timeout_sec), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=71, log= File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 664, in send, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=72, log= r = dispatch_hook('response', hooks, r, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=73, log= File "/usr/lib/python3.9/site-packages/requests/hooks.py", line 31, in dispatch_hook, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=74, log= _hook_data = hook(hook_data, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=75, log= File "/usr/local/lib/python3.9/site-packages/requests_credssp/credssp.py", line 199, in response_hook, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=82, log=[WinRMcpPython]: result code: 1, level=VERBOSE, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Operation state: Error
18:08:10 [xxx@hostname_1 1][VERBOSE] [workflow] finishExecuteNodeStep(hostname_1): NodeDispatch: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
18:08:10 [admin@xxx 1][ERROR] Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
18:08:10 [admin@xxx 1][VERBOSE] Failed dispatching to node hostname_1: com.dtolabs.rundeck.core.execution.workflow.steps.node.NodeStepException: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.DefaultScriptFileNodeStepUtils.executeScriptFile(DefaultScriptFileNodeStepUtils.java:134)
at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.ScriptFileNodeStepExecutor.executeNodeStep(ScriptFileNodeStepExecutor.java:81)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:207)
at com.dtolabs.rundeck.core.execution.ExecutionService$executeNodeStep$12.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.executeNodeStep(RunnerExecutionService.groovy:109)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:130)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:61)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodesWith(ExecutionServiceImpl.java:263)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:234)
at com.dtolabs.rundeck.core.execution.ExecutionService$dispatchToNodes$9.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.dispatchToNodes(RunnerExecutionService.groovy:116)
at com.dtolabs.rundeck.core.execution.workflow.steps.NodeDispatchStepExecutor.executeWorkflowStep(NodeDispatchStepExecutor.java:66)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:111)
at com.dtolabs.rundeck.core.execution.ExecutionService$executeStep$8.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.executeStep(RunnerExecutionService.groovy:98)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWFItem(BaseWorkflowExecutor.java:285)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWorkflowStep(BaseWorkflowExecutor.java:681)
at com.dtolabs.rundeck.core.execution.workflow.engine.StepCallable.apply(StepCallable.java:71)
at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:76)
at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:32)
at com.dtolabs.rundeck.core.rules.WorkflowEngineOperationsProcessor.lambda$beginOperation$1(WorkflowEngineOperationsProcessor.java:323)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.dtolabs.rundeck.core.execution.service.FileCopierException: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
at jdk.internal.reflect.GeneratedConstructorAccessor1471.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:72)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:286)
at com.rundeck.runner.filecopier.RunnerFileCopier.copyFileStream(RunnerFileCopier.groovy:148)
at com.rundeck.runner.filecopier.RunnerFileCopier.copyFile(RunnerFileCopier.groovy:155)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.fileCopyFile(ExecutionServiceImpl.java:320)
at com.dtolabs.rundeck.core.execution.ExecutionService$fileCopyFile$15.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.fileCopyFile(RunnerExecutionService.groovy:147)
at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.DefaultScriptFileNodeStepUtils.executeScriptFile(DefaultScriptFileNodeStepUtils.java:119)
... 26 more
18:08:10 [admin@xxx 1][VERBOSE] 1: Workflow step finished, result: NodeDispatchFailure: Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
18:08:10 [admin@xxx 1][VERBOSE] [workflow] Finish step: 1,NodeDispatch
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] OperationFailed: operation completed, success? false: OperationCompleted(identity=[1]PIWF2DIWF_SCDsftp_IN, stepNum=1, newState=DataState{state={step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}, stepResultCapture=StepResultCapture{stepResult=NodeDispatchFailure: Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1, stepSuccess=false, statusString='null', controlBehavior=null, resultData=MultiDataContextImpl(map={}, base=null)}, success=false)
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillProcessStateChange: state changes: [1]PIWF2DIWF_SCDsftp_IN DataState{state={step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {workflow.done=true, step.1.start=true, step.1.skip=true}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] DidProcessStateChange: applied state changes and rules (changed? true): [1]PIWF2DIWF_SCDsftp_IN - StateLogger{state=DataState{state={workflow.id=b3c20e64-e843-4973-acb2-8136baa4a21d, workflow.keepgoing=false, workflow.done=true, step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.start=true, after.step.1=true, step.1.state=failure, workflow.state=started, step.1.skip=true}}}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WorkflowEndState: Workflow end state reached.
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillShutdown: Workflow engine shutting down (interrupted? false)
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Complete: Workflow complete: [Step{stepNum=1, label='PIWF2DIWF_SCDsftp_IN'}: OperationCompleted(identity=[1]PIWF2DIWF_SCDsftp_IN, stepNum=1, newState=DataState{state={step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}, stepResultCapture=StepResultCapture{stepResult=NodeDispatchFailure: Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1, stepSuccess=false, statusString='null', controlBehavior=null, resultData=MultiDataContextImpl(map={}, base=null)}, success=false)]
18:08:10 [admin@xxx _][VERBOSE] [workflow] Finish execution: node-first: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [hostname_1: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, Node failures: {hostname_1=[COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, status: failed]
18:08:10 [admin@xxx _][VERBOSE] [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [hostname_1: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, Node failures: {hostname_1=[COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, status: failed]
18:08:10 [admin@xxx _][ERROR] Execution failed: 79992469 in project IWF: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [hostname_1: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, Node failures: {hostname_1=[COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, status: failed]
rac...@rundeck.com
Feb 16, 2024, 2:11:51 PMFeb 16
to rundeck-discuss
Hello, Nandhu.
Please contact the specialist Process Automation support staff, and they will help you ASAP :-)
Kind regards!
Please contact the specialist Process Automation support staff, and they will help you ASAP :-)
Kind regards!
nandhu
Feb 16, 2024, 6:24:29 PMFeb 16
to rundeck...@googlegroups.com
Pager duty support keep says target node windows is the problem. But in windows logs we do not see anything.
Any assistance on this would be appreciated.
--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/19ce4de7-428b-463d-ac50-52f7e82c7bf2n%40googlegroups.com.
Nandhu u
Feb 19, 2024, 8:15:26 AMFeb 19
to rundeck-discuss
Hi Team, Any help on this please. we keep hitting this intermittent issue.
Reply all
Reply to author
Forward
0 new messages