Occasional failures in Rundeck

42 views
Skip to first unread message

Nandhu u

unread,
Feb 16, 2024, 2:03:37 PMFeb 16
to rundeck-discuss
Hi Team,

we are running process automation (Rundeck) clustered instances in Linux with Runner in Linux environment. And our target node is Windows server.

This is a cyclic job where it run every 2 mins. Occasionally we are seeing this below error. When we checked network and Windows logs we do not see any problem. 

18:08:00 [admin@xxx _][VERBOSE] [workflow] Begin execution: node-first
18:08:00 [admin@xxx _][VERBOSE] preparing for sequential execution on 1 nodes
18:08:00 [admin@xxx _][VERBOSE] Executing command on node: hostname_1, NodeEntryImpl{tags=[IWF_AppServer], attributes={nodename=hostname_1, hostname=x.x.x.x, osFamily=windows, winrm-authtype=credssp, winrm-password-storage-path=keys/project/IWF/sp_ipmsscdctrm, description=IWF Prod App Server_ipmsscdctrm, node-executor=WinRMPython, file-copier=WinRMcpPython, username=xxx, tags=IWF_AppServer, winrm-user=xxx}, project='null'}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Start EngineWorkflowExecutor
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {before.step.1=true, after.step.1=false}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] start conditions for step [1]: []
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] skip conditions for step [1]: [(step.1.completed == 'true')]
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Create rule engine with rules: RuleEngine{ruleSet=[Rule: Conditions([(step.any.flowcontrol.halt == 'true')]) => DataState{state={workflow.done=true}}, Rule: Conditions([java.util.function.Predicate$$Lambda$801/0x0000000840916440@15cc348f]) => DataState{state={step.1.skip=true}}, Rule: Conditions([]) => DataState{state={step.1.start=true}}, Rule: Conditions([(workflow.keepgoing == 'false'), (step.any.state.failed == 'true')]) => DataState{state={workflow.done=true}}]}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Create workflow engine with state: StateLogger{state=DataState{state={workflow.id=b3c20e64-e843-4973-acb2-8136baa4a21d, workflow.keepgoing=false, before.step.1=true, after.step.1=false}}}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Begin: Workflow begin
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillProcessStateChange: state changes: init DataState{state={workflow.state=started}}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {workflow.state=started}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {step.1.start=true}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] DidProcessStateChange: applied state changes and rules (changed? true): init - StateLogger{state=DataState{state={workflow.id=b3c20e64-e843-4973-acb2-8136baa4a21d, workflow.keepgoing=false, before.step.1=true, step.1.start=true, after.step.1=false, workflow.state=started}}}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillRunOperation: operation starting: Step{stepNum=1, label='PIWF2DIWF_SCDsftp_IN'}
18:08:00 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] LoopProgress: Pending(1) => run(1), skip(0), remain(0)
18:08:00 [admin@xxx 1][VERBOSE] [workflow] Begin step: 1,NodeDispatch
18:08:00 [admin@xxx 1][VERBOSE] 1: Workflow step executing: ScriptFileItem{label='PIWF2DIWF_SCDsftp_IN', script=[2743 chars]}
18:08:00 [admin@xxx 1][VERBOSE] preparing for sequential execution on 1 nodes
18:08:00 [admin@xxx 1][VERBOSE] Executing command on node: hostname_1, NodeEntryImpl{tags=[IWF_AppServer], attributes={nodename=hostname_1, hostname=x.x.x.x, osFamily=windows, winrm-authtype=credssp, winrm-password-storage-path=keys/project/IWF/sp_ipmsscdctrm, description=IWF Prod App Server_ipmsscdctrm, node-executor=WinRMPython, file-copier=WinRMcpPython, username=xxx, tags=IWF_AppServer, winrm-user=xxx}, project='null'}
18:08:00 [xxx@hostname_1 1][VERBOSE] [workflow] beginExecuteNodeStep(hostname_1): NodeDispatch: ScriptFileItem{label='PIWF2DIWF_SCDsftp_IN', script=[2743 chars]}
18:08:04 [xxx@hostname_1 1][VERBOSE] Operation state: Queued
18:08:04 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=1, log=[WinRMcpPython] executing: [${config.interpreter}, -u, /opt/apps/runner/plugin-cache/py-winrm-plugin-2.1.1/winrm-filecopier.py, ${node.hostname}, /opt/apps/runner/tmp/dispatch18117215979559193780.tmp, C:\WINDOWS\TEMP\14613-79992469-hostname_1-dispatch-script.tmp.ps1], level=VERBOSE, meta=null)
18:08:04 [xxx@hostname_1 1][VERBOSE] Operation state: Running
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=3, log=  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=2, log=Traceback (most recent call last):, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=7, log=  File "<string>", line 3, in raise_from, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=8, log=  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=9, log=    httplib_response = conn.getresponse(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=10, log=  File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=11, log=    response.begin(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=12, log=  File "/usr/lib64/python3.9/http/client.py", line 320, in begin, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=13, log=    version, status, reason = self._read_status(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=14, log=  File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=15, log=    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1"), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=16, log=  File "/usr/lib64/python3.9/socket.py", line 704, in readinto, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=5, log=  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=4, log=    httplib_response = self._make_request(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=6, log=    six.raise_from(e, None), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=32, log=    httplib_response = self._make_request(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=33, log=  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=34, log=    six.raise_from(e, None), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=35, log=  File "<string>", line 3, in raise_from, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=37, log=    httplib_response = conn.getresponse(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=36, log=  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=38, log=  File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=17, log=    return self._sock.recv_into(b), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=18, log=ConnectionResetError: [Errno 104] Connection reset by peer, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=19, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=20, log=During handling of the above exception, another exception occurred:, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=21, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=22, log=Traceback (most recent call last):, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=23, log=  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=24, log=    resp = conn.urlopen(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=25, log=  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=26, log=    retries = retries.increment(, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=27, log=  File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=28, log=    raise six.reraise(type(error), error, _stacktrace), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=29, log=  File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 708, in reraise, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=30, log=    raise value.with_traceback(tb), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=31, log=  File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=51, log=  File "/opt/apps/runner/plugin-cache/py-winrm-plugin-2.1.1/winrm-filecopier.py", line 385, in <module>, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=40, log=  File "/usr/lib64/python3.9/http/client.py", line 320, in begin, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=41, log=    version, status, reason = self._read_status(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=42, log=  File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=43, log=    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1"), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=44, log=  File "/usr/lib64/python3.9/socket.py", line 704, in readinto, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=45, log=    return self._sock.recv_into(b), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=46, log=urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=47, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=48, log=During handling of the above exception, another exception occurred:, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=49, log=null, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=50, log=Traceback (most recent call last):, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=52, log=    copy.winrm_upload(remote_path=destination,, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=53, log=  File "/opt/apps/runner/plugin-cache/py-winrm-plugin-2.1.1/winrm-filecopier.py", line 187, in winrm_upload, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=54, log=    self.session.run_ps('if (!(Test-Path {0})) {{ New-Item -ItemType directory -Path {0} }}'.format(remote_path)), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=55, log=  File "/usr/local/lib/python3.9/site-packages/winrm/__init__.py", line 53, in run_ps, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=56, log=    rs = self.run_cmd('powershell -encodedcommand {0}'.format(encoded_ps)), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=57, log=  File "/usr/local/lib/python3.9/site-packages/winrm/__init__.py", line 40, in run_cmd, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=58, log=    shell_id = self.protocol.open_shell(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=59, log=  File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 166, in open_shell, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=60, log=    res = self.send_message(xmltodict.unparse(req)), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=61, log=  File "/usr/local/lib/python3.9/site-packages/winrm/protocol.py", line 243, in send_message, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=62, log=    resp = self.transport.send_message(message), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=63, log=  File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 309, in send_message, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=39, log=    response.begin(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=81, log=requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=76, log=    response = self.handle_401(response, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=77, log=  File "/usr/local/lib/python3.9/site-packages/requests_credssp/credssp.py", line 228, in handle_401, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=78, log=    response = response.connection.send(request, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=79, log=  File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 498, in send, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=80, log=    raise ConnectionError(err, request=request), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=64, log=    self.build_session(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=65, log=  File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 292, in build_session, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=66, log=    self.setup_encryption(), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=67, log=  File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 298, in setup_encryption, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=68, log=    self._send_message_request(prepared_request, ''), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=69, log=  File "/usr/local/lib/python3.9/site-packages/winrm/transport.py", line 327, in _send_message_request, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=70, log=    response = self.session.send(prepared_request, timeout=self.read_timeout_sec), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=71, log=  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 664, in send, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=72, log=    r = dispatch_hook('response', hooks, r, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=73, log=  File "/usr/lib/python3.9/site-packages/requests/hooks.py", line 31, in dispatch_hook, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=74, log=    _hook_data = hook(hook_data, **kwargs), level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=75, log=  File "/usr/local/lib/python3.9/site-packages/requests_credssp/credssp.py", line 199, in response_hook, level=ERROR, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Unknown Report Type: com.rundeck.sidecar.api.model.reports.log.LogEventReport: com.rundeck.sidecar.api.model.reports.log.LogEventData: LogEventData(line=82, log=[WinRMcpPython]: result code: 1, level=VERBOSE, meta=null)
18:08:10 [xxx@hostname_1 1][VERBOSE] Operation state: Error
18:08:10 [xxx@hostname_1 1][VERBOSE] [workflow] finishExecuteNodeStep(hostname_1): NodeDispatch: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
18:08:10 [admin@xxx 1][ERROR] Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
18:08:10 [admin@xxx 1][VERBOSE] Failed dispatching to node hostname_1: com.dtolabs.rundeck.core.execution.workflow.steps.node.NodeStepException: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.DefaultScriptFileNodeStepUtils.executeScriptFile(DefaultScriptFileNodeStepUtils.java:134)
at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.ScriptFileNodeStepExecutor.executeNodeStep(ScriptFileNodeStepExecutor.java:81)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:207)
at com.dtolabs.rundeck.core.execution.ExecutionService$executeNodeStep$12.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.executeNodeStep(RunnerExecutionService.groovy:109)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:130)
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:61)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodesWith(ExecutionServiceImpl.java:263)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:234)
at com.dtolabs.rundeck.core.execution.ExecutionService$dispatchToNodes$9.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.dispatchToNodes(RunnerExecutionService.groovy:116)
at com.dtolabs.rundeck.core.execution.workflow.steps.NodeDispatchStepExecutor.executeWorkflowStep(NodeDispatchStepExecutor.java:66)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:111)
at com.dtolabs.rundeck.core.execution.ExecutionService$executeStep$8.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.executeStep(RunnerExecutionService.groovy:98)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWFItem(BaseWorkflowExecutor.java:285)
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWorkflowStep(BaseWorkflowExecutor.java:681)
at com.dtolabs.rundeck.core.execution.workflow.engine.StepCallable.apply(StepCallable.java:71)
at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:76)
at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:32)
at com.dtolabs.rundeck.core.rules.WorkflowEngineOperationsProcessor.lambda$beginOperation$1(WorkflowEngineOperationsProcessor.java:323)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.dtolabs.rundeck.core.execution.service.FileCopierException: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
at jdk.internal.reflect.GeneratedConstructorAccessor1471.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:72)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:286)
at com.rundeck.runner.filecopier.RunnerFileCopier.copyFileStream(RunnerFileCopier.groovy:148)
at com.rundeck.runner.filecopier.RunnerFileCopier.copyFile(RunnerFileCopier.groovy:155)
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.fileCopyFile(ExecutionServiceImpl.java:320)
at com.dtolabs.rundeck.core.execution.ExecutionService$fileCopyFile$15.call(Unknown Source)
at com.rundeck.runner.service.RunnerExecutionService.fileCopyFile(RunnerExecutionService.groovy:147)
at com.dtolabs.rundeck.core.execution.workflow.steps.node.impl.DefaultScriptFileNodeStepUtils.executeScriptFile(DefaultScriptFileNodeStepUtils.java:119)
... 26 more

18:08:10 [admin@xxx 1][VERBOSE] 1: Workflow step finished, result: NodeDispatchFailure: Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1
18:08:10 [admin@xxx 1][VERBOSE] [workflow] Finish step: 1,NodeDispatch
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] OperationFailed: operation completed, success? false: OperationCompleted(identity=[1]PIWF2DIWF_SCDsftp_IN, stepNum=1, newState=DataState{state={step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}, stepResultCapture=StepResultCapture{stepResult=NodeDispatchFailure: Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1, stepSuccess=false, statusString='null', controlBehavior=null, resultData=MultiDataContextImpl(map={}, base=null)}, success=false)
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillProcessStateChange: state changes: [1]PIWF2DIWF_SCDsftp_IN DataState{state={step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Update conditional state: {workflow.done=true, step.1.start=true, step.1.skip=true}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] DidProcessStateChange: applied state changes and rules (changed? true): [1]PIWF2DIWF_SCDsftp_IN - StateLogger{state=DataState{state={workflow.id=b3c20e64-e843-4973-acb2-8136baa4a21d, workflow.keepgoing=false, workflow.done=true, step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.start=true, after.step.1=true, step.1.state=failure, workflow.state=started, step.1.skip=true}}}
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WorkflowEndState: Workflow end state reached.
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] WillShutdown: Workflow engine shutting down (interrupted? false)
18:08:10 [admin@xxx _][VERBOSE] [wf:b3c20e64-e843-4973-acb2-8136baa4a21d] Complete: Workflow complete: [Step{stepNum=1, label='PIWF2DIWF_SCDsftp_IN'}: OperationCompleted(identity=[1]PIWF2DIWF_SCDsftp_IN, stepNum=1, newState=DataState{state={step.1.completed=true, step.any.state.failed=true, before.step.1=false, step.1.state=failure, after.step.1=true}}, stepResultCapture=StepResultCapture{stepResult=NodeDispatchFailure: Failed dispatching to node hostname_1: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1, stepSuccess=false, statusString='null', controlBehavior=null, resultData=MultiDataContextImpl(map={}, base=null)}, success=false)]
18:08:10 [admin@xxx _][VERBOSE] [workflow] Finish execution:  node-first: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [hostname_1: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, Node failures: {hostname_1=[COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, status: failed]
18:08:10 [admin@xxx _][VERBOSE] [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [hostname_1: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, Node failures: {hostname_1=[COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, status: failed]
18:08:10 [admin@xxx _][ERROR] Execution failed: 79992469 in project IWF: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [hostname_1: COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, Node failures: {hostname_1=[COPY_ERROR: Reason: NonZeroResultCode[WinRMcpPython]: external script failed with exit code: 1]}, status: failed]

rac...@rundeck.com

unread,
Feb 16, 2024, 2:11:51 PMFeb 16
to rundeck-discuss
Hello, Nandhu.

Please contact the specialist Process Automation support staff, and they will help you ASAP :-)

Kind regards!

nandhu

unread,
Feb 16, 2024, 6:24:29 PMFeb 16
to rundeck...@googlegroups.com
Pager duty support keep says target node windows is the problem. But in windows logs we do not see anything. 

Any assistance on this would be appreciated.

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/19ce4de7-428b-463d-ac50-52f7e82c7bf2n%40googlegroups.com.

Nandhu u

unread,
Feb 19, 2024, 8:15:26 AMFeb 19
to rundeck-discuss
Hi Team, Any help on this please. we keep hitting this intermittent issue.
Reply all
Reply to author
Forward
0 new messages