Rundeck timeout

[Justin Gyles]

Hello,

Hoping someone here can provide some assistance. We currently have a job that runs against a windows node via the winrm plugin. It executes a CMD file on said node. But it is consistently failing after 8 hours. I have checked the ssh timeout settings in rundeck, and they are set to 0. Also adjusted the winrm plugin timeout settings as well. Is there somewhere else I should look within rundeck that might be causing it to timeout after 8 hours?

Thank you very much for your time here. 

[Reiner Acuña]

Hi Justin,

Can you share the job output in debug mode to see any clue about that job? Any clue checking the /var/log/rundeck/service.log file? (please change any potentially sensitive information)

Maybe you've some value in the "timeout" field in your job definition? Can you test with that?

https://docs.rundeck.com/docs/manual/creating-jobs.html#timeout

Also, which Rundeck version are you using? How did you set those timeouts? (pywinrm plugin for your windows node)

Regards!

[Reiner Acuña]

Also, check any Windows-side configuration like this:

https://docs.microsoft.com/en-us/windows/win32/winrm/installation-and-configuration-for-windows-remote-management#protocol-default-settings

Take a look:

https://gist.github.com/galenemery/434a95dd2583e8dedc27#file-winrm_setup-ps1-L7

Greetings!

On Tuesday, February 25, 2020 at 3:52:46 PM UTC-3, Justin G wrote:

[Justin G]

Thanks Reiner for the reply. Much appreciated!

Here is the useful information from the log in debug mode. The rundeck version we are on is: 3.2.1-20200113

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 500 1578 (connectionpool.py:396)[urllib3.connectionpool]

[DEBUG  ]  Starting new HTTP connection (1): servername (connectionpool.py:208)[urllib3.connectionpool]

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 401 0 (connectionpool.py:396)[urllib3.connectionpool]

[DEBUG  ]  Resetting dropped connection: servername (connectionpool.py:243)[urllib3.connectionpool]

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 401 0 (connectionpool.py:396)[urllib3.connectionpool]

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 200 0 (connectionpool.py:396)[urllib3.connectionpool]

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 200 1899 (connectionpool.py:396)[urllib3.connectionpool]

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 200 1104 (connectionpool.py:396)[urllib3.connectionpool]

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 200 5908 (connectionpool.py:396)[urllib3.connectionpool]

This then runs a cmd file which executes against a sql db. It normally takes 15 hours to run outside of rundeck. 

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 500 1578 (connectionpool.py:396)[urllib3.connectionpool]

these then get logged quite a bit up until the 8 hour mark. Then this gets logged:

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 401 0 (connectionpool.py:396)[urllib3.connectionpool]

[DEBUG  ]  Resetting dropped connection: servername (connectionpool.py:243)[urllib3.connectionpool]

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 401 0 (connectionpool.py:396)[urllib3.connectionpool]

[DEBUG  ]  servername:5985 "POST /wsman HTTP/1.1" 400 0 (connectionpool.py:396)[urllib3.connectionpool]

[ERROR  ]  Execution finished with the following error (winrm-exec.py:268)[root]

[ERROR  ]  Bad HTTP response returned from server. Code 400 (winrm-exec.py:269)[root]

I had set the timeout settings in the project config file, like this:

project.winrm-connection-timeout=72000000

project.winrm-timeout=PT72000.000s

(I know thats long, but the process runs forever)

Greatly appreciate the help on this. Please let me know if you need more info. 

[Reiner Acuña]

Hi Justin,

You're using winrm plugin, right? (https://github.com/rundeck-plugins/rundeck-winrm-plugin) Seems an issue with that plugin, take a look:

https://www.techpaste.com/2016/07/unexpected-http-response-500-winrm-rundeck/

Now, we recommend using PyWinRM plugin instead (out of the box with Rundeck 3.0/3.1 and 3.2 branches) because is the currently supported and it's easiest to configure :-)

https://github.com/rundeck-plugins/py-winrm-plugin

Test your job with PyWinRM plugin:

https://github.com/rundeck-plugins/py-winrm-plugin#configuration

And here the always-useful way to set a windows node:

https://github.com/rundeck-plugins/py-winrm-plugin#node-definition-example

Hope it helps!

[javed...@gmail.com]

Hello,

I run all my jobs on windows server. Along with this setting:

project.winrm-connection-timeout=72000000

project.winrm-timeout=PT72000.000s

You need to set the winrm remote shell session timeout on the windows server to the same amount. 

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/4db13b5c-0122-4c1b-8059-855ae28641b9%40googlegroups.com.

[Justin Gyles]

Thank you for all the replies. I made sure the session timeout matched on the windows node as well. The test is running right now and will report back with my findings. 

To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/CACQN2z%3DuVFAaSztu5NgB-2YY0kYA4zcrmQ%3D5FkW6kUwc3JC%2BJA%40mail.gmail.com.