Process not returning only when executed in Rundeck

[david.ke...@gmail.com]

Hi,

I am using rundeck 3.3.9 installed in a docker container.

I have a job that ssh's to a server and then executes bash script there:

____________________

cd path/to/py

python my_program.py

____________________

In this python file I need to parse the output of a binary:

____________________

from subprocess import Popen, PIPE
process = Popen([“mybinary”], stdout=PIPE, stderr=PIPE, shell=False)
stdout, stderr = process.communicate()

____________________

​

This works perfectly fine when I run it on the remote server. It also runs perfectly fine when I ssh from the rundeck host to the node and execute the my bash script.

But it doesn't work when I execute the contents of my bash script as an "inline script" in rundeck. The python program is started but it stops at the "communicate()" and hangs there forever.

I am not sure how to troubleshoot this and I would be glad if somebody could help me.

Thanks!

[rac...@rundeck.com]

Hi David,

Any clue running the job in Debug mode? Are you calling the same python interpreter in your Rundeck job?

I tested using this example in a rundeck job (calling python3 from bash script) and works ok.

Regards!

[david.ke...@gmail.com]

Hi Rainier,

I did some more digging to produce a minimal and reproducible non-working example but I didn't succeed. I was able to identify exactly when things break though:

I am calling this binary from python to refresh our Qlik apps: https://github.com/qlik-oss/qlik-cli

I can call this app and the call returns fine as long as the binary doesn't send calls to the Qlik server, meaning 

process = Popen(['qlik','reload','create','--appId'], stdout=PIPE, stderr=PIPE)

works and prints text to the logfile in rundeck (no calls are made from the binary to the server because these parameters are invalid). If I make a valid call :

process = Popen(['qlik','reload','create','--appId',myappid], stdout=PIPE, stderr=PIPE)

the process never returns. 

What I noticed is that no print()'s in the whole python file are ever shown in rundeck, meaning that even prints() that are before this call are shown in the rundeck log:

print("I will not be shown")

process = Popen(['qlik','reload','create','--appId',myappid], stdout=PIPE, stderr=PIPE)

So I can only guess that the binary does something to the output that the connection rundeck uses (SSH-2.0-JSCH-0.1.54) can't handle. Just to reiterate: This works on the server it should run and using ssh from the machine (docker container) rundeck is installed on.

Can I use another connection than SSH-2.0-JSCH-0.1.54 to check if that helps maybe?

And should I create a github issue for this? I guess I am the only one with this issue so I don't understand if you don't want to use resources on it.

Thanks for your help,

David

[david.ke...@gmail.com]

I did some more digging and I see the same behaviour without python, meaning if I call my binary within rundeck like this:

qlik reload create --appId c357b304-d061-411b-814f-5e303856b456

The job never ends.

This is the ssh version I am using in my container with which it works as expected:

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017

Is there a way to use openssh for some jobs? Or is there any other way I can debug this?

[rac...@rundeck.com]

Hi David,

I'm trying to reproduce your issue, it seems a specific problem with the qlik binary. If you try in this way (assigning the output to a variable) the job still running?

Greetings!

[david.ke...@gmail.com]

Hi Rainier,

thank you for your support!

I ran it like this (using bash, I don't have a windows server at hand atm)

VAR=`qlik reload create --appId c357b304-d061-411b-814f-5e303856b456`

and this didn't change the outcome.