Issues sending SIGINT to remote processes

[Danilo Tee]

Hello all,

I am currently preparing for a project will which require careful and coordinated execution of commands on around 50 servers, divided into 5 groups (groupA to groupE).

- Each of the groups need to receive differing commands or arguments and the commands have to be issued in parallel to them all.

I have a few solutions in mind and some initial testing has not persuaded me one way or the other since all have problems.

1. First approach is to use "pssh groupA & pssh groupB & pssh groupC......" to trigger pssh commands in parallel to each of the hosts. The problem with this approach is I cannot send a Ctrl+C to kill them all since the earlier instances are all in the background. Once I move them to the foreground they can be killed one by one (or with a killall pssh).

After testing this using a single command: pssh -P -h hosts.list ping xyz , I can see the output of the ping command and if I send a ctrl+C, pssh is killed as are the remote ping processes. Good.

If I try the same using hping, pssh -P -h hosts.list hping3 xyz,I receive no output from hping3 and after 30 seconds, I received a FAILURE: Timed out, Killed by signal 9 from pssh. Even worse, the hping process on the remote hosts is still running.

2. Second approach is to use pssh to run a shell script on the remote servers that executes a particular command based on the group assignment (checking it using hostname and a mapping). However, the same issues appear when trying to reliably ctrl-c and kill the remote script (and children processes of the script) started by pssh.

Any suggestions would be much appreciated as to how to execute this with the most control (and preferably also receiving output) over the remote processes. Thanks.

[Andrew McNabb]

On Sun, Feb 10, 2013 at 08:20:36PM -0800, Danilo Tee wrote:
>
> I am currently preparing for a project will which require careful and
> coordinated execution of commands on around 50 servers, divided into 5
> groups (groupA to groupE).
>
> - Each of the groups need to receive differing commands or arguments and
> the commands have to be issued in parallel to them all.

I have a couple of thoughts.

First, if I'm understanding correctly, there are specific commands that
need to be run on specific hosts. In other words, it's not that there
are 50 different commands that can be run on any arbitrary host, right?
So it sounds like the PSSH_NODENUM environment variable would not be
helpful for you.

Second, you may find it easiest to create a script that ties into
psshlib. Basically, you can copy what the pssh script does, but
customize the Task to have a different command for each host. I think
this will end up being a convenient solution for you, but let us know
how things go.

--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868