[slurm-users] srun with &&, |, and > oh my!
103 views
Skip to first unread message
Chandler
Jan 23, 2023, 9:23:28 PM1/23/23
to Slurm User Community List
I want to run a command like: cmd1 infile1 | cmd2 opt2 arg2 | cmd3 opt3 arg3 -- > outfile && cmd4 opt4 arg4
Which runs fine at any prompt.
I'm afraid to just put `srun` at the beginning though, would it run the whole set of commands on the compute node? I don't want to try it because it involves sending a bunch of data to stdout and then back into the other programs. I think it might only run `cmd1 infile1` on the compute node and then the rest will try and run on the head node?
Which runs fine at any prompt.
I'm afraid to just put `srun` at the beginning though, would it run the whole set of commands on the compute node? I don't want to try it because it involves sending a bunch of data to stdout and then back into the other programs. I think it might only run `cmd1 infile1` on the compute node and then the rest will try and run on the head node?
Williams, Gareth (IM&T, Black Mountain)
Jan 23, 2023, 9:55:50 PM1/23/23
to Slurm User Community List
Be brave and experiment! How far wrong can you go?
I will offer two related tips.
1) Since you want/need to use shell syntax, use srun to run a shell and pass the command as a string for that shell to interpret. Like:
srun bash -c "cmd1 infile1 | cmd2 opt2 arg2 | cmd3 opt3 arg3 -- > outfile && cmd4 opt4 arg4"
2) put the compound command into a shell script and use srun to run the script. Like
srun bash myscript.sh
That may help you to have a more readable/maintainable command.
The script could be static or dynamic (write it to a temporary file then use it straight away). The script could parse arguments if you like.
Happy experimenting.
Gareth
I will offer two related tips.
1) Since you want/need to use shell syntax, use srun to run a shell and pass the command as a string for that shell to interpret. Like:
srun bash -c "cmd1 infile1 | cmd2 opt2 arg2 | cmd3 opt3 arg3 -- > outfile && cmd4 opt4 arg4"
2) put the compound command into a shell script and use srun to run the script. Like
srun bash myscript.sh
That may help you to have a more readable/maintainable command.
The script could be static or dynamic (write it to a temporary file then use it straight away). The script could parse arguments if you like.
Happy experimenting.
Gareth
Chandler
Jan 23, 2023, 11:29:35 PM1/23/23
to Slurm User Community List
Williams, Gareth (IM&T, Black Mountain) wrote on 1/23/23 7:55 PM:
> srun bash -c "cmd1 infile1 | cmd2 opt2 arg2 | cmd3 opt3 arg3 -- > outfile && cmd4 opt4 arg4"
Yes this will work! Thanks!
> Be brave and experiment! How far wrong can you go?
Hmm I do love breaking and re-fixing things...
> srun bash -c "cmd1 infile1 | cmd2 opt2 arg2 | cmd3 opt3 arg3 -- > outfile && cmd4 opt4 arg4"
Davide DelVento
Jan 24, 2023, 10:49:03 AM1/24/23
to Slurm User Community List
Try first with small things like shell scripts you write which would
tell you where the thing is running (e.g. by using hostname). Keep in
mind that what would happen will most importantly depend on the shell.
For example, if you use "sudo" you know that using wildcards is
tricky, because your username you sudo from may not have reading
permissions to expand them, so will pass them literally to the
command, rather than expanding them as root.
Bottom line, good to be afraid. Understand shell syntax, but don't
trust and instead verify what happens with innocuous things before
opening the firehose :-)
tell you where the thing is running (e.g. by using hostname). Keep in
mind that what would happen will most importantly depend on the shell.
For example, if you use "sudo" you know that using wildcards is
tricky, because your username you sudo from may not have reading
permissions to expand them, so will pass them literally to the
command, rather than expanding them as root.
Bottom line, good to be afraid. Understand shell syntax, but don't
trust and instead verify what happens with innocuous things before
opening the firehose :-)
Reply all
Reply to author
Forward
0 new messages