singularity exec inside sbatch

[C T]

Hi All,

I am having problems trying to run singularity exec command passing it to sbatch. I can run the following command:

set -x;singularity exec simg.sif bwa 2>&1 | awk '{print $1}'

+ set -x

+ singularity exec simg.sif bwa

+ awk '{print $1}'

Program:

Version:

Contact:

Usage:

Command:

mem

fastmap

pemerge

aln

samse

sampe

bwasw

shm

fa2pac

pac2bwt

pac2bwtgen

bwtupdate

bwt2sa

Note:

There

`aln/samse/sampe'.

first.

However, if I ran it inside an sbatch, the $1 wasn't passed properly to awk:

$ sbatch --wrap "set -x && singularity exec $simg bwa 2>&1 | awk '{print $1}'"

Submitted batch job 1421870

$ cat slurm-1421870.out

+ awk '{print }'

+ singularity exec simg.sif bwa

Program: bwa (alignment via Burrows-Wheeler transformation)

Version: 0.7.17-r1188

Contact: Heng Li <l...@sanger.ac.uk>

Usage:   bwa <command> [options]

Command: index         index sequences in the FASTA format

         mem           BWA-MEM algorithm

         fastmap       identify super-maximal exact matches

         pemerge       merge overlapping paired ends (EXPERIMENTAL)

         aln           gapped/ungapped alignment

         samse         generate alignment (single ended)

         sampe         generate alignment (paired ended)

         bwasw         BWA-SW for long queries

         shm           manage indices in shared memory

         fa2pac        convert FASTA to PAC format

         pac2bwt       generate BWT from PAC

         pac2bwtgen    alternative algorithm for generating BWT

         bwtupdate     update .bwt to the new format

         bwt2sa        generate SA from BWT and Occ

Note: To use BWA, you need to first index the genome with `bwa index'.

      There are three alignment algorithms in BWA: `mem', `bwasw', and

      `aln/samse/sampe'. If you are not sure which to use, try `bwa mem'

      first. Please `man ./bwa.1' for the manual.

Can anyone please help me with how I'm supposed to do this command inside the sbatch to get the proper output? Thank you.

 

[Adrian Jackson]

This feels more like a slurm/bash issue than a singularity one. Have you tried escaping your quotation marks at all? i.e. 

sbatch --wrap "set -x && singularity exec $simg bwa 2>&1 | awk \'{print $1}\'"

To see if that makes a difference?

cheers

adrianj

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/d91b38e2-e334-4da0-95d1-8a0d0bdb37d3n%40lbl.gov.

[C T]

Thank you for your help. Escaping quotation always confuses me.

I've tried your suggestions but it still doesn't work.

$ sbatch --wrap "set -x && singularity exec simg.sif bwa 2>&1 | awk \'{print $1}\'"

Submitted batch job 1424154

$ cat slurm-1424154.out

+ awk ''\''{print' '}'\'''

awk: cmd. line:1: '{print

awk: cmd. line:1: ^ invalid char ''' in expression

+ singularity exec simg.sif bwa

[Dave Dykstra]

That's definitely not a singularity issue, the awk command is outside of
singularity. Try putting a backslash in front of the $1 instead. I can
get similar behavior with sh -c:

$ sh -c "set -x && echo 'hey there' | awk '{print $1}'"
+ echo 'hey there'
+ awk '{print }'
hey there
$ sh -c "set -x && echo 'hey there' | awk '{print \$1}'"
+ echo 'hey there'
+ awk '{print $1}'
hey

Dave

Dave

[C T]

Thank you so much, Dave! That works! Appreciate your help even though this is not singularity issue.

Actually, I intended to use the awk inside singularity.

Does that mean I need to do: 
sbatch --wrap "set -x && singularity exec simg.sif bwa 2>&1 | singularity exec awk '{print \$1}'"

Additionally, if I want to do multiple commands inside singularity, is the "&&" the correct way to do it?

sbatch --wrap "set -x && singularity exec simg.sif bwa 2>&1 && singularity exec awk '{print "test"}'"

Many times my commands are only two lines making it too cumbersome to create a script file.

[Dave Dykstra]

No, definitely don't run singularity twice. The singularity exec
command takes a container image and in your second case you didn't even
give it one.

It would be possible to put all your commands on the command line with
lots of good quoting and such, but the least confusing thing to do will
be to put all the commands into a script file and bind-mount it into
/bin with singularity -B. For example if "myscript" contains

set -x
bwa 2>&1 | awk '{print $1}'

and you do "chmod +x myscript" then do

singularity exec -B myscript:/bin/myscript simg.sif myscript

Dave

[C T]

Thank you for your help. 

Quotations always confuse me. Thanks for the suggestions. I'll start putting them in a script to avoid all these problems.

Thanks!