Command fails when using nextflow run but runs when using bash .command.run

[Matteo Schiavinato]

Hi everyone,

I just had a weird situation with Nextflow, and I need some help with that. I have a NF workflow that contains the following command inside a process:

samtools coverage --reference ${fasta} ${bam} > ${bam}.depth

The command works fine when used on a command line outside of nextflow. However, when used within the workflow, I get the error "unknown command "coverage"" which seems to be a samtools error. The "coverage" command is part of the samtools toolkit and works fine outside of nextflow, as stated above.

I navigated to the /work subdirectory where the command was run, and executed the .command.sh file as "sh .command.sh", which throws the same error.

I then executed the .command.run file using "bash .command.run" and it worked fine.

I don't get where the issue could be, I don't know where to start. Does anyone have any hint for me on where to start debugging? Why do .command.sh and .command.run have different behaviors? Does it all come down to the usage of sh instead of bash?

Thanks,

Matteo

[drhp...@gmail.com]

Hi Matteo,

The most obvious cause of issues like this are if you haven't specified how NF should use containers properly. Be great to have more information e.g. the command you are using to run the pipeline, whether you are using Docker/Singularity/Conda, a link to the repo (if public). If this is indeed the problem you can try to run "samtools version" instead of the "coverage" command to see if that finishes.

The reason it may be working is because you have "samtools" installed locally?

Cheers,

Harshil

[Matteo Schiavinato]

I'm using nextflow inside a sbatch file that I submit to a queue without docker / singularity / conda.

I'm going to try to remove my local samtools. However that should not be the issue, technically, because the samtools in using here is not my own installed one but one I import as a module. Which should be prefixed in the PATH.

--
You received this message because you are subscribed to a topic in the Google Groups "Nextflow" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nextflow/PSQVZ_ZuHjU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/7e2dd385-f7cc-4a29-ac74-7bc980b47528n%40googlegroups.com.

[Alan Hoyle]

Put some debug steps in your process's script above the samtools coverage line.  E.g. 

   which samtools  ; samtools help ; samtools version # etc...

That will make sure you're running the same binary in both circumstances and tell you what options are available.  Those should write to .command.log. 

It looks like samtools coverage wasn't introduced until version samtools 1.10:  https://github.com/samtools/samtools/releases

did you try "bash .command.sh"?  It might be some kind of $PATH inheritance thing. 

It's important to remember that nextflow requires bash and sh/bash do behave slightly differently.  

It sounds like your cluster uses modules, and the default samtools might be an out-of-date one.  Not sure how to work around that in Nextflow.  

--
  -  Alan Hoyle  -  al...@alanhoyle.com  -  http://www.alanhoyle.com/  -

You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/CAExV1eKUnQYak92erNhEkFaoorBCKjqub91tAzk2ewPV6vs2pg%40mail.gmail.com.

[drhp...@gmail.com]

Hi Guys,

I would advise against using environment modules locally for running NF for a number of reasons including reproducibility and the level of maintenance in terms of keeping software up-to-date etc.

In general, containers are much better for this sort of thing and Nextflow works seamlessly with executing processes in these isolated environments. We are now using Biocontainers for almost everything on nf-core. We don't specifically have a module for `samtools coverage` on nf-core/modules yet but we do have loads of other examples that may help:

https://github.com/nf-core/modules/tree/master/modules/samtools

Cheers,

Harshil