Specifying the conda installation location for Nextflow

[Steve]

In order to use conda with Nextflow, the docs say:

https://www.nextflow.io/docs/latest/conda.html

This feature requires Nextflow version 0.30.x or higher and the Conda or Miniconda package manager installed on your system.

How do you tell Nextflow where conda is installed to? Is it just assuming that the 'conda' command will be available on the system?

On our systems, I have several conda installations, and none of them are usable without first doing a few steps, such as:

module unload python
unset PYTHONHOME
unset PYTHONPATH
export PATH=/path/to/anaconda/bin:$PATH

 

How does this translate over to using conda with Nextflow? Even if I used a wrapper script to run these commands for the parent Nextflow process, it will not be propagated to HPC jobs that try to activate the provided conda env's for process execution, right? Do I have to manually bundle this in the process's `beforeScript` section?

The only thing listed here is the 'cacheDir':

https://www.nextflow.io/docs/latest/config.html#config-conda

But I do not see anything that declares the location of the conda to use.

[Paolo Di Tommaso]

> Is it just assuming that the 'conda' command will be available on the system?

Yes. 

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Steve]

It appears that if I launch Nextflow like this:

unset PYTHONHOME; unset PYTHONPATH; export PATH=/gpfs/home/kellys04/conda-NGS580/bin:$$PATH; \
    ./nextflow run main.nf -resume

with a process that looks like this:

process test_conda {
    echo true
    executor "slurm"
    input:
    val(x) from Channel.from('')

    script:
    """
    which conda
    """
}

Processes running on compute nodes still report the correct path to 'conda'.

This is a bit confusing, since I did not expect 'conda' to be in my $PATH when the process executes. It appears that the $PATH of the parent Nextflow process is being propagated to the environment in which the cluster jobs are running. Dont recall being aware of this feature, I am guessing this is intended? Is it described somewhere? If I run this:

$ grep 'conda' work/1c/d8ab76f86e82332dafb3375cdda92c/.command.*
work/1c/d8ab76f86e82332dafb3375cdda92c/.command.log:/gpfs/home/kellys04/conda-NGS580/bin/conda
work/1c/d8ab76f86e82332dafb3375cdda92c/.command.out:/gpfs/home/kellys04/conda-NGS580/bin/conda
work/1c/d8ab76f86e82332dafb3375cdda92c/.command.run:#SBATCH -J nf-test_conda_(1)
work/1c/d8ab76f86e82332dafb3375cdda92c/.command.run:# NEXTFLOW TASK: test_conda (1)
work/1c/d8ab76f86e82332dafb3375cdda92c/.command.sh:which conda

it is not clear how 'conda' is being propagated to the process's $PATH.

This also presents the problem that I may need to use multiple conda installations in a single pipeline, for example some tools only run under Miniconda2/Anaconda2, while others may need 3. Its not clear how I would implement this using the built-in conda implementation here. Any suggestions?

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

> This is a bit confusing, since I did not expect 'conda' to be in my $PATH when the process executes. It appears that the $PATH of the parent Nextflow process is being propagated to the environment in which the cluster jobs are running

This is generally done by the batch scheduler. Refer to the SLURM documentation for details. 

p

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.

[Steve]

Thanks, I tested this under SGE and this effect does not occur, was not aware that this was a 'feature' of SLURM.