Using conda envs as modules on a cluster

[Laurent Fournier]

Hi,

I am using Singularity 2.4.1.

laurent@NanoStation:~/atWork/Singularity$ cat Tool27

BootStrap: docker
From: continuumio/miniconda:latest

%post
    echo "nameserver 1.1.1.1" | tee /etc/resolv.conf > /dev/null
    export PATH=/opt/conda/bin
    conda config --add channels bioconda
    conda config --add channels conda-forge
    conda clean --index-cache --tarballs --packages --yes
   
%environment
    export PATH="/opt/conda/bin:/usr/local/bin:/usr/bin:/bin:"
    unset CONDA_DEFAULT_ENV
    export ANACONDA_HOME=/opt/conda

To use the tools inside, i created small shell scripts like that :

laurent@NanoStation:~/atWork/Singularity$ cat conda

#!/bin/sh
singularity exec my/path/to/pytools27.img conda $@

That works just fine. But when it comes to using conda envs...it don't work anymore.

First i had to gain access to bash instead of sh (which is working). 

Then i "conda create -n myenv" which work as well :

laurent@NanoStation:~/atWork/Singularity$ ./conda create -n py27 python=2.7

Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment /home/laurent/.conda/envs/py27:

But from that point, whenever i try "source activate" or even modify the calling script as follows :

laurent@NanoStation:~/atWork/Singularity$ cat activate

#!/bin/sh
singularity exec my/path/to/pytools27.img bash -c 'source activate $@'

Nothing happens :

laurent@NanoStation:~/atWork/Singularity$ source activate py27
laurent@NanoStation:~/atWork/Singularity$ ./activate py27
laurent@NanoStation:~/atWork/Singularity$

Any ideas ?

[Rémy Dernat]

Hi Laurent. 

I would try to debug it inside the container with :

singularity shell /path/to/your/singularity/image

Then you will be able to check what is missing to get it work. Maybe your issue is related to a problem with your env in your container.

BTW you can replace your small shell script :

#!/bin/sh
singularity exec my/path/to/pytools27.img conda $@

By adding a %runscript section in your singularity definition file.

Best regards,

Rémy.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Laurent Fournier]

Hi Rémi,

The thing is... i'm working on a cluster. 

The singularity container can't be accessed directly from our users therefore the reason i use the modules system (module load avail etc.). In that regard, i have to use this..weird technique with the shell scripts (they act as binaries, which i need to make those modules).

It seems that everything executed with the conda binary works because it writes outside of the container (cf. ~/.conda for each user) but when it comes to de/activate i *think* that it tries to apply to the inside of the container (where nothing is present because the environments are created into the user homes).

I've seen workaround for that kind of things on Dockers but...here we are singularity powered ;-)

[Tru Huynh]

Hi,

On Thu, Nov 30, 2017 at 02:38:55AM -0800, Laurent Fournier wrote:
> Hi,
>
...

> laurent@NanoStation:~/atWork/Singularity$ ./conda create -n py27 python=2.7
>
> Fetching package metadata ...............
> Solving package specifications: .
>
> Package plan for installation in environment /home/laurent/.conda/envs/py27:
>
>
> But from that point, whenever i try "source activate" or even modify the
> calling script as follows :
>
> laurent@NanoStation:~/atWork/Singularity$ cat activate
>
> #!/bin/sh
> singularity exec my/path/to/pytools27.img bash -c 'source activate $@'
>
>
> Nothing happens :
>
> laurent@NanoStation:~/atWork/Singularity$ source activate py27
> laurent@NanoStation:~/atWork/Singularity$ ./activate py27
> laurent@NanoStation:~/atWork/Singularity$
>
>
> Any ideas ?

your "singularity exec" is finished once the "source activate py27" exits
It' just a one line command... doing nothing and exiting.

[tru@elitebook840g3 singularity-docker-bioconda-forge]$ cat Singularity
#!/bin/bash
#
# Tru Huynh <t...@pasteur.fr>
# 2017-12-01: initial version
# use as baseline to build a container from miniconda
# added bioconda + conda-forge from Laurent Fournier <ma...@laurent-fournier.be>
# [Singularity] Using conda envs as modules on a cluster

BootStrap: docker
From: continuumio/miniconda

%runscript
echo "This is what happens when you run the container..."
export PATH=/opt/conda/bin:${PATH}
/bin/bash

%environment
export PATH=/opt/conda/bin:${PATH}
unset CONDA_DEFAULT_ENV
export ANACONDA_HOME=/opt/conda

%post
export PATH=/opt/conda/bin:${PATH}
echo "Hello from inside the container"
conda update -y conda
conda update --all
#

conda config --add channels bioconda
conda config --add channels conda-forge
conda clean --index-cache --tarballs --packages --yes

conda list > /conda.txt
touch /`date -u -Iseconds`

%labels
MAINTAINER truatpasteurdotfr

[tru@elitebook840g3 singularity-docker-bioconda-forge]$ sudo singularity build py27 Singularity
...
[tru@elitebook840g3 singularity-docker-bioconda-forge]$ singularity exec ./py27 bash -c 'conda create -n py27 python=2.7'
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/tru/.conda/envs/py27:

The following NEW packages will be INSTALLED:

ca-certificates: 2017.08.26-h1d4fec5_0
certifi: 2017.11.5-py27h71e7faf_0
libedit: 3.1-heed3624_0
libffi: 3.2.1-hd88cf55_4
libgcc-ng: 7.2.0-h7cc24e2_2
libstdcxx-ng: 7.2.0-h7a57d05_2
ncurses: 6.0-h9df7e31_2
openssl: 1.0.2m-h26d622b_1
pip: 9.0.1-py27ha730c48_4
python: 2.7.14-hdd48546_24
readline: 7.0-ha6073c6_4
setuptools: 36.5.0-py27h68b189e_0
sqlite: 3.20.1-hb898158_2
tk: 8.6.7-hc745277_3
wheel: 0.30.0-py27h2bc6bb2_1
zlib: 1.2.11-ha838bed_2

Proceed ([y]/n)? y

#
# To activate this environment, use:
# > source activate py27
#
# To deactivate an active environment, use:
# > source deactivate
#


[tru@elitebook840g3 singularity-docker-bioconda-forge]$ singularity exec py27 bash -c 'source activate py27; printenv|grep CONDA; conda list'
ANACONDA_HOME=/opt/conda
CONDA_PATH_BACKUP=/opt/conda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CONDA_PREFIX=/home/tru/.conda/envs/py27
CONDA_PS1_BACKUP=
CONDA_DEFAULT_ENV=py27
# packages in environment at /home/tru/.conda/envs/py27:
#
ca-certificates 2017.08.26 h1d4fec5_0
certifi 2017.11.5 py27h71e7faf_0
libedit 3.1 heed3624_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 7.2.0 h7cc24e2_2
libstdcxx-ng 7.2.0 h7a57d05_2
ncurses 6.0 h9df7e31_2
openssl 1.0.2m h26d622b_1
pip 9.0.1 py27ha730c48_4
python 2.7.14 hdd48546_24
readline 7.0 ha6073c6_4
setuptools 36.5.0 py27h68b189e_0
sqlite 3.20.1 hb898158_2
tk 8.6.7 hc745277_3
wheel 0.30.0 py27h2bc6bb2_1
zlib 1.2.11 ha838bed_2
[tru@elitebook840g3 singularity-docker-bioconda-forge]$

If you need your "activate" script to provide an interactive shell, you might need to spawn another shell:

[tru@elitebook840g3 singularity-docker-bioconda-forge]$ cat activate
#!/bin/sh
singularity exec /home/tru/git/truatpasteurdotfr/singularity-docker-bioconda-forge/py27 bash -c "source activate $@; bash"

[tru@elitebook840g3 singularity-docker-bioconda-forge]$ ./activate py27
Singularity.py27> tru@elitebook840g3:~/git/truatpasteurdotfr/singularity-docker-bioconda-forge$ printenv|grep CONDA
ANACONDA_HOME=/opt/conda
CONDA_PATH_BACKUP=/opt/conda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CONDA_PREFIX=/home/tru/.conda/envs/py27
CONDA_PS1_BACKUP=
CONDA_DEFAULT_ENV=py27
Singularity.py27> tru@elitebook840g3:~/git/truatpasteurdotfr/singularity-docker-bioconda-forge$

Cheers

Tru


--
Dr Tru Huynh | mailto:t...@pasteur.fr | tel/fax +33 1 45 68 87 37/19
https://research.pasteur.fr/en/team/structural-bioinformatics/
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France

[Rémy Dernat]

You could try it first on your laptop and then transfer it on the cluster. That is one aim of singularity.

Concerning your problem with the home directory, you can use either global options in singularity.conf to avoid to share the home between the host and the container or use another options at runtime (eg --contain or -H).

Rémy.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Laurent Fournier]

Amazing, directly from the creator of the recipe i use  ;-)

laurent@NanoStation:~/atWork/Singularity$ singularity exec py27 bash -c 'source activate py27; printenv|grep CONDA; conda list'

ANACONDA_HOME=/opt/conda
CONDA_PATH_BACKUP=/opt/conda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CONDA_PREFIX=/home/laurent/.conda/envs/py27
CONDA_PS1_BACKUP=
CONDA_DEFAULT_ENV=py27
# packages in environment at /home/laurent/.conda/envs/py27:
#
ca-certificates           2017.11.5                     0    conda-forge
certifi                   2017.11.5                py27_0    conda-forge
ncurses                   5.9                          10    conda-forge
openssl                   1.0.2m                        0    conda-forge
pip                       9.0.1                    py27_0    conda-forge
python                    2.7.14                        1    conda-forge
readline                  6.2                           0    conda-forge
setuptools                37.0.0                   py27_0    conda-forge
sqlite                    3.13.0                        1    conda-forge
tk                        8.5.19                        2    conda-forge
wheel                     0.30.0                     py_1    conda-forge
zlib                      1.2.11                        0    conda-forge

And :

laurent@NanoStation:~/atWork/Singularity$ cat activate

#!/bin/sh

singularity exec py27 bash -c "source activate $@; bash"

But when i run ./activate py27 the shell just...open an other shell, nested in the calling one (without the Singularity.py27> in front).

[Laurent Fournier]

Ok, my bad...this was a bashrc overwrite of my prompt !

I works like a charm :-)

Thank you so much !

Best regards,

Laurent.

[Tru Huynh]

On Fri, Dec 01, 2017 at 02:15:16AM -0800, Laurent Fournier wrote:
> Ok, my bad...this was a bashrc overwrite of my prompt !
> I works like a charm :-)
> Thank you so much !

:P
I would suggest using a specific $HOME for your scripts
(avoid to mix your host $HOME/{.bashrc,.condarc,.conda} and singularity based
ones (for channels, envs, ...) with the --home flags, of course ymmv.

[Laurent Fournier]

But so the envs will be shared amongst all the users isn't it ?

I try to set this for a 600+ users community so i'd like to keep it keen and compartimented.

Regards,

Laurent.

[Tru Huynh]

On Fri, Dec 01, 2017 at 02:45:44AM -0800, Laurent Fournier wrote:
> But so the envs will be shared amongst all the users isn't it ?

right, it's that the purpose?

> I try to set this for a 600+ users community so i'd like to keep it keen
> and compartimented.

then I would use a temporary home when "active" and destroy it after.
Just make sure that you don't end up wiping the actual host $HOME :P

d=`mktemp -d /dev/shm/XXXXXXX`
singularity exec -H $d <....>
/bin/rm -rf "$d"

[Laurent Fournier]

Hahah i bed that wouldn't be good.

I'm surounded by scientists not-so-linux-friendly .. this place was a mess when i came so..first, i try to avoid as much redundancies as i can :-) 

eg. the storage cluster is like at 84% of its 1Pb and in all that, i already found 10+Tb of duplicates (thanks to R, the .fq and reference genomes mostly).

I just set a new policy to encourage them to ask me installing on our frontends but just in case it seemed nice to keep the ability for them to avoid dependencies issues, in some cases.

Thanks for the "tmp home" tip anyway.

Good day and thank you again !

Best regards,

Laurent.