Mounting to Singularity container invoked by snakemake on HPC cluster

Dean Pettinga

unread,

Jun 4, 2019, 10:44:23 AM6/4/19

to singularity

I'm currently trying to implement a snakemake workflow on a HPC cluster using singularity containers for specific rules.

I'm having issues when mounting to the singularity container invoked by snakemake.

$ cd /path/to/smk_pipeline
$ snakemake --use-singularity
ERROR  : Failed to mount home directory /path/to/smk_pipeline -> /cm/local/apps/singularity/2.4.2/var/singularity/mnt/session/path/to/smk_pipeline: Permission denied

The rule being run invokes a container hosted at singularity-hub, but it errors out before it's able to do anything.

If i shell into a container interactively, no such problem occurs.

$ singularity shell rnaseq.simg
Singularity: Invoking an interactive shell within container...
Singularity rnaseq.simg:~>

I think it might be due to the fact that write permissions on /cm/local/apps/singularity/2.4.2/var/singularity/mnt/ are set to 0755.

One of our HPC admins suggested that I try to use a different mount point for singularity, but I haven't been able to find any information on this topic in the documentation...

I'm confused about why singularity is unable to mount any directories when invoked by snakemake, rather than by myself directly as a user. This may be a question more fit for the snakemake folks, but i thought i'd start here.

cheers,

Dean

v

unread,

Jun 4, 2019, 11:01:32 AM6/4/19

to singu...@lbl.gov

I've used Snakemake + Singularity (without the official --with-singularity) flag, here is the repo if it's helpful: https://github.com/sci-f/snakemake.scif

A few things to try:

- I would first shell into the container (that works) and see what binds you have, by default. If your cluster disabled some set of mounts, that could be the issue, or you might need to ask for them.

- I'm looking at the Singularity command group, and I wonder if you can try setting --singularity-args to make binds to your system that are needed? It does seem like something might be up with the session folder, others can comment too :)

- If your pipeline doesn't have needs beyond singularity, you could also try the approach I linked above, adding the snakemake deps into a container first, and then running it.

- Finally, I would recommend pulling an image first and then setting --singularity-dir to where it is, that way you can reuse the same image and not be pulling a gazillion if you run this in parallel.

Best,

Vanessa

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/4e3f0796-dbb8-48ac-a786-28ee3ff9f7f8%40lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

Dean Pettinga

unread,

Jun 4, 2019, 4:52:41 PM6/4/19

to singularity

Hey Vanessa,

to answer those few things:

- I'm able to shell into the container and bind anything i want with -B option

- I've been using --singularity-args in an attempt to bind, but thus far, it has not worked, leading me to believe it might be an issue to do with snakemake or the session folder.

- haven't looked at sci-f too closely, but i don't think it's going to be helpful for my situation.

- My pipeline uses at most 8-16 parallel processes, so i'm planning to pull the s-hub container as i'm hoping to make this easily reproducible for other users.

I'm still at a bit of an impasse...

-Dean

To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/4e3f0796-dbb8-48ac-a786-28ee3ff9f7f8%40lbl.gov.

v

unread,

Jun 4, 2019, 5:20:46 PM6/4/19

to singu...@lbl.gov

Can you try running with debug? I'm looking at the Snakemake source code and I see a few things. First, snakemake is creating a mountpoint for itself:

SNAKEMAKE_MOUNTPOINT = "/mnt/snakemake"

which is added as a bind to the container (and I haven't looked further but I'm guessing things like the Snakefile are bound here) 

args += " --bind {}:{}".format(SNAKEMAKE_SEARCHPATH, SNAKEMAKE_MOUNTPOINT)

It might be good to run a test and just list what is included that. What I'm also noticing for the shellcmd function (possibly this is used when you run a singularity command, since it uses exec despite it's name) is that the os.getcwd() is always bound as --home

    cmd = "{} singularity exec --home {} {} {} {} -c '{}'".format(
        envvars, os.getcwd(), args, img_path, shell_executable,
        cmd.replace("'", r"'\''"))
    logger.debug(cmd)

Are you working in a folder that is a subset of your actual home? It seems to be having trouble with that particular bind. Either way, if you might be able to run in debug so we see any commands that are printed? The full code is here, for those interested:

https://bitbucket.org/snakemake/snakemake/src/master/snakemake/singularity.py

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/ece8a25d-5717-496b-a8ef-aa60cca30391%40lbl.gov.

Dean Pettinga

unread,

Jun 10, 2019, 4:36:52 PM6/10/19

to singularity

Hey Vanessa,

I've run in debug mode, e.g.

snakemake --use-singularity --debug

i believe the relevant info from the output is below:

RuleException:
CalledProcessError in line 141 of /secondary/projects/bbc/research/smk_pipeline/Snakefile:
Command ' singularity exec --home /secondary/projects/bbc/research/smk_pipeline  --bind /secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/site-packages:/mnt/snakemake /secondary/projects/bbc/research/smk_pipeline/.snakemake/singularity/2f95debcc77a32fc723cdf8a4ad18fa2.simg bash -c 'set -euo pipefail;  Rscript -e '\''rmarkdown::render("scripts/edgeR_longReport.Rmd",output_format="html_document")'\'''' returned non-zero exit status 255.
  File "/secondary/projects/bbc/research/smk_pipeline/Snakefile", line 141, in __rule_edgeR_longReport
  File "/secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run

I am working in a subdirectory of my $HOME directory. It seems like this may be the root of the issue.

-Dean

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/ece8a25d-5717-496b-a8ef-aa60cca30391%40lbl.gov.

v

unread,

Jun 10, 2019, 5:34:11 PM6/10/19

to singu...@lbl.gov

And the final error is a permission denied? Snakemake is helping us a lot by providing the command that fails, so let's start by determining if it's the --home bind (or not). Try the same command, but remove the bind to the custom home:

singularity exec --bind /secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/site-packages:/mnt/snakemake /secondary/projects/bbc/research/smk_pipeline/.snakemake/singularity/2f95debcc77a32fc723cdf8a4ad18fa2.simg bash -c 'set -euo pipefail; Rscript -e '\''rmarkdown::render("scripts/edgeR_longReport.Rmd",output_format="html_document")'\'

If you are already sitting in your home, it's redundant anyway. And to simplify things a bit more, just exec a command like an echo or a which:

singularity exec --bind /secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/site-packages:/mnt/snakemake /secondary/projects/bbc/research/smk_pipeline/.snakemake/singularity/2f95debcc77a32fc723cdf8a4ad18fa2.simg bash -c 'echo hello'

And can you show the full output from debug? Snakemake seems to be cching images in the .snakemake folder, and what I want to also understand is why it's binding the Python packages base to /mnt/snakemake (do you know?).

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/1e966ec5-e71d-48bf-9ca4-ba43fd7fece3%40lbl.gov.

Dean Pettinga

unread,

Jun 11, 2019, 10:46:03 AM6/11/19

to singularity

ok. so here's a few more things:

exec command without the mount:

[dean.pettinga@node065 smk_pipeline]$ singularity exec /secondary/projects/bbc/research/smk_pipeline/.snakemake/singularity/2f95debcc77a32fc723cdf8a4ad18fa2.simg bash -c 'echo hello'
hello

with the mount:

[dean.pettinga@node065 smk_pipeline]$ singularity exec  --bind /secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/site-packages:/mnt/snakemake /secondary/projects/bbc/research/smk_pipeline/.snakemake/singularity/2f95debcc77a32fc723cdf8a4ad18fa2.simg bash -c 'echo hello'
ERROR  : There was an error binding the path /secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/site-packages: Permission denied
ABORT  : Retval = 255

different mount:

[dean.pettinga@node065 smk_pipeline]$ singularity exec --bind /secondary .snakemake/singularity/2f95debcc77a32fc723cdf8a4ad18fa2.simg bash -c 'echo hello'
hello

Full output from debug:

[dean.pettinga@node065 smk_pipeline]$ snakemake --use-singularity --debug --singularity-args '-B /secondary,/primary'
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
       count   jobs
       1       all
       1       edgeR_longReport
       1       edgeR_shortReport
       3

[Mon Jun 10 16:34:02 2019]
rule edgeR_longReport:
    input: deliverables/UniquelyMappingRates.txt, deliverables/UniquelyMappingReads.txt, deliverables/starMatrix.txt
    output: r_objects/topGenes.LPS_vs_PBS.rds, r_objects/topGenes.IL4_vs_PBS.rds, r_objects/topGenes.LPS_vs_IL4.rds, r_objects/volcano.LPS_vs_PBS.rds, r_objects/volcano.IL4_vs_PBS.rds, r_objects/volcano.LPS_vs_IL4.rds, r_objects/contrastMatrix.LPS_vs_PBS.rds, r_objects/contrastMatrix.IL4_vs_PBS.rds, r_objects/contrastMatrix.LPS_vs_IL4.rds, r_objects/pca_plot.rds, r_objects/var_plot.rds, r_objects/heatmap.rds, r_objects/DGE_design.rds, deliverables/LPS_vs_PBS.txt, deliverables/IL4_vs_PBS.txt, deliverables/LPS_vs_IL4.txt, deliverables/LPS_vs_PBS.cls, deliverables/IL4_vs_PBS.cls, deliverables/LPS_vs_IL4.cls, deliverables/LPS_vs_PBS.gct, deliverables/IL4_vs_PBS.gct, deliverables/LPS_vs_IL4.gct, deliverables/edgeR_longReport.html
    jobid: 54

Activating singularity image /secondary/projects/bbc/research/smk_pipeline/.snakemake/singularity/2f95debcc77a32fc723cdf8a4ad18fa2.simg
ERROR  : Failed to mount home directory /secondary/projects/bbc/research/smk_pipeline -> /cm/local/apps/singularity/2.4.2/var/singularity/mnt/session/secondary/projects/bbc/research/smk_pipeline: Permission denied
ABORT  : Retval = 255
[Mon Jun 10 16:34:02 2019]
Error in rule edgeR_longReport:
    jobid: 54
    output: r_objects/topGenes.LPS_vs_PBS.rds, r_objects/topGenes.IL4_vs_PBS.rds, r_objects/topGenes.LPS_vs_IL4.rds, r_objects/volcano.LPS_vs_PBS.rds, r_objects/volcano.IL4_vs_PBS.rds, r_objects/volcano.LPS_vs_IL4.rds, r_objects/contrastMatrix.LPS_vs_PBS.rds, r_objects/contrastMatrix.IL4_vs_PBS.rds, r_objects/contrastMatrix.LPS_vs_IL4.rds, r_objects/pca_plot.rds, r_objects/var_plot.rds, r_objects/heatmap.rds, r_objects/DGE_design.rds, deliverables/LPS_vs_PBS.txt, deliverables/IL4_vs_PBS.txt, deliverables/LPS_vs_IL4.txt, deliverables/LPS_vs_PBS.cls, deliverables/IL4_vs_PBS.cls, deliverables/LPS_vs_IL4.cls, deliverables/LPS_vs_PBS.gct, deliverables/IL4_vs_PBS.gct, deliverables/LPS_vs_IL4.gct, deliverables/edgeR_longReport.html


RuleException:
CalledProcessError in line 141 of /secondary/projects/bbc/research/smk_pipeline/Snakefile:

Command ' singularity exec --home /secondary/projects/bbc/research/smk_pipeline -B /secondary,/primary --bind /secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/site-packages:/mnt/snakemake /secondary/projects/bbc/research/smk_pipeline/.snakemake/singularity/2f95debcc77a32fc723cdf8a4ad18fa2.simg bash -c 'set -euo pipefail;  Rscript -e '\''rmarkdown::render("scripts/edgeR_longReport.Rmd",output_format="html_document")'\'''' returned non-zero exit status 255.

  File "/secondary/projects/bbc/research/smk_pipeline/Snakefile", line 141, in __rule_edgeR_longReport
  File "/secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /secondary/projects/bbc/research/smk_pipeline/.snakemake/log/2019-06-10T163401.115637.snakemake.log

permissions for /secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/site-packages: drwxr-xr-x

don't understand why it would need write permissions...

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/1e966ec5-e71d-48bf-9ca4-ba43fd7fece3%40lbl.gov.

v

unread,

Jun 11, 2019, 11:22:48 AM6/11/19

to singu...@lbl.gov

Can you run this:

$ singularity exec --bind /secondary/projects/bbc/tools/dean_miniconda3/lib/python3.6/site-packages:/mnt/snakemake /secondary/projects/bbc/research/smk_pipeline/.snakemake/singularity/2f95debcc77a32fc723cdf8a4ad18fa2.simg bash -c 'echo hello'

With --debug for singularity? Is there a simple dummy/toy example you can give me to reproduce on my local machine? That would be a lot easier / faster to figure out, if you are able!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/c8bc281b-1cf6-4bde-92f0-d847adf0ddb5%40lbl.gov.

Dean Pettinga

unread,

Sep 22, 2019, 4:09:01 PM9/22/19

to singularity

I spoke with my HPC admin and spent some time working through this permissions issue. it appears to be a problem with inherited permissions and POSIX/ACL cross-talk. Since the cluster-wide installation of Singularity is pretty old (2.4.2), we tested Singularity 3.4, which appears to solve this issue.

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/c8bc281b-1cf6-4bde-92f0-d847adf0ddb5%40lbl.gov.

Reply all

Reply to author

Forward