Singularity container + launching parallel jobs onto computing grid

[Mauricio Losilla]

Hello, 

I hope you are well. I have been running Trinity v2.11.0 from your singularity container. It runs very smoothly, thank you for this. 

I decided to try HPC GridRunner to launch butterfly jobs in parallel. This I could not get to work on our HPC (SLURM submission system). Although HPC GridRunner is not included in the singularity container (or I didn't find it) I installed it in our system and it seems to me that it communicates fine with Trinity. However, I wasn't able to submit any parallel jobs. 

After a few tests (without HPC GridRunner) it seems to me that I can't submit jobs from within the singularity container (I can submit jobs outside the container). I was wondering if there is a fix or workaround for this? Or maybe there is a setting I missed... I have never done this before. 

On a side note, I think that your Docker/Singularity build file (https://github.com/trinityrnaseq/trinityrnaseq/blob/master/Docker/Dockerfile) has two sets of instructions to install RSEM: lines 94-103 & 153-162

Thank you
Mau

[Brian Haas]

Hi Mau,

The singularity / HPC gridrunner compatibility isn't something that we've integrated but it's been on our unofficial todo list.  I just added it as a github issue so we'll tackle it for the next release.

Thx for pointing out the rsem redundant installation in the docker.  I'll take care of this too.

best,

~b

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/978d247d-f670-4352-88f3-1f91abd90efan%40googlegroups.com.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

[Mauricio Losilla]

Hi Brian,

Thank you. To clarify, my problem was that I couldn't submit any jobs from within the singularity container, independent of Trinity and HPC GridRunner. The problem is at the singularity - HPC level.

Thanks
Mau

[Brian Haas]

Right.  We probably won't have it submit from within singularity, but instead break it down into 3 steps.

1.  run singularity up to the point of the Trinity phase 2

2.  launch the Trinity phase 2 commands on the grid, but have those commands include running singularity at runtime.

3.  rerun the step (1) with the --FORCE option set so that it just collects all results and generates the final outputs (this requires that all (2) jobs complete successfully.

This process could be wrapped into another shell script or something so that it's one launch instead of 3 separate ones, but for now, just doing it in 3 separate steps should work ok with minimal effort.

To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/6d151aea-9ad1-46ef-b690-b499c6133892n%40googlegroups.com.