Can a singularity container execute programs "outside" of the container?

2,038 views
Skip to first unread message

Jonathan Greenberg

unread,
Jan 6, 2018, 2:47:26 PM1/6/18
to singularity
Weird question, I know, given the whole point of singularity is to avoid this sort of thing.  We are working with a SLURM queue on an HPC that requires singularity containers (fair enough).  My lab does a lot of R processing, and there is a package called "rslurm" that, in theory, send jobs to the queue.  However, this R is containerized so what it is currently attempting to do is send jobs to the containerized SLURM (which does not exist). I'd like it to send commands "up a level" to the level where the container was executed (the user's non-singularity environment) so it can be executed.  Is this possible?  If so, how do I accomplish this?

v

unread,
Jan 6, 2018, 3:46:42 PM1/6/18
to singu...@lbl.gov
Hey Jonathan,

The environment between the container and your host should be seamless, so if you have an "external" job manager, given that the command is on the path when you are inside the container (e.g., if you were to add something like --containall you would limit sharing from the host). It sounds like in your case you want to check the following things:

0. That R and rslurm is found within the container, period
1. That the R in your container is using the R library on the host with the correct RSlurm
2. That no "other" installation of Rslurm in the container (that isn't properly configured with your cluster) is being used instead
3. if it's the case that a container version of Rslurm is being used, that it is configured correctly.

So to start, I would simply do an interactive shell into the container, and start testing. 

singularity shell container.simg

which R
R

>> do some tests here for rslurm

Once you get started if you want to send progress back to the list there are probably enough R and slurm folk on here to help out. After you are confident about the things being used / called, try executing some working command to the container to test it out, eg:

singularity exec container.simg Rscript myscript.R

TLDR: you shouldn't need to send commands "up" one environment level because you are on the correct level - the environments are shared.

Best,

Vanessa

On Sat, Jan 6, 2018 at 11:47 AM, Jonathan Greenberg <jgr...@gmail.com> wrote:
Weird question, I know, given the whole point of singularity is to avoid this sort of thing.  We are working with a SLURM queue on an HPC that requires singularity containers (fair enough).  My lab does a lot of R processing, and there is a package called "rslurm" that, in theory, send jobs to the queue.  However, this R is containerized so what it is currently attempting to do is send jobs to the containerized SLURM (which does not exist). I'd like it to send commands "up a level" to the level where the container was executed (the user's non-singularity environment) so it can be executed.  Is this possible?  If so, how do I accomplish this?

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.



--
Vanessa Villamia Sochat
Stanford University '16

Jonathan Greenberg

unread,
Jan 6, 2018, 4:13:53 PM1/6/18
to singularity
Vanessa:

Thanks!  Is there something I have to do to specify the host environment in a definition when building an image?  How do I specify the path to host commands?

When I "which" squeue/sbatch (which is NOT in the container, rather, on the host running singularity):

which: no sbatch in (/opt/rh/devtoolset-7/root/usr/bin:/opt/rh/devtoolset-7/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
Singularity r-base-3.4.3.simg:/pathto/> 

It is only searching within the container's environment, it appears.

--j
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

v

unread,
Jan 6, 2018, 4:21:27 PM1/6/18
to singu...@lbl.gov
Is the sbatch installed on a filesystem that is different from where you are working? You might need to do something like bind the location directly. Let's say it's in /share/peanuts then try doing:

singularity shell --bind /share/peanuts/bin:/opt container.simg

and then make sure the opt/ path with the executable is on your path.

and you can change the location in the container to where makes sense, or make a folder just for the bind, etc. The above is just for an example! I've done similar things in the past and usually bind to some known location, then just add it to the path in my bash profile so it's found.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

Paolo Di Tommaso

unread,
Jan 7, 2018, 5:03:26 PM1/7/18
to singu...@lbl.gov
You should not invoke squeue/sbatch from within the container. Instead sbatch should run a script which executes your containerised R script. 


p

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

v

unread,
Jan 8, 2018, 3:04:15 PM1/8/18
to singu...@lbl.gov
But it should be ok to call sbatch on a job file within a container, no? I'm thinking of a scenario where you want to pass around submission scripts with a container of interest. For this use case it sounds like the job is being submit from R (with rslurm) that the user wants to provide in a container (and not install on the host) to still use the host sbatch. 

As for what is being submit to run, that would likely be a script in the container too, but care should be taken to ensure that the job launched loads any singularity module, and then equivalently runs the container. So something like:

0. have container with rslurm and some other analysis stuffs (in R?)
1. execute container on host --> uses rslurm to send jobs to host sbatch
2. job file that is executed again calls the container (and loads singularity if necessary)
3. final thing is script in container running on host.

I think, given that the lab doesn't want to (and can) run the submission step from the host, it would save the extra back and forth ness. But if the submission itself also is part of the "reproducible pipeline" then it makes sense for the container to also serve as a means to submit the job. What you would want to do is to provide very detailed instructions for how to do this. For example, the submission step (to slurm --> sbatch on host) might  be done with an exec command, but running the container call the end executable. That way, a user that doesn't have a job manager (or just wants to do the scientific thing) doesn't get confused if the container doesn't work because it doesn't have this external dependency. Does that make sense?

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

Jason Addison

unread,
Jan 23, 2018, 2:24:28 PM1/23/18
to singularity
How should the host command, being run from the container, access its libs?

I'm trying to do something very similar to Jonathan, I think.

I want to run qsub of the host (PBS scheduler) from a script within a container.

Using --bind, as you described, I am able to access the host qsub from within the container, though qsub is not able to find its libraries (ex. libssl.so.10 and, likely, others).

What is the best way to solve this?

--bind the dirs containing the missing libs? Would this create conflicts for stuff within the container?
See if the admins will containerize PBS?
try to install the missing libs within the container?
Something else?

Also, I want to run the host mpirun from scripts within my container. Is there a similar solution?

Background:

Our scripts do a lot of pre/post processing, setup/tear down, environment detection, and automate the PBS requests and MPI launching. We probably could un-automate the PBS request (qsub script and qsub call), though this would be unfortunate. We probably could not un-embed the mpirun call from within the script.

v

unread,
Jan 23, 2018, 2:30:49 PM1/23/18
to singu...@lbl.gov
I think likely you would do good to decouple the job scheduler for the thing actually being run. For example. if someone (with a different submission need / scheduler) wanted to use your software, it would be embedded with your scheduler. On the other hand, if the container is just an executable that can be run by a general scheduler (yours included) this goes much farther.

I'm not sure about the specifics for your software, but this is the approach that I would take. Trying tricks to get the scheduler (or a wrapper around it) working in the container doesn't seem like it would work nicely if moved to a different environment.


--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

Alan Sill

unread,
Jan 23, 2018, 3:14:20 PM1/23/18
to singu...@lbl.gov
Many schedulers (PBS Pro, Grid Engine, for example) have APIs. You could consider communicating with the API rather than trying to execute direct commands, fool with libraries and details of the local implementation, etc.

This would be a good time to get work started on updating the old slurm DRMAAv1 work to v2 (https://github.com/troeger/drmaav2-mock)!

Here is the one for PBS Pro:


And one for Univa’s Grid Engine:


Alan

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.



--
Vanessa Villamia Sochat
Stanford University '16

Alan Sill

unread,
Jan 23, 2018, 3:22:31 PM1/23/18
to singu...@lbl.gov
This might also be interesting:


"This is a Go API based on an open standard (Open Grid Forum DRMAA2) in order to submit and supervise workloads like OS processes, containers, PODs, tasks from a common interface.

It allows to develop and run job workflows in OS processes, and later easily switch to containers running as Cloud Foundry tasks, Docker containers, Grid Engine jobs, etc...

Its main pupose is supporting application developers with an abstraction layer on top of platforms, workload managers, and cluster schedulers, so that they don't require to deal with the underlaying details and differences when only simple operations (like starting a container and waiting until it is finished) are required.

It can be easily integrated in applications which create and execute job workflows."

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.


Jason Addison

unread,
Jan 23, 2018, 3:37:31 PM1/23/18
to singularity
OK. I'll give up on our PBS automation from within the container.

Thanks for the feedback.


v

unread,
Jan 23, 2018, 3:56:25 PM1/23/18
to singu...@lbl.gov
Would you share your recipe / work after? I'm interested to learn about how this works (and I'd bet others are too!)

On Tue, Jan 23, 2018 at 12:37 PM, Jason Addison <jrad...@gmail.com> wrote:
OK. I'll give up on our PBS automation from within the container.

Thanks for the feedback.


--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

Thomas Hartmann

unread,
Jan 24, 2018, 4:05:54 AM1/24/18
to singu...@lbl.gov
Hi Jason,

(not tried myself)
might it work just to bind the host lib paths to a distinct path and
just append it to the container's PATHs?
If there are overlapping libraries in the container and on the host, it
might be feasible to unset/set anew the PATHs to the bind dirs only
before executing the external bin??

Cheers,
Thomas
> --
> You received this message because you are subscribed to the Google
> Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to singularity...@lbl.gov
> <mailto:singularity...@lbl.gov>.

Jason Addison

unread,
Jan 25, 2018, 12:29:56 PM1/25/18
to singularity, thomas....@desy.de
Hi Thomas,

Yes. I think what you've described would be the next thing to try. It would probably be the approach to take to run host commands generally. Though, it also moves one further from the write-once-run-anywhere ideal.

I'll probably abandon attempts to call the host's scheduler from within the container. I'll still need to call the host's mpirun from within the container.

Jason

Message has been deleted

Jeff Kriske

unread,
Jan 31, 2018, 11:51:02 AM1/31/18
to singularity
Hey Alan, be careful with this as it can crash UGE's DRMAA2 implementation even though Daniel works for Univa. Even a simple hello world can wreak havoc on the scheduler at least it did on ours. I haven't tried it again for a while, but watch the DRMAA2 connections closely. Had zero issues with DRMAA1.

-Jeff
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

Reply all
Reply to author
Forward
0 new messages