No space left on device

[Chris Watkins]

Hey Team,

I am trying to run a Docker container directly in Singularity using:

singularity run docker://conatiner:tag

But when the image is being built I get a whole bunch of 

Cannot mkdir: No space left on device

errors. What is the typical cause of such an error? Is it Singularity or Docker?

Thanks in advance.

Chris

[Chris Watkins]

P.S. I just read this on the "private repo on DockerHub" thread,

I was testing this out again, and it's interesting that the tar error happens when you run / shell into the image, but not if you do a standard create --> import, or pull, even followed by shell. I think this hints at maybe it being related to the creation of a "on demand / temporary" container vs an actual one, which I know Greg was working on recently.

I would say my error is very related as I have seen exactly the same behaviour for both run and shell.

[vanessa s]

Hey Chris,

I usually see this error when the container I make (with create or pull) isn't large enough, and you are essentially trying to dump things into a container that isn't big enough. It's funny that you get it with run, that suggests that the temporary location isn't large enough for the layers being extracted. Can you try first creating the image (with create and --size to be very large) and see if that can successfully create the image? Then we would want to run --debug with run and see where the error originates. Then if we need to test beyond that, if you want to share the image name/tag we can test it out!

Best,

Vanessa

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[vanessa s]

oh and just seeing your second message - +1 that we need the wisdom of @gmk to comment on changes that were done for shell/run!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Chris Watkins]

Yeah,

So doing a create and then import seems to fix that build error.

So I would have to agree, that it might have something to do with the run and shell commands not creating a large enough image to begin with. Although as you say, I have no idea what is actually going on behind the scenes there.

Thanks for your help

Chris

[Gregory M. Kurtzer]

On Mon, May 29, 2017 at 11:52 PM, Chris Watkins <christoph...@me.com> wrote:

Yeah,

So doing a create and then import seems to fix that build error.

So I would have to agree, that it might have something to do with the run and shell commands not creating a large enough image to begin with. Although as you say, I have no idea what is actually going on behind the scenes there.

Technically, the action commands (run, shell, exec, test) do not create an image to run in, and instead they create a temporary container root directory by default in `/tmp/`. How much space do you have in tmp? (`df -h /tmp`)

Greg

 

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Chris Watkins]

Ooooo yer,

Filesystem                 Size  Used Avail Use% Mounted on

/dev/mapper/local_vg0-tmp  4.6G  4.6G     0 100% /tmp

I think the container I am trying build is around 10GB or so.

Thanks

On Tuesday, May 30, 2017 at 3:41:44 PM UTC+10, Chris Watkins wrote:

[vanessa s]

That's one fat container. :O

--

You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Chris Watkins]

Yeah,

It is a neural network model that contains all of the pre-trained weights. So I think they are quite large.

Does anyone know how to change that temp location?

I have also now encountered a problem with the singularity cache location when trying to import on another system. Can I change that somehow too?

My container is laarrrggee :-(

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[vanessa s]

Hey Chris,

Yes! Change the environment variable SINGULARITY_CACHEDIR to somewhere else, or set SINGULARITY_DISABLE_CACHE to Yes or True and it will use a temporary directory. You can see how all these variables get found and set here --> https://github.com/singularityware/singularity/blob/adf5259ac94d7b213c649f073dcda32459b0c8b7/libexec/python/defaults.py#L99 if you are interested!

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Chris Watkins]

Hi team,

Forgot to update.

That seems to have worked!

Now I can pull my container from DockerHub and run in an HPC environment without access to root.

Thanks a lot for your help.

Chris

[vanessa s]

Awesome! Glad to hear it :)

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.