Re: [Singularity] copying/moving container files to container directory in post section

[v]

Hey Simone!  

Could you give something to look at? Possibly what exists on the host plus your recipe file, and (if possible) a small toy example to play with. I'd be happy to help! These bugs and file paths... like peanut butter and jelly.

Best,

Vanessa

On Tue, Jan 9, 2018 at 2:22 AM, Simone Riggi <simone...@gmail.com> wrote:

Dear all,

I am creating a container in which I installed some libraries in the %post section. I am having troubles to copy files present in the container (source files) to an installation directory created in the container. NB: I do not want to copy files to/from my host system (I have seen the %files section is for that). Basically I have just this statement in %post:

cp $SOURCE_FILE $DEST_DIR

The cp (or mv) command seems ineffective because the file is not present in the desired directory after copying. Both file and destination dir exists.
I am sure I am doing some trivial mistake here.

Can you help me?

Many thanks

PS: Using latest Singularity version from git repo.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Simone Riggi]

Dear Vanessa,

thanks for your answer. I did some checks yesterday and I think I found my mistake. The destination directory was actually a file. The new version of a library I installed (compared to the previous) has changed the install directory layout in the new release and I have messed up things. I am able to copy file now. 

In case you find the recipe file at: https://github.com/simoneriggi/caesar_base.git

For some reasons I need to investigate why my builds on shub are always failing but locally I am able to build it. 

Thanks for your support,

Regards,

Simone 

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[v]

Hey Simone,

I got this one! There is a problem with the older builder you are using, and I need to deprecate the image. Give me a few minutes and I'll ping you with an update (and I see you started another, just let it be for now I'll take care of restarting it with an updated builder).

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[v]

Hey Simone,

I've updated the builder - would you mind triggering a build with your preferred method (either commit or manual) and then I'll be able to check on progress from there? Thanks!

Best,

Vanessa

[Simone Riggi]

Dear Vanessa,

thanks. I did yesterday a triggered build but it failed with this output:

Start Time: Wed Jan 10 22:38:06 UTC 2018.
Cloning into '/tmp/tmpnew0xsae'...
Already on 'master'
Your branch is up-to-date with 'origin/master'.
Return value of 137.
Killed: Thu Jan 11 00:38:06 UTC 2018.

How can I get more logs to understand what is going wrong? In my laptop I am able to build the container (around 1 GB in size at the end).

In case does shub offer a way to upload already build containers?

Thanks again,

Cheers,

Simone

[v]

Hey Simone,

Apologies for the delay - just getting back from a brief hiatus from my laptop and still sort of out of it! Could you point me again to the collection and repo so I can test this manually? What is happening is that the build seems to be timing out. (The kill happens after too much inactivity). I don't think it's an issue with your container, likely something to do with the updated Singularity or builder. Walking through the steps manually (I should be able to allocate some time over this weekend) should be a good first start I think.

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Simone Riggi]

Hi Vanessa,

no problem at all for delay. I send you the collection and github repo:

collection: https://www.singularity-hub.org/collections/411

container recipe: https://github.com/simoneriggi/caesar_base.git 

Many thanks for your support,

Regards,

Simone

[v]

Hey Simone!

I think what is going on is what the error message indicates - it times out and then gets killed. I want to suggest a different strategy - create a base image that is served on Docker Hub, and then use *that* as a base for the Singularity Hub build. Also make sure that you turn on debug mode in your Builder Settings to get as much output as we can. Is that something we can try?

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Simone Riggi]

Dear Vanessa,

thanks for your help. 

Not sure I understood correctly what you suggest. Is it possible to have singularity container being built in docker hub? Or to do that I need to re-write my singularity base recipe file to docker one? Hope this is not the case.

If I may ask, did you figure out why the build times out? Is it because there is a maximum time allowed for each build? If so, can this be increased? My container base has longer build time than normal because it build from sources some tools/libraries not available/suitable yet as system packages.

I also tried to turn on the debug mode but I was not able to actually see a verbose build output. I will check again.

Another question: is it possible to have singularity container files (built by users) hosted on the SingularityHub (say a maximum amount of space per user)? I mean no pipeline build required, only public storage on shub.

Thanks again,

SImone

[v]

Hey Simone,

Not sure I understood correctly what you suggest. Is it possible to have singularity container being built in docker hub?

You can have a Docker image (no Singularity in sight) that is stored on Docker Hub as tar.gz layers. Then with Singularity you can "pull" from Docker hub to assemble those layers into an image. Try it!

singularity pull docker://godlovedc/lolcow

 

For some use cases, it's easier to maintain one build recipe (a Dockerfile in this case) and then kill two birds with one stone by having both images available.

Or to do that I need to re-write my singularity base recipe file to docker one? Hope this is not the case.

If I may ask, did you figure out why the build times out? Is it because there is a maximum time allowed for each build? If so, can this be increased?

Yes, and unfortunately not at the moment.

 

My container base has longer build time than normal because it build from sources some tools/libraries not available/suitable yet as system packages.

I also tried to turn on the debug mode but I was not able to actually see a verbose build output. I will check again.

Another question: is it possible to have singularity container files (built by users) hosted on the SingularityHub (say a maximum amount of space per user)?

If your group wants to host a singularity registry, then you can push images to it!

https://singularityhub.github.io/sregistry/

 

I mean no pipeline build required, only public storage on shub.

That's a cool idea, but I'd probably need to charge people for that.

 

Thanks again,

:)

 

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.