copy a whole folder inside container

[M Noon]

Hi Vanessa, 

I know that we can copy individual files into a container in two ways:

# Both of the below are copied before %post
# 1. This is how to copy files for legacy < 2.3
%setup
    touch ${SINGULARITY_ROOTFS}/tacos.txt
    touch avocados.txt

# 2. This is how to copy files for >= 2.3
%files
    avocados.txt 

avocados.txt /opt 

I am using singularity 2.4 therefore, I am trying to using %files for this purpose. How can I copy a whole folder? I couldn't find an example anywhere. Can you please help me on this?

Regards,

M

[v]

Heyo!

I'm away from keyboard at the moment so I will need to get back to test this, but I think there are a few things that you can do. The easiest thing is in setup to just explicitly through the copy of the entire folder:

cp -R /path/host/folder $SINGULARITY_ROOTFS

And then the folder would be available at the root of the filesystem, and you could optionally name it something else. Using the files section you are also using the copy command but you would probably want to try something like this:

%files

/path/host/folder /folder

But if I remember correctly I had issues with this particular Syntax for some reason. You could also try removing the destination to see if that would work. My recommendation is to use set up in the first approach so you have more control.

I'm out adventuring I'll be back in a little bit and I can give you a more definitive answer!

Best,

Vanessa

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[v]

Heyo! Okay here is a more robust and complete answer. I'll describe this in two ways - the first is using %setup (my preference for control, interacts with host filesystem) and the second is using %files (still interacts with host, but relative to container, and less under your control.

Example Files

Here is how I prepared this example, creating the directory /tmp/data and saving the recipes to each of Singularity.setup and Singularity.files (shown below):

cd /tmp
touch /tmp/data/file1.txt
touch /tmp/data/file2.txt

our goal is to copy the directory data (in entirety) into our container. First, here is the setup recipe:

Setup

Bootstrap: docker
From: ubuntu:14.04

 

%setup
    # Copy and rename
    cp -R /tmp/data ${SINGULARITY_ROOTFS}/renamed
    # Copy to root with name "data"
    cp -R /tmp/data ${SINGULARITY_ROOTFS}
%post
    echo "Here is a listing at the root /"
    ls /
    echo "Here is a listing at /renamed"
    ls /renamed
    echo "Here is the content of /data"
    ls -1 /data

to build:

sudo singularity build setup.simg Singularity.setup

the output is the following - we can see that all three cases of what we did above we achieved what we wanted (it's up to you which is your preference). 

...
+ cp -R /tmp/data /usr/local/var/singularity/mnt/container/renamed
+ cp -R /tmp/data /usr/local/var/singularity/mnt/container
Running post scriptlet

 

+ echo Here is a listing at the root / 

Here is a listing at the root /    <--- note under this listing we see both renamed and data
+ ls /
bin   dev    home   media  proc   run        srv  usr
boot  environment  lib   mnt renamed  sbin        sys  var
data  etc    lib64  opt root   singularity  tmp
+ echo Here is a listing at /renamed

 

Here is a listing at /renamed
+ ls /renamed
file1.txt  file2.txt

 

+ echo Here is the content of /data
Here is the content of /data
+ ls /data
file1.txt  file2.txt

Files

The cleanest option (with just files) is to do something like this:

%files
    /tmp/data /opt

This will copy to /opt/data

Here is a listing at /opt

+ ls /opt

data

+ echo Here is a listing at /opt/data

+ ls /opt/data

file1.txt  file2.txt

You can also rename as we did before: 

 

%files
    /tmp/data /opt/custom

Here is a listing at /opt

+ ls /opt

custom

Here is a listing at /opt/custom

+ ls /opt/custom

file1.txt  file2.txt

If you need to move it to a place that doesn't exist when the container is fresh, you need to make the path in setup first.

%setup
 mkdir -p /opt/custom/more/data

And of course use mkdir -p because if you don't and build again, you will get an error that it exists.

Careful Here!

Finally, where I've seen weirdness is with respect to folders and paths that already exist. This should technically work:

%files
    /tmp/data
%post
    echo "Here is a tree at /tmp/data"
    tree /tmp/data

I would expect /tmp/data to be copied as /tmp/data. But it already exists (/tmp is bound, so data is there) so  I get this error:

Copying '/tmp/data' to '/tmp/data'
/bin/cp: cannot copy a directory, '/tmp/data', into itself, '/usr/local/var/singularity/mnt/container//tmp/data/data'
ERROR: Failed copying file(s) into container

And there are many derivations of that that pop up when you mess around with different binds, and also using a * or not. I just messed around a few times, and I got some crazy recursion going on:

 ls /tmp/data/data/data/
file1.txt  file2.txt

(and there are file1.txt and file2.txt in each of the data folders!)

!!!

A Third Strategy!

This isn't covered by %files or %setup, but what I like to do is to get data that should go in the container from some nice version controlled place, usually github :) This comes down to just cloning it (or wget, your choice) in the %post section:

%post
wget https://...
cd /opt && git clone https://...

This way, you can throw your recipe around, it can lose the folder it was in (with the dependencies on the host) and still have a good chance of working unless the files hosted on Github or elsewhere went away. It's also less burdensome to keep the data / recipe packaged together. Of course it's understandable if the files are too big for version control / download. but in this case I would question if you want to add them to the container, period. Big data like that might arguably be better to be mounted at runtime.

Summary

%files

The approach that I think is most sound using %files is to copy a single entire folder or files to the root, without specifying any directory paths already mounted that could lead to unexpected results and/or bugs.  Then do advanced manipulation of that single copy (folder or file) in %post

%setup

OR just doing everything in %setup where you have better control, relative to $SINGULARITY_ROOTFS.

Good practice

It's good practice to test your work, and never assume that things are what you think they should be. The derivation that you choose above is probably going to depend on your use case, and I definitely forget which of the derivations above work / don't work for the use case! So what to do? Just do some quick testing / sanity checks with prints and what not to be absolutely sure that your strategy is working.

Hope that helps!

Best,

Vanessa

%files
    /tmp/data
    /tmp/data /renamed
    /tmp/data/* /renamed

The first will (sort of) work because /tmp/data will go to /tmp/data in the container.

Now we can do the same trying to use the files section! Here is our recipe:

 

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[v]

Ignore the last section, it's a Vanessa-Copy-Paste-Fail :)

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Rutu Pandya]

Hi Vanessa,

I have bootstrapped a container from docker and have installed some more dependencies. Do not have a recipe file for it. Is it possible to copy a folder from the host into this already existing container? 

Would I need to create a recipe file for the container that includes building it in it's entirety or can I just use a cp command to copy the files and folders in an existing container?

Thanks!

Rutu

[v]

Hi Rutu,

You should generally write a recipe file. Your build, as it stands now without it, is not reproducible. Your future self (and others that might use the container) thank you for it.

Best,

Vanessa