Singularity Hub: remote build fails to find files in the GitHub repository

[Steven Lakin]

Hello,

My question is regarding how files and paths in the %files section need to be organized to properly build a remote container on Singularity Hub.

I have a GitHub directory structure that looks like this:

root -> containers -> Singularity
root -> data -> databases -> [files to be copied into the image]

My Singularity recipe looks like this for the %files section:

%files
    data/databases/Salmonella.tar.xz /opt
    data/databases/Plasmid.tar.xz /opt
    data/databases/Efaecalis.tar.xz /opt

The SHub builder cannot locate them, however they build fine locally running singularity build from the root repository directory:

Start Time: Sat Mar 24 19:40:29 UTC 2018.
Cloning into '/tmp/tmpi0l3609u'...
Already on 'master'
Your branch is up-to-date with 'origin/master'.
ERROR Return Code 1: b'Using container recipe deffile: /tmp/tmpi0l3609u/containers/Singularity
Building into existing container: /root/build
Using container recipe deffile: Singularity
Sanitizing environment
\x1b[0mAdding base Singularity environment to container
Docker image path: index.docker.io/library/debian:jessie-slim
Cache folder set to /root/.singularity/docker
[1/1] ||----------------------------------| 0.0%
[1/1] |===================================| 100.0%
Exploding layer: sha256:b0568b191983bc2844b2fdb48aeefa72452931bfead0a87e0515bfc602ea3b0c.tar.gz
Exploding layer: sha256:e441ed5dd1c0da8419a8806620561e4874afc56cc83d45f9b5ec10be1a6782ac.tar.gz
Adding files to container
Copying \'data/databases/Salmonella.tar.xz\' to \'/opt\'
/bin/cp: cannot stat \'data/databases/Salmonella.tar.xz\': No such file or directory
\x1b[31mERROR: Failed copying file(s) into container

Any suggestions would be appreciated.

Thanks,

Steven

[v]

Hey Steven,

The Singularity Hub builder is done with an isolated secure build, so it would be worth trying that first on your host. Here are the steps:

# Install branch with secure build
cd /tmp
git clone -b feature-squashbuild-secbuild-2.4.1 https://github.com/cclerget/singularity.git
/tmp/singularity && ./autogen.sh && ./configure --prefix=/usr/local && make && sudo make install && sudo make secbuildimg

Then to build, add the --isolated flag:

sudo singularity build --isolated container.simg Singularity

Let me know if that still works! The general things I would want to debug are:

 - are you in fact sitting in the directory that you think (e.g., is data there, along with subfolders)

 - do different versions of Singularity handle the copying of files (e.g., you are providing a source FILE and destination FOLDER)

 - and please check that you are indeed building the spec file you think you are!

If all else fails, you can use %setup to copy files more easily, or at least debug a bit to figure out what is going on. Hope that helps! It's hard for me to guess because you did not provide any links to the repo or container collection.

Best,

Vanessa

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[v]

Oh, and be sure to use branch for feature-squashbuild-secbuild-2.4.3, the one I posted is old :)

[v]

Ah we found the bug! You will want to ping @cclerget about this on The Sylabs team, he's the expert on the secure build. It is probably something easy to fix, fingers crossed!

On Sat, Mar 24, 2018 at 9:18 PM, Steven Lakin <steven....@gmail.com> wrote:

Hi Vanessa,

Thanks for your quick response.  I believe you are correct that the isolated environment option is what is causing issues; after reinstalling singularity as you instructed, the build gives the same error locally as on Singularity Hub:

sudo singularity build --isolated containers/recipes/container.simg containers/Singularity
Using container recipe deffile: containers/Singularity

Building into existing container: /root/build
Using container recipe deffile: Singularity
Sanitizing environment

Adding base Singularity environment to container
Docker image path: index.docker.io/library/debian:jessie-slim
Cache folder set to /root/.singularity/docker

[1/1] |===================================| 100.0%

Adding files to container
Copying 'data/databases/Salmonella.tar.xz' to '/opt'
/bin/cp: cannot stat 'data/databases/Salmonella.tar.xz': No such file or directory

ERROR: Failed copying file(s) into container

Without the isolated flag, the build proceeds without error:




sudo singularity build containers/recipes/bear-wgs.simg containers/Singularity
Using container recipe deffile: containers/Singularity
Sanitizing environment

Adding base Singularity environment to container
Docker image path: index.docker.io/library/debian:jessie-slim
Cache folder set to /root/.singularity/docker

Adding files to container
Copying 'data/databases/Salmonella.tar.xz' to '/opt'
Copying 'data/databases/Plasmid.tar.xz' to '/opt'
Copying 'data/databases/Efaecalis.tar.xz' to '/opt'
Running post scriptlet


I verified the build was performed from the repository root directory and that the file paths are in the correct format for the %files section.  Unfortunately I can't share the repository since it is being developed privately for government use for now.

My question is how this isolated environment handles the GitHub repository file paths after it clones from the master branch; it looks like it's creating its own build directory that is not the same as the root directory in the cloned repository, which means my relative pathing isn't helpful.  Is there another way to reference these files for the isolated environment?

Thanks,

Steven

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Cédric Clerget]

Hi Steven,

With SHub builder, all files you want to copy inside container must be in same folder than Singularity file, for your case, if you have :

root -> containers -> Singularity

data must be in containers folder like this :

root -> containers -> data -> databases

Best,
Cédric

[Steven Lakin]

Thanks everyone; that solved my problem.

Steven