using Singularity to add Ubuntu to a NPACI Rocks Cluster node

[Robert Kudyba]

I like that the quick start page makes things look simple: http://singularity.lbl.gov/quickstart

But in reality is that all it takes to add an OS to a node? Are there any considerations specific to running with in a cluster like this? Do I just mount an image (.img) file of Ubuntu some where and it's that simple? Can I just rename the Ubuntu .iso to .img? I take it the best practice is not to put it in /tmp as it'll get lost on reboot? Pardon the newbie questions...

[vanessa s]

Hey Robert,

I think it might be that simple? An iso file is not the same as a singularity image, so no you cannot just change the extensions and use simeotaneously. That said, I'm not actually familiar with iso format, and I'll take a look in free time with dinner! In case you are interested, the new quickstart (and all updated docs) are on docs/2.3 branch here:

https://github.com/singularityware/singularityware.github.io/blob/master/pages/docs/user-docs/docs-quick-start-installation.md

To answer your question, you would build an image however you like, and then just plop it on a cluster node, and shell / run / exec commands to it. You wouldn't mount it typically on your cluster, but you can bind your data paths to it with --bind/-B. And yes, don't put stuff in /tmp that you care about :)

Best,

Vanessa

On Thu, May 25, 2017 at 1:06 PM, Robert Kudyba <rku...@fordham.edu> wrote:

I like that the quick start page makes things look simple: http://singularity.lbl.gov/quickstart

But in reality is that all it takes to add an OS to a node? Are there any considerations specific to running with in a cluster like this? Do I just mount an image (.img) file of Ubuntu some where and it's that simple? Can I just rename the Ubuntu .iso to .img? I take it the best practice is not to put it in /tmp as it'll get lost on reboot? Pardon the newbie questions...

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Gregory M. Kurtzer]

Hi Robert,

Additionally to what Vanessa mentioned, you can't rename an .iso file to .img and expect it to work. You have to create the image, bootstrap or import into it, and then you can use it. Here is a more specific quick start to your use case:

git clone -b development https://github.com/singularityware/singularity.git

cd singularity

./autogen.sh

./configure --prefix=/usr/local

make

sudo make install

# now the latest development code is installed, so you can use it:

singularity create /tmp/ubuntu.img

singularity import /tmp/ubuntu.img docker://ubuntu:latest

singularity shell /tmp/ubuntu.img

Hope that helps!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Robert Kudyba]

Additionally to what Vanessa mentioned, you can't rename an .iso file to .img and expect it to work.

I based my hypothesis on these guides:
1) https://askubuntu.com/a/388321
2) http://www.ehow.com/info_8075301_differences-between-iso-img-files.html
3) http://en.wikipedia.org/wiki/IMG_%28file_format%29#Comparison_to_ISO_images

"There is no difference in the structure of ISO and IMG formats if the IMG file is uncompressed. It is possible for an IMG format file to be renamed with the ISO file extension and then opened in software that only recognizes the ISO file format. This is an effective way of accessing disc information in programs that do not handle the IMG format.

 

You have to create the image, bootstrap or import into it, and then you can use it.

What's the difference between bootstrap and import into it? Sorry bootstrap means a few things for me.

 

Here is a more specific quick start to your use case:

git clone -b development https://github.com/singularityware/singularity.git

cd singularity

./autogen.sh

./configure --prefix=/usr/local

make

sudo make install

# now the latest development code is installed, so you can use it:

singularity create /tmp/ubuntu.img

singularity import /tmp/ubuntu.img docker://ubuntu:latest

singularity shell /tmp/ubuntu.img

Hope that helps!

This is great. I take it leaving the Ubuntu image on a DVD would probably be too slow?

[vanessa s]

I based my hypothesis on these guides:
1) https://askubuntu.com/a/388321
2) http://www.ehow.com/info_8075301_differences-between-iso-img-files.html
3) http://en.wikipedia.org/wiki/IMG_%28file_format%29#Comparison_to_ISO_images

"There is no difference in the structure of ISO and IMG formats if the IMG file is uncompressed. It is possible for an IMG format file to be renamed with the ISO file extension and then opened in software that only recognizes the ISO file format. This is an effective way of accessing disc information in programs that do not handle the IMG format.

Ah, understand the confusion!  A singularity .img != a floppy disk style img. This is extension confusion on our part! 

 

You have to create the image, bootstrap or import into it, and then you can use it.

What's the difference between bootstrap and import into it? Sorry bootstrap means a few things for me.

You can think of import as "here is my image, dump layers into it" and bootstrap as a more robust creation, which will also add environment, labels, and a runscript. Bootstrap comes by way of a specification file that you create, and import is a one line command-line thing.

 

Here is a more specific quick start to your use case:

git clone -b development https://github.com/singularityware/singularity.git

cd singularity

./autogen.sh

./configure --prefix=/usr/local

make

sudo make install

# now the latest development code is installed, so you can use it:

singularity create /tmp/ubuntu.img

singularity import /tmp/ubuntu.img docker://ubuntu:latest

singularity shell /tmp/ubuntu.img

Hope that helps!

This is great. I take it leaving the Ubuntu image on a DVD would probably be too slow?

haha, please don't try using DVDs with Singularity - but if you do and figure out something interesting, please share with the list! Your fastest approach yes would be to use the ubuntu (docker) base as Greg has shown above. 

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Gregory M. Kurtzer]

On Fri, May 26, 2017 at 9:39 AM, vanessa s <vso...@gmail.com> wrote:

snip....

 

Ah, understand the confusion!  A singularity .img != a floppy disk style img. This is extension confusion on our part! 

This is a common problem that stems from the Window's world of determining the contents of a particular file based on the suffix. While, I agree it can be confusing, we only suffix a Singularity image with `.img` as a rough descriptor but there is nothing that specifies or requires this extension suffix. You could just as easily do:

$ singularity create /tmp/ubuntu.singularity

But that is kinda long,... perhaps instead:

$ singularity create /tmp/ubuntu.simg

Maybe the latter makes more sense in our documentation to further avoid this confusion.

 

 

You have to create the image, bootstrap or import into it, and then you can use it.

What's the difference between bootstrap and import into it? Sorry bootstrap means a few things for me.

You can think of import as "here is my image, dump layers into it" and bootstrap as a more robust creation, which will also add environment, labels, and a runscript. Bootstrap comes by way of a specification file that you create, and import is a one line command-line thing.

Yep, Vanessa nailed this. You can also think of it as `import` will take a pre-made container and stuff it into a container image, where `bootstrap` will create or rather install the operating system and environment into the container image via a defined recipe (the bootstrap definition file).

Hope that helps!

Greg

[Robert Kudyba]

git clone -b development https://github.com/singularityware/singularity.git

cd singularity

./autogen.sh

./configure --prefix=/usr/local

make

sudo make install

# now the latest development code is installed, so you can use it:

singularity create /tmp/ubuntu.img

singularity import /tmp/ubuntu.img docker://ubuntu:latest

singularity shell /tmp/ubuntu.img

Hope that helps!

So for Rocks I used:

 ./configure --prefix=/export/apps/singularity-install --sysconfdir=/etc

However that causes pathing problems when running the singularity command:

./singularity import /export/apps/ubuntu.simg docker://ubuntu:latest

Docker image path: index.docker.io/library/ubuntu:latest

Cache folder set to /root/.singularity/docker

Importing: base Singularity environment

ERROR  : Session directory does not exist: /export/apps/var/singularity/mnt/session

ABORT  : Retval = 255 

Are there other command line path options to use when running the above command?

[Robert Kudyba]

I think it might be that simple? An iso file is not the same as a singularity image, so no you cannot just change the extensions and use simeotaneously. That said, I'm not actually familiar with iso format, and I'll take a look in free time with dinner! In case you are interested, the new quickstart (and all updated docs) are on docs/2.3 branch here:

https://github.com/singularityware/singularityware.github.io/blob/master/pages/docs/user-docs/docs-quick-start-installation.md

To answer your question, you would build an image however you like, and then just plop it on a cluster node, and shell / run / exec commands to it. You wouldn't mount it typically on your cluster, but you can bind your data paths to it with --bind/-B. And yes, don't put stuff in /tmp that you care about :)

OK I got it going to this point working on the head node:

 singularity shell /export/apps/ubuntu.simg

Singularity: Invoking an interactive shell within container...

Singularity ubuntu.simg:/etc/profile.d> 

But when I go to the compute node: 

./singularity 

Error loading functions: /export/apps/libexec/singularity/functions

So I’m not sure on the usage I guess from a node and how to "shell / run / exec commands to it”.

[vanessa s]

Is singularity installed on the compute node with the same permissions and paths? For our cluster, we have it in a /share location, and then it's loaded via a module.

--

You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Robert Kudyba]

On Jun 5, 2017, at 3:57 PM, vanessa s <vso...@gmail.com> wrote:

Is singularity installed on the compute node with the same permissions and paths? For our cluster, we have it in a /share location, and then it's loaded via a module.

Not exactly per the Rocks manual applications are installed as follows:

Adding Applications to Compute Nodes

If you have code you'd like to share among the compute nodes, but your code isn't in an RPM (or in a roll), then this procedure describes how you can share it with NFS.

On the frontend, go to the directory /export/apps.

# cd /export/apps

Then add the files you'd like to share within this directory.

All files will be available on the compute nodes under: /share/apps. For example:

# cd /export/apps # touch myapp # ssh compute-0-0 # cd /share/apps # ls myapp

[Rick Wagner]

Robert,

That’s an NFS export for the Rocks head node that may be mounted nosuid

—Rick

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Robert Kudyba]

On Jun 5, 2017, at 4:08 PM, Rick Wagner <richard....@gmail.com> wrote:

Robert,

That’s an NFS export for the Rocks head node that may be mounted nosuid

Right in /etc/fstab for other applications we already have this:

UUID=xxxxx /export                 ext4    defaults        1 2

So what else needs to be done?

[Rick Wagner]

One option is to build an RPM using the spec file in the Singularity repo and put that in contrib for your Rocks distro along with a package tag in extend compute. That’s my preference because I don’t like serving applications over NFS, especially ones with setuid.

WARNING: the following will allow any executable with a setuid bit in that NFS file system to be run.

You can try changing defaults to defaults,suid and remounting the file system on a compute node.

—Rick

[Robert Kudyba]

On Jun 5, 2017, at 4:30 PM, Rick Wagner <richard....@gmail.com> wrote:

On Jun 5, 2017, at 4:13 PM, Robert Kudyba <rku...@fordham.edu> wrote:

On Jun 5, 2017, at 4:08 PM, Rick Wagner <richard....@gmail.com> wrote:

Robert,

That’s an NFS export for the Rocks head node that may be mounted nosuid

Right in /etc/fstab for other applications we already have this:

UUID=xxxxx /export                 ext4    defaults        1 2

So what else needs to be done?

One option is to build an RPM using the spec file in the Singularity repo and put that in contrib for your Rocks distro along with a package tag in extend compute. That’s my preference because I don’t like serving applications over NFS, especially ones with setuid.

WARNING: the following will allow any executable with a setuid bit in that NFS file system to be run.

You can try changing defaults to defaults,suid and remounting the file system on a compute node.

Is this an “or” suggestion? I don’t have a problem with changing the /etc/fstab. I did that and ran:

  rocks sync config

  rocks sync users

Also made sure to use the config option as /share/apps (not /export/apps as that’s just for the head node)

Still getting:

/share/apps/bin

./singularity shell /share/apps/ubuntu.simg

[Rémy Dernat]

Hi Robert,

You just have to install singularity on a NFS export; Replace:

```

 ./configure --prefix=/export/apps/singularity-install --sysconfdir=/etc

```

With:

```

 ./configure --prefix=/export/apps/singularity-install

```

Then, it should works as expected.

Do not forget that some singularity commands, like `bootstrap` in 2.3 or many more in previous release need sudo/root privileges.

Best regards,
Rémy

--

You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Robert Kudyba]

OK I did that but still getting the bathing error about the conf file. I did find a previous thread mentioning this https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/mTPunMFQF4k but there’s no solution on how to point to an alternative location, like a -c option. I updated the init file to add /export/apps/singularity-install to the $PATH.

./singularity --debug create /export/apps/ubuntu.simg

Enabling debugging

Ending argument loop

Singularity version: 2.3-master.gda9cd20

Exec'ing: /export/apps/singularity-install/libexec/singularity/cli/create.exec

VERBOSE [U=0,P=17149]      message_init()                            Set messagelevel to: 5

VERBOSE [U=0,P=17149]      singularity_config_parse()                Initialize configuration file: /etc/singularity/singularity.conf

ERROR   [U=0,P=17149]      singularity_config_parse()                Specified configuration file /etc/singularity/singularity.conf does not appear to be a normal file.

ERROR   [U=0,P=17149]      singularity_config_parse()                Could not open configuration file /etc/singularity/singularity.conf: No such file or directory

VERBOSE [U=0,P=17149]      singularity_registry_init()               Initializing Singularity Registry

VERBOSE [U=0,P=17149]      singularity_registry_set()                Adding value to registry: 'WRITABLE' = '1'

DEBUG   [U=0,P=17149]      singularity_registry_set()                Returning singularity_registry_set(WRITABLE, 1) = 0

VERBOSE [U=0,P=17149]      singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/export/apps/singularity-install/libexec'

DEBUG   [U=0,P=17149]      singularity_registry_set()                Returning singularity_registry_set(libexecdir, /export/apps/singularity-install/libexec) = 0

VERBOSE [U=0,P=17149]      singularity_registry_set()                Adding value to registry: 'COMMAND' = 'create'

DEBUG   [U=0,P=17149]      singularity_registry_set()                Returning singularity_registry_set(COMMAND, create) = 0

VERBOSE [U=0,P=17149]      singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'

DEBUG   [U=0,P=17149]      singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0

VERBOSE [U=0,P=17149]      singularity_registry_set()                Adding value to registry: 'VERSION' = '2.3-master.gda9cd20'

DEBUG   [U=0,P=17149]      singularity_registry_set()                Returning singularity_registry_set(version, 2.3-master.gda9cd20) = 0

VERBOSE [U=0,P=17149]      singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/export/apps/singularity-install/var'

DEBUG   [U=0,P=17149]      singularity_registry_set()                Returning singularity_registry_set(localstatedir, /export/apps/singularity-install/var) = 0

VERBOSE [U=0,P=17149]      singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/export/apps/singularity-install/etc'

DEBUG   [U=0,P=17149]      singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /export/apps/singularity-install/etc) = 0

VERBOSE [U=0,P=17149]      singularity_registry_set()                Adding value to registry: 'BINDIR' = '/export/apps/singularity-install/bin'

DEBUG   [U=0,P=17149]      singularity_registry_set()                Returning singularity_registry_set(bindir, /export/apps/singularity-install/bin) = 0

VERBOSE [U=0,P=17149]      singularity_registry_set()                Adding value to registry: 'IMAGE' = '/export/apps/ubuntu.simg'

DEBUG   [U=0,P=17149]      singularity_registry_set()                Returning singularity_registry_set(IMAGE, /export/apps/ubuntu.simg) = 0

DEBUG   [U=0,P=17149]      singularity_registry_get()                Returning NULL on 'HOME'

DEBUG   [U=0,P=17149]      singularity_registry_get()                Returning NULL on 'TARGET_UID'

DEBUG   [U=0,P=17149]      singularity_registry_get()                Returning NULL on 'TARGET_GID'

DEBUG   [U=0,P=17149]      singularity_priv_init()                   Initializing user info

DEBUG   [U=0,P=17149]      singularity_priv_init()                   Set the calling user's username to: root

DEBUG   [U=0,P=17149]      singularity_priv_init()                   Marking uinfo structure as ready

DEBUG   [U=0,P=17149]      singularity_priv_init()                   Obtaining home directory

VERBOSE [U=0,P=17149]      singularity_priv_init()                   Set home (via getpwuid()) to: /root

VERBOSE [U=0,P=17149]      singularity_suid_init()                   Running SUID program workflow

VERBOSE [U=0,P=17149]      singularity_suid_init()                   Checking program has appropriate permissions

VERBOSE [U=0,P=17149]      singularity_suid_init()                   Checking configuration file is properly owned by root

ABORT   [U=0,P=17149]      singularity_suid_init()                   Running in privileged mode, root must own the Singularity configuration file: /etc/singularity/singularity.conf

ABORT   [U=0,P=17149]      singularity_suid_init()                   Retval = 255

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Rémy Dernat]

Hi,

Try to remove every previous install and the install src directory. Then, logout/login and clone singularity again. Finally, build it from a clean environment.

Let us know if it solves your issue.

Otherwise, it could be a bug. Try to create a symbolic link to configuration file where it is searching (/etc/singularity/singularity.conf).

Best regards,

Rémy

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Robert Kudyba]

I was able to delete everything except the var directory: 

rmdir session/

rmdir: failed to remove `session/': Device or resource busy

pwd

/share/apps/var/singularity/mnt

But deleting the source and recloning did get me further. On the head node I can run singularity commands. But again on the desired node we’d like to run this on we get this:

./singularity 

Error loading functions: /export/apps/singularity-install/libexec/singularity/functions

On the Rocks mailing list I as reminded to do this:

"You need to reference singularity (or any app for that matter) as /share/apps and never /export/apps

/export/apps only exists on the head node. "

So the good news is once I redid everything to use /share/apps I was able to get the singularity command to work on the compute node.

So once I tried a command like: ./singularity run /share/apps/ubuntu.simg 

groups: cannot find name for group ID 426

It looks like I’m in a shell:

ls -l /singularity 

lrwxrwxrwx 1 root root 24 Jun  6 14:46 /singularity -> .singularity.d/runscript

But is that error/warning normal?

[Rémy Dernat]

I think you built your image on a machine with uid "426" which is not present on the cluster, so, that's normal. You just have to enter in the container with '-w' flag in root, and edit permissions on your runscript to fit your user on the cluster side.

BTW, of course, normally, on Rocks, you had to build your image with:

```

./configure --prefix=/share/apps/singularity-install

```

Instead of:

```

./configure --prefix=/export/apps/singularity-install

```

Specially if some binaries contain the path of some files like singularity.conf

Sorry for this mistake.

Best,

Rémy

[Robert Kudyba]

is this the correct usage of -w?

./singularity run -w /share/apps/ubuntu.simg 

ERROR  : Could not open image /share/apps/ubuntu.simg: Permission denied

ABORT  : Retval = 255

[vanessa s]

Almost! To use -w you need to run as root:

sudo singularity run -w /share/apps/ubuntu.simg

[vanessa s]

And if you want to shell inside (and run doesn't open a shell) you should just use shell instead:

sudo singularity shell --writable /share/apps/ubuntu.simg

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Robert Kudyba]

It’s all coming together! Since we don’t really care about the Ubuntu image I made it chmod 777 to allow writeable to all. Also does anyone know how to export the $PATH to all Rocks compute nodes to include /share/apps/singularity-install/bin/singularity to allow singularity to work without being in the bin/ directory? I see the init file in etc but that doesn’t seem to work before calling singularity.

[Rick Wagner]

On each compute node create a singularity.sh and singularity.csh file in /etc/profile.d/. You can put these on the front end and distribute them via 411.

Example for adding some IB utilities to the PATH:

[ux455215@comet-ln2 profile.d]$ pwd
/etc/profile.d

[ux455215@comet-ln2 profile.d]$ cat ibutils.sh 
if ! echo ${PATH} | grep -q /opt/ibutils/bin ; then
        PATH=${PATH}:/opt/ibutils/bin
fi

[ux455215@comet-ln2 profile.d]$ cat ibutils.csh 
if ( "${path}" !~ */opt/ibutils/bin* ) then
        set path = ( $path /opt/ibutils/bin )
endif
[ux455215@comet-ln2 profile.d]$ 

—Rick

[Rémy Dernat]

I suggest the use of a modulefile in /etc/modulefiles/

You can create a file called singularity-<version> and then, add it to your list of files to synchronize in /var/411/Files.mk

Then run:

cd /var/411

make clean && make force

rocks sync config

rocks sync users

Hope that helps

Rémy.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

-- 
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

-- 
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

-- 
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

-- 
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

-- 
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

-- 
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

-- 

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

-- 

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

-- 
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.