First use of Singularity; bind mount problems, some other minor issues (2.1.2, CentOS 7.x and 6.x)

[Ryan Novosielski]

So a few things in no particular order -- thanks for this software, BTW -- I finally have had a use case for it:

1) I downloaded 2.0.9 from EPEL and my Lustre file system (mounted at /HPCTMP_NOBKUP) gave an error if you try to use the image from one of the directories, and you can't work with any files from the whole tree. I discovered that there are bind path settings to use, but this 2.0.9 RPM doesn't appear to have a singularity.conf file, and doesn't appear to pay any attention to one if you add one to /etc/singularity (which does exist).

2) I downloaded 2.1.2 as a .tar.gz and went through the instructions to create an RPM. It creates a non-ideally-named RPM: singularity-2.1-0.1.el6.x86_64.rpm. Shouldn't it be 2.1.2-0.1 or something like that?

3) Still having trouble using files in my Lustre directories with 2.1.2. I can now see the current directory well enough it seems. /HPCTMP_NOBKUP is still empty. So I tried adding it to the now-existing singularity.conf file. Then I started getting "WARNING: Non existant 'bind point' in container: '/HPCTMP_NOBKUP'" without it working any better.

4) Is there any way to sign up for this list with a regular e-mail address? My work has a Google domain but I'm not allowed to use it as my primary e-mail (a restriction placed on some staff -- long stupid story). I can't seem to figure out a way to sign up as my real work address, without I guess creating another non-Gmail Google account using my work e-mail address. Is there something smarter?

Thanks again. If you wouldn't mind copying novo...@rutgers.edu, I'd appreciate it.

[Gregory M. Kurtzer]

Hi Ryan,

1. Yes, we are aware that the EPEL version needs to be updated and Bennet is working on that. Hopefully it will be updated with the release of 2.2.

2. Yes, bug and fixed but not in the 2.1.2 release. Sorry, my bad!

3. Because bind points occur as bind mounts, the target must be available. So you will need to create ./HPCTMP_NOBKUP directory within the container. The 2.2 release has a solution for this, but it only works on new'ish kernels (e.g. RHEL7).

4. I've been considering that... And wondering how best to handle. I asked some other projects if we could leverage their existing mailman implementations, but was unable to secure an email list home. I am also considering www.group.io. Does anyone have experience with them?

Thanks Ryan!

Greg

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Gregory M. Kurtzer

HPC Systems Architect and Technology Developer

Lawrence Berkeley National Laboratory HPCS
University of California Berkeley Research IT
Singularity Linux Containers (http://singularity.lbl.gov/)

Warewulf Cluster Management (http://warewulf.lbl.gov/)

GitHub: https://github.com/gmkurtzer, Twitter: https://twitter.com/gmkurtzer

[Ryan Novosielski]

Thanks for your quick reply:

3) I /swear/ tried this both ways actually (found that answer in earlier reading), but it's now working as expected.

Thanks again.

[Gregory M. Kurtzer]

Haha, creepy!

Let me/us know how things go and let me know if you have any ideas about alternative email list solutions!

On Thursday, September 8, 2016, Ryan Novosielski <novo...@scarletmail.rutgers.edu> wrote:

Thanks for your quick reply:

3) I /swear/ tried this both ways actually (found that answer in earlier reading), but it's now working as expected.

Thanks again.

On Thursday, September 8, 2016 at 7:17:06 PM UTC-4, Gregory M. Kurtzer wrote:

Hi Ryan,

1. Yes, we are aware that the EPEL version needs to be updated and Bennet is working on that. Hopefully it will be updated with the release of 2.2.

2. Yes, bug and fixed but not in the 2.1.2 release. Sorry, my bad!

3. Because bind points occur as bind mounts, the target must be available. So you will need to create ./HPCTMP_NOBKUP directory within the container. The 2.2 release has a solution for this, but it only works on new'ish kernels (e.g. RHEL7).

4. I've been considering that... And wondering how best to handle. I asked some other projects if we could leverage their existing mailman implementations, but was unable to secure an email list home. I am also considering www.group.io. Does anyone have experience with them?

Thanks Ryan!

Greg

On Thursday, September 8, 2016, Ryan Novosielski <novo...@scarletmail.rutgers.edu> wrote:

So a few things in no particular order -- thanks for this software, BTW -- I finally have had a use case for it:

1) I downloaded 2.0.9 from EPEL and my Lustre file system (mounted at /HPCTMP_NOBKUP) gave an error if you try to use the image from one of the directories, and you can't work with any files from the whole tree. I discovered that there are bind path settings to use, but this 2.0.9 RPM doesn't appear to have a singularity.conf file, and doesn't appear to pay any attention to one if you add one to /etc/singularity (which does exist).

2) I downloaded 2.1.2 as a .tar.gz and went through the instructions to create an RPM. It creates a non-ideally-named RPM: singularity-2.1-0.1.el6.x86_64.rpm. Shouldn't it be 2.1.2-0.1 or something like that?

3) Still having trouble using files in my Lustre directories with 2.1.2. I can now see the current directory well enough it seems. /HPCTMP_NOBKUP is still empty. So I tried adding it to the now-existing singularity.conf file. Then I started getting "WARNING: Non existant 'bind point' in container: '/HPCTMP_NOBKUP'" without it working any better.

4) Is there any way to sign up for this list with a regular e-mail address? My work has a Google domain but I'm not allowed to use it as my primary e-mail (a restriction placed on some staff -- long stupid story). I can't seem to figure out a way to sign up as my real work address, without I guess creating another non-Gmail Google account using my work e-mail address. Is there something smarter?

Thanks again. If you wouldn't mind copying novo...@rutgers.edu, I'd appreciate it.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Gregory M. Kurtzer

HPC Systems Architect and Technology Developer

Lawrence Berkeley National Laboratory HPCS
University of California Berkeley Research IT
Singularity Linux Containers (http://singularity.lbl.gov/)

Warewulf Cluster Management (http://warewulf.lbl.gov/)

GitHub: https://github.com/gmkurtzer, Twitter: https://twitter.com/gmkurtzer

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Ryan Novosielski]

Also, in case you're curious what I'm doing, I have the same problem as this person:
http://stackoverflow.com/questions/38644259/how-to-install-glibc-2-14-without-admin-right

There does not appear to be a reasonable solution to this otherwise. I tried compiling GLIBC 2.17 and placing it elsewhere, and then doing the magic incantation to make sure that the regular library files didn't interfere with this GLIBC, but then of course some of those libraries in /lib64 are required by the program that requires the new GLIBC, which then leaves you to build those as well outside of the system tree, and meh, forget it. 

The real solution is probably for these folks to not unnecessarily depend on a GLIBC that's excessively new.

[Bernard Li]

Have you tried building gdc-client from source or does that not work?
If so then using Singularity for that is a good use case unless you
can convince your sysadmins to build you an Ubuntu cluster :-)

Cheers,

Bernard

>>>> an email to singularity...@lbl.gov.

>>>
>>>
>>>
>>> --
>>> Gregory M. Kurtzer
>>> HPC Systems Architect and Technology Developer
>>> Lawrence Berkeley National Laboratory HPCS
>>> University of California Berkeley Research IT
>>> Singularity Linux Containers (http://singularity.lbl.gov/)
>>> Warewulf Cluster Management (http://warewulf.lbl.gov/)
>>> GitHub: https://github.com/gmkurtzer, Twitter:
>>> https://twitter.com/gmkurtzer
>>>

> --
> You received this message because you are subscribed to the Google Groups
> "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to singularity...@lbl.gov.

[Bernard Li]

4. Try sending an email to <singularit...@lbl.gov> -- see if that works.

Cheers,

Bernard

>> email to singularity...@lbl.gov.

>
>
>
> --
> Gregory M. Kurtzer
> HPC Systems Architect and Technology Developer
> Lawrence Berkeley National Laboratory HPCS
> University of California Berkeley Research IT
> Singularity Linux Containers (http://singularity.lbl.gov/)
> Warewulf Cluster Management (http://warewulf.lbl.gov/)
> GitHub: https://github.com/gmkurtzer, Twitter: https://twitter.com/gmkurtzer
>

> --
> You received this message because you are subscribed to the Google Groups
> "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to singularity...@lbl.gov.

[Bernard Li]

Hi Ryan:

Just wondering if you were able to subscribe to the Google Group using
the method I outlined.

Thanks,

Bernard

[Dave Love]

Ryan Novosielski <novo...@scarletmail.rutgers.edu> writes:

> So a few things in no particular order -- thanks for this software, BTW --
> I finally have had a use case for it:
>
> 1) I downloaded 2.0.9 from EPEL

?? It's not released for EPEL (and it's unfortunate that it got into
Fedora). I need to consult on what to do about that.

> and my Lustre file system (mounted at
> /HPCTMP_NOBKUP) gave an error if you try to use the image from one of the
> directories,

As far as I remember, it requires flock, so won't work on a parallel
filesystem tuned to be fast. (People may be saved by flock being
necessary for MPI-IO on Lustre, at least using ROMIO.)

The problems I've had with mounts/loopback in EL6 seem to be connected
with dbus interfering with loop devices in some way I've not figured
out. Restarting dbus freed devices shown by losetup -a that were
apparently associated with dead processes.

[Dave Love]

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> Hi Ryan,
>
> 1. Yes, we are aware that the EPEL version needs to be updated and Bennet
> is working on that. Hopefully it will be updated with the release of 2.2.

No-one has contacted the package point-of-contact about that.

> 4. I've been considering that... And wondering how best to handle. I asked
> some other projects if we could leverage their existing mailman
> implementations, but was unable to secure an email list home. I am also
> considering www.group.io. Does anyone have experience with them?

How about groups.io, which is different, and also Sourceforge under new
management?

[Dave Love]

Ryan Novosielski <novo...@scarletmail.rutgers.edu> writes:

> Also, in case you're curious what I'm doing, I have the same problem as
> this person:
> http://stackoverflow.com/questions/38644259/how-to-install-glibc-2-14-without-admin-right
>
> There does not appear to be a reasonable solution to this otherwise. I
> tried compiling GLIBC 2.17 and placing it elsewhere, and then doing the
> magic incantation to make sure that the regular library files didn't
> interfere with this GLIBC, but then of course some of those libraries in
> /lib64 are required by the program that requires the new GLIBC, which then
> leaves you to build those as well outside of the system tree, and meh,
> forget it.
>
> The real solution is probably for these folks to not unnecessarily depend
> on a GLIBC that's excessively new.

The solution is not to try to run binaries directly which are
incompatible with your operating system and to have free software which
is easy to build and install.

[Ryan Novosielski]

It seems to have worked; thank you for the suggestion (didn’t get to try it right away — was on vacation). I guess we can list that as a way to do this (one can’t just click the subscribe button in the e-mail though when accepting the subscription or you wind up in the same place as trying to subscribe via the web).

--
____
|| \\UTGERS, |---------------------------*O*---------------------------
||_// the State | Ryan Novosielski - novo...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark
`'

[Gregory M. Kurtzer]

On Tue, Sep 13, 2016 at 6:52 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:

Ryan Novosielski <novo...@scarletmail.rutgers.edu> writes:

> So a few things in no particular order -- thanks for this software, BTW --
> I finally have had a use case for it:
>
> 1) I downloaded 2.0.9 from EPEL

??  It's not released for EPEL (and it's unfortunate that it got into
Fedora).  I need to consult on what to do about that.

Why is it unfortunate that it got into Fedora?

 

> and my Lustre file system (mounted at
> /HPCTMP_NOBKUP) gave an error if you try to use the image from one of the
> directories,

As far as I remember, it requires flock, so won't work on a parallel
filesystem tuned to be fast.  (People may be saved by flock being
necessary for MPI-IO on Lustre, at least using ROMIO.)

Singularity no longer requires flock() for the image. It does require it within the session directory which should always be configured to be a local file system.

 

The problems I've had with mounts/loopback in EL6 seem to be connected
with dbus interfering with loop devices in some way I've not figured
out.  Restarting dbus freed devices shown by losetup -a that were
apparently associated with dead processes.

That is good to know. Are you running a desktop installation? I've seen some GUI environments also try to display the Singularity image as a mounted filesystem in the file viewer/desktop and then not want to release it. Perhaps that is related to the dbus issue you've seen.

[Bennet Fauber]

Dave,

I haven't wanted to contact anyone just yet because I don't think I'm
quite up to speed, and because I offered to 'help out' but not really
to take the lead on anything. I am hoping someone else can lead, and
I can be a trusty sidekick for some of the more mundane tasks, thus
all of my communication so far has been with Greg and really only
about the spec file itself and whether basic tests of the resulting
.rpm work.

-- bennet

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.