/etc/localtime required more than 50 bind mounts

[Robert Lugg]

Hi all,  on our CentOS 7 machines, a TensorFlow Serving container runs fine.  On a CentOS6.6 machine, I get the error:

.../singularity/3.1.0/bin/singularity  run serving.sif

WARNING: underlay of /etc/localtime required more than 50 (68) bind mounts

FATAL: kernel too old

I can dig up and post serving.sif if it would be helpful, but from this error can anyone diagnose what is wrong?  Apparently the '50 check' is hard-coded into Singularity.  Could that be disabled, increased, or parameterized?  Or is there a workaround that I might use?

Thanks for any guidance.

[Rémy Dernat]

Hi,

Your issue with /etc/locatime is just a warnkng.

The bind mounts you see are due to the "--nv" option which bind every single file from cuda (libs, binaries). The kernel too old is, however, another issue due to the host operating system. Indeed, it means that your glibc on the host side, is too old. So, try to build your container from an older release; eg: if you build from centos7, try with centos6.

Best regards,

Rémy

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Jason Stover]

> FATAL: kernel too old

Are you using an Ubuntu 18 image? If so, that won't run on a RHEL 6.x
based OS. When they compiled GLibc they set the lowest Kernel version
to 3.3.0 (I think... in the 3's anyway).

That's just not something we can work around. :/

-J

[Robert Lugg]

Hi Remy,

DrDaveD made the change back in October with:

https://github.com/sylabs/singularity/commit/fed0bb5a93b4efe704954332814ce0cd2c31cfc7#diff-d4e781e3ba1f9b682f75e98ccfcaa97c

to

src/pkg/util/fs/layout/layer/underlay/underlay.go

I will ignore the warning.  But, I wonder what its purpose is.  If its a common occurance should the (warning) limit be increased?  Do you know the purpose of "50" as the warning limit?

Thank you on your advice regarding building using an older OS.  I am not sure all my software can be built using an older OS, but I haven't tried that one specifically.

Regards.

[David Dykstra]

Hi Robert,

Remy's not right about the warning; in your case, it is indeed
/etc/localtime causing the warning, not --nv. That's just another case
that can cause the warning.

The problem is that in order for underlay to add a bind mount point
inside a big directory such as /etc, it has to also bind mount every
other file in the directory from the original image. That may or may
not be a problem, but that's why there's a warning. I haven't seen any
problems caused by a large number of bind mount points when testing as a
single user on one machine, but I don't know if there might be limits
with many containers running on a large many-core system. I also don't
know how it affects performance, but it is reasonable to think that it
has some impact.

You can avoid the warning by including an /etc/localtime bind mount
point inside the image, if you have control over that. There's also a
singularity -s option to silence all warnings. There's not currently
any parameter to change the warning point; I didn't think that was worth
doing.

Dave

On Wed, Apr 24, 2019 at 12:11:25PM -0700, Robert Lugg wrote:
> Hi Remy,
>
> DrDaveD made the change back in October with:
>
> https://github.com/sylabs/singularity/commit/fed0bb5a93b4efe704954332814ce0cd2c31cfc7#diff-d4e781e3ba1f9b682f75e98ccfcaa97c
>
> to
>
> src/pkg/util/fs/layout/layer/underlay/underlay.go

> <https://github.com/sylabs/singularity/commit/fed0bb5a93b4efe704954332814ce0cd2c31cfc7#diff-d4e781e3ba1f9b682f75e98ccfcaa97c>

>
> I will ignore the warning. But, I wonder what its purpose is. If its a
> common occurance should the (warning) limit be increased? Do you know the
> purpose of "50" as the warning limit?
>
>
> Thank you on your advice regarding building using an older OS. I am not
> sure all my software can be built using an older OS, but I haven't tried
> that one specifically.
>
> Regards.
>
>
> On Wednesday, April 24, 2019 at 10:36:45 AM UTC-7, Robert Lugg wrote:
> >
> > Hi all, on our CentOS 7 machines, a TensorFlow Serving container runs
> > fine. On a CentOS6.6 machine, I get the error:
> >
> > .../singularity/3.1.0/bin/singularity run serving.sif
> >
> > WARNING: underlay of /etc/localtime required more than 50 (68) bind mounts
> >
> > FATAL: kernel too old
> >
> >
> > I can dig up and post serving.sif if it would be helpful, but from this
> > error can anyone diagnose what is wrong? Apparently the '50 check' is
> > hard-coded into Singularity. Could that be disabled, increased, or
> > parameterized? Or is there a workaround that I might use?
> >
> >
> > Thanks for any guidance.
> >
>

[Rémy Dernat]

Hi,

> Do you know the purpose of "50" as the warning limit?

Do you use any special / parallel file system ?

Best regards,

[Robert Lugg]

Sadly, I am very IT illiterate.  I don't believe I'm using anything special other than a very old OS :(

> To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.