bind at image build time?

[Noah]

Hi,

   Are there any parts of the host filesystem exposed during image build %post phase?  Can I make something visible with a bind -like argument?

After reviewing the documentation it seems like the answer is no.

My main purpose at hand is to build a container that holds a complex application.  The build process uses spack, cmake, and other fun things.   I've been attempting to build the application through the image %post phase.

Two consequences have been in front of me on a Cray system supporting Singularity (--fakeroot):

   1)  I can't point the build at at lots of libraries & utilities installed on the host at build time (i.e. %post time)   As an example, this would be a desirable alternative to the MPI "Bind" approach described in the Singularity documentation (where there the simple app is built on the host). (and admittedly somewhat contrary to some container principles)

   2). For some reason Spack sees itself running on platform type "linux" during the image build, but when I run the container the same Spack install now sees its platform type as "cray" and can't find any compiler that was installed during %post.

`spack arch` returns either linux-ubuntu20.04-zen2 or cray-ubuntu20.04-x86_64 depending on that context on the same node.

Thanks for your thoughts.

-Noah

[David Trudgian]

Hi Noah,

Using `--bind` at `singularity build` time has been possible since v3.8.0, with the caveat that the directory/file to which the bind is being made must exist in the container root filesystem, before the `%post` script runs.

https://docs.sylabs.io/guides/3.11/user-guide/build_a_container.html#bind (we do need to improve this documentation)

You can add directories / files to the container root filesystem before `%post` from the `%setup` section in a definition file, e.g.:

%setup

    mkdir -p ${SINGULARITY_ROOTFS}/my/directory

Then you can `singularity build --bind /my/directory ...`

Be very careful using `%setup` as it runs on the host. If you make a mistake, you will be modifying the host's own filesystem.

Regarding (2)... you might want to inquire with the Spack developers how the cray detection is performed. There is likely a way we can get Spack to detect cray in the container build if we now how that is working.

Cheers,

DT

[Noah]

Thank you David.

Here I realize I've generally been reading the Singularity 3.0 documentation while this Cray system is running Singularity 3.8.4.  I now see the --bind argument to the build command.  Now I need to think about how I'd want to use it -- philosophically what am I trying to containerize and what am I not.

For the second matter with Spack's platform determination, I figured out that this is happening through environment variable inspection.  I don't know which one(s) precisely, but the host has lots of CRAY_* type environment variables set.

I learned about singularity exec's --cleanenv argument.  Using --cleanenv, at runtime spack is determining its platform is "linux" matching the build-time determination.

Again, I don't really know what I want the answer to be.  I might want the opposite, "cray" at build time because its is probably unrealistic that this container image would ever be performance portable to a non-cray.

I'm going to investigate the last discrepancy.  "x86_64" vs. "zen2".  In this context of a large HPC application, I do want the application build to optimize for the actual CPU architecture taking full advantage of any extended instruction set.  I'm not sure how Spack is identifying the CPU type and may ask that community.

Regards,

-Noah

[David Trudgian]

Hi Noah,

Thanks for the information about the CRAY_ env vars etc... which matches with...

https://spack.readthedocs.io/en/latest/getting_started.html#using-linux-containers-on-cray-machines

During build time Singularity is not propagating host environment, so it makes sense it doesn't identify it as a Cray system.

It does look like there is also a module based component of cray compiler detection, though...

https://spack.readthedocs.io/en/latest/getting_started.html#cray-compiler-detection

The `x86_64` vs `zen2` detection would be interesting to know more about. I'm not clear why that would be environment based. Perhaps a conditional off of a CRAY_ var. Looks like doing the spack build with `target=zen2` might be sufficient to force it, though?

Cheers,

DT