How to override an environment variable during container run time?

[Samy]

Hello everyone,

I'm using singularity 2.6 . I have this variable to setup the out put folder path like this:

    %environment

    OUTPUT=${HOME}

    APPNAME="lammps"

    RESULTS_DIR=${OUTPUT}/${APPNAME}

    %apprun lammps

    <runcommand> |tee -a $RESULTS_DIR/

I wold like to override this directory depending the situation.

I tried:

      export SINGULARITYENV_OUTPUT=/localdisk/

      singularity run --app lammps container.simg

But that still writes the files to the $HOME directory.

Any leads please how to get this variable customized without changing and rebuilding the container every time? thank you.

[v]

The variable RESULTS_DIR is already defined (with a previous $OUTPUT) so you would probably do better to try changing RESULTS_DIR directly:

$ SINGULARITYENV_RESULTS_DIR=something singularity exec busybox.simg env | grep RESULTS_DIR
RESULTS_DIR=something

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Samy]

This still doesn't override RESULTS_DIR original value :(

Any other ideas?

Thanks,

To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

[v]

Can you provide a (dummy) recipe with a similar entrypoint? I installed Singularity 2.6 for you this morning so I'm ready to go! It will suck if it's an old bug that doesn't pass the variables to the SCIF apps.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Samy]

Thanks for doing that. I attached a simplified recipe with the same content of the #environment section i have in the original recipe.

On Wednesday, May 8, 2019 at 10:59:44 AM UTC-7, vanessa wrote:

Can you provide a (dummy) recipe with a similar entrypoint? I installed Singularity 2.6 for you this morning so I'm ready to go! It will suck if it's an old bug that doesn't pass the variables to the SCIF apps.

[Jason Stover]

This is how it works... `SINGULARITYENV_xxxx` is going to pass `xxxx`
to the singularity execution *before* anything from the container side
happens. *But* the 90-environment.sh file is sourced when you `exec`,
`run`, etc... so the variable is going to be set to whatever you
define there in the environment section. i.e. That is going to
overwrite what's given in the `SINGULARITYENV_` variable value you
give.

You'll want to do something like:
---- start cut ----
if [ -n "$MYRES" ]; then
RESULTS_DIR=$MYRES
else
RESULTS_DIR=${HOME}/${APPNAME}
fi
---- end cut ----

And then use: `SINGULARITYENV_MYRES=/some/path singularity [....]`

So, if we have `MYRES` set, then use that value for RESULT_DIR,
otherwise default to `${HOME}/${APPNAME}`.

-J

> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[v]

Hey Samy,

You are using the SCIF app in a weird way - when you define any section for "lammps" it will automatically generate a home for it under /scif/apps/llamps, and environment variables for the name, base, etc. are available when you run it.

In either case, I tweaked your recipe with Jason' suggestion to demonstrate the fix:

https://gist.github.com/vsoch/7a7d41d166a846badf723b1462a9db46

Here is an example running without changing the results directory:

$ singularity run --app lammps lamps.simg 
lammps is running.
Single node Output file is vanessa-ThinkPad-T460s_lammps_2019_05_08.results and log is vanessa-ThinkPad-T460s_lammps_2019_05_08 and results dir is /home/vanessa/lammps

and here is with a custom results folder:

$ SINGULARITYENV_CUSTOM_RESULTS_DIR=/tmp/pancakes singularity run --app lammps lamps.simg 
lammps is running.
Single node Output file is vanessa-ThinkPad-T460s_lammps_2019_05_08.results and log is vanessa-ThinkPad-T460s_lammps_2019_05_08 and results dir is /tmp/pancakes

Notice how $SCIF_APPNAME can be used when running llamps i(the echo for the appname) nstead of setting a global environment variable. Try this to see all the envars that are available to you for lammps:

$ singularity exec --app lammps lamps.simg env | grep SCIF_APP
SCIF_APPROOT_lammps=/scif/apps/lammps
SCIF_APPMETA=/scif/apps/lammps/scif
SCIF_APPRUN_lammps=/scif/apps/lammps/scif/runscript
SCIF_APPLABELS_lammps=/scif/apps/lammps/scif/labels.json
SCIF_APPDATA=/scif/data/lammps
SCIF_APPROOT=/scif/apps/lammps
SCIF_APPLIB_lammps=/scif/apps/lammps/lib
SCIF_APPDATA_lammps=/scif/data/lammps
SCIF_APPINPUT=/scif/data/lammps/input
SCIF_APPBIN_lammps=/scif/apps/lammps/bin
SCIF_APPS=/scif/apps
SCIF_APPOUTPUT=/scif/data/lammps/output
SCIF_APPMETA_lammps=/scif/apps/lammps/scif
SCIF_APPNAME=lammps

Good team work, y'all! :)

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Samy]

thank you both for the solution. I'm trying it now.

Vanessa, could you clarify how I'm using SCIF in a weird way? I'm using the same style for multiple recipes and would like to fix them all.

Also, I'm using WORKDIR=$SINGULARITY_ROOTFS and you're using WORKDIR=${SCIF_APPBASE}: what is the different between $SINGULARITY_ROOTFS and SCIF_APPBASE please? doesn't it matter which one i use? (they both seem to work similarly)

[v]

Sure thing! Setting SINGULARITY_ROOTFS to anything in any location other than %setup doesn't make sense - it's only valid for that section as the base (root) of where the build is happening before it's squashed up. I set it to be the app base because that is the (apps) personal base that is built into the folder. If you want somewhere to write data (meaning bind from the host) then you probably want the SCIF_APPDATA, which would be under /scif/data/<appname>.

Actually I might have gotten it wrong, looks like it's SCIF_APPROOT (not SCIF_APPBASE)

SCIF_APPROOT_lammps=/scif/apps/lammps

My mistake! That's probably why you see similar result (neither are defined). Give that one a try - along with SCIF_APPDATA. Both should be defined as the locations for lammps when it's running.

The purpose of SCIF is to allow for modular environments, so setting any global variables for an app doesn't make sense, unless of course you have something specific in mind. In your case, you could just reference SCIF_APPNAME when lammps is running to get lammps (and this would work for any other app). 

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Samy]

Perfect! working now :)

Thanks to both of you.

On Wednesday, May 8, 2019 at 1:28:25 PM UTC-7, vanessa wrote:

Sure thing! Setting SINGULARITY_ROOTFS to anything in any location other than %setup doesn't make sense - it's only valid for that section as the base (root) of where the build is happening before it's squashed up. I set it to be the app base because that is the (apps) personal base that is built into the folder. If you want somewhere to write data (meaning bind from the host) then you probably want the SCIF_APPDATA, which would be under /scif/data/<appname>.

Actually I might have gotten it wrong, looks like it's SCIF_APPROOT (not SCIF_APPBASE)

SCIF_APPROOT_lammps=/scif/apps/lammps

My mistake! That's probably why you see similar result (neither are defined). Give that one a try - along with SCIF_APPDATA. Both should be defined as the locations for lammps when it's running.

The purpose of SCIF is to allow for modular environments, so setting any global variables for an app doesn't make sense, unless of course you have something specific in mind. In your case, you could just reference SCIF_APPNAME when lammps is running to get lammps (and this would work for any other app).