Singularity environment definition script broken since 3.6

104 views
Skip to first unread message

Manuel Schwarze

unread,
Aug 31, 2021, 6:32:50 AM8/31/21
to singularity
Dear Singularity community and developers,

we have some older containers that we have built with Singularity 2.6.1. They define the file /.singularity.d/env/91-myenv.sh to always define the proper environment for each call into the container:

export CONDA_PREFIX=
export CONDA_PATH_BACKUP=
export CONDA_PS1_BACKUP=
export CONDA_DEFAULT_ENV=
export MYPYTHON_ROOT=/mypython
export PATH=/mypython/miniconda3/bin:/mypython/miniconda3/bin:/mypython/miniconda3/condabin:/mypython/miniconda3/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
export LANG=en_US.utf-8
export LC_ALL=en_US.utf-8
unset RDBASE
unset RDCONFIGFILE
export PYTHONNOUSERSITE=true
source activate mypython
export PYTHONPATH=${PYTHONPATH_PREPEND}:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_descs/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_sklearn/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_predict_core/scripts:${PYTHONPATH_APPEND}
export MYPYTHON_SCRIPT_PATH=/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_descs/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_sklearn/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_predict_core/scripts
[ -z "$OMP_NUM_THREADS" ] && export OMP_NUM_THREADS=1
[ -z "$MKL_NUM_THREADS" ] && export MKL_NUM_THREADS=1

Until Singularity 3.5 this was working perfectly fine. When calling for instance singularity with a python command, I could trust that the line "source activate mypython" was executed as part of the environment setup. Since Singularity 3.6 this is broken. When invoking the container, it shows the following error message:

source: open /home/myuserid/activate: no such file or directory

Interesting is that the script was actually executed, because I can see the environment variables set properly, e.g. when using a shell to invoke the container. Only the "source activate mypython" call fails, although the PATH to find the "activate" command was set one line before (/mypython/miniconda3/bin). Also, when manually calling the same command when I am "in the container" (by shell or by other commands) the command call works just fine.

We had started using Singularity about 3 years ago with the promise to our users that our containers will work just fine even in 10 years, and they can trust the reproducibility of their results when using the containers. I am disappointed that Singularity changes are breaking such basic things now.

But putting these feelings aside, I am really curious how to handle this now. I would like to know what from today's perspective is wrong, and what would be the right way to do it. Ideally, I would like to learn what I need to do in a way that works with Singularity < 3.6 and >= 3.6. Then I would consider - only if necessary, of course - patching my old containers and change the build process for the new ones accordingly.

Any feedback and potential resolution approaches are highly appreciated - thanks in advance for your time.

Kind regards,
Manuel Schwarze
NIBR Informatics, Basel, Switzerland

Thomas Hartmann

unread,
Aug 31, 2021, 7:28:47 AM8/31/21
to singu...@lbl.gov
Hi Manuel,

there has been changes to the shell interpreter with 3.6 that affect
bash specifics - might be the cause for your issue as well
https://github.com/hpcng/singularity/issues/5541

not sure how
*source activate mypython*
would be interpreted anyway in bash - is this a shell function or a
shell script (has it a specific she-bang pointing to the correct shell?
might be problematic, if you use ksh-specifics or so without including
it in the container etc.)

For functions I made a short summary of the issue thread, that worked for me
https://confluence.desy.de/pages/viewpage.action?pageId=175914754

Cheers,
Thomas


On 31/08/2021 12.32, Manuel Schwarze wrote:
> Dear Singularity community and developers,
>
> we have some older containers that we have built with Singularity 2.6.1.
> They define the file /.singularity.d/env/91-myenv.sh to always define
> the proper environment for each call into the container:
>
> export CONDA_PREFIX=
> export CONDA_PATH_BACKUP=
> export CONDA_PS1_BACKUP=
> export CONDA_DEFAULT_ENV=
> export MYPYTHON_ROOT=/mypython
> export
> PATH=/mypython/miniconda3/bin:/mypython/miniconda3/bin:/mypython/miniconda3/condabin:/mypython/miniconda3/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
> export LANG=en_US.utf-8
> export LC_ALL=en_US.utf-8
> unset RDBASE
> unset RDCONFIGFILE
> export PYTHONNOUSERSITE=true
> *source activate mypython*
> export
> PYTHONPATH=${PYTHONPATH_PREPEND}:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_descs/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_sklearn/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_predict_core/scripts:${PYTHONPATH_APPEND}
> export
> MYPYTHON_SCRIPT_PATH=/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_descs/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_sklearn/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_predict_core/scripts
> [ -z "$OMP_NUM_THREADS" ] && export OMP_NUM_THREADS=1
> [ -z "$MKL_NUM_THREADS" ] && export MKL_NUM_THREADS=1
>
> Until Singularity 3.5 this was working perfectly fine. When calling for
> instance singularity with a python command, I could trust that the line
> "*source activate mypython*" was executed as part of the environment
> setup. Since Singularity 3.6 this is broken. When invoking the
> container, it shows the following error message:
>
> source: open /home/myuserid/activate: no such file or directory
>
> Interesting is that the script was actually executed, because I can see
> the environment variables set properly, e.g. when using a shell to
> invoke the container. Only the "source activate mypython" call fails,
> although the PATH to find the "activate" command was set one line before
> (/mypython/miniconda3/bin). Also, when manually calling the same command
> when I am "in the container" (by shell or by other commands) the command
> call works just fine.
>
> We had started using Singularity about 3 years ago with the promise to
> our users that our containers will work just fine even in 10 years, and
> they can trust the reproducibility of their results when using the
> containers. I am disappointed that Singularity changes are breaking such
> basic things now.
>
> But putting these feelings aside, I am really curious how to handle this
> now. I would like to know what from today's perspective is wrong, and
> what would be the right way to do it. Ideally, I would like to learn
> what I need to do in a way that works with Singularity < 3.6 *and *>=
> 3.6. Then I would consider - only if necessary, of course - patching my
> old containers and change the build process for the new ones accordingly.
>
> Any feedback and potential resolution approaches are highly appreciated
> - thanks in advance for your time.
>
> Kind regards,
> Manuel Schwarze
> NIBR Informatics, Basel, Switzerland
>
> --
> You received this message because you are subscribed to the Google
> Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to singularity...@lbl.gov
> <mailto:singularity...@lbl.gov>.
> To view this discussion on the web visit
> https://groups.google.com/a/lbl.gov/d/msgid/singularity/d164a8a6-a0ac-4e55-8269-be277162f1d6n%40lbl.gov
> <https://groups.google.com/a/lbl.gov/d/msgid/singularity/d164a8a6-a0ac-4e55-8269-be277162f1d6n%40lbl.gov?utm_medium=email&utm_source=footer>.

Message has been deleted
Message has been deleted
Message has been deleted

Manuel Schwarze

unread,
Sep 7, 2021, 9:27:52 AM9/7/21
to singularity
  Thanks Thomas for your reply and sharing your workaround.
In my case I have found an even easier workaround: It was enough to use absolute path names to overcome that apparently the PATH variable is not honored at that time. When calling
  source /mypython/miniconda3/bin/activate mypython
it actually works again. Nevertheless, the main issue would be to patch old containers now in a way that changes just the single line of code in my environment script. I would really like to avoid this!

@Singularity Developers: I understand that software needs to be changed, and sometimes also decisions have to be taken to break contracts and APIs. But for Singularity as container solution that we gave to our users promising them that Singularity containers can be still used in 10 years to reliably run them to reproduce scientific results, it just feels like a huge disappointment. Going back to our users telling them that a 2 year old container will not work anymore just seems strange. I personally find that breaking something like this is a bug, not a feature. I would suggest reconsidering how old containers are treated, and to modify the code that old containers will continue to work just fine as before. When building a container with Singularity 3.6+ (with the breaking behavior), then I would find it ok if the handling is done differently, but not for old containers. Could you please discuss this in your architecture meetings to see, if it would be possible to mitigate the issue and fix that bug and also consider such an architecture pattern for future API breaking changes? We need to trust Singularity, otherwise what is the point of using it for long-term reproducability?

Another related question: For Singularity 3.6 I found the following documentation discussing how to define environments for future containers (3.6+): https://sylabs.io/guides/3.7/user-guide/environment_and_metadata.html#changes-in-singularity-3-6 With all the options discussed there, I am somehow missing the most wanted details how I would define in the future my *pre-defined* container environment (inside the container, not passed from the outside!) that gets always setup when a shell is invoked or a command executed (exec). If the files /.singularity.d/env files shall not be used anymore in the future, how should it be done instead *without* having the user to know or do anything? The complete logic, including activating a conda environment, should be part of the container - we explicitly do not pass in any environment from the host system (-clearenv parameter), we use only very specific SINGULARITY_ env variables, so that we have full control. Everything else was defined in the file I posted in my starting message of this thread. How would that look like in Singularity 3.6+? 

Kind regards,
Manuel  

Manuel Schwarze

unread,
Sep 13, 2021, 7:30:32 PM9/13/21
to singu...@lbl.gov

Dear Dave, dear Vanessa,

 

my I kindly ask you for some feedback for my questions in this thread (see below) and for my concerns about the broken container invocation functionality since Singularity 3.6? It would be highly appreciated to understand the way forward for our users when using Singularity containers for our application that requires long-term reproducibility of scientific results, which we were realizing with Singularity containers.

 

Thanks in advance for your time and your support!

 

Kind regards,

Manuel

--

You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/9fc47e04-b0b5-410e-a83e-6126cacb8ad9n%40lbl.gov.

v

unread,
Sep 13, 2021, 7:45:13 PM9/13/21
to Manuel Schwarze, singu...@lbl.gov
Hi Manuel,

I'm not a Singularity maintainer so I can't offer any insight as to future development or what to do, but I do hope you figure out a workaround - in the past you would set %environment in a Singularity definition file, and then SINGULARITY_ENV and then binding files in /singularity.d/env, but you are saying none of those are working? So I'm not sure I have a good suggestion, but hopefully a maintainer does. The promise of Singularity early on was to ensure this kind of long term reproduciiblity, and I'm sorry that hasn't been the case.

I think the real answer is that perfect reproducibility is something we can strive for, but in practice very challenging to achieve. The most real world scenario is that when there are little changes in the design of a container, you don't really have any choice but to change your practices or rebuild, noting the importance of always keeping build recipes and workflows.

Best,

Vanessa

Dave Dykstra

unread,
Sep 14, 2021, 7:05:43 PM9/14/21
to Manuel Schwarze, singu...@lbl.gov
Hi Manuel,

I also don't have an in depth knowledge about the details of the change
that was made to the environment handling. I think we need to hear from
Cedric or possibly Greg. I do know that it was understood at the time
that it was not completely backward compatible, and I think the
intention was to try to make it more sustainable going forward.

Dave
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov<mailto:singularity...@lbl.gov>.
> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/9fc47e04-b0b5-410e-a83e-6126cacb8ad9n%40lbl.gov <https://groups.google.com/a/lbl.gov/d/msgid/singularity/9fc47e04-b0b5-410e-a83e-6126cacb8ad9n%40lbl.gov?utm_medium=email&utm_source=footer >.

Cédric Clerget

unread,
Oct 12, 2021, 2:51:20 AM10/12/21
to singularity, Dave Dykstra, singu...@lbl.gov, Manuel Schwarze
Hi Manuel,

Based on your explanation, it looks like the Singularity shell interpreter doesn't handle the possibility to load a sourced script from PATH. I opened this ticket https://github.com/hpcng/singularity/issues/6220, please comment in the github ticket so we could ping you to test a fix

Best regards,
Cédric
Reply all
Reply to author
Forward
0 new messages