Dear Singularity community and developers,
we have some older containers that we have built with Singularity 2.6.1. They define the file /.singularity.d/env/91-myenv.sh to always define the proper environment for each call into the container:
export CONDA_PREFIX=
export CONDA_PATH_BACKUP=
export CONDA_PS1_BACKUP=
export CONDA_DEFAULT_ENV=
export MYPYTHON_ROOT=/mypython
export PATH=/mypython/miniconda3/bin:/mypython/miniconda3/bin:/mypython/miniconda3/condabin:/mypython/miniconda3/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
export LANG=en_US.utf-8
export LC_ALL=en_US.utf-8
unset RDBASE
unset RDCONFIGFILE
export PYTHONNOUSERSITE=true
source activate mypython
export PYTHONPATH=${PYTHONPATH_PREPEND}:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_descs/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_sklearn/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_predict_core/scripts:${PYTHONPATH_APPEND}
export MYPYTHON_SCRIPT_PATH=/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_descs/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_sklearn/scripts:/mypython/miniconda3/envs/mypython/lib/python3.8/site-packages/mypython_predict_core/scripts
[ -z "$OMP_NUM_THREADS" ] && export OMP_NUM_THREADS=1
[ -z "$MKL_NUM_THREADS" ] && export MKL_NUM_THREADS=1
Until Singularity 3.5 this was working perfectly fine. When calling for instance singularity with a python command, I could trust that the line "source activate mypython" was executed as part of the environment setup. Since Singularity 3.6 this is broken. When invoking the container, it shows the following error message:
source: open /home/myuserid/activate: no such file or directory
Interesting is that the script was actually executed, because I can see the environment variables set properly, e.g. when using a shell to invoke the container. Only the "source activate mypython" call fails, although the PATH to find the "activate" command was set one line before (/mypython/miniconda3/bin). Also, when manually calling the same command when I am "in the container" (by shell or by other commands) the command call works just fine.
We had started using Singularity about 3 years ago with the promise to our users that our containers will work just fine even in 10 years, and they can trust the reproducibility of their results when using the containers. I am disappointed that Singularity changes are breaking such basic things now.
But putting these feelings aside, I am really curious how to handle this now. I would like to know what from today's perspective is wrong, and what would be the right way to do it. Ideally, I would like to learn what I need to do in a way that works with Singularity < 3.6 and >= 3.6. Then I would consider - only if necessary, of course - patching my old containers and change the build process for the new ones accordingly.
Any feedback and potential resolution approaches are highly appreciated - thanks in advance for your time.
Kind regards,
Manuel Schwarze
NIBR Informatics, Basel, Switzerland