Singularity and "development mode" python package
797 views
Skip to first unread message
Laurent C.
May 18, 2017, 5:01:54 PM5/18/17
to singu...@lbl.gov
Hello all,
I'm developing a python package that is published on pypi. On my
computer I have it installed in "editable mode" [0] to test it.
However, I sometime need to run the last stable version of the
software. Although I can do it with virtualenv, I thought of creating
a Singularity container to allow me to experiment with Singularity.
I created a container that install the last version of the software
from pypi. But when I'm running it with the user (ie, not root), the
version of the software that it uses is the one on the host, not the
one from the container. When the container is run with root, it uses
the stable version from the container.
What could be the reason of this behaviour, and how could I fix this?
I build singularity from git, last commit 26e35168c49
Best regards,
Laurent
[0] https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs
I'm developing a python package that is published on pypi. On my
computer I have it installed in "editable mode" [0] to test it.
However, I sometime need to run the last stable version of the
software. Although I can do it with virtualenv, I thought of creating
a Singularity container to allow me to experiment with Singularity.
I created a container that install the last version of the software
from pypi. But when I'm running it with the user (ie, not root), the
version of the software that it uses is the one on the host, not the
one from the container. When the container is run with root, it uses
the stable version from the container.
What could be the reason of this behaviour, and how could I fix this?
I build singularity from git, last commit 26e35168c49
Best regards,
Laurent
[0] https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs
David Godlove
May 18, 2017, 5:31:43 PM5/18/17
to singu...@lbl.gov
Just a shot in the dark here. You might get that behavior if you installed pypi in root's home directory. If that's the case, you should try installing it somewhere else. Maybe /usr/local? Then you should be good to go.
--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
CARLOS EDUARDO ARANGO GUTIERREZ
May 18, 2017, 5:35:27 PM5/18/17
to singu...@lbl.gov
Hi Laurent
2 things.
> Could you run
$ Singularity --version
And show us the output thx
> Can you show us the step by step of how you installed the software ( your .def file is able)
Best regards
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
------------------------------------------------------------------------------------------------------------------------
CARLOS EDUARDO ARANGO GUTIERREZ.
Cel : +57 3182545671
Computer Science PhD StudentLaboratorio de redes y sistemas distribuidos - LASCILAB
vanessa s
May 18, 2017, 6:14:25 PM5/18/17
to singu...@lbl.gov
Hey Laurent,
If you want the latest version of Singularity, you do it from development branch. Like:
git clone -b development https://www.github.com/singularityware/singularity.git
cd singularity
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install
but this is probably not related to your issue. Since Singularity also includes your system python, you are probably calling that. When you run the python, make sure to specify the entire path to the executable, eg:
exec /usr/bin/python
and not
python
If you could share you spec file for building, and then the specific way you install the software, I'd be glad to give you more details! Basically, you are either probably installing to or calling the wrong python from within the container.
Best,
Vanessa
On Thu, May 18, 2017 at 5:01 PM, Laurent C. <lrn...@gmail.com> wrote:
--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
Gregory M. Kurtzer
May 18, 2017, 6:31:37 PM5/18/17
to singu...@lbl.gov
There are only a few cases where it would even be possible to be calling the host's python from within the container:
* The container is not actually running (unlikey if you are calling with something like `singularity exec ....`).
* You have python installed into a shared location (e.g. `$HOME/bin` or any other path shared between the containers (in which case, Vanessa's suggestion will fix)
* Your bootstrap file did something very weird. I don't know what, but if so it must be weird and I'd like to see it. haha
Hope that helps!
Greg
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
--
Rémy Dernat
May 19, 2017, 2:02:31 AM5/19/17
to singu...@lbl.gov
Like said before, I think your package is installed in your HOME, that is why het that behaviour. Indeed, by default, the HOME is shared between your host and your container.
Did you add the flag '--user' to your pip install or do you use pyenv or virtualenv or another thing (conda ?) ?
Best regards
Remy
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
John Hearns
May 19, 2017, 3:39:13 AM5/19/17
to singu...@lbl.gov
I think Remy has hit the nail on the head here.
I am not going to express myself very well here, but in general does this point up the need for some 'good practice' guidelines
for installing Python (or other similar) packages within containers?
Ie something like, very crudely:
Create a directory called /apps/python
Put your virtualenv there
Priedhorsky, Reid
May 19, 2017, 10:24:46 AM5/19/17
to singu...@lbl.gov
> On May 19, 2017, at 12:02 AM, Rémy Dernat <rem...@gmail.com> wrote:
>
> Did you add the flag '--user' to your pip install or do you use pyenv or virtualenv or another thing (conda ?) ?
HTH,
Reid
Laurent C.
May 19, 2017, 10:54:37 AM5/19/17
to singu...@lbl.gov
Hello all,
Thanks for all your answers. I'll try to give the information you've asked.
$ singularity --version
2.2.99-development.g26e35168
Bootstrap file:
BootStrap: docker
From: ubuntu:16.10
%setup
%post
apt-get update && apt-get install -y \
grass-core \
python-pip
pip install --disable-pip-version-check --upgrade pip
pip install itzi
%runscript
exec itzi "$@"
I now think the main point here is that on the host, I've installed
the development version of itzi as a user. It resides in ~/.local/bin/
on the host.
On the container, it is installed as root (see def file above).
Therefore, as mentioned by Rémy, it might be the shared $HOME that
create the issue. So, if I want to make sure that the python packages
installed on the host by the user do not interfere with packages
inside the container, should I create a virtualenv inside the
container?
Regards,
Laurent
Thanks for all your answers. I'll try to give the information you've asked.
$ singularity --version
2.2.99-development.g26e35168
Bootstrap file:
BootStrap: docker
From: ubuntu:16.10
%setup
%post
apt-get update && apt-get install -y \
grass-core \
python-pip
pip install --disable-pip-version-check --upgrade pip
pip install itzi
%runscript
exec itzi "$@"
I now think the main point here is that on the host, I've installed
the development version of itzi as a user. It resides in ~/.local/bin/
on the host.
On the container, it is installed as root (see def file above).
Therefore, as mentioned by Rémy, it might be the shared $HOME that
create the issue. So, if I want to make sure that the python packages
installed on the host by the user do not interfere with packages
inside the container, should I create a virtualenv inside the
container?
Regards,
Laurent
Gregory M. Kurtzer
May 19, 2017, 11:07:23 AM5/19/17
to singu...@lbl.gov
I am not much of a Python programmer, but if the path which includes `~/.local/bin` can be influenced by an environment variable, you can do this by setting (or unsetting) it in `/etc/singularity/init` or `~/.singularity-init` or you can variable transposition within Singularity to do things like:
on the host:
SINGULARITYENV_HELLO=WORLD
export SINGULARITYENV_HELLO
within the container, that will be transposed to `HELLO=WORLD`. And if you set a variable to nothing `...=""` it will unset it from within the host as follows:
HELLO=WORLD
SINGULARITYENV_HELLO=""
export HELLO SINGULARITYENV_HELLO
Within the container, neither "HELLO" nor "SINGULARITYENV_HELLO" will be defined.
Hope that helps!
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
Laurent C.
May 19, 2017, 12:31:53 PM5/19/17
to singu...@lbl.gov
Hello,
I believe that which program is run is influenced by the PATH variable.
On the host:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
In the container:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
On the host, ~/.local/bin/ is not in the PATH. So when running my
software from the command line on the host, it actually call
/usr/local/bin/itzi, but then, through Python's entry points system,
run the files on my development folder in my home directory.
This for the Python thing. I still don't understand how to circumvent
this behaviour and make sure that when running the container, it does
not run the file from $HOME.
Regards,
Laurent
>> > an email to singularity...@lbl.gov.
I believe that which program is run is influenced by the PATH variable.
On the host:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
In the container:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
On the host, ~/.local/bin/ is not in the PATH. So when running my
software from the command line on the host, it actually call
/usr/local/bin/itzi, but then, through Python's entry points system,
run the files on my development folder in my home directory.
This for the Python thing. I still don't understand how to circumvent
this behaviour and make sure that when running the container, it does
not run the file from $HOME.
Regards,
Laurent
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "singularity" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to singularity...@lbl.gov.
>> --
>> You received this message because you are subscribed to the Google Groups
>> "singularity" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to singularity...@lbl.gov.
>
> --
> You received this message because you are subscribed to the Google Groups
> "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an
vanessa s
May 19, 2017, 4:34:53 PM5/19/17
to singu...@lbl.gov
Hey Laurent,
I think best practice is to just be very specific about the executable that is called, and rely on path variables as minimally as possible. If the itzi that is installed to your container is reliably importing modules from it's same installed python (if not we again need to check pythonpath) then you would just be specific to have your spec file look like this:
BootStrap: docker
From: ubuntu:16.10
%setup
%post
apt-get update && apt-get install -y \
grass-core \
python-pip
pip install --disable-pip-version-check --upgrade pip
pip install itzi
%runscript
exec /usr/local/bin/itzi "$@"
and then running:
./itzi.img --help
usage: itzi [-h] {run,version} ...
A dynamic, fully distributed hydraulic and hydrologic model.
positional arguments:
{run,version}
run run a simulation
version display software version number
optional arguments:
-h, --help show this help message and exit
Of course the only difference is calling the itzi in the container directly! to check this, I just added "which itzi" to the %post section to see the version installed, and then added that path to the runscript. I ran into this exact issue when I started using Singularity, and I've found this to be the most straightforward approach. It should actually be taken for most executable calls, because you can never be sure if/when the person using your container has the same executable somewhere within the containers sight.
Best,
Vanessa
>> > an email to singularity+unsubscribe@lbl.gov.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "singularity" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to singularity+unsubscribe@lbl.gov.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to singularity+unsubscribe@lbl.gov.
--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
Laurent C.
May 23, 2017, 4:38:36 PM5/23/17
to singu...@lbl.gov
Hello Vanessa,
Thanks for your answer. The problem is that the executable of the
"development version" of itzi is actually in /usr/local/bin/itzi.
This executable is a Python script that call the python files inside
the user HOME directory.
cat /usr/local/bin/itzi
#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'itzi','console_scripts','itzi'
__requires__ = 'itzi'
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.exit(
load_entry_point('itzi', 'console_scripts', 'itzi')()
So it seems that the trouble happen because pkg_resources says where
are the actual files to be executed.
Unfortunately I haven't found information on how to circumvent this.
My guess is that is defined by some kind of Python environment.
Regards,
Laurent
>> >> > an email to singularity...@lbl.gov.
Thanks for your answer. The problem is that the executable of the
"development version" of itzi is actually in /usr/local/bin/itzi.
This executable is a Python script that call the python files inside
the user HOME directory.
cat /usr/local/bin/itzi
#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'itzi','console_scripts','itzi'
__requires__ = 'itzi'
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.exit(
load_entry_point('itzi', 'console_scripts', 'itzi')()
So it seems that the trouble happen because pkg_resources says where
are the actual files to be executed.
Unfortunately I haven't found information on how to circumvent this.
My guess is that is defined by some kind of Python environment.
Regards,
Laurent
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "singularity" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> email to singularity...@lbl.gov.
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "singularity" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "singularity" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to singularity...@lbl.gov.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "singularity" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "singularity" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to singularity...@lbl.gov.
>> --
>> You received this message because you are subscribed to the Google Groups
>> "singularity" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
> --
> You received this message because you are subscribed to the Google Groups
> "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to singularity...@lbl.gov.
> You received this message because you are subscribed to the Google Groups
> "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an
Laurent C.
May 26, 2017, 1:56:20 PM5/26/17
to singu...@lbl.gov
Hello,
Some more test.
When running the container with --contain it launches the software
installed in the container, and not the one in the host $HOME.
But none of the host $HOME is available. This is not desirable if
needed to run the software on host data, which is the likely use case.
It seems like the solution might be to create a virtualenv inside the container.
Regards,
Laurent
Some more test.
When running the container with --contain it launches the software
installed in the container, and not the one in the host $HOME.
But none of the host $HOME is available. This is not desirable if
needed to run the software on host data, which is the likely use case.
It seems like the solution might be to create a virtualenv inside the container.
Regards,
Laurent
Gregory M. Kurtzer
May 26, 2017, 2:19:14 PM5/26/17
to singu...@lbl.gov
How about the usage of the `-H` option to create a virtual home?
>>> >> > an email to singularity+unsubscribe@lbl.gov.
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> >> Groups
>>> >> "singularity" group.
>>> >> To unsubscribe from this group and stop receiving emails from it, send
>>> >> an
>>> >> email to singularity+unsubscribe@lbl.gov.
>>> >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "singularity" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an
>>> > email to singularity+unsubscribe@lbl.gov.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "singularity" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to singularity+unsubscribe@lbl.gov.
>>
>>
>>
>>
>> --
>> Vanessa Villamia Sochat
>> Stanford University '16
>> (603) 321-0676
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "singularity" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to singularity+unsubscribe@lbl.gov.
--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
Rémy Dernat
May 27, 2017, 10:28:46 AM5/27/17
to singu...@lbl.gov
Hi Laurent
You can also change your PYTHONHOME or your PYTHONPATH...:
Best regards
Rémy.
>>> >> > an email to singularity...@lbl.gov.
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> >> Groups
>>> >> "singularity" group.
>>> >> To unsubscribe from this group and stop receiving emails from it, send
>>> >> an
>>> >> email to singularity...@lbl.gov.
>>> >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "singularity" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an
>>> > email to singularity...@lbl.gov.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "singularity" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to singularity...@lbl.gov.
>>
>>
>>
>>
>> --
>> Vanessa Villamia Sochat
>> Stanford University '16
>> (603) 321-0676
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "singularity" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to singularity...@lbl.gov.
--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
Laurent C.
Aug 8, 2017, 5:58:35 PM8/8/17
to singu...@lbl.gov
Hello,
Today I tried again to sort this out.
What I did:
- create a virtualenv inside the container: I did not find out how to
activate it a runtime with a normal user
- set PYTHONPATH inside the container: no change
- run singularity with the '-H' option: It works!
However, it would be nice if there was a way in the bootstrap
instructions to make sure that the container is always separated from
local python packages in the HOME directory of the user.
Regards,
Laurent
Today I tried again to sort this out.
What I did:
- create a virtualenv inside the container: I did not find out how to
activate it a runtime with a normal user
- set PYTHONPATH inside the container: no change
- run singularity with the '-H' option: It works!
However, it would be nice if there was a way in the bootstrap
instructions to make sure that the container is always separated from
local python packages in the HOME directory of the user.
Regards,
Laurent
Reply all
Reply to author
Forward
0 new messages