PYTHONHOME woes (virtualenv, uWSGI)

[davidf...@gmail.com]

Hi all,

First of all: miniconda saves me a lot of trouble. My ansible playbook for building a server is a lot shorter without all installing development libraries and compiling numpy, scipy, pytables, and the like. But now I'm having trouble...

This is a first-warning sign, I think:

$ pip install virtualenv

$ virtualenv foo

New python executable in foo/bin/python

foo/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

ERROR: The executable foo/bin/python is not functioning

ERROR: It thinks sys.prefix is u'/home/hisparc' (should be u'/home/hisparc/foo')

ERROR: virtualenv is not compatible with this system or executable

I guess that conda does something arcane in how it handles relocating the python interpreter. My actual problem is trying to install and use uWSGI in an actual conda environment, not a virtualenv. I did:

$  conda create -p /srv/publicdb/env python pip --yes

$  /srv/publicdb/env/bin/pip install uwsgi

$  echo "import sys; print '\n'.join(sys.path)" >app.py

$  /srv/publicdb/env/bin/uwsgi --python app.py

*** Starting uWSGI 2.0.2 (64bit) on [Fri Mar  7 21:07:49 2014] ***

compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-4) on 07 March 2014 21:03:05

os: Linux-2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 20:37:17 CST 2013

nodename: localhost.localdomain

machine: x86_64

clock source: unix

detected number of CPU cores: 2

current working directory: /home/hisparc

detected binary path: /srv/publicdb/env/bin/uwsgi

!!! no internal routing support, rebuild with pcre support !!!

your processes number limit is 1024

your memory page size is 4096 bytes

detected max file descriptor number: 1024

lock engine: pthread robust mutexes

thunder lock: disabled (you can enable it with --thunder-lock)

Python version: 2.7.6 |Continuum Analytics, Inc.| (default, Jan 17 2014, 10:13:17)  [GCC 4.1.2 20080704 (Red Hat 4.1.2-54)]

*** Python threads support is disabled. You can enable it with --enable-threads ***

Python main interpreter initialized at 0x23fd960

your mercy for graceful operations on workers is 60 seconds

mapped 72760 bytes (71 KB) for 1 cores

*** Operational MODE: command ***

*** uWSGI is running in multiple interpreter mode ***

spawned uWSGI worker 1 (and the only) (pid: 32451, cores: 1)

.

/home/hisparc

/opt/miniconda/lib/python27.zip

/opt/miniconda/lib/python2.7

/opt/miniconda/lib/python2.7/plat-linux2

/opt/miniconda/lib/python2.7/lib-tk

/opt/miniconda/lib/python2.7/lib-old

/opt/miniconda/lib/python2.7/lib-dynload

/opt/miniconda/lib/python2.7/site-packages

/opt/miniconda/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg

The problem is clear: python does *not* look for libraries in the conda environment. It looks for libraries in the system-wide conda environment. If I remove miniconda from my path and use the system python to create a virtualenv and install uWSGI, the sys.path is:

.

/home/hisparc

/srv/publicdb/env/lib/python27.zip

/srv/publicdb/env/lib/python2.7

/srv/publicdb/env/lib/python2.7/plat-linux2

/srv/publicdb/env/lib/python2.7/lib-tk

/srv/publicdb/env/lib/python2.7/lib-old

/srv/publicdb/env/lib/python2.7/lib-dynload

/srv/publicdb/env/lib/python2.7/site-packages

/srv/publicdb/env/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg

The same (correct) behavior happens when I activate the conda environment before running uWSGI, like this:

$ source activate /srv/publicdb/env/

Why does it not correctly set the environment when I call the uWSGI binary without activating the environment? It works for virtualenvs, why not for conda environments?

Best regards,

David

[Aaron Meurer]

This is expected behavior. virtualenvs work by symlinking certain
parts of Python into a directory, so that things act reasonably
isolated. conda environments are much simpler. You should think of
each conda environment as a completely independent prefix. Something
you do to one environment should not affect the other. Therefore, it's
not reasonable for you to expect the python from the root conda
environment to find anything in the new environment. If you want it
to, just activate the environment (all this does is put it in the
front of your PATH), or call the python from the environment
explicitly, like /srv/publicdb/env/bin/python.

Aaron Meurer

> --
> Anaconda Community Support Group Brought to you by Continuum Analytics
> ---
> You received this message because you are subscribed to the Google Groups
> "Anaconda - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to anaconda+u...@continuum.io.
> To post to this group, send email to anac...@continuum.io.
> Visit this group at http://groups.google.com/a/continuum.io/group/anaconda/.

[David Fokkema]

But that's it, exactly. I *am* calling the uwsgi binary from my conda environment. I'm not calling the root environment. The binary is linked with libpython inside the conda environment. However, uwsgi only looks for packages in the root environment.

David

Verstuurd vanaf mijn iPad

> You received this message because you are subscribed to a topic in the Google Groups "Anaconda - Public" group.
> To unsubscribe from this topic, visit https://groups.google.com/a/continuum.io/d/topic/anaconda/JVxrCz9TZlI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to anaconda+u...@continuum.io.

[Aaron Meurer]

Based on your subject line, though, I'm assuming you have PYTHONHOME
set. I'm guessing this is the source of the problem. Does it work if
you unset it?

Aaron Meurer

[David Fokkema]

That's me staring too long at a computer screen trying to solve the same problem. Some information is clearly missing. I have *not* set PYTHONHOME, but my *root* conda environment is in the PATH (/opt/mini conda/bin). My conda environment is *not* (/srv/publicdb/publicdb_env/bin/).

Setting PYTHONHOME (or PATH) to my conda environment solves the problem. However, my old setup (using virtualenvs) relies on the fact that I can call *any* binary, without setting environment variables, like this:

$ /path/to/my/condaenv/bin/program options

and that python somehow figures out the correct path to the libraries installed in the conda environment or virtualenv. While plain python indeed does this, uwsgi does not. Since conda also has problems with creating a plain virtualenv, my guess was that this is a conda problem and not necessarily a uwsgi problem.

I tried everything I could think of to do two things:
1. Break uwsgi in my old virtualenv by degrading my virtualenv (i.e. removing stuff)
2. Getting uwsgi to work in my new conda environment (i.e. adding stuff)

I only managed to break uwsgi by removing the interpreter. I couldn't find hidden settings files or something, or a weird custom directory layout.

Other than how the relocation of the interpreter is managed in a condaenv and a virtualenv I can't think of any crucial differences. I might be wrong there, of course, and maybe uwsgi is doing some very specific things. But at the end of the day, uwsgi can figure out the correct python path when called in a virtualenv, but not when called in a conda environment.

And I *really* like that I can use the path to a binary to select an environment. That saves setting up environment variables in things like cron or supervisor and results in clean, expected behavior.

Thanks for your time!

David

[Travis Oliphant]

On Fri, Mar 7, 2014 at 3:13 PM, <davidf...@gmail.com> wrote:

Hi all,

First of all: miniconda saves me a lot of trouble. My ansible playbook for building a server is a lot shorter without all installing development libraries and compiling numpy, scipy, pytables, and the like. But now I'm having trouble...

Glad you are finding the tools useful. 

 

This is a first-warning sign, I think:

$ pip install virtualenv

$ virtualenv foo

New python executable in foo/bin/python

foo/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

ERROR: The executable foo/bin/python is not functioning

ERROR: It thinks sys.prefix is u'/home/hisparc' (should be u'/home/hisparc/foo')

ERROR: virtualenv is not compatible with this system or executable

I guess that conda does something arcane in how it handles relocating the python interpreter.

Theoretically virtualenv should work inside conda environments, but it's not the recommended way to go.  It's not clear to me why you would want to create a virtualenv inside a conda environment instead of just another conda environment (flat is better than nested and all that...).   

conda is not arcane at all --- it's actually very simple.  virtualenv is *far more* complicated and brittle in how it tries to manage environments --- with a few special-cases that it gets wrong in the case of conda environments. 

conda will get confused if PYTHONHOME is set.  It can also be problematic if you have LD_LIBRARY_PATH set (or the equivalent on Mac).    

what does conda info --system tell you? 

The problem seems to be with uwsgi.   uwsgi is hard-coding it's dependency on system libpython rather than using the one available in the environment conda is providing.   

I suspect it's something in the pip install uwsgi....   

.

/home/hisparc

/srv/publicdb/env/lib/python27.zip

/srv/publicdb/env/lib/python2.7

/srv/publicdb/env/lib/python2.7/plat-linux2

/srv/publicdb/env/lib/python2.7/lib-tk

/srv/publicdb/env/lib/python2.7/lib-old

/srv/publicdb/env/lib/python2.7/lib-dynload

/srv/publicdb/env/lib/python2.7/site-packages

/srv/publicdb/env/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg

The same (correct) behavior happens when I activate the conda environment before running uWSGI, like this:

$ source activate /srv/publicdb/env/

Why does it not correctly set the environment when I call the uWSGI binary without activating the environment? It works for virtualenvs, why not for conda environments?

Again, I believe it's because you pip installed uWSGI and it is not setting up the paths correctly when the pip install builds uWSGI --- basically the build script for uWSGI is not respecting the actual location of libpython for the environment but picking up the system or root build of python. 

The best option is to build a conda package for uwsgi.   There are several ways to do this.  The easiest is to try: 

conda skeleton pypi uWSGI     # This will make a recipe named uwsgi in your current working directory

conda build uwsgi  --no-test      # This will build the recipe but not run the tests (which break here). 

uWSGI embeds python and how it finds the libpython it needs is getting confused is my guess.   Building the conda package with conda build will fix the bad linking step that is happening by using pip install to build. 

Best,

-Travis

 

Best regards,

David

--
Anaconda Community Support Group Brought to you by Continuum Analytics
---
You received this message because you are subscribed to the Google Groups "Anaconda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to anaconda+u...@continuum.io.
To post to this group, send email to anac...@continuum.io.
Visit this group at http://groups.google.com/a/continuum.io/group/anaconda/.

--

Travis Oliphant

CEO

Continuum Analytics, Inc.

http://www.continuum.i

[Travis Oliphant]

You can also use the uwsgi package I just uploaded to my channel on binstar to see if that works for you: 

conda install -c travis uwsgi

I have linux-64 and mac osx packages up there.

-Travis

Travis Oliphant

CEO

Continuum Analytics, Inc.

http://www.continuum.io

[David Fokkema]

On 8 mrt. 2014, at 10:38 AM, Travis Oliphant <tra...@continuum.io> wrote:

Glad you are finding the tools useful. 

Me, too, :-)

 

Theoretically virtualenv should work inside conda environments, but it's not the recommended way to go.  It's not clear to me why you would want to create a virtualenv inside a conda environment instead of just another conda environment (flat is better than nested and all that...).   

I wanted to create a virtualenv that was as close to a conda environment as possible. So I installed the virtualenv package in the root conda environment, and used the virtualenv binary to create a new, empty, virtualenv. That did not work.

conda is not arcane at all --- it's actually very simple.  virtualenv is *far more* complicated and brittle in how it tries to manage environments --- with a few special-cases that it gets wrong in the case of conda environments. 

I'll take your word on this!

conda will get confused if PYTHONHOME is set.  It can also be problematic if you have LD_LIBRARY_PATH set (or the equivalent on Mac).    

what does conda info --system tell you? 

sys.version: 2.7.6 |Continuum Analytics, Inc.| (defau...

sys.prefix: /opt/miniconda

sys.executable: /opt/miniconda/bin/python

conda location: /opt/miniconda/lib/python2.7/site-packages/conda

conda-build: /opt/miniconda/bin/conda-build

conda-convert: /opt/miniconda/bin/conda-convert

conda-index: /opt/miniconda/bin/conda-index

conda-skeleton: /opt/miniconda/bin/conda-skeleton

CIO_TEST: <not set>

CONDA_DEFAULT_ENV: <not set>

CONDA_ENVS_PATH: <not set>

LD_LIBRARY_PATH: <not set>

PATH: /opt/miniconda/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hisparc/bin

PYTHONPATH: <not set>

At first, I suspected this too. However, ldd tells me:

$ ldd /srv/publicdb/publicdb_env/bin/uwsgi

linux-vdso.so.1 =>  (0x00007fff86bff000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbbe02d3000)

libm.so.6 => /srv/publicdb/publicdb_env/lib/libm.so.6 (0x00000032b9600000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007fbbe00ce000)

libz.so.1 => /srv/publicdb/publicdb_env/lib/libz.so.1 (0x00007fbbdfeb9000)

libssl.so.1.0.0 => /srv/publicdb/publicdb_env/lib/libssl.so.1.0.0 (0x00007fbbdfc55000)

libcrypto.so.1.0.0 => /srv/publicdb/publicdb_env/lib/libcrypto.so.1.0.0 (0x00007fbbdf892000)

libutil.so.1 => /lib64/libutil.so.1 (0x00007fbbdf68f000)

libpython2.7.so.1.0 => /srv/publicdb/publicdb_env/lib/libpython2.7.so.1.0 (0x00007fbbdf2be000)

libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fbbdf087000)

libc.so.6 => /lib64/libc.so.6 (0x00007fbbdecf3000)

/lib64/ld-linux-x86-64.so.2 (0x00007fbbe04f7000)

libfreebl3.so => /lib64/libfreebl3.so (0x00007fbbdea90000)

It seems to pick up the correct libpython.

The best option is to build a conda package for uwsgi.   There are several ways to do this.  The easiest is to try: 

conda skeleton pypi uWSGI     # This will make a recipe named uwsgi in your current working directory

conda build uwsgi  --no-test      # This will build the recipe but not run the tests (which break here). 

uWSGI embeds python and how it finds the libpython it needs is getting confused is my guess.   Building the conda package with conda build will fix the bad linking step that is happening by using pip install to build. 

Using these instructions, I build a package and installed it using --use-local.  The libraries have slightly different paths (including env/bin/../lib/, instead of just env/lib) but it still does not work. It still wants to import libraries from /opt/miniconda.

The problem can be with uwsgi, but I don't understand how. A virtualenv works, but in a conda environment it does not. What's the difference?

Thanks!

Best regards,

David

[David Fokkema]

On 8 mrt. 2014, at 10:44 AM, Travis Oliphant <tra...@continuum.io> wrote:

> You can also use the uwsgi package I just uploaded to my channel on binstar to see if that works for you:
>
> conda install -c travis uwsgi
>
> I have linux-64 and mac osx packages up there.
>
> -Travis
>

I'm testing this in scientific linux 6.4, which uses an older libc library and the binary won't run. Using the instructions in your previous mail I could build my own. I do marvel at the ease of use of conda channels (and skeleton, and build).

David

[ianoz...@gmail.com]

I've just worked through a solution for Anaconda + Apache + Flask + uWSGI (using Travis' conda package), it is documented on my blog: http://ianozsvald.com/2014/06/19/flask-mod_uwsgi-apache-continuums-anaconda/
HTH, Ian.

[Derek Hohls]

I know this is a very old thread (so I will start a new one if needed), but the OP's core question is "My actual problem is trying to install and use uWSGI in an actual conda environment," and it seems that the problem was not solved by end-of-thread.  Has anyone created a working approach for this?

Thanks

Derek

[David Fokkema]

> On 23 May 2015, at 16:18, Derek Hohls <game...@gmail.com> wrote:
>
> I know this is a very old thread (so I will start a new one if needed), but the OP's core question is "My actual problem is trying to install and use uWSGI in an actual conda environment," and it seems that the problem was not solved by end-of-thread. Has anyone created a working approach for this?
>
> Thanks
> Derek
>

Hi Derek,

I finally decided to install uWSGI using pip, in my virtualenv, and set PYTHONHOME to the path of my virtualenv. That did the trick. Not very elegant, but it works.

Best regards,
David

[jwdo...@gmail.com]

Hello David,

I threw out a Stackoverflow question a week ago: https://stackoverflow.com/questions/49988674/run-application-with-uwsgi-in-a-conda-virtualenv

Now I found this discussion. Did you eventually solved this somehow the last 3 years?

Best regards,


Joost

[David Fokkema]

Hi Joost,

Yes, I did. I don’t like the solution much, and maybe there’s a better one using current versions of software and tools. However, we still use the following on our production servers: first, set the PYTHONHOME environment variable to your Conda environment. Then call uwsgi like you did. I hope this works for you too.

For reference, our full provisioning scripts are available here: https://github.com/HiSPARC/publicdb/tree/master/provisioning/roles/publicdb

We use supervisor to start uwsgi. 

Best,

David

Verstuurd vanaf mijn iPhone

[Joost Döbken]

Aha, I get it, that is exactly what I have now!

However, if I want to run multiple apps with different environments on the same server; this will not work anymore.

btw, with 'sudo systemctl enable uwsgi' you can start uwsgi at boot

Joost

[David Fokkema]

On 1 May 2018, at 22:37, Joost Döbken <jwdo...@gmail.com> wrote:

Aha, I get it, that is exactly what I have now!

However, if I want to run multiple apps with different environments on the same server; this will not work anymore.

You don’t have to start all apps in the same environment. You can even set environment variables for the duration of one command only, like:

$ PYTHONHOME=/srv/my_python_env python script.py

Many init-scripts or config files for supervisor-like programs also allow you to specify environment variables per app.

btw, with 'sudo systemctl enable uwsgi' you can start uwsgi at boot

Well, not on the pretty old CT-supported distributions we have to work with… We have to pip install uwsgi into a conda environment.

David