SSL support not compiled into conda psycopg2 package?

[David Moss]

I recently switched away from virtualenv/pip to using conda for a new project and I ran into the following problem.

The official psycopg2 2.6 package seems broken on OS X. The library was installed in a fresh conda environment using :-

conda install psycopg2

It seems to function normally until you attempt to use the DSN connection string flags that enable SSL. At this point, you run into the following exception:

...

with psycopg2.connect(**connect_args) as conn:

File "/Users/drkjam/miniconda/envs/psycopg2/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect

conn = _connect(dsn, connection_factory=connection_factory, async=async)

psycopg2.OperationalError: sslmode value "require" invalid when SSL support is not compiled in

Where connect_args is a dictionary containing the entry {'sslmode': 'require'} (one of the options that forms part of an SSL connection).

This error is being generated from somewhere within libpq.so.

As a workaround I uninstalled the native conda package and tried using the pip version instead:

conda uninstall psycopg2

pip install psycopg2

This gave me a different error:

$ python

Python 2.7.9 |Continuum Analytics, Inc.| (default, Dec 15 2014, 10:37:34)

...

>>> import psycopg2

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "/Users/drkjam/miniconda/envs/psycopg2/lib/python2.7/site-packages/psycopg2/__init__.py", line 50, in <module>

from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID

ImportError: dlopen(/Users/drkjam/miniconda/envs/psycopg2/lib/python2.7/site-packages/psycopg2/_psycopg.so, 2): Library not loaded: libssl.1.0.0.dylib

Referenced from: /Users/drkjam/miniconda/envs/psycopg2/lib/python2.7/site-packages/psycopg2/_psycopg.so

Reason: image not found

However, it was easy to fix as the paths in  _psycopg.so to libssl and libcrypto were pointing to the wrong locations.

I updated them by modifying the paths like so:

install_name_tool -change "libssl.1.0.0.dylib" "@loader_path/../../../libssl.1.0.0.dylib" ${CONDA_HOME}/envs/psycopg2/lib/python2.7/site-packages/psycopg2/_psycopg.so

install_name_tool -change "libcrypto.1.0.0.dylib" "@loader_path/../../../libcrypto.1.0.0.dylib" ${CONDA_HOME}/envs/psycopg2/lib/python2.7/site-packages/psycopg2/_psycopg.so

This fixed my problem and the SSL connections are now working fine.

Can someone take a look at the build for this package and re-release it with SSL enabled so I don't have to apply this workaround going forward?

Best regards,

David Moss

[Michal]

Hi,

I came across the same issue.

Thanks David for this post!

It would be great if someone could fix this so that we don't need to do this workaround every time.

Also when I am using the conda environments should I be doing the same whole thing for each of environments with modified paths?

Thank you and best regards,

Michal

[Ilan Schnell]

We recompiled psycopg2 with have_ssl=1 (in setup.cfg).  I hope this fixes the problem.  Use:

$ conda update psycopg2

to get the update.
On OSX, we now have already:
$ otool -L _psycopg.so
_psycopg.so:
    @loader_path/../../../libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    @loader_path/../../../libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.0.0)

- Ilan

--
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+un...@continuum.io.
To post to this group, send email to co...@continuum.io.
Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

[Andrew Harris]

This still isn't working in python 3.4? Is it possible to get SSL support compiled in the python3 version of psycopg2?

[michal szczecinski]

Hi,

I did that and now I am getting this error:

   conn = _connect(dsn, connection_factory=connection_factory, async=async)

psycopg2.OperationalError: missing "=" after "postgres://blablabla:blabla..." in connection info string

Just to provide some background I am using this for Luigi enabled log history saved to my postgres db.

Thanks and best regards,

Michal

[Michal]

My apologies, this error I think was caused by something else when I updated my conda packages and python version.

However I second Andrew's post, it still does not seem to work for Python 3.4, is it possible to roll it out there too?

Michal

To unsubscribe from this group and stop receiving emails from it, send an email to conda+unsubscribe@continuum.io.

[Adrian Palacios]

I just ran into the same problem. With condo I uninstalled psycopg2 w/ conda, then reinstalled and even ran update for this package but still getting a similar error.

Only difference is I am using a different dialect, specifically, a library to connect to redshift: https://pypi.python.org/pypi/redshift-sqlalchemy

So, my error is just slightly different:

OperationalError: (OperationalError) sslmode value "require" invalid when SSL support is not compiled in

 None None

I'm new to python/anaconda so I am not sure if this is on the conda side or if it's the redshift package?

Thanks!

On Thursday, June 4, 2015 at 2:47:37 PM UTC-4, Ilan wrote:

[Samantha Zeitlin]

ok, so this will be a long reply, but I'm having the same problem. In the process of trying to fix it, I further screwed up my SSL, so although I've now fixed it (?) at least partly (see below), but something is still wrong:

(myenv)~$ conda list
# packages in environment at /Users/szeitlin/anaconda/envs/myenv:
#

Traceback (most recent call last):

  File "/Users/szeitlin/anaconda/envs/myenv/lib/python3.4/site-packages/pip-1.4.1-py3.4.egg/pip/backwardcompat/__init__.py", line 123, in <module>
    from ssl import match_hostname, CertificateError
  File "/Users/szeitlin/anaconda/envs/myenv/lib/python3.4/ssl.py", line 97, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: dlopen(/Users/szeitlin/anaconda/envs/myenv/lib/python3.4/lib-dynload/_ssl.so, 2): Symbol not found: _SSLv2_method
  Referenced from: /Users/szeitlin/anaconda/envs/myenv/lib/python3.4/lib-dynload/_ssl.so
  Expected in: /Users/szeitlin/anaconda/envs/myenv/lib/python3.4/lib-dynload/../../libssl.1.0.0.dylib

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/szeitlin/anaconda/envs/myenv/bin/pip", line 4, in <module>
    from pip import main
  File "/Users/szeitlin/anaconda/envs/myenv/lib/python3.4/site-packages/pip-1.4.1-py3.4.egg/pip/__init__.py", line 9, in <module>
    from pip.log import logger
  File "/Users/szeitlin/anaconda/envs/myenv/lib/python3.4/site-packages/pip-1.4.1-py3.4.egg/pip/log.py", line 7, in <module>
    from pip import backwardcompat
  File "/Users/szeitlin/anaconda/envs/myenv/lib/python3.4/site-packages/pip-1.4.1-py3.4.egg/pip/backwardcompat/__init__.py", line 125, in <module>
    from ssl_match_hostname import match_hostname, CertificateError
ImportError: No module named 'ssl_match_hostname'
# Warning: subprocess call to pip failed

However, after hours of digging through SO, I was able to determine that part of the problem was due to python3. 

So I fixed my problem outside of conda by reinstalling python3 using homebrew, and then I was able to install psycopg2 with pip3. 

sudo brew install python3 --with-brewed-openssl --build-from-source

pip3 install --upgrade pip

Successfully installed pip-7.1.0

pip3 install psycopg2

psycopg2 in /usr/local/lib/python3.4/site-packages

But I still get: 

(myenv)~$ conda install psycopg2
Fetching package metadata: Could not connect to https://repo.continuum.io/pkgs/pro/noarch/
.Could not connect to https://repo.continuum.io/pkgs/pro/osx-64/
.Could not connect to https://repo.continuum.io/pkgs/free/noarch/
.Could not connect to https://repo.continuum.io/pkgs/free/osx-64/
.
Solving package specifications: .
Package plan for installation in environment /Users/szeitlin/anaconda/envs/myenv:
The following packages will be downloaded:
    package                    |            build
    ---------------------------|-----------------
    psycopg2-2.6               |           py34_1         287 KB
The following NEW packages will be INSTALLED:
    psycopg2: 2.6-py34_1
Proceed ([y]/n)? y
Fetching packages ...
Could not connect to https://repo.continuum.io/pkgs/free/osx-64/psycopg2-2.6-py34_1.tar.bz2
Error: Connection error: Can't connect to HTTPS URL because the SSL module is not available.: https://repo.continuum.io/pkgs/free/osx-64/psycopg2-2.6-py34_1.tar.bz2

I finally gave up and just installed it in this env using pip3, and that worked. 

But when I try to run django, it still says:

ImportError: dlopen(/Users/szeitlin/anaconda/envs/myenv/lib/python3.4/site-packages/psycopg2/_psycopg.so, 2): Library not loaded: libssl.1.0.0.dylib
  Referenced from: /Users/szeitlin/anaconda/envs/myenv/lib/python3.4/site-packages/psycopg2/_psycopg.so
  Reason: image not found

So I'm not sure if the problem is specific to conda per se, because it seems like conda is not talking to pip, and there's still some confusion about where the various ssl pieces need to be for homebrew and conda to both find them (and I've run into some incompatibility with those symlinks before). 

I don't know how to tell django to look where the working versions of libssl.1.0.0.dylib are, short of making a new env. In the past when I've had problems like this, I wasn't able to find a better workaround. 

I may still have to uninstall conda and then reinstall it, because right now I can't use it to install any new packages. 

So I don't know if that helps... 

[Adrian Palacios]

Wow thanks for such a thorough response. I'm curious, with the last remaining error you have...I remember at one point I had to fix my symlinks as well, something like:

$ sudo rm /Users/lib/libssl.1.0.0.dylib

$ sudo rm /Users/lib/libcrypto.1.0.0.dylib

$ sudo ln -s /Users/adrianp/anaconda/lib/libssl.1.0.0.dylib

$ sudo ln -s /Users/adrianp/anaconda/lib/libcrypto.1.0.0.dylib

More details here: http://stackoverflow.com/questions/27264574/import-psycopg2-library-not-loaded-libssl-1-0-0-dylib

Would that fix your last error?

[Samantha Zeitlin]

Yes, I had tried that symlink trick before. Maybe the secret is to first install psycopg2, and then redo the symlinks, since that's more or less what I did.

So... some (?) things are sort of (?) working now, in the sense that I can get django and psycopg2 to import into python both inside and outside of conda and pip environments.

But, and this is a big problem for me, anaconda is still not happy. If I try to install the latest iPython notebook via the Launcher:

[RuntimeError(u"Connection error: Can't connect to HTTPS URL because the SSL module is not available.: https://repo.continuum.io/pkgs/free/osx-64/yaml-0.1.6-0.tar.bz2\n",)]

It sure would be nice if there were an easy way to tell what anaconda is looking for, and where it's looking for it. In other words, where do I look up what version(s) of SSL files are the right ones to have, and the path? And how do I check the versions of these dylib files (I'm guessing that may not be possible once they're compiled into binaries-?)?

sam

[VictorF]

Folks, I had the same issue as well, and it took me a WHOLE day to fix it, it's working now!!!

It's a super weird "BUG", and I had a super simple and hacky solution: 

My env: Python 2.7.10 |Anaconda 2.1.0 (x86_64)| (default, May 28 2015, 17:04:42)  [GCC 4.2.1 (Apple Inc. build 5577)] on darwin

• Fix the .so dylib file linking : 
•   sudo ln -s /Users/YOURUSERNAME/anaconda/lib/libssl.1.0.0.dylib /usr/lib
•   sudo ln -s /Users/YOURUSERNAME/anaconda/lib/libcrypto.1.0.0.dylib /usr/lib
• Do not use "conda install psycopg2" 
• Use "pip install psycopg2"

Check if it's installed : 

conda list

psycopg2                  2.6                      py27_1

THEN, bingo!!! 

I am now able to connect to the Heroku postgres using SSL mode, and insert / delete rows.!! 

[David Moss]

Is there any chance in the near future of the Continuum (Anaconda) version of the psycopg2 package being updated to support SSL properly so we can stop relying on pip and all these hacks?

[Lucas Chapin]

For posterity, a workaround that worked for me without having to rely on symlinks:

As others have suggested, install with pip:

$ conda uninstall psycopg2

$ pip install psycopg2

Python will throw an error when trying to import psycopg2, so run this:

$ export DYLD_FALLBACK_LIBRARY_PATH=$HOME/anaconda/lib/:$DYLD_FALLBACK_LIBRARY_PATH

Credit: http://stackoverflow.com/questions/27264574/import-psycopg2-library-not-loaded-libssl-1-0-0-dylib

[och...@sbcglobal.net]

Thank you!  This solved my problem after many hours of trying other things.

[Michael Sarahan]

While I'm glad that pip has gotten you working, you should be very careful with this approach in general.  Pip lacks a dependency solver, and as such it happily creates "inconsistent" environments, where one dependency may conflict with another.  These inconsistencies may directly break your software, or it may cause problems with conda down the road, because conda does have a solver, and in the face of inconsistency, conda can do unintuitive things.  The more things you install with pip, the more likely some kind of conflict is.  Sometimes they are harmless, but sometimes they totally break an environment.  It is definitely a good idea to use conda environments, so that if things break, you can easily re-create your environment and limit your downtime.

Good luck, and may all your environments be well-behaved.

Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/conda/feab34c3-d97e-42a3-a204-575edb8d55c7%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.