What do i need to do so that mod_wsgi will find libmysqlclient.16.dylib? I don't seem to be able to set DYLD_LIBRARY_PATH

261 views
Skip to first unread message

mathew davis

unread,
Nov 18, 2011, 4:41:50 PM11/18/11
to mod...@googlegroups.com
I am trying to run django on osx 10.7 (lion) with apache mod_wsgi and virtualenv.
My site works if I use the django testing server:

>     (baseline)otter:hello mathew$ python manage.py runserver

but it doesn't work when I run apache. The core of the error seems to be

>     Library not loaded: libmysqlclient.16.dylib

I think its to do with the path apache is using to locate libmysqlclient.16.dylib

when I run otool in the lib directory it looks good

>     otter:lib mathew$ pwd
>         /usr/local/mysql/lib
>        
>     otter:lib mathew$ otool -L libmysqlclient.16.dylib
>         libmysqlclient.16.dylib:
>             libmysqlclient.16.dylib (compatibility version 16.0.0, current version 16.0.0)
>             /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1)
>  

but from outside it can't find it

>     otter:lib mathew$ cd /
>     otter:/ mathew$ otool -L libmysqlclient.16.dylib
>     otool: can't open file: libmysqlclient.16.dylib (No such file or directory)

if i manually set DYLD_LIBRARY_PATH otool works

> otter:lib mathew$ DYLD_LIBRARY_PATH=/usr/local/mysql/lib otter:lib
> mathew$ otool -L libmysqlclient.16.dylib libmysqlclient.16.dylib:
>     libmysqlclient.16.dylib (compatibility version 16.0.0, current
> version 16.0.0)     /usr/lib/libSystem.B.dylib (compatibility version
> 1.0.0, current version 125.0.1)

When I run the django testing server, my .bash_profile sets up the virtualenv and the path to the mysql dynamic library

>     export DYLD_LIBRARY_PATH=/usr/local/mysql/lib/:$DYLD_LIBRARY_PATH
>     export PATH

When i run apache it finds my virtualenv paths, but it doesn't seem to find the dynamic library path.

I tried adding this path to /usr/sbin/envvars

>     DYLD_LIBRARY_PATH="/usr/lib:/usr/local/mysql/lib:$DYLD_LIBRARY_PATH"
>     export DYLD_LIBRARY_PATH

and to /private/etc/paths.d/libmysql

>     /usr/local/mysql/lib

then restarted the machine
but that has not changed the error message.

>     Error loading MySQLdb module: dlopen(/usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/_mysql.so,
> 2):
>         Library not loaded: libmysqlclient.16.dylib


I don't think is a permissions issue:

>     -rwxr-xr-x  1 root  wheel  3787328  4 Dec  2010 libmysqlclient.16.dylib
>    
>     drwxr-xr-x  39 root    wheel   1394 18 Nov 21:07 /
>     drwxr-xr-x@ 15 root    wheel    510 24 Oct 22:10 /usr
>     drwxrwxr-x  20 root    admin    680  2 Nov 20:22 /usr/local
>     drwxr-xr-x  20 mathew  admin    680  9 Nov 21:58 /usr/local/python_virtualenv
>     drwxr-xr-x   6 mathew  admin    204  2 Nov 21:36 /usr/local/python_virtualenv/baseline
>     drwxr-xr-x   4 mathew  admin    136  2 Nov 21:26 /usr/local/python_virtualenv/baseline/lib
>     drwxr-xr-x  52 mathew  admin   1768  2 Nov 21:26 /usr/local/python_virtualenv/baseline/lib/python2.7
>     drwxr-xr-x  18 mathew  admin    612  4 Nov 21:20 /usr/local/python_virtualenv/baseline/lib/python2.7/site-packages
>     -rwxr-xr-x   1 mathew  admin  66076  2 Nov 21:18 /usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/_mysql.so

         

**What do i need to do so that mod_wsgi will find libmysqlclient.16.dylib?**


apache and mysql are both 64 bit:

> otter:lib mathew$ file /usr/sbin/httpd /usr/sbin/httpd: Mach-O
> universal binary with 2 architectures /usr/sbin/httpd (for
> architecture x86_64):    Mach-O 64-bit executable x86_64 /usr/sbin/httpd
> (for architecture i386):    Mach-O executable i386 otter:lib mathew$
>
> otter:lib mathew$ file /usr/local/mysql/lib/libmysqlclient.16.dylib
> /usr/local/mysql/lib/libmysqlclient.16.dylib: Mach-O 64-bit
> dynamically linked shared library x86_64


here's some more detail of the error:


> Request URL:    http://hello.local/ Django Version:    1.3.1 Exception
> Type:    ImproperlyConfigured Exception Value:     Error loading MySQLdb
> module:
> dlopen(/usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/_mysql.so,
> 2): Library not loaded: libmysqlclient.16.dylib   Referenced from:
> /usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/_mysql.so
> Reason: image not found Exception
> Location:    /usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/django/db/backends/mysql/base.py
> in <module>, line 14 Python Executable:    /usr/sbin/httpd Python
> Version:    2.7.2 Python Path:   
> ['/usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg',
> '/usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg',  '/usr/local/python_virtualenv/baseline/lib/python27.zip',
> '/usr/local/python_virtualenv/baseline/lib/python2.7',
> '/usr/local/python_virtualenv/baseline/lib/python2.7/plat-darwin',
> '/usr/local/python_virtualenv/baseline/lib/python2.7/plat-mac',
> '/usr/local/python_virtualenv/baseline/lib/python2.7/plat-mac/lib-scriptpackages',
> '/usr/local/python_virtualenv/baseline/lib/python2.7/lib-tk',
> '/usr/local/python_virtualenv/baseline/lib/python2.7/lib-old',
> '/usr/local/python_virtualenv/baseline/lib/python2.7/lib-dynload',
> '/usr/local/Cellar/python/2.7.2/lib/python2.7',
> '/usr/local/Cellar/python/2.7.2/lib/python2.7/plat-darwin',
> '/usr/local/Cellar/python/2.7.2/lib/python2.7/lib-tk',
> '/usr/local/Cellar/python/2.7.2/lib/python2.7/plat-mac',
> '/usr/local/Cellar/python/2.7.2/lib/python2.7/plat-mac/lib-scriptpackages',
> '/usr/local/python_virtualenv/baseline/lib/python2.7/site-packages',
> '/usr/local/python_virtualenv/baseline/lib/python2.7/site-packages/PIL',
> '/Users/mathew/dev',  '/usr/local/mysql/lib/']


Graham Dumpleton

unread,
Nov 18, 2011, 7:33:41 PM11/18/11
to mod...@googlegroups.com
This is an issue with how MySQLdb Python module is built. Can you tell
me how you installed it?

Did you use pip, or did you download package as tar.gz and run setup.py in it?

I have tried twice before to sort this out with people on IRC but they
would not do what I asked them to help me solve it properly. I am at
least most of the way along sorting out what the issue is however. :-)

Graham

> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/modwsgi/-/YIrdgT4hh-oJ.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to
> modwsgi+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/modwsgi?hl=en.
>

Simon Kesenci

unread,
Nov 18, 2011, 8:23:48 PM11/18/11
to mod...@googlegroups.com, mod...@googlegroups.com
I've solved this with install_name_tool:

sudo install_name_tool -change libmysqlclient.16.dylib /full/path/to/libmysqlclient.16.dylib {file which can't load dylib}

--S.

mathew davis

unread,
Nov 18, 2011, 9:58:24 PM11/18/11
to modwsgi
hmm,

I installed MySQL with the download image

I used brew to install mod_wsgi

and pip to install Django and MySQL-python

as I couldn't remember whether I already had DYLD_LIBRARY_PATH set in
my bash profile when I did this,
I have reinstalled mysql, MySQL-python and mod_wsgi


On Nov 19, 11:33 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:


> This is an issue with how MySQLdb Python module is built. Can you tell
> me how you installed it?
>
> Did you use pip, or did you download package as tar.gz and run setup.py in it?
>
> I have tried twice before to sort this out with people on IRC but they
> would not do what I asked them to help me solve it properly. I am at
> least most of the way along sorting out what the issue is however. :-)
>
> Graham
>

> >> Request URL:   http://hello.local/Django Version:    1.3.1 Exception

mathew davis

unread,
Nov 18, 2011, 9:59:56 PM11/18/11
to modwsgi
i'm doing something wrong, i'm getting a usage warning that my
install_name_tool syntax is wrong

(baseline)otter:lib mathew$ pwd
/usr/local/mysql/lib
(baseline)otter:lib mathew$ sudo install_name_tool -change
libmysqlclient.18.dylib /usr/local/mysql/lib/libmysqlclient.18.dylib
Password:
Usage: install_name_tool [-change old new] ... [-rpath old new] ... [-
add_rpath new] ... [-delete_rpath old] ... [-id name] input
(baseline)otter:lib mathew$ otool -L libmysqlclient.18.dylib
libmysqlclient.18.dylib:
libmysqlclient.18.dylib (compatibility version 18.0.0, current
version 18.0.0)


/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 125.0.1)

On Nov 19, 12:23 pm, Simon Kesenci <genera...@gmail.com> wrote:
> I've solved this with install_name_tool:
>
> sudo install_name_tool -change libmysqlclient.16.dylib /full/path/to/libmysqlclient.16.dylib {file which can't load dylib}
>
> --S.
>

> On Nov 18, 2011, at 16:33, Graham Dumpleton <graham.dumple...@gmail.com> wrote:
>
>
>
>
>
>
>
> > This is an issue with how MySQLdb Python module is built. Can you tell
> > me how you installed it?
>
> > Did you use pip, or did you download package as tar.gz and run setup.py in it?
>
> > I have tried twice before to sort this out with people on IRC but they
> > would not do what I asked them to help me solve it properly. I am at
> > least most of the way along sorting out what the issue is however. :-)
>
> > Graham
>

> >>> Request URL:    http://hello.local/Django Version:    1.3.1 Exception

Simon Kesenci

unread,
Nov 18, 2011, 10:19:13 PM11/18/11
to mod...@googlegroups.com
You left out the last argument, {file which can't load dylib}, probably the MySQLdb Python module.

--S.

mathew davis

unread,
Nov 18, 2011, 10:35:07 PM11/18/11
to modwsgi
ok now I see

sudo install_name_tool -change libmysqlclient.18.dylib /usr/local/
mysql/lib/libmysqlclient.18.dylib /usr/local/python_virtualenv/
baseline/lib/python2.7/site-packages/_mysql.so

i was wondering how install_name_tool was globally changing references
to libmysqlclient.18.dylib, now I see it is only patching _mysql.so

Well that's fixed my problem, but it would be interesting to
understand why _mysql.so couldn't find libmysqlclient.18.dylib in the
first place


thank you for all you help,

let me know if there is anything I can do if you still wish to get to
the bottom of this problem

thank you

Mathew

> >>>>> Request URL:    http://hello.local/DjangoVersion:    1.3.1 Exception

Simon Kesenci

unread,
Nov 18, 2011, 11:31:56 PM11/18/11
to mod...@googlegroups.com
/usr/local/mysql/lib apparently does not appear in your library path. I haven't installed a Python MySQL module in some years but earlier this year when I installed PHP with MySQL the PHP executables would not link unless I ran all the make commands with DYLD_LIBRARY_PATH="/usr/local/mysql/lib". And then I had to use install_name_tool as mentioned to get PHP to run. Funny enough, otool showed all the other libraries with their full path names. I haven't encountered this problem with any other package.

--S.

mathew davis

unread,
Nov 19, 2011, 6:06:02 AM11/19/11
to modwsgi
the thing is I have tried to add the strange thing is I did try to set
DYLD_LIBRARY_PATH, but it didn't work.

for now I'm happy that configuration is work, but next time I install
python mysql, I'll experiment again and seee if I can find a way to
make it work without install_name_tool

thank you simon

> ...
>
> read more »

Graham Dumpleton

unread,
Nov 19, 2011, 2:45:02 PM11/19/11
to mod...@googlegroups.com
Sorry for not replying, but I have lost Internet access at home and may not get it back for a few more days.

Right now you are treating the symptom and not the cause.

What you have may still not work reliably because the extension is only linked with single threaded version of MySQL library and not the re entrant _r version.

Not linking with _r version would be caused by MySQL config script not being found or not running properly when pip was run to install module.

I would like to sort this out properly still when I get my Internet back.

Graham

mathew davis

unread,
Nov 20, 2011, 7:00:51 AM11/20/11
to modwsgi
Thanks Graham,

the current solution at least allows me to get on with other
development work,
and I'd be happy to try and help resolve this when you can.

Mat

On Nov 20, 6:45 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:


> Sorry for not replying, but I have lost Internet access at home and may not
> get it back for a few more days.
>
> Right now you are treating the symptom and not the cause.
>
> What you have may still not work reliably because the extension is only
> linked with single threaded version of MySQL library and not the re entrant
> _r version.
>
> Not linking with _r version would be caused by MySQL config script not
> being found or not running properly when pip was run to install module.
>
> I would like to sort this out properly still when I get my Internet back.
>
> Graham
>

> On Saturday, 19 November 2011, Simon Kesenci <genera...@gmail.com> wrote:
> > I've solved this with install_name_tool:
>
> > sudo install_name_tool -change libmysqlclient.16.dylib
>
> /full/path/to/libmysqlclient.16.dylib {file which can't load dylib}
>
>
>
>
>
>
>
>
>
> > --S.
>

> > On Nov 18, 2011, at 16:33, Graham Dumpleton <graham.dumple...@gmail.com>


> wrote:
>
> >> This is an issue with how MySQLdb Python module is built. Can you tell
> >> me how you installed it?
>
> >> Did you use pip, or did you download package as tar.gz and run setup.py
> in it?
>
> >> I have tried twice before to sort this out with people on IRC but they
> >> would not do what I asked them to help me solve it properly. I am at
> >> least most of the way along sorting out what the issue is however. :-)
>
> >> Graham
>

Graham Dumpleton

unread,
Nov 21, 2011, 7:22:57 PM11/21/11
to mod...@googlegroups.com
Some details on this finally.

FWIW, this may be a MySQL 5.5 issue.

At least I used to do this on MySQL 5.1 and had no problems at all. I
have though since upgraded to MySQL 5.5 and see potential for
problems.

If you try and build MySQL client lib and don't have MySQL bin
directory in path it will fail with:

$ python setup.py build
sh: mysql_config: command not found
Traceback (most recent call last):
File "setup.py", line 15, in <module>
metadata, options = get_config()
File "/Users/graham/Packages/MySQL-python-1.2.3/setup_posix.py",
line 43, in get_config
libs = mysql_config("libs_r")
File "/Users/graham/Packages/MySQL-python-1.2.3/setup_posix.py",
line 24, in mysql_config
raise EnvironmentError("%s not found" % (mysql_config.path,))
EnvironmentError: mysql_config not found

If however the MySQL bin directory is in path you would on MacOS X
10.6 (Snow Leopard) get:

$ PATH=/usr/local/mysql/bin:$PATH python setup.py build
running build
running build_py
creating build
creating build/lib.macosx-10.6-universal-2.6
copying _mysql_exceptions.py -> build/lib.macosx-10.6-universal-2.6
creating build/lib.macosx-10.6-universal-2.6/MySQLdb
copying MySQLdb/__init__.py -> build/lib.macosx-10.6-universal-2.6/MySQLdb
copying MySQLdb/converters.py -> build/lib.macosx-10.6-universal-2.6/MySQLdb
copying MySQLdb/connections.py -> build/lib.macosx-10.6-universal-2.6/MySQLdb
copying MySQLdb/cursors.py -> build/lib.macosx-10.6-universal-2.6/MySQLdb
copying MySQLdb/release.py -> build/lib.macosx-10.6-universal-2.6/MySQLdb
copying MySQLdb/times.py -> build/lib.macosx-10.6-universal-2.6/MySQLdb
creating build/lib.macosx-10.6-universal-2.6/MySQLdb/constants
copying MySQLdb/constants/__init__.py ->
build/lib.macosx-10.6-universal-2.6/MySQLdb/constants
copying MySQLdb/constants/CR.py ->
build/lib.macosx-10.6-universal-2.6/MySQLdb/constants
copying MySQLdb/constants/FIELD_TYPE.py ->
build/lib.macosx-10.6-universal-2.6/MySQLdb/constants
copying MySQLdb/constants/ER.py ->
build/lib.macosx-10.6-universal-2.6/MySQLdb/constants
copying MySQLdb/constants/FLAG.py ->
build/lib.macosx-10.6-universal-2.6/MySQLdb/constants
copying MySQLdb/constants/REFRESH.py ->
build/lib.macosx-10.6-universal-2.6/MySQLdb/constants
copying MySQLdb/constants/CLIENT.py ->
build/lib.macosx-10.6-universal-2.6/MySQLdb/constants
running build_ext
building '_mysql' extension
creating build/temp.macosx-10.6-universal-2.6
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv
-Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -pipe
-Dversion_info=(1,2,3,'final',0) -D__version__=1.2.3
-I/usr/local/mysql/include
-I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
-c _mysql.c -o build/temp.macosx-10.6-universal-2.6/_mysql.o -Os -g
-fno-common -fno-strict-aliasing -arch x86_64
In file included from _mysql.c:36:
/usr/local/mysql/include/my_config.h:329:1: warning: "SIZEOF_SIZE_T" redefined
In file included from
/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:9,
from pymemcompat.h:10,
from _mysql.c:29:
/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/pymacconfig.h:33:1:
warning: this is the location of the previous definition
In file included from _mysql.c:36:
/usr/local/mysql/include/my_config.h:422:1: warning: "HAVE_WCSCOLL" redefined
In file included from
/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:8,
from pymemcompat.h:10,
from _mysql.c:29:
/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/pyconfig.h:803:1:
warning: this is the location of the previous definition
gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup
build/temp.macosx-10.6-universal-2.6/_mysql.o -L/usr/local/mysql/lib
-lmysqlclient_r -lpthread -o
build/lib.macosx-10.6-universal-2.6/_mysql.so -arch x86_64

Important to note here is that it links against "-lmysqlclient_r".

The "_r" version is reentrant and is what should be used if running
multithread process.

Which library is linked is dependent on code defined in
'setup_posix.py' of the MySQL package.

In that file it has:

def get_config():
import os, sys
from setup_common import get_metadata_and_options, enabled,
create_release_file

metadata, options = get_metadata_and_options()

if 'mysql_config' in options:
mysql_config.path = options['mysql_config']

extra_objects = []
static = enabled(options, 'static')
if enabled(options, 'embedded'):
libs = mysql_config("libmysqld-libs")
client = "mysqld"
elif enabled(options, 'threadsafe'):
libs = mysql_config("libs_r")
client = "mysqlclient_r"
if not libs:
libs = mysql_config("libs")
client = "mysqlclient"
else:
libs = mysql_config("libs")
client = "mysqlclient"

The flags actually come from site.cfg which has by default:

[options]
# embedded: link against the embedded server library
# threadsafe: use the threadsafe client
# static: link against a static library (probably required for embedded)

embedded = False
threadsafe = True
#threadsafe = False
static = False

So, by default if build package from source yourself (not using pip)
it should find threadsafe and link against _r version of library.

For MySQL 5.5 at least:

$ ls -las /usr/local/mysql/lib
total 368624
0 drwxr-xr-x 12 root wheel 408 31 Oct 14:23 .
0 drwxr-xr-x 16 root wheel 544 12 Oct 23:21 ..
7400 -rwxr-xr-x 1 root wheel 3787624 12 Oct 23:19 libmysqlclient.18.dylib
19552 -rw-r--r-- 1 root wheel 10008760 12 Oct 23:19 libmysqlclient.a
8 lrwxr-xr-x 1 root wheel 23 31 Oct 14:23
libmysqlclient.dylib -> libmysqlclient.18.dylib
8 lrwxr-xr-x 1 root wheel 20 31 Oct 14:23
libmysqlclient_r.18.dylib -> libmysqlclient.dylib
8 lrwxr-xr-x 1 root wheel 16 31 Oct 14:23
libmysqlclient_r.a -> libmysqlclient.a
8 lrwxr-xr-x 1 root wheel 20 31 Oct 14:23
libmysqlclient_r.dylib -> libmysqlclient.dylib
175552 -rw-r--r-- 1 root wheel 89879064 12 Oct 23:16 libmysqld-debug.a
166064 -rw-r--r-- 1 root wheel 85023840 12 Oct 23:19 libmysqld.a
24 -rw-r--r-- 1 root wheel 8488 12 Oct 23:19 libmysqlservices.a
0 drwxr-xr-x 14 root wheel 476 12 Oct 23:21 plugin

Under MySQL 5.1 am pretty sure the _r library wasn't a symlink.

Changing site.cfg to not say threadsafe and ensure -l option matches
library used makes no difference.

So basically this looks like an issue with how MySQL 5.5 libraries are
built. There didn't used to be an issue with MySQL 5.1.

FWIW, I now get:

$ otool -L build/lib.macosx-10.6-universal-2.6/_mysql.so
build/lib.macosx-10.6-universal-2.6/_mysql.so:


libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current

version 125.2.11)

Under 5.1 it used to resolve the library no problem.

I will have to rebuild my MySQL Python client library properly at some
point and will look more at the problem then.

Graham

Graham Dumpleton

unread,
Dec 5, 2012, 8:11:36 PM12/5/12
to modwsgi
Someone early in the thread said that it can be fixed by using install_name_tool.

sudo install_name_tool -change libmysqlclient.16.dylib /full/path/to/libmysqlclient.16.dylib {file which can't load dylib}

Stuff elsewhere I have read has suggested that something like this is the only way. When they built the shared library they screwed up with the naming or something.

Another link about it is:


but I have seen other pages somewhere which describe the whole problem better, I just don't remember where.

Graham


On 6 December 2012 11:39, Joe Siravo <joe.s...@gmail.com> wrote:
Graham, i found this searching google and was wondering how you fixed this issue? i am having the same issue with libmysqlclient_r.so.16 and i am not sure what to do... any suggestions? just a fyi i cannot get root access from my hosting provider. 
To view this discussion on the web visit https://groups.google.com/d/msg/modwsgi/-/bPsejT2rbusJ.
Reply all
Reply to author
Forward
0 new messages