centos 7, compiled python 3.10, undefined symbol GENERAL_NAME_free

764 views
Skip to first unread message

Lou King

unread,
Jun 6, 2022, 6:43:46 AM6/6/22
to modwsgi
I'm getting an error trying to start a service which uses mod_wsgi 4.9.1 in a virtual environment.

On my centos 7 system, I compiled python 3.10, which is altinstalled to /usr/local/bin/python3.10

I have a virtualenv, but may have initially built mod_wsgi with an earlier version of python 3.10 (I was having trouble getting it to build correctly). However, my last attempt I used 

pip install --ignore-installed --no-cache-dir mod_wsgi==4.9.1 --no-binary mod_wsgi

to try to get it to build with the latest python3.10 (I tried several pip installs, starting with just --no-cache-dir)

Note I am trying to upgrade several applications to python3.10, so for the most part I have left the original files created by mod_wsgi-express setup-server alone under the centos7 yum installed python 3.6, mod_wsgi 4.7.0

I've hand-edited /etc/mod_wsgi-express/sandbox.routes.loutilities.com/httpd.conf as follows

I'm not sure this is sufficient to "shift" to the new python version, but I was hoping to touch as little as possible rather than setting everything up again for 8 applications.
I see the following in journalctl when I try to start the service.

Jun 05 08:55:02 loutility-server-digitalocean apachectl[23870]: httpd (mod_wsgi-express): Syntax error on line 159 of /etc/mod_wsgi-express/sandbox.routes.loutilities.com/httpd.conf: Cannot load /var/www/sandbox.routes.loutilities.com/venv/lib64/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so into server: /var/www/sandbox.routes.loutilities.com/venv/lib64/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so: undefined symbol: GENERAL_NAME_free

I notice the following from the build directory

Python-3.10.4]$ grep -R GENERAL_NAME_free . ./Modules/_ssl.c: sk_GENERAL_NAME_pop_free(names, GENERAL_NAME_free); 
Binary file ./Modules/_ssl.o matches 
Binary file ./python matches 
Binary file ./libpython3.10.a matches Binary file ./Programs/_testembed matches

and libpython3.10.a is in /usr/local/lib/python3.10/config-3.10-x86_64-linux-gnu:

$ find /usr/local/lib -name libpython3.10.a
/usr/local/lib/libpython3.10.a
/usr/local/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.a

What other debugging steps should I take? Do I need to re-run mod_wsgi-express? setup-server. Is mod_wsgi not being built correctly? Or something else?

Lou King

unread,
Jun 7, 2022, 1:50:50 PM6/7/22
to modwsgi
FWIW, I tried rerunning mod_wsgi-express setup-server, and get the same results

$ sudo LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/custom-openssl/lib /root/bin/init-mod_wsgi-express runningroutes sandbox.routes.loutilities.com routesmgr routesmgr 8002
Server URL         : http://proxysvr.loutilities.com:8002/
Server Root        : /etc/mod_wsgi-express/sandbox.routes.loutilities.com
Server Conf        : /etc/mod_wsgi-express/sandbox.routes.loutilities.com/httpd.conf
Error Log File     : /etc/mod_wsgi-express/sandbox.routes.loutilities.com/error_log (warn)
Rewrite Rules      : /etc/mod_wsgi-express/sandbox.routes.loutilities.com/rewrite.conf
Environ Variables  : /etc/mod_wsgi-express/sandbox.routes.loutilities.com/envvars
Control Script     : /etc/mod_wsgi-express/sandbox.routes.loutilities.com/apachectl
Operating Mode     : daemon
Request Capacity   : 5 (1 process * 5 threads)
Request Timeout    : 60 (seconds)
Startup Timeout    : 15 (seconds)
Queue Backlog      : 100 (connections)
Queue Timeout      : 45 (seconds)
Server Capacity    : 20 (event/worker), 20 (prefork)
Server Backlog     : 500 (connections)
Locale Setting     : en_US.UTF-8
$ sudo /etc/mod_wsgi-express/sandbox.routes.loutilities.com/apachectl start
httpd (mod_wsgi-express): Syntax error on line 163 of /etc/mod_wsgi-express/sandbox.routes.loutilities.com/httpd.conf: Cannot load /var/www/sandbox.routes.loutilities.com/venv/lib/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so into server: /var/www/sandbox.routes.loutilities.com/venv/lib/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so: undefined symbol: GENERAL_NAME_free

where

$ sudo cat /root/bin/init-mod_wsgi-express
#!/bin/bash
if [[ $# -lt 5 ]] ; then
   echo "usage:"
   echo "    init-mod_wsgi-express project servername user group port"
   exit 0
fi

source /var/www/$2/venv/bin/activate
mod_wsgi-express setup-server --server-name proxysvr.loutilities.com --port $5 --user $3 --group $4 /var/www/$2/$1/$1/$1.wsgi --working-directory /var/www/$2/$1/$1/ --server-root /etc/mod_wsgi-express/$2
deactivate

Graham Dumpleton

unread,
Jun 16, 2022, 4:49:29 PM6/16/22
to mod...@googlegroups.com
Your Apache installation and Python installations are compiled against different SSL library versions. You can't have this, they need to match. You would need to build any custom Python version against same system SSL libraries that system Apache is using.

Note I am trying to upgrade several applications to python3.10, so for the most part I have left the original files created by mod_wsgi-express setup-serveralone under the centos7 yum installed python 3.6, mod_wsgi 4.7.0


I've hand-edited /etc/mod_wsgi-express/sandbox.routes.loutilities.com/httpd.conf as follows

I'm not sure this is sufficient to "shift" to the new python version, but I was hoping to touch as little as possible rather than setting everything up again for 8 applications.
I see the following in journalctl when I try to start the service.

Jun 05 08:55:02 loutility-server-digitalocean apachectl[23870]: httpd (mod_wsgi-express): Syntax error on line 159 of /etc/mod_wsgi-express/sandbox.routes.loutilities.com/httpd.conf: Cannot load /var/www/sandbox.routes.loutilities.com/venv/lib64/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so into server: /var/www/sandbox.routes.loutilities.com/venv/lib64/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so: undefined symbol: GENERAL_NAME_free

I notice the following from the build directory

Python-3.10.4]$ grep -R GENERAL_NAME_free . ./Modules/_ssl.c: sk_GENERAL_NAME_pop_free(names, GENERAL_NAME_free); 
Binary file ./Modules/_ssl.o matches 
Binary file ./python matches 
Binary file ./libpython3.10.a matches Binary file ./Programs/_testembed matches

and libpython3.10.a is in /usr/local/lib/python3.10/config-3.10-x86_64-linux-gnu:

$ find /usr/local/lib -name libpython3.10.a
/usr/local/lib/libpython3.10.a
/usr/local/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.a

What other debugging steps should I take? Do I need to re-run mod_wsgi-express? setup-server. Is mod_wsgi not being built correctly? Or something else?

-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/ca6481d3-bded-4b98-9437-5d10992762ffn%40googlegroups.com.

Lou King

unread,
Jun 16, 2022, 5:18:16 PM6/16/22
to mod...@googlegroups.com
Wow, thanks. This is great information.

I think this means I can't go to python 3.10 because as I understand, it requires an openssl upgrade. 

You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/KZZQHpFclGA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/29311AFA-71AC-4621-AE08-DA0DAC5C1219%40gmail.com.

Lou King

unread,
Jun 18, 2022, 12:27:25 PM6/18/22
to modwsgi
I built python 3.9.13, which uses the system openssl (same as apache, I assue). Now I see the following

Lou King

unread,
Jun 18, 2022, 12:52:26 PM6/18/22
to modwsgi
On the off chance, I followed https://modwsgi.readthedocs.io/en/master/user-guides/installation-issues.html#undefined-forkpty-on-fedora-7 and edited envvars as follows

LD_PRELOAD=/usr/lib64/libutil.so
export LD_PRELOAD

now I see 


I see something at https://stackoverflow.com/questions/23719304/error-on-apache2-restart-using-mod-python-undefined-symbol-sincos and wondering if I should have done anything differently while building python?

Graham Dumpleton

unread,
Jun 18, 2022, 5:25:55 PM6/18/22
to mod...@googlegroups.com
If you are building your own Python make sure you read:


Ignore that it talks about docker.

In the past at least, Python out of the box with no configuration doesn't build how you would want it to. Not sure if same is still the case.

Also be wary of using pyenv to create Python installations, it has a history of not building things in a way that works with embedded systems.

Lou King

unread,
Jun 23, 2022, 1:53:30 PM6/23/22
to modwsgi
Success!

Based on your blog post, I looked at how centos 7 python 3.6 was compiled, removed all the switches about redhat build and host, and referencing specific directories and used the remaining for my ./configure execution

$ diff python3.6-options.txt python3.6-options-shortened.txt
1,3d0
< '--build=x86_64-redhat-linux-gnu'
< '--host=x86_64-redhat-linux-gnu'
< '--program-prefix='
5,17d1
< '--prefix=/usr'
< '--exec-prefix=/usr'
< '--bindir=/usr/bin'
< '--sbindir=/usr/sbin'
< '--sysconfdir=/etc'
< '--datadir=/usr/share'
< '--includedir=/usr/include'
< '--libdir=/usr/lib64'
< '--libexecdir=/usr/libexec'
< '--localstatedir=/var'
< '--sharedstatedir=/var/lib'
< '--mandir=/usr/share/man'
< '--infodir=/usr/share/info'
29,30d12
< 'build_alias=x86_64-redhat-linux-gnu'
< 'host_alias=x86_64-redhat-linux-gnu'

so my resulting ./configure was as follows

./configure '--disable-dependency-tracking' '--enable-ipv6' '--enable-shared' '--with-computed-gotos=yes' '--with-dbmliborder=gdbm:ndbm:bdb' '--with-system-expat' '--with-system-ffi' '--enable-loadable-sqlite-extensions' '--with-dtrace' '--with-ssl-default-suites=openssl' '--without-ensurepip' '--enable-optimizations' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv  ' 'LDFLAGS=-Wl,-z,relro  -g  ' 'CPPFLAGS= ' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

It turns out on centos 7  --disable-dependency-tracking and  --with-system-ffi were unneeded, but I left them in because I had other issues I thought might have been caused by them and leaving them in seemed benign. (apparently I had left some garbage from an interim build which apparently wasn't cleaned up from make clean, so I had to start my build from scratch, but this became apparent right after removing those switches)

    configure: WARNING: unrecognized options: --disable-dependency-tracking
    configure: WARNING: --with(out)-system-ffi is ignored on this platform

Note for my configuration, I need LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib so python3.9 can find the shared library.

Graham Dumpleton

unread,
Jun 23, 2022, 6:24:34 PM6/23/22
to mod...@googlegroups.com
It is odd that you need to force /usr/local/lib into LD_LIBRARY_PATH at run time. I thought that was a standard directory Linux systems would check. Looks like someone has mucked up the ld.so.cache.

Lou King

unread,
Jun 23, 2022, 7:55:03 PM6/23/22
to modwsgi
I'll look at that. Actually I built the server myself on a digitalocean "droplet" (virtual server), and am pretty sure I wouldn't have changed anything along those lines. I don't remember exactly how I put the Centos 7 on the droplet, but checking now I see that digitalocean has a way to create the droplet with a specific image. 

Off topic, but I have to do some research before I go to a newer linux version (and in doing so allow myself to follow later python versions) -- I think Centos is doing something weird with their "stream" images. I see "Centos 8 stream" and "Centos 9 stream" as options now. IIRC, RockyLinux has "fixed" versions which act like earlier Centos versions (yup: https://rockylinux.org/)



Reply all
Reply to author
Forward
0 new messages