ImportError caused by wrong library path in flask app using WSGI daemon

[Nathan Wendt]

I have a flask application that is run on apache using mod_wsgi. I have a python environment that I set up using miniconda. Initially, mod_wsgi was installed using conda. Using a mod_wsgi daemon process with a properly set python-home, I was able to run a very basic "hello, world" app to verify things were working properly. I then began to add pieces to the app to process data and output images. I began to see ImportError in the apache logs when trying to run the app. The specific error was:

ImportError: /lib64/libz.so.1: version `ZLIB_1.2.9` not found (required by ~/miniconda3/envs/......./libpng16.so.16)

This error is triggered by importing the rasterio package (which depends on GDAL, which needs libpng). When I look at the libpng referred to by the error with ldd, however, it is properly linked to the libz that is in the conda environment. Somehow the library path only includes the system paths within the daemon process. If I am to use my conda environment python on the command line to import rasterio, no such error occurs. It only occurs within the flask app. I next built mod_wsgi from source using the conda environment python just to be sure I was having that all linked correctly. The same error still occurred in the app. When checking the sys.path from within the app, that all looks appropriate as well.

Just based on what I am seeing, it seems like something is going on specifically with the mod_wsgi daemon process. Do I need to set something other than just python-home? There is an SO question with a very similar issue (https://stackoverflow.com/questions/33497639/why-is-wsgi-looking-for-a-library-in-lib64-when-the-correct-version-is-in-the-p), but the proposed solution seems like bad practice to me. I am hoping someone has an idea of what might be happening here.

mod_wsgi: 4.9.3

apache: 2.4.6

python: 3.10.6

flask: 2.2.2

[Graham Dumpleton]

Conda distributions of Python don't play nicely with embedded systems because they ship their own versions of common libraries which are also shipped with the operating system, but where the Conda version can be an incompatible version.

This causes problems when Python is being embedded in an application (Apache in this case), where the application (Apache) is already linking in the system version of the library.

Because the dynamic linker only looks at the shared library name, it will see that libz.so is already linked into the application process and use whatever that is, which will then not be the version your Python extension expects and will fail.

For Apache usually the problem is that you also have mod_php loaded into Apache and it is preloading a lot of PHP extensions which have references to some library which drags in the conflicting system library. This is a known issue with some graphics libraries such as libpng.so pulled in by PHP.

So the only real way around this in this case is not to load PHP into Apache if you don't need PHP to be enabled. Do that and it may then work.

If you must have PHP loaded into your primary Apache, then use mod_wsgi-express for your Python application instead and configure the front end Apache where PHP runs to proxy requests through to the mod_wsgi-express instance as necessary.

So try the following two things.

1. Try and run your application using mod_wsgi-express instead of your primary Apache instance and see if that works.

2. Disable PHP in your primary Apache instance, completely stop Apache and start it again (not just a reload), and see if it then works.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/9dd85566-ec3f-42f4-bfe3-3032e6d486den%40googlegroups.com.

[Nathan Wendt]

Thanks. Taking the php module out still produced the same error. Perhaps there is still a module that requires those system libraries that I am not thinking of? In terms of  mod_wsgi-express, I am not sure how to use that in the setup that I have. Running it on my local machine and accessing it with localhost makes sense. However, my application is running on an external server with its own apache/security configurations. I was not successful trying to access the express server instance. I also ended up trying to set LD_LIBRARY_PATH and then start apache again as well as directly loading the libraries in the config files to no avail. I've seen some of your suggestions to set LD_RUN_PATH and build libraries from source. Unfortunately, I think I would have to build several packages in order for that to work. The reason for using conda was to easily get a newer version of python than the RHEL system comes packaged with. As long as I did not miss a step in any of your suggestions, perhaps there is not a good way to get this to work.

[Graham Dumpleton]

If your system had "pldd" command, as root user, run it and supply the process ID of the Apache parent process (the one running as root). Check that no PHP module is loaded and see if the problem library is also preloaded still.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/c6a94048-148d-4a6d-8d2e-4d7881e57a9fn%40googlegroups.com.

[Nathan Wendt]

Using pldd we did confirm that the PHP module was not loaded. An internal server error was still present with the same import error. I ended up digging into the other modules that were still loaded and found that there were 4 others that linked to the system libz:

• mod_deflate
• mod_ssl
• mod_systemd
• mod_security

We aren't going to be able to turn those off. We may just have to use the older system python unless there are other clever solutions I am not aware of.