pkg_resources.py2_warn error with Pyinstaller 4.0 if built with docker

[Kim Neunert]

Hi, i hope i can get some help or hints here.

Over at https://github.com/cryptoadvance/specter-desktop/issues/373 we're trying to automate the release-process for our binary artifact. Before, we created the binaries manually on our desktop. Now, we want to automate that on gitlab/docker-runners.

We're using python:3.8 which is debian buster-based.

Here is the output of the broken build having this Error on pyinstaller 4.0:

5056 ERROR: Hidden import 'pkg_resources.py2_warn' not found:

```
39 INFO: PyInstaller: 4.0
39 INFO: Python: 3.8.5
43 INFO: Platform: Linux-5.4.0-47-generic-x86_64-with-glibc2.2.5
45 INFO: UPX is not available.
48 INFO: Extending PYTHONPATH with paths
['/mnt/pyinstaller']
54 INFO: checking Analysis
54 INFO: Building Analysis because Analysis-00.toc is non existent
54 INFO: Initializing module dependency graph...
55 INFO: Caching module graph hooks...
63 INFO: Analyzing base_library.zip ...
2871 INFO: Processing pre-find module path hook distutils from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
2885 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.8'
4549 INFO: Caching module dependency graph...
4666 INFO: running Analysis Analysis-00.toc
4680 INFO: Analyzing specter_desktop.py
4938 INFO: Analyzing hidden import 'pkg_resources.py2_warn'
5056 ERROR: Hidden import 'pkg_resources.py2_warn' not found
5056 INFO: Processing module hooks...
5057 INFO: Loading module hook 'hook-PyQt5.QtGui.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5121 INFO: Loading module hook 'hook-PyQt5.QtWebEngineWidgets.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5343 INFO: Loading module hook 'hook-encodings.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5382 INFO: Loading module hook 'hook-PyQt5.QtQuick.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5465 INFO: Loading module hook 'hook-PyQt5.QtCore.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5481 INFO: Loading module hook 'hook-PyQt5.QtWidgets.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5512 INFO: Loading module hook 'hook-pkg_resources.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
6152 INFO: Processing pre-safe import module hook win32com from '/usr/local/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
6154 WARNING: Hidden import "pkg_resources.py2_warn" not found!
6155 WARNING: Hidden import "pkg_resources.markers" not found!
6155 INFO: Excluding import '__main__'
6156 INFO:   Removing import of __main__ from module pkg_resources
6157 INFO: Loading module hook 'hook-PyQt5.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
6182 WARNING: Hidden import "sip" not found!
...
```
Note how it contains the error but imports the hook some lines later.

The working-build looks like this:
```
30 INFO: PyInstaller: 4.0
31 INFO: Python: 3.8.2
78 INFO: Platform: Linux-5.4.0-47-generic-x86_64-with-glibc2.27
80 INFO: UPX is not available.
84 INFO: Extending PYTHONPATH with paths
['/home/kim/src/specter-desktop/pyinstaller']
90 INFO: checking Analysis
90 INFO: Building Analysis because Analysis-00.toc is non existent
90 INFO: Initializing module dependency graph...
91 INFO: Caching module graph hooks...
96 INFO: Analyzing base_library.zip ...
1960 INFO: Processing pre-find module path hook distutils from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1963 INFO: distutils: retargeting to non-venv dir '/home/kim/.pyenv/versions/3.8.2/lib/python3.8'
3758 INFO: Caching module dependency graph...
3845 INFO: running Analysis Analysis-00.toc
3852 INFO: Analyzing specter_desktop.py
4068 INFO: Analyzing hidden import 'pkg_resources.py2_warn'
4153 INFO: Processing module hooks...
4154 INFO: Loading module hook 'hook-PyQt5.QtGui.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4240 INFO: Loading module hook 'hook-PyQt5.QtWebEngineWidgets.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4583 INFO: Loading module hook 'hook-encodings.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4628 INFO: Loading module hook 'hook-PyQt5.QtQuick.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4693 INFO: Loading module hook 'hook-PyQt5.QtCore.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4710 INFO: Loading module hook 'hook-PyQt5.QtWidgets.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4748 INFO: Loading module hook 'hook-_tkinter.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4796 INFO: checking Tree
4796 INFO: Building Tree because Tree-00.toc is non existent
4796 INFO: Building Tree Tree-00.toc
4801 INFO: checking Tree
4801 INFO: Building Tree because Tree-01.toc is non existent
4801 INFO: Building Tree Tree-01.toc
4803 INFO: Loading module hook 'hook-pkg_resources.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
5115 INFO: Processing pre-safe import module hook win32com from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
5285 INFO: Processing pre-safe import module hook six.moves from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
5323 WARNING: Hidden import "pkg_resources.markers" not found!
5324 INFO: Excluding import '__main__'
5325 INFO:   Removing import of __main__ from module pkg_resources
5325 INFO: Loading module hook 'hook-PyQt5.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
5339 WARNING: Hidden import "sip" not found!
...
```

So this doesn't contain the error but also doesn't process the hook.


This created this binary:
https://github.com/cryptoadvance/specter-desktop/releases/download/v0.7.3-pre1/specter_desktop-v0.7.3-pre1-x86_64-linux-gnu.tar.gz

Unfortunately, it's failing to execute properly like this:

```
➜  specter_desktop-v0.7.3-pre1-x86_64-linux-gnu ./Specter                                                  
/usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so: undefined symbol: g_date_time_format_iso8601
Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
libGL error: MESA-LOADER: failed to open iris (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
libGL error: failed to load driver: iris
libGL error: MESA-LOADER: failed to open iris (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
libGL error: failed to load driver: iris
libGL error: MESA-LOADER: failed to open swrast (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
libGL error: failed to load driver: swrast
QGLXContext: Failed to create dummy context
WebEngineContext used before QtWebEngine::initialize() or OpenGL context creation failed.
[2020-09-09 18:22:24,471] INFO in specterd: Logging configured
[2020-09-09 18:22:24,472] INFO in server: pyinstaller based instance running in /tmp/_MEI1JDnf0/specterd
[2020-09-09 18:22:24,473] INFO in server: Initializing LoginManager
[2020-09-09 18:22:24,473] INFO in server: Initializing Specter
[2020-09-09 18:22:24,479] INFO in server: Login disabled
[2020-09-09 18:22:24,479] INFO in server: Initializing Controller ...
* Started Specter daemon...
[2020-09-09 18:22:24,507] INFO in _internal:  * Running on http://127.0.0.1:25441/ (Press CTRL+C to quit)
Failed to create OpenGL context for format QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize 24, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize -1, stencilBufferSize 8, samples 0, swapBehavior QSurfaceFormat::DefaultSwapBehavior, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
[1]    283287 abort (core dumped)  ./Specter
➜  specter_desktop-v0.7.3-pre1-x86_64-linux-gnu
```

Any ideas what might be wrong here?

Cheers

Kim

[bwoodsend]

The 5056 ERROR: Hidden import 'pkg_resources.py2_warn' not found: shouldn’t really be labelled an error - it’s just because setuptools removed that hidden module in version 49 (ish). Certainly it’s not why your CI is failing.

I had an issue on both Github actions and TravisCI which turned out just to be that neither VM’s come with opengl32.dll or the OS equivalent (it should be a system library). That ugly mess of a traceback is the kind of error message you’d get when one DLL loading goes wrong - so it could be opengl32 or some other binary your app uses. Have you checked that running normally on CI (without PyInstaller) works?

​

[Kim Neunert]

Hi bwoodsend,

Thanks for answering.

> I had an issue on both Github actions and TravisCI which turned out just to be that neither VM’s come with

> opengl32.dll or the OS equivalent (it should be a system library).

i tried to install every mesa/opengl package i could get my hands on, namely

build-essential libgl1-mesa-dev libgl1-mesa-dev libglew-dev libsdl2-dev libsdl2-image-dev libglm-dev libfreetype6-dev mesa-utils wine libegl-mesa0 libegl1-mesa libgl1-mesa-dri libgbm1 libgl1-mesa-dev libgl1-mesa-dri libgl1-mesa-glx libgles2-mesa libd3dadapter9-mesa libd3dadapter9-mesa-dev libegl-mesa0 libegl1-mesa libegl1-mesa-dev libgbm-dev libgbm1 libgl1-mesa-dev libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgles2-mesa libgles2-mesa-dev libglx-mesa0 libosmesa6 libosmesa6-dev libwayland-egl1-mesa libxatracker-dev libxatracker2 mesa-common-dev mesa-opencl-icd mesa-va-drivers mesa-vdpau-drivers mesa-vulkan-drivers

... but that didn't fixed it unfortunately. Do you use a docker-image or a buildscript which i could investigate?

> Have you checked that running normally on CI (without PyInstaller) works?

The app is a python-flask app which serves via http. Yes, it works there without any issue. We just want to have it as a desktop-application.

Btw, sorry for the double-post.

[bwoodsend]

If it runs ok on CI then Gitlab CI has everything it needs so missing opengl DLL isn’t the issue. The fault is definitely somewhere in the PyInstaller-ing part. Can you clarify what you are doing? It looks you are using PyInstaller in a docker image on CI, then downloading the executable and testing it locally - is that right? You’re not trying to test the executable on CI?

I’d still guess the issue is missing binaries related - probably a DLL that PyInstaller fails to collect. You can use psutil to work out which binaries your code needs:

# Put me at the end of your code.
import pustil
print(psutil.Process().memory_maps())

Then compare the output with what PyInstaller includes in your bundle.

​

[Kim Neunert]

> You can use psutil to work out which binaries your code needs:

Thank you, that helped a lot but couldn't solve yet may issue.

I figured out the files in /usr/lib which got missing in the broken binary and did a "dpkg -S /usr/lib..." for each of them and compiled a list of packages for that. This lead to:

apt install libgl1-mesa-dri gvfs gvfs-libs libdrm-amdgpu1 libdrm-nouveau2 libdrm-radeon1 libedit2 libelf1 libllvm10 libsensors5 libvulkan1 libzstd1

Looking again at the difference showed me e.g. that this file is missing:

/usr/lib/x86_64-linux-gnu/dri/iris_dri.so (which looks promising to me)

Searching for it on my ubuntu-system got me that package:

dpkg -S /usr/lib/x86_64-linux-gnu/dri/iris_dri.so                         
libgl1-mesa-dri:amd64: /usr/lib/x86_64-linux-gnu/dri/iris_dri.so

checking the content of the debian-buster package didn't contained that file:

dpkg -L libgl1-mesa-dri | grep iris # empty result

I guess this might point to better use ubuntu as a build-image?

Just that you get a better idea what i'm doing. This is what effectively happens in the CI:

# instead of directly doing:

# pyinstaller specter_desktop.spec

# i'm doing

docker run -it -v $(pwd):/mnt  registry.gitlab.com/cryptoadvance/specter-desktop/python-bitcoind:v0.19.1-20200909 bash
# Now, inside docker:
cd /mnt
source .env/bin/activate
cd pyinstaller

pyinstaller specter_desktop.spec

The above image (python-bitcoind:v0.19.1-20200909) is built upon python:3.8-buster.

The python-dockerimage is not available in an ubuntu-flavor:

https://hub.docker.com/_/python?tab=description

So the best option is creating that image myself? I'd love to avoid that if i can, though. Any other better option?

thanks so far.

[Kim Neunert]

So, i've built the ubuntu-based image and tested my theory. And it worked but just after adding these lines to my spec:

    binaries.extend([("/usr/lib/x86_64-linux-gnu/dri/iris_dri.so","."),\
                     ("/usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so","."),\
                     ("/usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so","."),\
                     ("/usr/lib/x86_64-linux-gnu/",".")])

# and later

a = Analysis(['specter_desktop.py'],
             binaries=binaries,

...

When it got built, i got these warnings:

19093 INFO: Building PKG (CArchive) PKG-00.pkg
19116 WARNING: One binary added with two internal names.
19116 WARNING: ('gio/modules/libgvfsdbus.so',
 '/usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so',
 'BINARY')
19116 WARNING: was placed previously at
19116 WARNING: ('libgvfsdbus.so',
 '/usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so',
 'BINARY')
19122 WARNING: One binary added with two internal names.
19122 WARNING: ('dri/iris_dri.so', '/usr/lib/x86_64-linux-gnu/dri/iris_dri.so', 'BINARY')
19123 WARNING: was placed previously at
19123 WARNING: ('iris_dri.so', '/usr/lib/x86_64-linux-gnu/dri/iris_dri.so', 'BINARY')
19124 WARNING: One binary added with two internal names.
19124 WARNING: ('libgvfscommon.so',
 '/usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so',
 'BINARY')
19124 WARNING: was placed previously at
19124 WARNING: ('gvfs/libgvfscommon.so',
 '/usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so',
 'BINARY')
222799 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.

There are also now a lot of font-issues which were not there before like:

Fontconfig warning: "/etc/fonts/fonts.conf", line 5: unknown element "its:rules"
Fontconfig warning: "/etc/fonts/fonts.conf", line 6: unknown element "its:translateRule"
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'translate'
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'selector'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'xmlns:its'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'version'
Fontconfig warning: "/etc/fonts/fonts.conf", line 9: unknown element "description"
Fontconfig warning: "/etc/fonts/conf.d/10-hinting-slight.conf", line 4: unknown element "its:rules"
Fontconfig warning: "/etc/fonts/conf.d/10-hinting-slight.conf", line 5: unknown element "its:translateRule"