Pyinstaller not able to bundle module which is internally used by other module in python script

[Alok Singh Mahor]

I have simple python script which is using pyhrv module. pyhrv itself is using another module spectrum. Here is my script

alok.py

import pyhrv

print('Hello using pyinstaller')
print('pyhrv version:', pyhrv.__version__)

output on running this script is

(ecg) alok@alok-HP-Laptop-14s-cr1xxx:~/tmp/python/test$ python alok.py 
Hello using pyinstaller
pyhrv version: 0.4.0

Now I am trying to create bundle which would be having all dependencies using pyinstaller.

(ecg) alok@alok-HP-Laptop-14s-cr1xxx:~/tmp/python/test$ pyinstaller alok.py 
22 INFO: PyInstaller: 3.6
22 INFO: Python: 3.7.3
22 INFO: Platform: Linux-5.0.0-38-generic-x86_64-with-Ubuntu-19.04-disco
22 INFO: wrote /home/alok/tmp/python/test/alok.spec
24 INFO: UPX is not available.
25 INFO: Extending PYTHONPATH with paths
['/home/alok/tmp/python/test', '/home/alok/tmp/python/test']
25 INFO: checking Analysis
25 INFO: Building Analysis because Analysis-00.toc is non existent
25 INFO: Initializing module dependency graph...
26 INFO: Caching module graph hooks...
29 INFO: Analyzing base_library.zip ...
1633 INFO: Caching module dependency graph...
1682 INFO: running Analysis Analysis-00.toc
1701 INFO: Analyzing /home/alok/tmp/python/test/alok.py
1737 INFO: Processing pre-safe import module hook   six.moves
2004 INFO: Processing pre-find module path hook   distutils
2004 INFO: distutils: retargeting to non-venv dir '/usr/lib/python3.7'
2589 INFO: Processing pre-safe import module hook   setuptools.extern.six.moves
2774 INFO: Processing pre-find module path hook   site
2775 INFO: site: retargeting to fake-dir '/home/alok/mypyenv/ecg/lib/python3.7/site-packages/PyInstaller/fake-modules'
9509 INFO: Processing module hooks...
9510 INFO: Loading module hook "hook-scipy.linalg.py"...
9510 INFO: Loading module hook "hook-scipy.special._ellip_harm_2.py"...
9510 INFO: Loading module hook "hook-matplotlib.backends.py"...
10024 INFO:   Matplotlib backend "GTK3Agg": ignored
    cairo backend requires that pycairo>=1.11.0 or cairocffiis installed
10314 INFO:   Matplotlib backend "GTK3Cairo": ignored
    cairo backend requires that pycairo>=1.11.0 or cairocffiis installed
10728 INFO:   Matplotlib backend "MacOSX": ignored
    cannot import name '_macosx' from 'matplotlib.backends' (/home/alok/mypyenv/ecg/lib/python3.7/site-packages/matplotlib/backends/__init__.py)
11093 INFO:   Matplotlib backend "nbAgg": ignored
    No module named 'IPython'
11649 INFO:   Matplotlib backend "Qt4Agg": ignored
    Failed to import any qt binding
12142 INFO:   Matplotlib backend "Qt4Cairo": ignored
    cairo backend requires that pycairo>=1.11.0 or cairocffiis installed
12811 INFO:   Matplotlib backend "Qt5Agg": ignored
    Failed to import any qt binding
13315 INFO:   Matplotlib backend "Qt5Cairo": ignored
    cairo backend requires that pycairo>=1.11.0 or cairocffiis installed
13829 INFO:   Matplotlib backend "TkAgg": ignored
    No module named 'tkinter'
14341 INFO:   Matplotlib backend "TkCairo": ignored
    No module named 'tkinter'
14940 INFO:   Matplotlib backend "WebAgg": ignored
    Traceback (most recent call last):
  File "/home/alok/mypyenv/ecg/lib/python3.7/site-packages/matplotlib/backends/backend_webagg.py", line 26, in <module>
    import tornado
ModuleNotFoundError: No module named 'tornado'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 12, in <module>
  File "/home/alok/mypyenv/ecg/lib/python3.7/site-packages/matplotlib/backends/backend_webagg.py", line 28, in <module>
    raise RuntimeError("The WebAgg backend requires Tornado.")
RuntimeError: The WebAgg backend requires Tornado.
15684 INFO:   Matplotlib backend "WX": ignored
    No module named 'wx'
16207 INFO:   Matplotlib backend "WXAgg": ignored
    No module named 'wx'
16719 INFO:   Matplotlib backend "WXCairo": ignored
    No module named 'wx'
17221 INFO:   Matplotlib backend "agg": added
17732 INFO:   Matplotlib backend "cairo": ignored
    cairo backend requires that pycairo>=1.11.0 or cairocffiis installed
18490 INFO:   Matplotlib backend "pdf": added
19092 INFO:   Matplotlib backend "pgf": added
19533 INFO:   Matplotlib backend "ps": added
20003 INFO:   Matplotlib backend "svg": added
20811 INFO:   Matplotlib backend "template": added
21226 INFO: Loading module hook "hook-xml.py"...
21299 INFO: Loading module hook "hook-setuptools.py"...
21973 INFO: Loading module hook "hook-pkg_resources.py"...
22511 INFO: Processing pre-safe import module hook   win32com
22518 INFO: Excluding import '__main__'
22521 INFO:   Removing import of __main__ from module pkg_resources
22522 INFO: Loading module hook "hook-encodings.py"...
22591 INFO: Loading module hook "hook-scipy.py"...
22592 INFO: Loading module hook "hook-scipy.io.matlab.py"...
22593 INFO: Loading module hook "hook-numpy.core.py"...
22865 INFO: Loading module hook "hook-scipy.sparse.csgraph.py"...
22869 INFO: Loading module hook "hook-matplotlib.py"...
23349 INFO: Loading module hook "hook-numpy.py"...
23350 INFO: Loading module hook "hook-distutils.py"...
23353 INFO: Loading module hook "hook-lib2to3.py"...
23354 INFO: Loading module hook "hook-pydoc.py"...
23355 INFO: Loading module hook "hook-sysconfig.py"...
23367 INFO: Loading module hook "hook-sklearn.metrics.cluster.py"...
23368 WARNING: Hidden import "sklearn.utils.sparsetools._graph_validation" not found!
23368 WARNING: Hidden import "sklearn.utils.sparsetools._graph_tools" not found!
23368 WARNING: Hidden import "sklearn.utils.lgamma" not found!
23371 INFO: Loading module hook "hook-scipy.special._ufuncs.py"...
23447 INFO: Looking for ctypes DLLs
23587 WARNING: library user32 required via ctypes not found
23717 INFO: Analyzing run-time hooks ...
23738 INFO: Including run-time hook 'pyi_rth_mplconfig.py'
23740 INFO: Including run-time hook 'pyi_rth_mpldata.py'
23742 INFO: Including run-time hook 'pyi_rth_pkgres.py'
23743 INFO: Including run-time hook 'pyi_rth_multiprocessing.py'
23781 INFO: Looking for dynamic libraries
26340 INFO: Looking for eggs
26340 INFO: Python library not in binary dependencies. Doing additional searching...
26361 INFO: Using Python library /lib/x86_64-linux-gnu/libpython3.7m.so.1.0
26383 INFO: Warnings written to /home/alok/tmp/python/test/build/alok/warn-alok.txt
26511 INFO: Graph cross-reference written to /home/alok/tmp/python/test/build/alok/xref-alok.html
26556 INFO: checking PYZ
26556 INFO: Building PYZ because PYZ-00.toc is non existent
26556 INFO: Building PYZ (ZlibArchive) /home/alok/tmp/python/test/build/alok/PYZ-00.pyz
28307 INFO: Building PYZ (ZlibArchive) /home/alok/tmp/python/test/build/alok/PYZ-00.pyz completed successfully.
28334 INFO: checking PKG
28334 INFO: Building PKG because PKG-00.toc is non existent
28334 INFO: Building PKG (CArchive) PKG-00.pkg
28370 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
28371 INFO: Bootloader /home/alok/mypyenv/ecg/lib/python3.7/site-packages/PyInstaller/bootloader/Linux-64bit/run
28371 INFO: checking EXE
28371 INFO: Building EXE because EXE-00.toc is non existent
28371 INFO: Building EXE from EXE-00.toc
28371 INFO: Appending archive to ELF section in EXE /home/alok/tmp/python/test/build/alok/alok
28388 INFO: Building EXE from EXE-00.toc completed successfully.
28390 INFO: checking COLLECT
28391 INFO: Building COLLECT because COLLECT-00.toc is non existent
28391 INFO: Building COLLECT COLLECT-00.toc
28602 INFO: Building COLLECT COLLECT-00.toc completed successfully.

Now I will try to execute created binary.

(ecg) alok@alok-HP-Laptop-14s-cr1xxx:~/tmp/python/test$ ls
alok.py  alok.spec  build  dist  __pycache__

(ecg) alok@alok-HP-Laptop-14s-cr1xxx:~/tmp/python/test$ ./dist/alok/alok 
/home/alok/mypyenv/ecg/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py:623: MatplotlibDeprecationWarning: 
The MATPLOTLIBDATA environment variable was deprecated in Matplotlib 3.1 and will be removed in 3.3.
  exec(bytecode, module.__dict__)
Traceback (most recent call last):
  File "alok.py", line 1, in <module>
    import pyhrv
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/home/alok/mypyenv/ecg/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "pyhrv/__init__.py", line 36, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/home/alok/mypyenv/ecg/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "pyhrv/hrv.py", line 47, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/home/alok/mypyenv/ecg/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "pyhrv/frequency_domain.py", line 46, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/home/alok/mypyenv/ecg/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "spectrum/__init__.py", line 19, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/home/alok/mypyenv/ecg/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "spectrum/datasets.py", line 148, in <module>
  File "spectrum/datasets.py", line 137, in spectrum_data
  File "pkg_resources/__init__.py", line 481, in get_distribution
  File "pkg_resources/__init__.py", line 357, in get_provider
  File "pkg_resources/__init__.py", line 900, in require
  File "pkg_resources/__init__.py", line 786, in resolve
pkg_resources.DistributionNotFound: The 'spectrum' distribution was not found and is required by the application
[27542] Failed to execute script alok

I dont know why its not able to include spectrum in distribution. How can I make this working?

[Hartmut Goebel]

Am 04.03.20 um 09:00 schrieb Alok Singh Mahor:

I dont know why its not able to include spectrum in distribution. How can I make this working?

Thi reason is explained in the manual.

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer
Information Security Management, Security Governance, Secure Software Development

Goebel Consult, Landshut
http://www.goebel-consult.de

Blog: https://www.goe-con.de/blog/bin-offiziell-entdecker-einer-sicherheitslucke
Kolumne: https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2010-09-mut-zur-beschraenkung