pyinstaller + librosa: "Module not found" for EXE, even w/ hidden import?

[Scott Hawley]

Hi, I have a package called "myapp", for which I was able to successfully build a working executable using pyinstaller on Windows yesterday.   Great! 

Today I'm trying to make a Linux executable (on the Pop!_OS distribution, which is an Ubuntu variant) using an Anaconda environment like I did for Windows.  But the Linux executable was giving me this error  when I ran it...

source/ $ pyinstaller -w --onefile " myapp.py

source/ $ cd dist

dist/ $ ./myapp

/home/mcskwayrd/anaconda3/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py:623: MatplotlibDeprecationWarning: 

The MATPLOTLIBDATA environment variable was deprecated in Matplotlib 3.1 and will be removed in 3.3.

  exec(bytecode, module.__dict__)

Traceback (most recent call last):

  File "myapp.py", line 28, in <module>

    import librosa

ModuleNotFoundError: No module named 'librosa'

[30822] Failed to execute script myapp

...so I tried specifying librosa as a hidden input:

source/ $ pyinstaller -w --onefile --hidden-import="librosa" myapp.py

...but this had no effect; the same error persists.   Any suggestions?   (I didn't have this problem on Windows.) 

Thanks!

PS- 

Since I know it's recommended for debugging, I tried doing a --onedir build instead.  But that yielded a different error: 

dist/myapp$ ./myapp 

[31669] Error loading Python lib '/home/mcskwayrd/myapp/source/dist/myapp/libpython3.6m.so.1.0': dlopen: /home/mcskwayrd/myapp/source/dist/myapp/libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

...this seems to be trying to access a file that doesn't exist, because the real file has no ".1.0" ending:

dist/myapp/ $ ls libpython3.6m*

libpython3.6m.so

Adding a --debug when running the executable didn't give any new information. 

PPS- 

My environment/install looks like this:

  conda create --name myapp python=3.7

  conda activate myapp

  conda install -c numba numba      # numba is optional, actually

  conda install -c conda-forge librosa

  conda install pyqt pillow pyaudio

  pip install pyinstaller                       # conda's version is 3.5, pip's is 3.6

my setuptools version is 41.4.0...perhaps a downgrade could help?  

  

[Abasi Brown]

1. Try opening the spec file and after the line that says 'block_cipher = None' put:

def get_librosa_path():

     import librosa

     librosa_path = sklearn.__path__[0]

     return librosa_path

2. Then after the lines that say 'pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)' put:

dict_tree = Tree(get_librosa_path(), prefix='librosa', excludes=["*.pyc"])
a.datas += dict_tree
a.binaries = filter(lambda x: 'librosa' not in x[0], a.binaries)

3. Then go to your command line and run the following command:

pyinstaller your_file_name.spec --specpath=test

Then run the executable.  You will likely discover you need to import other libraries.  Any library that doesn't work as a hidden import can be imported this way just replace 'librosa' with the name of the library you are trying to include.

[Abasi Brown]

revise step one:

def get_librosa_path():

     import librosa

     librosa_path = librosa.__path__[0]

     return librosa_path

[Hartmut Goebel]

Am 03.04.20 um 01:04 schrieb Scott Hawley:

Any suggestions?

See the "hooks" section in the manual

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer
Information Security Management, Security Governance, Secure Software Development

Goebel Consult, Landshut
http://www.goebel-consult.de

Blog: https://www.goe-con.de/blog/nicht-pgp-ist-gescheitert-die-entwickler-haben-versagt
Kolumne: https://www.goe-con.de/hartmut-goebel/cissp-gefluester/2011-02-fleisige-datensammler-fur-lukratives-geschaeftsmodell-gesucht

[Scott Hawley]

Thanks very much for your suggestions!   Following those steps (i.e. editing myapp.spec) changes the amount of output provided before the "Module not found" error message appears, but this has moved the error message from the final executable earlier to the pyinstaller build attempt:

source/ $ pyinstaller myapp.spec --specpath=test

Traceback (most recent call last):

  File "/home/mcskwayrd/anaconda3/bin/pyinstaller", line 10, in <module>

    sys.exit(run())

  File "/home/mcskwayrd/anaconda3/lib/python3.6/site-packages/PyInstaller/__main__.py", line 114, in run

    run_build(pyi_config, spec_file, **vars(args))

  File "/home/mcskwayrd/anaconda3/lib/python3.6/site-packages/PyInstaller/__main__.py", line 65, in run_build

    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)

  File "/home/mcskwayrd/anaconda3/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 734, in main

    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))

  File "/home/mcskwayrd/anaconda3/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 681, in build

    exec(code, spec_namespace)

  File "myapp.spec", line 25, in <module>

    dict_tree = Tree(get_librosa_path(), prefix='librosa', excludes=["*.pyc"])

  File "myapp.spec", line 6, in get_librosa_path

    import librosa

ModuleNotFoundError: No module named 'librosa'

$ 

[Scott Hawley]

SOLVED!

This actually, apparently was entirely a 'conda' environment error.  What fixed it was:

conda deactivate myapp

conda activate myapp 

conda uninstall librosa

conda install -c conda-forge librosa 

There were some extra hidden targets that neeeded defining before the whole thing would build, which were...

hiddenimports=['librosa', 'scipy._lib.messagestream', 'sklearn.tree', 'sklearn.neighbors.typedefs', 'sklearn.neighbors.quad_tree', 'sklearn.tree._utils', 'sklearn.utils._cython_blas'],

...but the main thing as "turning it off and on again" with respect to the conda environment.  

Thanks!

On Friday, April 3, 2020 at 8:06:55 AM UTC-5, Abasi Brown wrote:

[Chris Barker]

it seems you've solved your issue, but a note:

My environment/install looks like this:

  conda create --name myapp python=3.7

  conda activate myapp

  conda install -c numba numba      # numba is optional, actually

  conda install -c conda-forge librosa

  conda install pyqt pillow pyaudio

  pip install pyinstaller                       # conda's version is 3.5, pip's is 3.6

conda-forge has 3.6. As a rule, if you want the latest and greatest, conda-forge is the way to go. I've found it works well to add the conda-forge channel first in your config, and go from there.

It also works better to always give conda the entire list of dependencies at once: put them in a requirements file, and pass that to conda. IN your example, that would be:

conda_requirements.txt:

python=3.7

numba 

librosa

pyqt

pillow

pyaudio

pyinstaller              

Then you can do: 

conda create -n myapp --file conda_requriements.txt

or you can create the env, and do:

conda install --file conda_requriements.txt

If requirements change: either a new one, or an updated version, you still want to pass them all in at once.

This lets the conda requirements resolvet use all the information to build an optimal environment, even with only one change.

Better yet, make an environment.yaml file with all the versions pinned, and use that to build the environment for your pyinstaller build. You are assured consistency that way.

HTH,

-CHB

 

my setuptools version is 41.4.0...perhaps a downgrade could help?  

  

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyinstaller...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/900beedb-ffa7-4322-9969-3824a37531cd%40googlegroups.com.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov