Compiling using pyinstaller external pythonpath that is in another program module

[António Alexandre Fraga Dias]

I have created a program in python where it call's an external python module from a commercial program. As far i have come in my analyses, I have to mentioned I am not a programmer or an expert, this module is a python over C. The lib is in pyd extension and in order to be call in the python script it has to have a environment variable call pythonpath an only can be used in python 3.4 or 3.5. I manage to circumvent this by using sys.path.insert(0, path) where path is the path to the module to be load or call without going through environment variable or the need to have it defined. The module structure is as follows:

caris\ 

         __init__.py 

         _py_caris.pyd

caris\

         __pycache__\

                                ......

caris\

         bathy\

                    .....

caris\

         coverage\

                          ....

caris\

         ui\

              ....

My script has a PyQt5 gui and runs over or is a "wrapper" over this module. When compiling with pyinstaller the processed starts with a bat file.

"C:\Program Files\Python35\Scripts\pyinstaller.exe" -n "Point Cloud" --clean -y ^ -c --debug=imports ^ --exclude-module=caris ^ --add-data="ReadAPointCloudFromAPLYFileExample_01.png;." ^ ^ --icon=Point_Cloud.ico ^ PC.py

Have tried some variations but no improvement.


This is snip from the output if i do not use --exclude-module=caris when compiling.

import search # PyInstaller PYZ
# caris not found in PYZ
# bytecode is stale for 'caris'
# code object from C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris\__init__.py
# caris._py_caris not found in PYZ
Traceback (most recent call last):
  File "PC.py", line 19, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "c:\program files\python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "search.py", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 697, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris\__init__.py", line 15, in <module>
    from ._py_caris import *
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 577, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 938, in create_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: DLL load failed: Impossível localizar o procedimento especificado.
[9712] Failed to execute script PC

My questions are:

Why PYZ has a problem with caris._py_caris? What I'm doing wrong or is this a bug?

caris._py_caris is the caris._py_caris.pyd could this be a problem for Pyinstaller?

Is this a correct why to do it?

[bwoodsend]

I take it you are putting the sys.path.insert in your PC.py? In which case that isn't enough for PyInstaller to find and bundle it properly.




I believe you can't do this using the command line. Run your bat file once. It should create a file called PC.spec. In it is all the information you would normally pass via command line arguments. To build you can just call

PyInstaller PC.spec

rather than running your bat file in future.




Now to modify the build you should modify the spec file. In your spec there should be an argument `pathex=["folder/containing/your/script"]`. Add "C:/Program Files/CARIS/BASE Editor/5.3/python/3.5/" to that list noting the use of /s rather than \s in the path.




That should do it.





Brénainn


[António Alexandre Fraga Dias]

First thank you.

I think i have done it correctly, the adding of the path that you suggested. You can see the file.

# -*- mode: python ; coding: utf-8 -*-


block_cipher = None




a = Analysis(['PC.py'],
             pathex=['D:\\Python\\Novo\\Point Cloud Ver2','C:\\Program Files\\CARIS\\BASE Editor\\5.3\\python\\3.5'],
             binaries=[],
             datas=[('ReadAPointCloudFromAPLYFileExample_01.png', '.')],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=['caris'],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [('v', None, 'OPTION')],
          exclude_binaries=True,
          name='Point Cloud',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True , icon='Point_Cloud.ico')
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='Point Cloud')

After running it the result was the same. I have to had that the path C:\\Program Files\\CARIS\\BASE Editor\\5.3\\python\\3.5 is a variable path since it corresponds to the install directory and I retrieve it using a class.

In a way i have also to refer that the caris module has DLL hidden in the pyd, do not know if this is the reason for having this error. It's clear when running without the

--exclude-module=caris

command.

If i do it i get the DLL list that pyinstaller sees but does not finds it. 

Any ideas? 

[bwoodsend]

Yes your changes to the spec file were exactly what I meant.

Hmm - running short of ideas. I've never had an issue caused by pyd files - I think PyInstaller handles them OK.

This block at the top of the traceback:

import search # PyInstaller PYZ
# caris not found in PYZ
# bytecode is stale for 'caris'
# code object from C:\Program Files\CARIS\BASE Editor\5.3\python\3.5\caris\__init__.py
# caris._py_caris not found in PYZ


That looks like it's still trying to run the original caris and didn't include it at build time. Try putting your `sys.path.insert(0, path)` line at the top of your spec file, before the `a = Analysis(...)` bit.

Also could you translate the following to English. I think I know what it says but I'm not sure.

_call_with_frames_removed
ImportError: DLL load failed: Impossível localizar o procedimento especificado.


Brénainn

 

[António Alexandre Fraga Dias]

Hello

Yes the translation would be:

_call_with_frames_removed
ImportError: DLL load failed: impossible to locate specified procedure.

As far for the `sys.path.insert(0, path)` do you mean use the class where i retrieve the path? In this path is a variable that is dependent of the installation dir of Caris Software.

Thank you

Alex

[bwoodsend]

Ok I'm not sure we're talking about the same thing here. Do you want the Caris software to be copied into your executable or do you want your executable to find and interact with the Caris installation at run time? I originally assumed the first but it looks like you actually mean the second.

I've done the 1st before and I just tried the second on a .pyd and they've both worked fine so either should be possible.

I'd maybe try finding and running Caris's python interpreter and see if what it's loading (using sys.modules) and if it's using any environment variables (os.environ).

Just to confirm - you are using Python 3.5 to run PyInstaller? pyd files must be run on the same minor version they are compiled for.

Brénainn

[António Alexandre Fraga Dias]

Hi

I would like the second.

 executable to find and interact with the Caris installation at run time

 

This is because is a commercial software and i do not have permission to include it.

I'd maybe try finding and running Caris's python interpreter and see if what it's loading (using sys.modules)

I didn't knew this was possible, I would like to try it and I will look in to it, can you give me some tips? 

On the environment variables only one that is the pythonpath set manually. I mange no to use it by using sys.path.insert(0, path) and it works. TTo point out that spyder or many other can' load the caris module. To run I use the python interpreter that come with it.

I use python 3.5 and compile on it. I have 3.5 ant 3.7 install but I call on compiling execution the v3.5.

Thank you

Alex

[bwoodsend]

Unless they've taken steps to hide it you can run the Caris python interpreter just by searching inside the Caris installation for python.exe and running it. But if you can run the Caris modules from your normal Python then there is no need.

I've tried googling around for what might trigger a The specified procedure could not be found Exception and it looks like it could be almost anything. One of the more likely looking ones is missing dependencies. Try running the following snippet in a fresh console. Don't use a console in Spyder for this because IPython imports lots of extra stuff on startup - just run from command. :

import sys
sys.path.insert(0, "C:/Program Files/CARIS/BASE Editor/5.3/python/3.5/")
import caris
print(sys.modules.keys())


This should list all the modules Caris uses. Post that list here. Hopefully there is something in there that needs to be added to PyInstaller's hiddenimports option.

Brénainn

[António Alexandre Fraga Dias]

Hi Brénainn,

I have run with different configurations and no success. 

I got when I used 

sys.path.insert(0, "C:/Program Files/CARIS/BASE Editor/5.3/python/3.5/caris")

was:

import search # PyInstaller PYZ

# caris not found in PYZ

Traceback (most recent call last):

  File "PC.py", line 19, in <module>

  File "<frozen importlib._bootstrap>", line 968, in _find_and_load

  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked

  File "c:\program files\python35\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module

    exec(bytecode, module.__dict__)

  File "search.py", line 7, in <module>

  File "<frozen importlib._bootstrap>", line 968, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

ImportError: No module named 'caris'

[7940] Failed to execute script PC

So must likely problem is that loading of caris module is not fully done. 

So I started to do research  based on your snippet  and the look for, in `sys`, something more so I used also `sys.modules.items() sys.path_importer_cache`

When doing this I found that in cache has 3 paths, as follow:

C:/Program Files/CARIS/BASE Editor/5.3/python/3.5/ 

C:/Program Files/CARIS/BASE Editor/5.3/python/3.5\caris 

C:/Program Files/CARIS/BASE Editor/5.3/python/3.5\caris\coverage 

The module itself are 4 as you can see from sys.modules.values() below:

('caris', <module 'caris' from 'C:/Program Files/CARIS/BASE Editor/5.3/python/3.5\\caris\\__init__.py'>) 

('caris._py_caris', <module 'caris._py_caris' from 'C:/Program Files/CARIS/BASE Editor/5.3/python/3.5\\caris\\_py_caris.pyd'>) 

('caris.coverage', <module 'caris.coverage' from 'C:/Program Files/CARIS/BASE Editor/5.3/python/3.5\\caris\\coverage\\__init__.py'>) 

('caris.coverage._py_caris_coverage', <module 'caris.coverage._py_caris_coverage' from 'C:/Program Files/CARIS/BASE Editor/5.3/python/3.5\\caris\\coverage\\_py_caris_coverage.pyd'>) 

Now I do call caris using:

import caris

from caris.coverage import *

As you can see I do import 2 modules but it comes 4, 2 for each. 

caris 

caris._py_caris 

caris.coverage 

caris.coverage._py_caris_coverage 

The caris._py_caris  and caris.coverage._py_caris_coverage are load from a pyd and caris and caris.coverage with __init__.py that can be easily seen from the sys.modules.values()  results. 

I hope this gives some idea over what is happening. I can see why this is occurring unless is something hidden preventing to fully comprehend this.

So in my view the exe is lacking something that the python interpreter, I used the  IDLE (Python 3.5 64-bit) for this and for this program and no problem, it works perfectly.

Sorry for this long explanation  but I',m really frustrated.

Thank you for your help.

Alex 

`