Re: [PyInstaller] ImportError: The _imaging C module is not installed

[Martin Zibricky]

Hi,

on IRC you wrote:

I've gone through and explicitly included anything PIL._imaging
seems to need.
pyinstaller.py does seem to have a problem finding any .so files
that start with @rpath

How did you install PIL?

It seems like your PIL._imaging.so has some dependencies starting with
@rpath.
What is your output of command

otool -L PIL._imaging.so

My is:

PIL/_imaging.so:
/opt/local/lib/libjpeg.8.dylib (compatibility version 12.0.0, current
version 12.0.0)
/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current
version 1.2.6)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 159.1.0)

PyInstaller is able to handle only @lib_path and @executable_path.

Question is: What path prefix to use instead of @rpath?

In other words, where should Pyinstaller look for missing .so/.dylib
files? PyInstaller needs to know the real path, not @rpath.

mmiscavage píše v Po 18. 06. 2012 v 11:29 -0700:
> I am running on Mac osX 10.7.4 using the
> pyinstaller-pyinstaller-2145d84.tar.gz. I was able to get this project
> working on Windows 7 but cannot get past this error on Mac. I am able
> to 'from PIL import _imaging' from the Python interactive console. The
> pyinstaller.py successfully finds the _imaging.so and puts it in the
> dist folder with all other lib files. Is this something I should be
> looking to make a hook for ? Or is something else wrong? Any help or
> nudge in the right direction would be appreciated.

[mmiscavage]

Here is my otool -L output:

@rpath/libjpeg.7.dylib (compatibility version 8.0.0, current version 8.0.0)

/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)

/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.0.0)

I have added those files specifically to the .spec file's a.binaries. 

I'm not sure what @rpath is supposed to be or if I can change it to something else. Any ideas on that?

My python install as well as PIL are from Enthought Python 7.2. I'm using Enthought Python because I was having a hard time getting wxPython and H5py working together properly on the Mac. 

[Martin Zibricky]

What are the absolute paths to your ./PIL/_imaging.so and to
libjpeg.7.dylib from your python distribution?

mmiscavage píše v Út 19. 06. 2012 v 08:50 -0700:

> Here is my otool -L output: @rpath/libjpeg.7.dylib (compatibility
> version 8.0.0, current version 8.0.0) /usr/lib/libz.1.dylib
> (compatibility version 1.0.0, current version
> 1.2.3) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current
> version 1.0.0) /usr/lib/libSystem.B.dylib (compatibility version
> 1.0.0, current version 111.0.0)
>
> I have added those files specifically to the .spec file's
> a.binaries.
>
> I'm not sure what @rpath is supposed to be or if I can change it to
> something else. Any ideas on that?

What pyinstaller does is replacing absolute paths by '@lib_path'. In
your case the output from otool for the _imaging.so from dist dir would
look like:
@lib_path/libjpeg.7.dylib
/usr/lib/libz.1.dylib
/usr/lib/libgcc_s.1.dylib
/usr/lib/libSystem.B.dylib

The issue is where pyinstaller should look for libjpeg.7.dylib.

I suppose that with the absolute paths you provide we'll able to fix
that.

[mmiscavage]

The absolute path to libjpeg.7.dylib is: /Library/Frameworks/Python.framework/Versions/7.2/lib/libjpeg.7.dylib

The absolute path to _imaging.so is: /Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/PIL/_imaging.so

[Martin Zibricky]

mmiscavage píše v Út 19. 06. 2012 v 10:43 -0700:

> The absolute path to libjpeg.7.dylib
> is: /Library/Frameworks/Python.framework/Versions/7.2/lib/libjpeg.7.dylib The absolute path to _imaging.so is: /Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/PIL/_imaging.so

When looking for @rpath info I found out that:
- osx uses MachO binary format
- we should read MachO command LC_RPATH from the _imaging.so binary
- value of LC_RPATH should be in your case '../../../' or
'../../../../lib'
- then we need to replace @rpath by this relative path and get absolute
path for it:

path_to_jpglib = os.path.join(DIR_TO_IMAGING_SO, '../../../../lib',
'libjpeg.7.dylib)

- PyInstaller uses macholib for manipulation with binaries on osx.
- I think macholib should be able to read the value of LC_RPATH from
_imaging.so

[mmiscavage]

Do I need to add 'path_to_jpglib = os.path.join(DIR_TO_IMAGING_SO, '../../../../lib',  'libjpeg.7.dylib) ' to my spec file ? Or are you adding a new case for Pyinstaller to handle? Or am I missing something else?

[Martin Zibricky]

mmiscavage píše v Út 19. 06. 2012 v 11:54 -0700:

> Do I need to add 'path_to_jpglib = os.path.join(DIR_TO_IMAGING_SO,
> '../../../../lib', 'libjpeg.7.dylib) ' to my spec file ? Or are you
> adding a new case for Pyinstaller to handle? Or am I missing something
> else?

This is a theoretical way how it could be properly fixed in pyinstaller.

As a temporary workaround you need to include 'path_to_jpglib' in your
spec file. And probably more .dylib. I suppose there will be more
unresolved dylib dependencies.

[mmiscavage]

I was able to get my executable working by adding all necessary dylibs to the a.binaries list and then editing line 224 of dylib.py to reflect my lib directory and not an @ variable. Not the nicest solution but it got it working for me. Thanks for your help.

[Martin Zibricky]

mmiscavage píše v Út 19. 06. 2012 v 14:50 -0700:

> I was able to get my executable working by adding all necessary dylibs
> to the a.binaries list and then editing line 224 of dylib.py to
> reflect my lib directory and not an @ variable. Not the nicest
> solution but it got it working for me. Thanks for your help.

I made some changes in pyinstaller development branch. Could you please
test it if it fixes the issue with @rpath for you?

Code diff:

https://github.com/pyinstaller/pyinstaller/commit/132b9b7dce9958f6651803300e5f7070ff0c9e63

[Don Dwiggins]

On 6/26/12 5:54 AM, Martin Zibricky wrote:
> mmiscavage píše v Út 19. 06. 2012 v 14:50 -0700:
>> I was able to get my executable working by adding all necessary dylibs
>> to the a.binaries list and then editing line 224 of dylib.py to
>> reflect my lib directory and not an @ variable. Not the nicest
>> solution but it got it working for me. Thanks for your help.
> I made some changes in pyinstaller development branch. Could you please
> test it if it fixes the issue with @rpath for you?

It looks like I'm having a similar problem as mmiscavage, but on Windows
XP. I get the same message when trying to execute one of my
applications. Martin, would your change apply on Windows as well?


Thanks,

--

Don Dwiggins
Advanced Publishing Technology

[Martin Zibricky]

Don Dwiggins píše v St 27. 06. 2012 v 13:37 -0700:

The @rpath fix is related only to Mac OS X.

Pyinstaller Tests pass on windows. So I suppose this change didn't break
anything for windows.

[Don Dwiggins]

On 6/27/12 2:15 PM, Martin Zibricky wrote:
> Don Dwiggins píše v St 27. 06. 2012 v 13:37 -0700:

>> It looks like I'm having a similar problem as mmiscavage, but on
>> Windows
>> XP. I get the same message when trying to execute one of my
>> applications. Martin, would your change apply on Windows as well?
> The @rpath fix is related only to Mac OS X.
>
> Pyinstaller Tests pass on windows. So I suppose this change didn't break
> anything for windows.

This prompted me to see if I could reproduce the problem on my machine.
I succeeded, taking the test_PIL test and putting it in a multipackage
with a dummy executable. I've submitted this as a new ticket, #431
(under moderation). (Unfortunately, I found the hard way that the
upload of the package of files needed to reproduce the error is too
large to upload, and I can't add files to the ticket until it's released
from moderation. I've attached just the needed files to this message in
a zip.)

[Don Dwiggins]

On 6/28/12 12:05 PM, Don Dwiggins wrote:
>
> This prompted me to see if I could reproduce the problem on my
> machine. I succeeded, taking the test_PIL test and putting it in a
> multipackage with a dummy executable. I've submitted this as a new
> ticket, #431 (under moderation). (Unfortunately, I found the hard way
> that the upload of the package of files needed to reproduce the error
> is too large to upload, and I can't add files to the ticket until it's
> released from moderation. I've attached just the needed files to this
> message in a zip.)
>

I don't know where I got the number 431 for the ticket. but it's wrong.
The real ticket is #573. I've just attached the files needed to
reproduce it.

[Matteo Boscolo]

Hi all,
I try to create a com object dll and I found some trouble:

I'm using the code provided from the git repository

1) the documentation say to use the MakeComServer.py but this file is
under the utils folder so at least python utils/MakeCOMServer.py options

2) The MakeComServer.py dose not work at all, I've fix it adding the
following line at the main section:

before:
try:
print
print epilog
except KeyboardInterrupt:
after:
try:
create(args, opts.debug, opts.verbose, opts.workdir, opts.ascii)
print epilog
except KeyboardInterrupt:

3) once the new .py file is genereted and a spec file too, if I try to
register the dll with regsvr32 [dll genereted ] I get an error: The
module ...mymodule.. whose loaded but the entry-point DllRegisterServer
was not found.
but if I look at the dll.. with dependencywolker I could find only the
dllRegisterServerEx and the DllUnregisterServerEx

any help is appreciated

regards,
Matteo

[hlcd...@gmail.com]

Forget about that.

Pyinstaller COM support is for now completely buggy.

To overcome your problem, here is how I process:
1) Download pyinstaller source code and extract it
2) Install Visual C++ express 2008
3) Inside MSVC console, go to bootloader source directory

DllRegisterServer is not exported in bootloaders because, with msvc (notice: not with gcc), you have to explicitely telling him that dllmain.def has to be taken into account during link step, if you want the relevant 4 functions to be exported. As I have not found any clue to tell waf how to include this dependency, I do something else.

4) Modify C:\Program Files\Microsoft SDKs\Windows\v6.0A\Include\ObjBase.h and C:\Program Files\Microsoft SDKs\Windows\v6.0A\Include\OleCtl.h to modify DllCanUnloadNow, DllGetClassObject, DllRegisterServer, DllUnregisterServer and precede their prototype with __declspec(dllexport).
5) It's not enough. In ..\PyInstaller-2.1\bootloader\windows, you have to modify dllmain.c and add :
#pragma comment( linker, "/export:DllCanUnloadNow=_DllCanUnloadNow@0")
#pragma comment( linker, "/export:DllGetClassObject=_DllGetClassObject@12")
#pragma comment( linker, "/export:DllRegisterServer=_DllRegisterServer@0" )
#pragma comment( linker, "/export:DllUnregisterServer=_DllUnregisterServer@0")
the goal is to force linker to export the corresponding functions without the _ and @n decorations.

6) Recompile and put the 4 win32 generated DLLs in ..Python27\Lib\site-packages\PyInstaller\bootloader\Windows-32bit

7) Unfortunately, it is not enough because startUp function inside dllmain.c is buggy. There is a totally meaningless memset inside it. You have to replace it with code borrowed from ..\PyInstaller-2.1\bootloader\common\main.c. More explicitely:

ARCHIVE_STATUS *archive_status = NULL;
archive_status = (ARCHIVE_STATUS *) calloc(1,sizeof(ARCHIVE_STATUS));

8) Unfortunately, this is not enough. When Iregister mycom object, there was still errors:

LOADER: Manipulating Python's sys.path
  File "<string>", line 1
    sys.path.append(r"..\test_com\")

                                                       ^
SyntaxError: EOL while scanning string literal

and

Installed ZlibsLOADER: Running scripts
Traceback (most recent call last):
  File "<string>", line 15, in <module>
AttributeError: 'module' object has no attribute '_MEIPASS'
LOADER: RC: -1 from pyi_rth_Tkinter
Loading PythoncomPythoncom failed to load

Hope this help, and will give hints to others to continue to debug COM functionnality of pyinstaller.

Hope Pyinstaller will read this post and won't drop COM support in future release, because it is so useful.