Care to share your numpy or OpenCV import hacks?

[Brian]

With PyInstaller 1.5, I was using the line
import numpy.core.multiarray
in my source to cause the numpy.core.multiarray.pyd to be copied into
the dist folder for the sake of the OpenCV code that needed it.

Now, with PyInstaller r1608, numpy.core.multiarray.pyd is still being
copied, but the exception
numpy.core.multiarray failed to import
is being thrown.

I created hook-numpy.py, with
hiddenimports = ["numpy.core", "numpy.core.multiarray"]
inside, but that failed to do the trick. There is no warnproject.txt
file to be found.

[Brian]

I've added "C:/Python27/Lib/site-packages/numpy" to the pathex of my
spec file, so the numpy pyd files appear in the dist directory, but
the “ImportError: numpy.core.multiarray failed to import” message
still appears. Is there something I can edit by hand just to get the
executable to recognize that the pyds are in the same directory?

[Martin Zibricky]

Could you please try Pyinstaller r1609?

If that works for you?

Thanks in advance.

Brian píše v Po 03. 10. 2011 v 16:22 -0700:

[Brian]

Thanks again for your quick response, Martin.

I created simple test projects for numpy and OpenCV.

My numpy test is getnumpy.py:
import numpy

print numpy.vander.__doc__

I ran:
python pyinstaller.py ../src/getnumpy.py
and then ran getnumpy.exe. r1609 works where r1608 failed!

My test for OpenCV is getopencv.py:
import cv
print cv.Trace.__doc__ # print cv.trace.__doc__ in the case of
OpenCV 2.2

I tried this with both OpenCV 2.2 and 2.3. In both cases, the result
was something like:
PS C:\users\b\p\tst\d\pyinstaller-trunk-r1609> .\getopencv\dist
\getopencv\getopencv.exe

ImportError: numpy.core.multiarray failed to import

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\PyInstaller\iu.py",
line 424, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\PyInstaller\iu.py",
line 514, in doimport
exec co in mod.__dict__
File "C:\users\b\p\tst\d\pyinstaller-trunk-r1609\getopencv\build
\pyi.win32\getopencv\outPYZ1.pyz\cv", line 1, in <module>
File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\PyInstaller\iu.py",
line 424, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\PyInstaller\iu.py",
line 493, in doimport
mod = director.getmod(nm)
File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\PyInstaller\iu.py",
line 285, in getmod
mod = owner.getmod(nm)
File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\PyInstaller\iu.py",
line 94, in getmod
mod = imp.load_module(nm, fp, attempt, (ext, mode, typ))

ImportError: numpy.core.multiarray failed to import

PS C:\users\b\p\tst\d\pyinstaller-trunk-r1609>

The OpenCV "build process", if I can call it that as I'm doing it, is
a little kludgy. For OpenCV 2.3, I copy cv.py and cv2.pyd from C:
\OpenCV2.3\build\python\2.7 to C:\Python27\Lib\site-packages. But, I
can open Python and call:
import cv
print cv.Trace.__doc__
and it works, so PyInstaller may be at fault, here.

Brian

[Martin Zibricky]

Brian píše v Út 04. 10. 2011 v 08:25 -0700:

> I tried this with both OpenCV 2.2 and 2.3. In both cases, the result
> was something like:

Where could I download opencv package?

[Brian]

I got it from http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.3.1/.

The installation instructions are at http://opencv.willowgarage.com/wiki/InstallGuide.
I used CMake-GUI to configure it and generate the Makefile. I think
the instructions for using cmake (not CMake-GUI) may be out of date.
Then, I used mingw32-make to build it. I'm not even sure that making
it is necessary, because the timestamps of cv.py and cv2.pyd in C:
\OpenCV2.3\build\python\2.7 are from August.

Thanks for looking into this.

[Martin Zibricky]

Brian píše v Út 04. 10. 2011 v 10:01 -0700:

> Thanks for looking into this.

Your getopencv example is working when I add there explicit import of
numpy module.

Pyintaller is not able detect opencv dependencie because cv2.pyd is a
native binary not a compiled python code.

We would need to create an import hook for cv module.

[Brian]

Ah, you're right. The explicit import works. I'll try this on the real
code. Thanks, Martin.

[Martin Zibricky]

Brian píše v Út 04. 10. 2011 v 15:40 -0700:

> Ah, you're right. The explicit import works. I'll try this on the real
> code. Thanks, Martin.

Could you please try to create import hook hook-cv.py with numpy or
maybe other necessary modules as hidden imports?

If it will work we could include it with other pyinstaller hooks.

[Brian]

Would hook-cv.py currently look like this?:

hiddenimports = ["numpy"]


On another note, I have not researched this yet, but I've encountered
a regression and PyInstaller is one of the view things that has
changed between versions. Is it possible that a recent change could
result in something strange going on with multiprocessing, especially
multiprocessing queues? Don't research this just yet, because I have
yet to confirm that pyinstaller is the only variable.

[Martin Zibricky]

Brian píše v St 05. 10. 2011 v 15:04 -0700:

>
> Would hook-cv.py currently look like this?:
>
> hiddenimports = ["numpy"]

Yes, that's exact.

>
>
> On another note, I have not researched this yet, but I've encountered
> a regression and PyInstaller is one of the view things that has
> changed between versions. Is it possible that a recent change could
> result in something strange going on with multiprocessing, especially
> multiprocessing queues? Don't research this just yet, because I have
> yet to confirm that pyinstaller is the only variable.

Are you experiencing issues like:

http://www.pyinstaller.org/ticket/182
http://www.pyinstaller.org/ticket/146

[Brian]

The traceback I get when running an r1609-compiled instance of my code
is:

Process SendeventProcess-1:

Traceback (most recent call last):

File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\gui\build
\pyi.win32\gui\outPY
Z1.pyz\multiprocessing.process", line 232, in _bootstrap
File "utils.pyx", line 2855, in utils.SendeventProcess.run (utils.c:
57485)
File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\gui\build
\pyi.win32\gui\outPY
Z1.pyz\multiprocessing.queues", line 75, in put
WindowsError: [Error 5] Access is denied

Line 2855 in utils.pyx reads:
self.resultsQueue.put((o, e))

SendeventProcess is a multiprocessing.Process.

I tried building the code using both hook-cv.py and, alternately,
"import numpy". With hook-cv.py, I saw the above traceback, as I did
with "import numpy", but I also, on other occasions, saw:

Traceback (most recent call last):

File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\gui\build
\pyi.win32\gui\outPY
Z1.pyz\multiprocessing.queues", line 238, in _feed
IOError: [Errno 6] The handle is invalid

and on other times I didn't see either of the two errors, but an
error, I don't know what, was caught within the code. My guess is that
I would have also seen these other two errors with the "import numpy"
approach had I run it enough times. The code invariably fails with
r1609 regardless of whether hook-cv.py or "import cv" is used.

To compile the code with r1609, I call:
python pyinstaller.py --onedir --icon=path/to/icon1.ico <moduleString>

I don't see any errors when running the code with 1.5.1 on Windows 7.
(Remember, it was broken with 1.5.1 on XP). To get the code to run
with 1.5.1, I add the line:
import numpy.core.multiarray
and the "import numpy" line was removed. That is the only difference
in the source code.

To compile it with 1.5.1, I call:
python Configure.py
python Makespec.py --onedir --icon=path/to/icon1.ico <moduleString>
python Build.py gui/gui.spec

I do have multiprocessing.freeze_support() at the very start of my
code, but I don't have any multiprocessing.Manager's.

[Martin Zibricky]

Brian píše v St 05. 10. 2011 v 21:40 -0700:

> I do have multiprocessing.freeze_support() at the very start of my
> code, but I don't have any multiprocessing.Manager's.

Could you please put together a small multiprocessing example which
would reproduce the issue?

BTW, I didn't know there is freeze_support() function in python. We
should mention that in the manual.

[Brian]

Here's an example:

import multiprocessing

class SendeventProcess(multiprocessing.Process):
def __init__(self, resultQueue):
self.resultQueue = resultQueue

multiprocessing.Process.__init__(self)
self.start()

def run(self):
self.resultQueue.put((1,2))


if __name__ == '__main__':
multiprocessing.freeze_support()
print 'pyb'
resultQueue = multiprocessing.Queue()
SendeventProcess(resultQueue)
print 'pyb'


When I run the compiled executable, I variously get the "handle is
invalid" error, the "Access is denied" error, and a successful run
with no errors. The un-compiled script runs well.

[Martin Zibricky]

Brian píše v Čt 06. 10. 2011 v 07:21 -0700:

> When I run the compiled executable, I variously get the "handle is
> invalid" error, the "Access is denied" error, and a successful run
> with no errors. The un-compiled script runs well.

The same is valid for me. Thanks for the example. I'll see what I can do
about it.

[Martin Zibricky]

Brian píše v Čt 06. 10. 2011 v 07:21 -0700:

> Here's an example:

I can confirm that your issue is related to this bug:

http://www.pyinstaller.org/ticket/182

If I did the following change and recompilled bootloader then your
example seems working.

Could you please recompile bootloader with the foolowing change and let
me know if your app works then?

If you want me to send you compilled bootloader with this change, let me
know.

--- source/common/launch.c (revision 1609)
+++ source/common/launch.c (working copy)
@@ -1296,7 +1296,7 @@
* as subprocess, this subprocess will not fooled into thinking that
it
* is already unpacked.
*/
- unsetenv("_MEIPASS2");
+ //unsetenv("_MEIPASS2");

/* Iterate through toc looking for scripts (type 's') */
while (ptoc < status->tocend) {

[Brian]

I tried compiling the bootloader with the version of waf in r1609, and
got this:

PS C:\Users\b\p\tst\d\pyinstaller-trunk-r1609m\source> python waf
configure build install
Windows-32bit detected
Checking for program gcc or cc : C:\MinGW\bin\gcc.exe
Checking for program cpp : C:\MinGW\bin\cpp.exe
Checking for program ar : C:\MinGW\bin\ar.exe
Checking for program ranlib : C:\MinGW\bin\ranlib.exe
Checking for program python : C:\Python27\python.exe
Checking for Python version : 2.7.1
Checking for library python27 : yes
Checking for program python2.7-config : not found
Checking for program python-config-2.7 : not found
Checking for custom code : Could not find the python
development headers
C:\Users\b\p\tst\d\pyinstaller-trunk-r1609m\source\wscript:111: error:
the configuration failed (see 'C:\\Users\\b\\p\\tst\\d\\pyinstaller-
trunk-r1609m\\source\\build\\config.log')
PS C:\Users\b\p\tst\d\pyinstaller-trunk-r1609m\source>


The last bit of C:\\Users\\b\\p\\tst\\d\\pyinstaller-trunk-r1609m\
\source\\build\\config.log is:
----------------------------------------
Checking for program python2.7-config
find program=['python2.7-config'] paths=[] var='PYTHON_CONFIG'
-> ''

----------------------------------------
Checking for program python-config-2.7
find program=['python-config-2.7'] paths=[] var='PYTHON_CONFIG'
-> ''
Include path for Python extensions (found via distutils module): 'C:\\\
\Python27\\\\include'

----------------------------------------
Checking for custom code
==>

#include "Python.h"
#ifdef __cplusplus
extern "C" {
#endif
void Py_Initialize(void);
void Py_Finalize(void);
#ifdef __cplusplus
}
#endif
int main()
{
Py_Initialize();
Py_Finalize();
return 0;
}

<==
[1/2] [32mcc: build\.conf_check_0\test.c -> build
\.conf_check_0\testbuild\default\test_1.o
[0mcc1.exe: warnings being treated as errors
..\test.c:6:7: error: 'Py_Initialize' redeclared without dllimport
attribute: previous dllimport ignored
..\test.c:7:7: error: 'Py_Finalize' redeclared without dllimport
attribute: previous dllimport ignored
['C:\\MinGW\\bin\\gcc.exe', '-Wdeclaration-after-statement', '-
Werror', '-fno-strict-aliasing', '-IC:\\\\Python27\\\\include', '..\
\test.c', '-c', '-o', 'default\\test_1.o']
command returned 'Build failed: -> task failed (err #1): \n\t{task:
cc test.c -> test_1.o}'Could not find the python development headers

I found very little on Google about this problem. I do have a
populated c:\Python27\include folder, and I don't know why it's
failing with the above error.

Brian

[Hartmut Goebel]

Am 07.10.2011 18:12, schrieb Brian:
> I tried compiling the bootloader with the version of waf in r1609, and
> got this:

While this may show another issue about compiling r1609, there should be
a solution which does not require recompiling the loader.

Please check my today's comment on http://www.pyinstaller.org/ticket/182
and the attachment. (I can not test it myself, since I have no Windows
machine at hand).

--
Schönen Gruß - Regards
Hartmut Goebel
Dipl.-Informatiker (univ.), CISSP, CSSLP

Goebel Consult
Spezialist für IT-Sicherheit in komplexen Umgebungen
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/
Goebel Consult ist Mitglied bei http://www.7-it.de

[Brian]

Thanks, Hartmut. There are some errors in the solution you posted that
you'll be able to discover using your current platform before it's
necessary to test on Windows.

Some comments:
there is no multiprocessing.forking.Popen class
choose either the name "_Popen" or "MyOpen"
in the call to super(), the name "_Popen" apparently can't be
referenced, perhaps because it is a nested class

I made these modifications, but the result is:
PS C:\users\b\p\tst\d\src> python .\test_multiprocess_22.py
pyb

Traceback (most recent call last):

File ".\test_multiprocess_22.py", line 36, in <module>
SendeventProcess(resultQueue)
File ".\test_multiprocess_22.py", line 24, in __init__
self.start()
File "C:\Python27\lib\multiprocessing\process.py", line 104, in
start
self._popen = Popen(self)
File ".\test_multiprocess_22.py", line 10, in __init__
super(_Popen, self).__init__(*args, **kw)
File "C:\Python27\lib\subprocess.py", line 672, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 841, in _execute_child
args = list2cmdline(args)
File "C:\Python27\lib\subprocess.py", line 573, in list2cmdline
for arg in seq:
TypeError: 'SendeventProcess' object is not iterable
PS C:\users\b\p\tst\d\src>

Here's the current version:
import multiprocessing
import os
import subprocess


class _Popen(subprocess.Popen):
def __init__(self, *args, **kw):
os.putenv('_MEIPASS2', '--dummy--')
try:
super(_Popen, self).__init__(*args, **kw)
finally:
os.unsetenv('_MEIPASS2')

# this is *completly* untested
class Process(multiprocessing.Process):
_Popen = _Popen


class SendeventProcess(Process):

def __init__(self, resultQueue):
self.resultQueue = resultQueue

multiprocessing.Process.__init__(self)
self.start()

def run(self):

print 'SendeventProcess'
self.resultQueue.put((1, 2))
print 'SendeventProcess'

if __name__ == '__main__':
multiprocessing.freeze_support()
print 'pyb'
resultQueue = multiprocessing.Queue()
SendeventProcess(resultQueue)
print 'pyb'

On Oct 7, 5:09 pm, Hartmut Goebel <h.goe...@goebel-consult.de> wrote:
> Am 07.10.2011 18:12, schrieb Brian:
>
> > I tried compiling the bootloader with the version of waf in r1609, and
> > got this:
>
> While this may show another issue about compiling r1609, there should be
> a solution which does not require recompiling the loader.
>

> Please check my today's comment onhttp://www.pyinstaller.org/ticket/182

> and the attachment. (I can not test it myself, since I have no Windows
> machine at hand).
>
> --
> Schönen Gruß - Regards
> Hartmut Goebel
> Dipl.-Informatiker (univ.), CISSP, CSSLP
>
> Goebel Consult

> Spezialist für IT-Sicherheit in komplexen Umgebungenhttp://www.goebel-consult.de

[Hartmut Goebel]

Am 08.10.2011 05:52, schrieb Brian:
> Thanks, Hartmut. There are some errors in the solution you posted that
> you'll be able to discover using your current platform before it's
> necessary to test on Windows.

Sure, but i was in a hurry ;-) I wanted to post the basic idea anyway.

> there is no multiprocessing.forking.Popen class

These is, see `pydoc multiprocessing.forking.Popen`, just the import was
wrong. One needs to import multiprocessing.forking explicitly.

You must not inherit from subprocess.Popen, which, is a different
implementation. See multiprocessing.forking. This is why your code fails.

I've just uploaded an new version which works (unpackaged) on Linux.
Please try this.

[Brian]

I changed the code to this:

import multiprocessing.forking
import os

class _Popen(multiprocessing.forking.Popen):

def __init__(self, *args, **kw):
os.putenv('_MEIPASS2',

os.path.dirname(os.path.abspath(__file__)) + ' ')

try:
super(_Popen, self).__init__(*args, **kw)
finally:
os.unsetenv('_MEIPASS2')

class Process(multiprocessing.Process):
pass
Process._Popen = _Popen

class SendeventProcess(Process):
def __init__(self, resultQueue):
self.resultQueue = resultQueue

multiprocessing.Process.__init__(self)
self.start()

def run(self):
print 'SendeventProcess'
self.resultQueue.put((1, 2))
print 'SendeventProcess'


if __name__ == '__main__':
multiprocessing.freeze_support()
print 'pyb'
resultQueue = multiprocessing.Queue()
SendeventProcess(resultQueue)
print 'pyb'

I couldn't define Process this way:

class Process(multiprocessing.Process):
_Popen = _Popen

Doing so caused an error (from PyInstaller code, IIRC) about a key not
being in a dictionary ('_Popen' being the key, and the class the
dictionary?)

I don't know what value of _MEIPASS2 should be used. Notice that I
added a space at the end, because PyInstaller was truncating the value
by one.

I eventually ran my production code on Windows 7 and Windows XP using
r1609 with these changes incorporated. Windows 7 was fine. XP gave an
error

Traceback (most recent call last):

File "<string>", line 31, in <module>
File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\trialInstaller\build
\pyi.win32\trialInstaller\outPYZ1.pyz\os", line 150, in makedirs
File "C:\Users\b\p\tst\d\pyinstaller-trunk-r1609\trialInstaller\build
\pyi.win32\trialInstaller\outPYZ1.pyz\os", line 157, in makedirs
WindowsError: [Error 5] Access is denied: 'C:\\Program Files\\support'

A second executable in the production code fell into an infinite loop.
I've never seen these two errors on Windows 7, but it's possible they
are my own errors. I don't have full-time access to the XP machine to
dig into the problems.

Brian

On Oct 8, 4:52 am, Hartmut Goebel <h.goe...@goebel-consult.de> wrote:
> Am 08.10.2011 05:52, schrieb Brian:
>
> > Thanks, Hartmut. There are some errors in the solution you posted that
> > you'll be able to discover using your current platform before it's
> > necessary to test on Windows.
>
> Sure, but i was in a hurry ;-) I wanted to post the basic idea anyway.
>
> > there is no multiprocessing.forking.Popen class
>
> These is, see `pydoc multiprocessing.forking.Popen`, just the import was
> wrong. One needs to import multiprocessing.forking explicitly.
>
> You must not inherit from subprocess.Popen, which, is a different
> implementation. See multiprocessing.forking. This is why your code fails.
>
> I've just uploaded an new version which works (unpackaged) on Linux.
> Please try this.
>
> --
> Schönen Gruß - Regards
> Hartmut Goebel
> Dipl.-Informatiker (univ.), CISSP, CSSLP
>
> Goebel Consult

> Spezialist für IT-Sicherheit in komplexen Umgebungenhttp://www.goebel-consult.de

[Hartmut Goebel]

Hi Brain,

I changed the code to this:

Thanks, but I'm still missing quite a lot of information:

• Is Martins code (test_multiprocess.py) a vailid testcase? (This is:works unpackaged, works packages as --onedir, but fails packages as --onfile)
• Does my code (test_multiprocess_2.2.py) works if run unpackaged? Does is run when packages as --onedir?
• If not: what are the exact error messages?
  

Doing so caused an error (from PyInstaller code, IIRC) about a key not

This is quite a difference! You should first make the code running unpackaged. There is no use in packaging code which does not even work unpackaged.

being in a dictionary ('_Popen' being the key, and the class the
dictionary?)

I plain do not understand this. Please post the exact error message.

I don't know what value of _MEIPASS2 should be used. Notice that I

The value is not of any matter. The boadloader code just checks for the existence of this environment variable.

[Brian]

test_multiprocess.py works unpackaged



test_multiprocess.py does not work when built with --onedir
PS C:\Users\b\p\tst\d\pyinstaller-r1641> .\_test_multiprocess\dist
\_test_multiprocess\_test_multiprocess.exe
pyb
pyb
PS C:\Users\b\p\tst\d\pyinstaller-r1641> SendeventProcess
SendeventProcess

Traceback (most recent call last):

File "C:\Users\b\p\tst\d\pyinstaller-r1641\_test_multiprocess\build
\pyi.win32\_test_multiprocess\outPYZ1.pyz\multiprocessing.queues",

line 238, in _feed
IOError: [Errno 6] The handle is invalid

PS C:\Users\b\p\tst\d\pyinstaller-r1641> .\_test_multiprocess\dist
\_test_multiprocess\_test_multiprocess.exe
pyb
pyb
PS C:\Users\b\p\tst\d\pyinstaller-r1641> SendeventProcess
Process SendeventProcess-1:

Traceback (most recent call last):

File "C:\Users\b\p\tst\d\pyinstaller-r1641\_test_multiprocess\build
\pyi.win32\_test_multiprocess\outPYZ1.pyz\multiprocessing.process",
line 232, in _bootstrap
File "<string>", line 13, in run
File "C:\Users\b\p\tst\d\pyinstaller-r1641\_test_multiprocess\build
\pyi.win32\_test_multiprocess\outPYZ1.pyz\multiprocessing.queues",
line 75, in put

WindowsError: [Error 5] Access is denied

test_multiprocess.py does not work when built with --onefile
PS C:\Users\b\p\tst\d\pyinstaller-r1641> .\_test_multiprocess\dist
\_test_multiprocess.exe
pyb
pyb
PS C:\Users\b\p\tst\d\pyinstaller-r1641> SendeventProcess
SendeventProcess

Traceback (most recent call last):

File "C:\Users\b\p\tst\d\pyinstaller-r1641\_test_multiprocess\build
\pyi.win32\_test_multiprocess\outPYZ1.pyz\multiprocessing.queues",

line 238, in _feed
IOError: [Errno 6] The handle is invalid

PS C:\Users\b\p\tst\d\pyinstaller-r1641> .\_test_multiprocess\dist
\_test_multiprocess.exe
pyb
pyb
PS C:\Users\b\p\tst\d\pyinstaller-r1641> SendeventProcess
Process SendeventProcess-1:

Traceback (most recent call last):

File "C:\Users\b\p\tst\d\pyinstaller-r1641\_test_multiprocess\build
\pyi.win32\_test_multiprocess\outPYZ1.pyz\multiprocessing.process",
line 232, in _bootstrap
File "<string>", line 13, in run
File "C:\Users\b\p\tst\d\pyinstaller-r1641\_test_multiprocess\build
\pyi.win32\_test_multiprocess\outPYZ1.pyz\multiprocessing.queues",
line 75, in put

WindowsError: [Error 5] Access is denied

test_multiprocess_2.2.py works unpackaged



test_multiprocess_2.2.py does not work when packaged with --onedir
PS C:\Users\b\p\tst\d\pyinstaller-r1641> .\_test_multiprocess_22\dist
\_test_multiprocess_22\_test_multiprocess_22.exe
pyb
pyb
PS C:\Users\b\p\tst\d\pyinstaller-r1641> Traceback (most recent call
last):
File "<string>", line 3, in <module>
File "C:\Users\b\p\tst\d\pyinstaller-r1641\PyInstaller\iu.py", line

424, in importHook
mod = _self_doimport(nm, ctx, fqname)

File "C:\Users\b\p\tst\d\pyinstaller-r1641\PyInstaller\iu.py", line

493, in doimport
mod = director.getmod(nm)

File "C:\Users\b\p\tst\d\pyinstaller-r1641\PyInstaller\iu.py", line

285, in getmod
mod = owner.getmod(nm)

File "C:\Users\b\p\tst\d\pyinstaller-r1641\PyInstaller\archive.py",
line 439, in getmod
localpath:ExtInPkgImporter(localpath, nm)},
File "C:\Users\b\p\tst\d\pyinstaller-r1641\PyInstaller\archive.py",
line 456, in __init__
iu.DirOwner.__init__(self, path)
File "C:\Users\b\p\tst\d\pyinstaller-r1641\PyInstaller\iu.py", line
71, in __init__
raise OwnerError("%s is not a directory" % path)
iu.OwnerError: <OwnerError --dummy- is not a directory>



When I change the line in test_multiprocess_2.2.py containing 'dummy'
to this:

os.putenv('_MEIPASS2', os.path.dirname(os.path.abspath(__file__)) + '
')

the executable works with --onedir, but this message appears with --
onefile:
PS C:\Users\b\p\tst\d\pyinstaller-r1641> .\_test_multiprocess_22b\dist
\_test_multiprocess_22b.exe
pyb
pyb
Error loading Python DLL: C:\Users\b\p\tst\d\pyinstaller-r1641
python27.dll (error code 126)

When I change the line to this:
os.putenv('_MEIPASS2', os.path.dirname(os.path.abspath(__file__)))
the --onefile build fails like this:
PS C:\Users\b\p\tst\d\pyinstaller-r1641> .\_test_multiprocess_22c\dist
\_test_multiprocess_22c.exe
pyb
pyb
Error loading Python DLL: C:\Users\b\p\tst\d\pyinstaller-
r1641python27.dll (error code 126)

Notice the presence of a space between "pyinstaller-r1641" and
"python27.dll" and the lack of one in the error message of the second
build.

I have not been able to reproduce the error about the key.

On Oct 9, 1:15 pm, Hartmut Goebel <h.goe...@goebel-consult.de> wrote:
> Hi Brain,
>
> > I changed the code to this:
>
> Thanks, but I'm still missing quite a lot of information:
>

>     * Is Martins code (test_multiprocess.py) a vailid testcase? (This

>       is:works unpackaged, works packages as --onedir, but fails
>       packages as --onfile)

>     * Does my code (test_multiprocess_2.2.py) works if run unpackaged?

>       Does is run when packages as --onedir?

>     * If not: what are the exact error messages?

>
> > Doing so caused an error (from PyInstaller code, IIRC) about a key not
>
> This is quite a difference! You should first make the code running
> unpackaged. There is no use in packaging code which does not even work
> unpackaged.
>
> > being in a dictionary ('_Popen' being the key, and the class the
> > dictionary?)
>
> I plain do not understand this. Please post the exact error message.
>
> > I don't know what value of _MEIPASS2 should be used. Notice that I
>
> The value is not of any matter. The boadloader code just checks for the
> existence of this environment variable.
>
> --
> Schönen Gruß - Regards
> Hartmut Goebel
> Dipl.-Informatiker (univ.), CISSP, CSSLP
>
> Goebel Consult

> Spezialist für IT-Sicherheit in komplexen Umgebungenhttp://www.goebel-consult.de

[Brian]

I forgot to mention that all of this testing of r1641 of PyInstaller
was done with the hook I created for OpenCV in place. hook-cv.py
consists only of the line:
hiddenimports = ["numpy"]

[Hartmut Goebel]

Am 09.10.2011 21:45, schrieb Brian:
> test_multiprocess.py works unpackaged
Fine.

> test_multiprocess.py does not work when built with --onedir

You said, your original program did run when built with --onedir, didn't
it? So we already have a problem here.

> test_multiprocess.py does not work when built with --onefile

This is what I expected

> test_multiprocess_2.2.py works unpackaged
This is good.

> test_multiprocess_2.2.py does not work when packaged with --onedir

This is not so good :-(

> File "C:\Users\b\p\tst\d\pyinstaller-r1641\PyInstaller\iu.py", line
> 71, in __init__
> raise OwnerError("%s is not a directory" % path)
> iu.OwnerError:<OwnerError --dummy- is not a directory>

Thanks for your other tests here. Somebody has to look much deeper into
the code and follow the control-flow. I'll not have time for this
before the end of this week -- and I still do not have a Windows
development environment :-)

Perhaps you can try to find out a bit more. One central point is
source/common/launch.c and as it looks PyInstaller\iu.py, too.

You may try this patch to solve the '--dummy--' problem:

===================================================================
--- PyInstaller/archive.py (Revision 1649)
+++ PyInstaller/archive.py (Arbeitskopie)
@@ -428,7 +428,7 @@
except AttributeError:
raise ImportError, "PYZ entry '%s' (%s) is not a valid
code object" % (nm, repr(co))
if ispkg:
- if '_MEIPASS2' in _environ:
+ if '_MEIPASS2' in _environ and _environ['_MEIPASS2'] !=
'--dummy--']:
localpath = _environ['_MEIPASS2'][:-1]
else:
localpath = iu._os_path_dirname(self.path)

[Brian]

In summary of the problem so far, importing OpenCV is broken until the
OpenCV hook is added (I haven't checked that in the past ~12 hours).
Users also have to be told how they have to modify their
multiprocessing code to get it to work with PyInstaller, and the
limited testing I've been able to do on Windows XP shows that
PyInstaller may be broken for that OS despite making changes to the
target source code.

Martin, do you have time to look into this problem again?

> Spezialist für IT-Sicherheit in komplexen Umgebungenhttp://www.goebel-consult.de

[Martin Zibricky]

Brian píše v Po 10. 10. 2011 v 08:32 -0700:

> In summary of the problem so far, importing OpenCV is broken until the
> OpenCV hook is added (I haven't checked that in the past ~12 hours).
> Users also have to be told how they have to modify their
> multiprocessing code to get it to work with PyInstaller, and the
> limited testing I've been able to do on Windows XP shows that
> PyInstaller may be broken for that OS despite making changes to the
> target source code.
>
> Martin, do you have time to look into this problem again?

So the issue here is to get test_multiprocess_2.2.py working on WinXP
when frozen, right?

I think I could find some time this evening or tomorrow.

[Brian]

Yes, that's the issue. I know my code breaks on XP, but I haven't
tried test_multiprocess_22.py. I'll verify that it fails later today
or tomorrow and report the results to save you the hassle in case it's
just a problem with my code.

[Martin Zibricky]

Brian píše v Po 10. 10. 2011 v 10:23 -0700:

> Yes, that's the issue. I know my code breaks on XP, but I haven't
> tried test_multiprocess_22.py. I'll verify that it fails later today
> or tomorrow and report the results to save you the hassle in case it's
> just a problem with my code.

I tried test_multiprocess_22.py on winxp. But it didn't work. It was
failing because of '--dummy--' in PYTHONPATH.

I'm not sure where to start with debugging.

[Brian]

Hartmut provided a patch a few posts ago. You could also try replacing
'--dummy--' with
os.path.dirname(os.path.abspath(__file__)) + ' '
which is what I did.

Harmtut also suggested where to look to start debugging in his most
recent post.

[Martin Zibricky]

Hartmut Goebel píše v Po 10. 10. 2011 v 15:56 +0200:

>
> You may try this patch to solve the '--dummy--' problem:
>
> ===================================================================
> --- PyInstaller/archive.py (Revision 1649)
> +++ PyInstaller/archive.py (Arbeitskopie)

When trying the patch I got the following traceback:

-------------------------------------------------------

Z:\tmp\pyi_trunk>.\test_multiprocess_2.2\dist\test_multiprocess_2.2
\test_multipr
ocess_2.2.exe
_MEIPASS2 is NULL
Extracting binaries
Executing self as child with Setting up to run child
Creating child process
Waiting for child process to finish...
_MEIPASS2 is Z:\tmp\pyi_trunk\test_multiprocess_2.2\dist
\test_multiprocess_2.2\
Already in the child - running!
manifestpath: Z:\tmp\pyi_trunk\test_multiprocess_2.2\dist
\test_multiprocess_2.2\
test_multiprocess_2.2.exe.manifest
Activation context created
Activation context activated
Z:\tmp\pyi_trunk\test_multiprocess_2.2\dist\test_multiprocess_2.2
\python27.dll
Manipulating evironment
PYTHONPATH=Z:/tmp/pyi_trunk/test_multiprocess_2.2/dist/test_multiprocess_2.2
importing modules from CArchive
extracted iu
extracted struct
extracted archive
Installing import hooks
outPYZ1.pyz
Running scripts
pyb

Traceback (most recent call last):

File "<string>", line 33, in <module>
File "<string>", line 21, in __init__
File "Z:\tmp\pyi_trunk\test_multiprocess_2.2\build\pyi.win32
\test_multiprocess
_2.2\outPYZ1.pyz\multiprocessing.process", line 130, in start
File "<string>", line 7, in __init__
OSError: [Errno 42] Illegal byte sequence
RC: -1 from test_multiprocess_2.2
OK.
Deactivating activation context
Releasing activation context
Done
Back to parent...
Freeing status for Z:\tmp\pyi_trunk\test_multiprocess_2.2\dist
\test_multiprocess
_2.2\test_multiprocess_2.2.exe

[Brian]

Remove the patch and try replacing '--dummy--' with
os.path.dirname(os.path.abspath(__file__)) + ' '

It worked for me on Windows 7.

[Martin Zibricky]

Hartmut Goebel píše v Po 10. 10. 2011 v 15:56 +0200:

> You may try this patch to solve the '--dummy--' problem:
>
> ===================================================================
> --- PyInstaller/archive.py (Revision 1649)
> +++ PyInstaller/archive.py (Arbeitskopie)

Previously I run wrong code example. When applying patch for --dummy--
this is the output. There is only error in activating context.

---------------------------------------------------

pyb

OK.
Deactivating activation context
Releasing activation context
Done
Back to parent...
Freeing status for Z:\tmp\pyi_trunk\test_multiprocess_2.2\dist
\test_multiprocess
_2.2\test_multiprocess_2.2.exe

_MEIPASS2 is --dummy--

Z:\tmp\pyi_trunk>Already in the child - running!
manifestpath: --dummy--test_multiprocess_2.2.exe.manifest
Error activating the context

Z:\tmp\pyi_trunk\test_multiprocess_2.2\dist\test_multiprocess_2.2
\python27.dll
Manipulating evironment

PYTHONPATH=--dummy-;Z:/tmp/pyi_trunk/test_multiprocess_2.2/dist/test_multiproces
s_2.2

importing modules from CArchive
extracted iu
extracted struct
extracted archive
Installing import hooks
outPYZ1.pyz
Running scripts

SendeventProcess
SendeventProcess

[Martin Zibricky]

Brian píše v Po 10. 10. 2011 v 14:17 -0700:
> os.path.dirname(os.path.abspath(__file__))

os.path.dirname(os.path.abspath(sys.executable)) could also work.

[Brian]

Martin, are you saying that you've tested it on XP and it experiences
an error in activating the context? Is the error meaningful or is it
something that can be ignored?

[Martin Zibricky]

Brian píše v Po 10. 10. 2011 v 16:11 -0700:

> Martin, are you saying that you've tested it on XP and it experiences
> an error in activating the context? Is the error meaningful or is it
> something that can be ignored?

That's what I'm not sure. For this I would like to get some input from
someone more experienced.

[Martin Zibricky]

Brian píše v Po 10. 10. 2011 v 16:11 -0700:

I tried to set _MEIPASS2 to
os.path.dirname(os.path.abspath(sys.executable)) and it seems to work
correctly.

Attached is the multiprocess code that works for me on winxp without any
context error. Could you test it please?

If it works we should create from this example a test case for
pyinstaller.

Output when running on winxp:
--------------------------------------------------
Z:\tmp\pyi_trunk>.\test_multiprocess_2_2\dist\test_multiprocess_2_2
\test_multiprocess_2_2.exe

_MEIPASS2 is NULL
Extracting binaries
Executing self as child with Setting up to run child
Creating child process
Waiting for child process to finish...

_MEIPASS2 is Z:\tmp\pyi_trunk\test_multiprocess_2_2\dist
\test_multiprocess_2_2\

Already in the child - running!

manifestpath: Z:\tmp\pyi_trunk\test_multiprocess_2_2\dist
\test_multiprocess_2_2\
test_multiprocess_2_2.exe.manifest

Activation context created
Activation context activated

Z:\tmp\pyi_trunk\test_multiprocess_2_2\dist\test_multiprocess_2_2
\python27.dll
Manipulating evironment
PYTHONPATH=Z:/tmp/pyi_trunk/test_multiprocess_2_2/dist/test_multiprocess_2_2

importing modules from CArchive
extracted iu
extracted struct
extracted archive
Installing import hooks
outPYZ1.pyz
Running scripts
pyb
pyb
OK.
Deactivating activation context
Releasing activation context
Done
Back to parent...

Freeing status for Z:\tmp\pyi_trunk\test_multiprocess_2_2\dist
\test_multiprocess
_2_2\test_multiprocess_2_2.exe

Z:\tmp\pyi_trunk>_MEIPASS2 is Z:\tmp\pyi_trunk\test_multiprocess_2_2
\dist\test_m
ultiprocess_2_2\

Already in the child - running!

manifestpath: Z:\tmp\pyi_trunk\test_multiprocess_2_2\dist
\test_multiprocess_2_2\
test_multiprocess_2_2.exe.manifest

Activation context created
Activation context activated

Z:\tmp\pyi_trunk\test_multiprocess_2_2\dist\test_multiprocess_2_2
\python27.dll
Manipulating evironment
PYTHONPATH=Z:/tmp/pyi_trunk/test_multiprocess_2_2/dist/test_multiprocess_2_2

[Brian]

Martin,

Due to a shift in task focus, it could be a while before I get back to
PyInstaller. Thanks for your work. Dealing with you has been a
pleasure.

Brian

>  test_multiprocess_2_3.py
> < 1KViewDownload

[Hartmut Goebel]

Am 11.10.2011 00:49, schrieb Martin Zibricky:
> PYTHONPATH=--dummy-;Z:/tmp/pyi_trunk/test_multiprocess_2.2/dist/test_multiproces
> s_2.2

IC, this is a lace I've overseen since I was grepping for _MEIPASS2

[Hartmut Goebel]

Am 11.10.2011 15:42, schrieb Martin Zibricky:
> I tried to set _MEIPASS2 to
> os.path.dirname(os.path.abspath(sys.executable)) and it seems to work
> correctly.

Great. Thanks for debugging.

[Hartmut Goebel]

Am 11.10.2011 15:42, schrieb Martin Zibricky:

> I tried to set _MEIPASS2 to
> os.path.dirname(os.path.abspath(sys.executable)) and it seems to work
> correctly.

Documented in http://www.pyinstaller.org/wiki/Recipe/Multiprocessing

[Martin Zibricky]

Brian píše v Út 11. 10. 2011 v 21:50 -0700:

>
> Due to a shift in task focus, it could be a while before I get back to
> PyInstaller. Thanks for your work. Dealing with you has been a
> pleasure.
>
> Brian

Brian, if the recipe (example) does not work for you

http://www.pyinstaller.org/wiki/Recipe/Multiprocessing

please reopen the following ticket:

http://www.pyinstaller.org/ticket/182

[Martin Zibricky]

Hartmut Goebel píše v Čt 13. 10. 2011 v 11:40 +0200:
> Documented in http://www.pyinstaller.org/wiki/Recipe/Multiprocessing

It works fine in --onedir mode. However for --onefile there are some
issues.

One minor issue was that we need to get correct _MEIPASS value. And we
can't get it from dirname(sys.executable). I fixed the Recipe to get the
right value from PYTHONPATH.

However, another issue is that the main process is not waiting for the
multiprocess subprocesses to finish their activity!

- not frozen python code is waiting for multiprocess subprocesses)
- the multiprocess subprocess then fails becase it can't find
libpython.dll
- that's because at that time main process is gone and files unpacked
into tmp dir are already deleted

We need to find a way how to wait in main process for multiprocess
subprocesses.

I reopened the ticket #182.

Martin

[Hartmut Goebel]

Am 13.10.2011 14:15, schrieb Martin Zibricky:
> Hartmut Goebel píše v Čt 13. 10. 2011 v 11:40 +0200:
>> Documented in http://www.pyinstaller.org/wiki/Recipe/Multiprocessing
> It works fine in --onedir mode. However for --onefile there are some
> issues.

Argl. Reading this thread I thought, it does.

> One minor issue was that we need to get correct _MEIPASS value. And we
> can't get it from dirname(sys.executable). I fixed the Recipe to get the
> right value from PYTHONPATH.

This will fail if sys.path was changed in the meantime. I propose
changing the loader to store the value somewhere else. E.e. in sys._MEIPASS.

> However, another issue is that the main process is not waiting for the
> multiprocess subprocesses to finish their activity!

So we need dome kind of `wait`. Maybe this could help at the end of
__main__:

while multiprocessing.active_children():
multiprocessing.active_children()[0].join()

[Martin Zibricky]

Hartmut Goebel píše v Čt 13. 10. 2011 v 15:27 +0200:

> This will fail if sys.path was changed in the meantime. I propose
> changing the loader to store the value somewhere else. E.e. in
> sys._MEIPASS.

This would be fine.

[Martin Zibricky]

Hartmut Goebel píše v Čt 13. 10. 2011 v 15:27 +0200:

> while multiprocessing.active_children():
> multiprocessing.active_children()[0].join()

This works. I'll update the recipe.

I think we have a working example.

[Hartmut Goebel]

Am 13.10.2011 23:48, schrieb Martin Zibricky:
> This works. I'll update the recipe.

Fine.

> I think we have a working example

Please mind closing #182 then,

[Martin Zibricky]

Hartmut Goebel píše v Pá 14. 10. 2011 v 11:47 +0200:

> > I think we have a working example
> Please mind closing #182 then,

what about the sys._MEIPASS?

[Hartmut Goebel]

Am 14.10.2011 14:58, schrieb Martin Zibricky:
> what about the sys._MEIPASS

I forgot. Unfortunalty I'm not the C-programmer in our team ;-)

[Martin Zibricky]

Hartmut Goebel píše v Pá 14. 10. 2011 v 16:02 +0200:

>
> Am 14.10.2011 14:58, schrieb Martin Zibricky:
> > what about the sys._MEIPASS

We could unset _MEIPASS in the iu.py file during bootstrap process and
not directly in the loader?

> I forgot. Unfortunalty I'm not the C-programmer in our team ;-)

Me too :)

[Hartmut Goebel]

Am 14.10.2011 22:31, schrieb Martin Zibricky:
> We could unset _MEIPASS in the iu.py file during bootstrap process and
> not directly in the loader?

Good idea. This should work (while I have to look at the source first.)
Moving as much logic into Python code as possible eases maintenance.

>> I forgot. Unfortunalty I'm not the C-programmer in our team ;-)
> Me too :

*calling* Gioooovaaaanniiiiii ;-)