Support for NumPy?
Chris Ball
PyInstaller looks like a very handy way to distribute Python software,
so I'd like to use it. I have a question, though: does PyInstaller
support NumPy?
The reason I ask is that NumPy is not listed at
http://www.pyinstaller.org/wiki/SupportedPackages, but matplotlib is
(which depends on NumPy, doesn't it?). I searched this mailing list,
and I found hints that people are using NumPy with PyInstaller, but
apparently no explicit discussion of it. Sorry if I've missed
something obvious!
From reading PyInstaller's manual, I see there an issue with "hidden"
imports (which I understand to mean that e.g. just having "import
numpy" in my script does not allow PyInstaller to find various names
that numpy actually exposes). I found that PyInstaller has hooks to
support this kind of thing, but is it possible to tell PyInstaller to
include all of numpy without trying to figure out which parts my
script actually uses? I couldn't see anything about that in the
manual.
Thanks,
Chris
Daniel Hyams
pyinstaller. Just give it a whirl...
> --
> You received this message because you are subscribed to the Google Groups "PyInstaller" group.
> To post to this group, send email to pyins...@googlegroups.com.
> To unsubscribe from this group, send email to pyinstaller...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pyinstaller?hl=en.
>
>
--
Daniel Hyams
dhy...@gmail.com
Chris Ball
> I've never had a problem packaging numpy nor matplotlib with
> pyinstaller. Just give it a whirl...
That sounds very encouraging.
I've now tried it, but it didn't work for me immediately. Again, I'm
sorry if I've missed something. Maybe you can see I've done something
stupid from the information below? Thanks!
Here is my test script:
import numpy
a = numpy.array([1,2,3])
Here is how I ran pyinstaller:
$ python pyinstaller-svn/pyinstaller.py test1.py
In build/pyi.linux2/test1/warntest1.txt, I see 78 warnings. The file
begins like this:
W: no module named PyQt4._qt (top-level import by PyQt4.QtXml)
W: no module named matplotlib.rcParams (top-level import by matplotlib.figure)
W: no module named glib.Idle (top-level import by gobject)
W: no module named numpy.not_equal (top-level import by numpy.numarray.ufuncs)
When I run the resulting script (dist/test1/test1), I see this
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 434, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 524, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test1/outPYZ1.pyz/numpy", line 130, in <module>
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 434, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 524, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test1/outPYZ1.pyz/numpy.add_newdocs", line
9, in <module>
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 434, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 524, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test1/outPYZ1.pyz/numpy.lib", line 4, in <module>
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 434, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 524, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test1/outPYZ1.pyz/numpy.lib.type_check",
line 8, in <module>
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 434, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 524, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test1/outPYZ1.pyz/numpy.core", line 34, in <module>
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 434, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 524, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test1/outPYZ1.pyz/numpy.testing", line 10,
in <module>
File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 453, in importHook
raise ImportError("No module named %s" % fqname)
ImportError: No module named decorators
I have pyinstaller r1700 from svn. I'm using Ubuntu 10.04 LTS (which
includes Python 2.6.5 and numpy 1.3.0).
Thanks,
Chris
Chris Ball
...
> I have pyinstaller r1700 from svn. I'm using Ubuntu 10.04 LTS (which
> includes Python 2.6.5 and numpy 1.3.0).
Sorry, just to add: I first tried the current release of PyInstaller
(1.5.1), using this script:
import pylab
pylab.plot([1,2,3])
I got what looks like a very similar error, though:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/pylab", line 1, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/matplotlib", line 129, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/matplotlib.rcsetup", line
19, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/matplotlib.colors", line
52, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/numpy", line 130, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/numpy.add_newdocs", line
9, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/numpy.lib", line 4, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/numpy.lib.type_check",
line 8, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/numpy.core", line 34, in <module>
File "./pyinstaller-1.5.1/iu.py", line 436, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "./pyinstaller-1.5.1/iu.py", line 521, in doimport
exec co in mod.__dict__
File "./build/pyi.linux2/test2/outPYZ1.pyz/numpy.testing", line 10,
in <module>
File "./pyinstaller-1.5.1/iu.py", line 455, in importHook
raise ImportError, "No module named %s" % fqname
ImportError: No module named decorators
Chris
Hartmut Goebel
> import numpy
> a = numpy.array([1,2,3])
> File "./build/pyi.linux2/test1/outPYZ1.pyz/numpy.testing", line 10,
> in <module>
> File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 453, in importHook
> raise ImportError("No module named %s" % fqname)
> ImportError: No module named decorators
This looks like some module is not packages. But I can not reproduce
this, here the module is packaged correctly. Please follow the
instructions in
<http://www.pyinstaller.org/wiki/HowtoReportBugs#Beforesubmittingareport:Makesureeverythingispackaged>
to track this down. Thanks.
--
Schönen Gruß - Regards
Hartmut Goebel
Dipl.-Informatiker (univ.), CISSP, CSSLP
Goebel Consult
Spezialist für IT-Sicherheit in komplexen Umgebungen
http://www.goebel-consult.de
Monatliche Kolumne: http://www.cissp-gefluester.de/
Goebel Consult ist Mitglied bei http://www.7-it.de
Hartmut Goebel
> import pylab
> pylab.plot([1,2,3])
Well, it helps a lot using a *minimal* example. And since numpy fails,
you are better of dbugging you other test-script.
Chris Ball
<h.go...@goebel-consult.de> wrote:
> Am 21.10.2011 13:02, schrieb Chris Ball:
>> import numpy
>> a = numpy.array([1,2,3])
> [...]
>> File "./build/pyi.linux2/test1/outPYZ1.pyz/numpy.testing", line 10,
>> in <module>
>> File "./pyinstaller-svn/PyInstaller/loader/iu.py", line 453, in importHook
>> raise ImportError("No module named %s" % fqname)
>> ImportError: No module named decorators
>
> This looks like some module is not packages. But I can not reproduce
> this, here the module is packaged correctly. Please follow the
> instructions in
> <http://www.pyinstaller.org/wiki/HowtoReportBugs#Beforesubmittingareport:Makesureeverythingispackaged>
> to track this down. Thanks.
Thanks for the help.
Since you said you couldn't reproduce the problem, I tried on another
system and had no trouble this time. Again, I tried with PyInstaller
1.5.1 on 32-bit Ubuntu 10.04 LTS (same as the first system, with the
same versions of python and numpy). So, that's great! I'll continue
trying to package my application and see how I get on.
Meanwhile, I have two questions.
The first is: can I tell PyInstaller to include a module/package even
if it is not explicitly imported by my script?
The second (not so important to me, so feel free to ignore) is why
didn't PyInstaller work with numpy on the first system?
The dist/ directory on both systems appears to contain the same
numpy-related files. Using ArchiveViewer to inspect the outPYZ1.pyz
files on both, I can see that numpy.testing.decorators is missing from
the first system's but not the second's:
System 1:
...
'numpy.random': (True, 1567857L, 1283),
'numpy.random.info': (False, 899404L, 1341),
'numpy.testing': (True, 3559330L, 416),
'numpy.testing.info': (False, 3148191L, 596),
'numpy.testing.numpytest': (False, 2954259L, 999),
'numpy.testing.parametric': (False, 2850802L, 3551),
'numpy.testing.utils': (False, 3159099L, 8359),
'numpy.version': (False, 1666492L, 393),
System 2:
...
'numpy.random': (True, 678742L, 1283),
'numpy.random.info': (False, 1238003L, 1343),
'numpy.testing': (True, 1239346L, 416),
'numpy.testing.decorators': (False, 997868L, 2377),
'numpy.testing.noseclasses': (False, 1125914L, 4013),
'numpy.testing.nosetester': (False, 1508182L, 3889),
'numpy.testing.numpytest': (False, 1451974L, 1000),
'numpy.testing.utils': (False, 1522985L, 8360),
'numpy.version': (False, 885498L, 395),
...
On System 1, I can import numpy.testing.decorators without a problem.
I wonder why the numpy modules are not the same on both systems?
Thanks again,
Chris
Hartmut Goebel
> Thanks for the help.
You're welcome.
> The second (not so important to me, so feel free to ignore) is why
> didn't PyInstaller work with numpy on the first system?
> files on both, I can see that numpy.testing.decorators is missing from
> the first system's but not the second's:
investigate it.
In build/logdict*.log you'll find the dependencies. If the module is
missing here, some dependency did not get resolved. Please search for
"decorators", nor for the qualified module name. This may expose the error.
In build/pyi.*/YourProject/warn*.txt you'll find the warnings, esp.
missing modules. If the module is listed here,
Hartmut Goebel
> if it is not explicitly imported by my script?
Currently the only way is to write a hook. When using SVn trunk, you may
use the option --additional-hooks-dir=HOOKSPATH.
Further, there is <http://www.pyinstaller.org/ticket/305> to implement
some way for specifying hidden imports in the .spec file.
Chris Ball
<h.go...@goebel-consult.de> wrote:
> Am 23.10.2011 20:41, schrieb Chris Ball:
>>
>> Thanks for the help.
>
> You're welcome.
>>
>> The second (not so important to me, so feel free to ignore) is why
>> didn't PyInstaller work with numpy on the first system?
>
> [...]
>>
>> files on both, I can see that numpy.testing.decorators is missing from
>> the first system's but not the second's:
>
> This is the question you started this thread for, so we should try to
> investigate it.
>
> In build/logdict*.log you'll find the dependencies.
(I found logdict2.6.5.final.0-1.log in the same directory as build/
and dist/, not in build/)
> If the module is missing
> here, some dependency did not get resolved. Please search for "decorators",
> nor for the qualified module name. This may expose the error.
On system 1 (not working), "decorators" does not appear. On system 2
(working), "decorators" does appear:
numpy.testing: None -> <Module numpy.testing
/usr/lib/python2.6/dist-packages/numpy/testing/__init__.pyc
imports=[('unittest', 0, 0, -1), ('unittest.TestCase', 0, 0, -1),
('decorators', 0, 0, -1), ('utils', 0, 0, -1), ('utils.*', 0, 0, -1),
('numpytest', 0, 0, -1), ('numpytest.*', 0, 0, -1), ('nosetester', 0,
0, -1), ('nosetester.NoseTester', 0, 0, -1), ('nosetester', 0, 0, -1),
('nosetester.run_module_suite', 0, 0, -1)] binaries=[] datas=[]>
and
numpy.testing.decorators: None -> <Module numpy.testing.decorators
/usr/lib/python2.6/dist-packages/numpy/testing/decorators.pyc
imports=[('nose', 1, 0, -1), ('nose', 1, 0, -1), ('noseclasses', 1, 0,
-1), ('noseclasses.KnownFailureTest', 1, 0, -1)] binaries=[] datas=[]>
> In build/pyi.*/YourProject/warn*.txt you'll find the warnings, esp. missing
> modules. If the module is listed here,
On both systems, there are many warnings (warntest1.txt is 426 lines
on system 2, and 584 on system 1). On system 1, "decorators" does not
appear; on system 2, "decorators" appears:
W: no module named nose (delayed import by numpy.testing.decorators)
The contents of the log and warnings files prompted me to check if
nose is installed on system 2 but not system 1. This turned out to be
the case. So, the problem was just not having the python-nose package
installed. Sorry about that! Still, this might be helpful to know for
someone else in the future.
Thanks again,
Chris
Hartmut Goebel
> On system 1 (not working), "decorators" does not appear. On system 2
> (working), "decorators" does appear:
So on system 1 we have a problem, since "decorators" is not found as a
dependency.
Please send me (as private mail) logdict*.log of both systems, so I can
have a look into.
> So, the problem was just not having the python-nose package
> installed.
I doubt, this was the problem. since
a) nose does not depend on numpy.testing.decorators, but the other way round
b) numpy.testing.decorators should have been included anyway.
Or am I wrong here?
Chris Ball
<h.go...@goebel-consult.de> wrote:
> Am 24.10.2011 19:35, schrieb Chris Ball:
>>
>> On system 1 (not working), "decorators" does not appear. On system 2
>> (working), "decorators" does appear:
>
> So on system 1 we have a problem, since "decorators" is not found as a
> dependency.
>
> Please send me (as private mail) logdict*.log of both systems, so I can have
> a look into.
>
>> So, the problem was just not having the python-nose package
>> installed.
>
> I doubt, this was the problem. since
> a) nose does not depend on numpy.testing.decorators, but the other way round
> b) numpy.testing.decorators should have been included anyway.
>
> Or am I wrong here?
Unfortunately, all I can say is what I did: after "sudo apt-get remove
python-numpy" followed by "sudo apt-get install python-nose" and then
"sudo apt-get install python-numpy", I no longer had the problem on
system 1. I assumed that meant missing python-nose was the problem.
However, it seems you were right to question that explanation. I just
did "sudo apt-get remove python-nose", and then went through the
pyinstaller procedure again from the beginning (as I have done each
time, starting from command to un-tar pyinstaller). The resulting
binary runs without error! So if the problem was not related to
python-nose, perhaps it was the removal of python-numpy followed by
its re-installation that fixed things? I have no idea why that would
be; before removing python-numpy, I checked that I could import
numpy.testing.decorators.
Now I can't reproduce the original problem. I guess this means there
was never any problem with pyinstaller. Sorry!
Chris
Chris Ball
<h.go...@goebel-consult.de> wrote:
> Am 23.10.2011 20:41, schrieb Chris Ball:
>>
>> The first is: can I tell PyInstaller to include a module/package even
>> if it is not explicitly imported by my script?
>
> Currently the only way is to write a hook. When using SVn trunk, you may use
> the option --additional-hooks-dir=HOOKSPATH.
Ok, thanks.
Chris
Hartmut Goebel
> So if the problem was not related to
> python-nose, perhaps it was the removal of python-numpy followed by
> its re-installation that fixed things?
seams to be fixed in deed, I do not really car :-)
> Now I can't reproduce the original problem. I guess this means there
> was never any problem with pyinstaller. Sorry!
Never mind. Thanks for helping debugging, do not hesitate to mail next
time again.