Pyrex / py2exe import interaction
Clark C . Evans
of pyrex and py2exe; the problem occurs when the module
compiled with pyrex has an "import". The program then
works just fine until the calling script is packaged
up with py2exe, then you get the following:
Traceback (most recent call last):
File "<string>", line 1, in ?
File "imputil.pyc", line 103, in _import_hook
File "<string>", line 52, in _import_top_module
File "imputil.pyc", line 216, in import_top
File "imputil.pyc", line 267, in _import_one
File "<string>", line 163, in get_code
File "imputil.pyc", line 89, in _import_hook
File "imputil.pyc", line 152, in _determine_import_context
AttributeError: 'module' object has no attribute 'get'
Anyway, both of these programs play all sorts of tricks to
get importing to work, so I figure that interaction of the
two importing strategies is causing difficulty.
Following is the source code for the various files that
I'm using. Further, these files are packged up at
http://clarkevans.com/tmp/prob.tgz ; the build was performed
with ActiveState 2.2.1 on Win2K using py2exe 0.3.3 and
pyrex 0.4 compiled using mingw32, gcc 2.95.3-6
Any help would be very cool. My actual setup is much more
complicated than this, but I tried to make the smallest
example which exhibits the behavior. Also, I've had to
split "setup.py" into two distinct files since I didn't
know how to make them both work together.
Oh yes, both of these are *fantastic* tools. Thank you
for them and your help!
Clark
P.S. Great tools!
--------------- Makefile -------------------
test: clean dist
cd dist/test; ./test.exe
testmod.pyd: testmod.pyx
python build.py build_ext --inplace --compiler=mingw32
dist: test.py testmod.pyd
python dist.py py2exe
clean:
rm -f *.c *.o *.so *.cpp *.pyd *.pyc *~ *.bak core
rm -rf build dist
--------------- build.py -------------------
from distutils.core import setup
from distutils.extension import Extension
from Pyrex.Distutils import build_ext
setup(
name = 'testmod',
ext_modules=[
Extension("testmod",["testmod.pyx"]),
],
cmdclass = {'build_ext': build_ext}
)
--------------- dist.py --------------------
from distutils.core import setup
import py2exe
setup(
name = 'Demos',
scripts = ["test.py"],
)
--------------- testmod.pyx ----------------
import string
def go():
for x in string.split("one,two",","):
print x
--------------- test.py --------------------
from testmod import go
go()
--------------- OUTPUT ---------------------
$ make test
python build.py build_ext --inplace --compiler=mingw32
running build_ext
building 'testmod' extension
testmod.cpp: In function `struct PyObject * __pyx_f_go(PyObject *, PyObject *, PyObject *)':
testmod.cpp:73: warning: label `__pyx_L3' defined but not used
testmod.cpp:59: warning: label `__pyx_L2' defined but not used
creating build
creating build\temp.win32-2.2
creating build\temp.win32-2.2\Release
C:\msys\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -Ic:\Python22\include -c testmod.cpp -o build\temp.win32-2.2\Release\testmod.o
writing build\temp.win32-2.2\Release\testmod.def
C:\msys\mingw\bin\gcc.exe -mno-cygwin -mdll -static -s build\temp.win32-2.2\Release\testmod.o build\temp.win32-2.2\Release\testmod.def -Lc:\Python22\libs -lpython22 -o testmod.pyd
python dist.py py2exe
running py2exe
running build
running build_scripts
creating build\scripts-2.2
copying test.py -> build\scripts-2.2
running install_scripts
creating build\bdist.win32
creating build\bdist.win32\winexe
creating build\bdist.win32\winexe\lib
creating build\bdist.win32\winexe\lib\Python22
creating build\bdist.win32\winexe\lib\Python22\Scripts
copying build\scripts-2.2\test.py -> build\bdist.win32\winexe\lib\Python22\Scripts
creating dist
+----------------------------------------------------
| Processing script test.py with py2exe-0.3.3
+----------------------------------------------------
creating dist\test
creating build\bdist.win32\winexe\collect
creating build\bdist.win32\winexe\collect\test
creating build\bdist.win32\winexe\collect\test\Scripts.py2exe
Searching modules needed to run 'test.py' on path:
['c:\\cygwin\\home\\cce\\prob\\build\\bdist.win32\\winexe\\lib\\Python22\\Lib\\site-packages', '', 'C:\\Python22\\Lib\\site-packages\\Pythonwin', 'C:\\Python22\\Lib\\site-packages\\win32', 'C:\\Python22\\Lib\\site-packages\\win32\\lib', 'C:\\Python22\\Lib\\site-packages', 'c:\\Python22\\DLLs', 'c:\\Python22\\lib', 'c:\\Python22\\lib\\lib-tk', 'c:\\Python22']
force_imports =
Resolving binary dependencies:
C:\WINNT\System32\python22.dll
testmod.pyd
C:\Python22\Lib\site-packages\py2exe\run.exe
ext_mapping = {
'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))
}
copying C:\Python22\Lib\site-packages\py2exe\support.py -> build\bdist.win32\winexe\collect\test\Scripts.py2exe
byte-compiling c:\Python22\lib\imputil.py to imputil.pyc
copying test.py -> build\bdist.win32\winexe\collect\test\Scripts.py2exe\__main__.py
changing into 'build\bdist.win32\winexe\collect\test'
c:\cygwin\bin\zip.exe -rq c:\cygwin\home\cce\prob\build\bdist.win32\winexe\test.zip .
changing back to 'c:\cygwin\home\cce\prob'
creating dist\test\test.exe
copying C:\WINNT\System32\python22.dll -> dist\test
copying testmod.pyd -> dist\test
removing 'build\bdist.win32\winexe\collect\test' (and everything under it)
Built File dist\test\test.exe
removing 'build\bdist.win32\winexe' (and everything under it)
warning: py2exe: could not parse version number ''
No VersionInfo will be created
cd dist/test; ./test.exe
Traceback (most recent call last):
File "<string>", line 1, in ?
File "imputil.pyc", line 103, in _import_hook
File "<string>", line 52, in _import_top_module
File "imputil.pyc", line 216, in import_top
File "imputil.pyc", line 267, in _import_one
File "<string>", line 163, in get_code
File "imputil.pyc", line 89, in _import_hook
File "imputil.pyc", line 152, in _determine_import_context
AttributeError: 'module' object has no attribute 'get'
--
Clark C. Evans Axista, Inc.
http://www.axista.com 800.926.5525
XCOLLA Collaborative Project Management Software
Greg Ewing
> of pyrex and py2exe; the problem occurs when the module
> compiled with pyrex has an "import".
I'll look into this if I can, but I don't know if
there's much I can do, as I don't have easy access
to a Windows machine to debug this sort of thing on.
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury, | A citizen of NewZealandCorp, a |
Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. |
gr...@cosc.canterbury.ac.nz +--------------------------------------+
Clark C . Evans
| > Hello. I'm having difficulty with using a combination
| > of pyrex and py2exe; the problem occurs when the module
| > compiled with pyrex has an "import".
|
| I'll look into this if I can, but I don't know if
| there's much I can do, as I don't have easy access
| to a Windows machine to debug this sort of thing on.
I've tracked it down a bit further. The _import_one is being
called on 'testmod', which is the binary module compiled by
pyrex. Then, in some override code for "get_code" which appears
to be provided as a dynamically generated string, it calls
_import_hook on the following items: imp, marshal, sys, and
then finally string. In the cases of imp, marshal, and sys
the 'globals' parameter of _import_hook is a mapping. But
then somehow, when it goes to import string, globals is
actually a module object instead of a mapping.
So, somehow the get_code operator provided somehow is
passing a module to _import_hook instead of a mapping...
Traceback (most recent call last):
File "<string>", line 1, in ?
File "imputil.pyc", line 103, in _import_hook
File "<string>", line 52, in _import_top_module
File "imputil.pyc", line 216, in import_top
File "imputil.pyc", line 267, in _import_one
File "<string>", line 163, in get_code
File "imputil.pyc", line 89, in _import_hook
File "imputil.pyc", line 152, in _determine_import_context
AttributeError: 'module' object has no attribute 'get'
Here is my trace that I generated...
imputil.py#085 IN _import_hook(fqname='testmod',globals={'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None},locals={'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None},fromlist=('go',))
imputil.py#088 _determine_import_context(globals={'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None})
imputil.py#151 {'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None}.get('__importer__')
imputil.py#102 _import_top_module('testmod')
imputil.py#215 _import_one(None,'testmod','testmod')
imputil.py$266 get_code(None,'testmod','testmod')
imputil.py#215 _import_one(None,'testmod','testmod')
imputil.py$266 get_code(None,'testmod','testmod')
imputil.py#085 IN _import_hook(fqname='imp',globals={'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None},locals=None,fromlist=None)
imputil.py#088 _determine_import_context(globals={'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None})
imputil.py#151 {'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None}.get('__importer__')
imputil.py#085 IN _import_hook(fqname='marshal',globals={'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None},locals=None,fromlist=None)
imputil.py#088 _determine_import_context(globals={'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None})
imputil.py#151 {'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None}.get('__importer__')
imputil.py#085 IN _import_hook(fqname='sys',globals={'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None},locals=None,fromlist=None)
imputil.py#088 _determine_import_context(globals={'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None})
imputil.py#151 {'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None}.get('__importer__')
imputil.py#085 IN _import_hook(fqname='string',globals=<module 'testmod' (built-in)>,locals=None,fromlist=[])
imputil.py#088 _determine_import_context(globals=<module 'testmod' (built-in)>)
imputil.py#151 <module 'testmod' (built-in)>.get('__importer__')
Clark C . Evans
during a call to imp.load_module on line 163 of support.py of py2exe
The tail of my trace now looks like...
imputil.py#151 {'__builtins__': <module '__builtin__' (built-in)>, '_extensions_mapping': {'testmod': ('testmod.pyd', ('.pyd', 'rb', 3))}, '__name__': '__main__', '_ModuleType': <type 'module'>, '__doc__': None}.get('__importer__')
AT support.py#160
CALLING imp.load_module(fqname='testmod', fp=<open file 'c:\cygwin\home\cce\prob\dist\test\testmod.pyd', mode 'rb' at 0x007E9390>, pathname='c:\\cygwin\\home\\cce\\prob\\dist\\test\\testmod.pyd', desc=('.pyd', 'rb', 3))
imputil.py#085 IN _import_hook(fqname='string',globals=<module 'testmod' (built-in)>,locals=None,fromlist=[])
imputil.py#088 _determine_import_context(globals=<module 'testmod' (built-in)>)
imputil.py#151 <module 'testmod' (built-in)>.get('__importer__')
Traceback (most recent call last):
File "<string>", line 1, in ?
File "imputil.pyc", line 103, in _import_hook
File "<string>", line 52, in _import_top_module
File "imputil.pyc", line 216, in import_top
File "imputil.pyc", line 267, in _import_one
File "<string>", line 164, in get_code
File "imputil.pyc", line 89, in _import_hook
File "imputil.pyc", line 152, in _determine_import_context
AttributeError: 'module' object has no attribute 'get'
...
Now, I just tried imp.load_module directly, and it worked. So,
somehow one of those other "hooks" is firing and doing something
strange... I don't know where to put trace code to peek
ActivePython 2.2.1 Build 222 (ActiveState Corp.) based on
Python 2.2.1 (#34, Apr 15 2002, 09:51:39) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import imp
>>> name = "c:\\cygwin\\home\\cce\\prob\\dist\\test\\testmod.pyd"
>>> file = open(fn,'rb')
>>> imp.load_module('testmod',file,name,('.pyd','rb',3))
<module 'testmod' from 'c:\cygwin\home\cce\prob\dist\test\testmod.pyd'>
>>> ^Z
So. I just don't know where else to look at this point.
Here's hoping!
Clark
| --
| http://mail.python.org/mailman/listinfo/python-list
Greg Ewing
> module compiled by pyrex.
What puzzles me most about this is that there's no such
thing as a "binary module" -- Pyrex modules are perfectly
ordinary module objects, the same as the interpreter uses.
I have a feeling this may have something to do with the
"current module". When I was figuring out how to implement
the import statement, I did some poking about to find out
how the interpreter does it. I can't remember all the
details, but I think I found that I couldn't emulate it
exactly, because it made use of a notion of a "current
module" that only applies when interpreted code is being
executed.
Maybe the import hooks are using this to find out what
package the importing module belongs to? If that's the
case, then it's probably going to get whatever was the
last interpreted module that a function was executed
from. Now, if the Pyrex module is being invoked as the
main module by py2exe without going through an
interpreted module first... maybe that's what the
root cause of the problem is?
I'm just guessing wildly here, and may be completely
wrong.
Thomas Heller
> Hello. I'm having difficulty with using a combination
> of pyrex and py2exe; the problem occurs when the module
> works just fine until the calling script is packaged
> up with py2exe, then you get the following:
>
> File "<string>", line 1, in ?
> File "imputil.pyc", line 103, in _import_hook
> File "<string>", line 52, in _import_top_module
> File "imputil.pyc", line 216, in import_top
> File "imputil.pyc", line 267, in _import_one
> File "<string>", line 163, in get_code
> File "imputil.pyc", line 89, in _import_hook
> File "imputil.pyc", line 152, in _determine_import_context
> AttributeError: 'module' object has no attribute 'get'
>
> get importing to work, so I figure that interaction of the
> two importing strategies is causing difficulty.
>
> Following is the source code for the various files that
> I'm using. Further, these files are packged up at
> http://clarkevans.com/tmp/prob.tgz ; the build was performed
> with ActiveState 2.2.1 on Win2K using py2exe 0.3.3 and
> pyrex 0.4 compiled using mingw32, gcc 2.95.3-6
>
> Any help would be very cool. My actual setup is much more
> complicated than this, but I tried to make the smallest
> example which exhibits the behavior. Also, I've had to
> split "setup.py" into two distinct files since I didn't
> know how to make them both work together.
-----
from distutils.core import setup, Extension
import py2exe
from Pyrex.Distutils import build_ext
setup(
name = "Demos",
scripts=["test.py"],
ext_modules=[Extension("testmod", ["testmod.pyx"])],
cmdclass = {"build_ext": build_ext}
)
-----
but now to the more interesting questions:
It seems to be a problem (as you also have discovered in later posts)
in the way Pyrex does the importing.
A nice way to debug this (which also works without py2exe, and
without windows) is to check it with using imputil's own import
hook in the interactive Python interpreter:
C:\evans\prob>py22
Python 2.2.1 (#34, Apr 9 2002, 19:34:33) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import imputil
>>> imputil._test_revamp() # install the hook
>>> import testmod
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\python22\lib\imputil.py", line 103, in _import_hook
top_module = self._import_top_module(parts[0])
File "C:\python22\lib\imputil.py", line 187, in _import_top_module
module = self.fs_imp.import_from_dir(item, name)
File "C:\python22\lib\imputil.py", line 536, in import_from_dir
result = self._import_pathname(_os_path_join(dir, fqname), fqname)
File "C:\python22\lib\imputil.py", line 569, in _import_pathname
return importFunc(filename, finfo, fqname)
File "C:\python22\lib\imputil.py", line 602, in import_file
module = imp.load_module(fqname, fp, filename, self.desc)
File "C:\python22\lib\imputil.py", line 89, in _import_hook
parent = self._determine_import_context(globals)
File "C:\python22\lib\imputil.py", line 152, in _determine_import_context
if not globals or not globals.get('__importer__'):
AttributeError: 'module' object has no attribute 'get'
>>> import pdb
>>> pdb.pm()
> c:\python22\lib\imputil.py(152)_determine_import_context()
-> if not globals or not globals.get('__importer__'):
(Pdb) l
147 will be looked for within that package. The context could also be None,
148 meaning there is no context -- the module should be looked for as a
149 "top-level" module.
150 """
151
152 -> if not globals or not globals.get('__importer__'):
153 # globals does not refer to one of our modules or packages. That
154 # implies there is no relative import context (as far as we are
155 # concerned), and it should just pick it off the standard path.
156 return None
157
(Pdb) print globals
<module 'testmod' (built-in)>
(Pdb)
So we see 'globals' is a module, not a dictionary (as it should be).
IMO Pyrex should call PyImport_ImportModuleEx(), which AFAIK uses the import hook,
and is also easier to use than to retrieving the __import__ from builtins,
and it should probably call it with the result from PyEval_GetGlobals(),
but I'll leave this to Greg.
HTH,
Thomas