A hack to let Pyrex 0.3.x users build extension classes on Win32
Graham Fawcett
I downloaded and am using version 0.3.2 of the fantastic Pyrex
extension language (
http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ ) and am happily
speeding up some crucial code using this marvellous product.
However, I ran into a hitch when trying to build an extension class on
my Win32 machine: the compiler complained with "initializer not a
constant" errors. This is a known issue in Python; see the FAQ at
http://www.python.org/cgi-bin/faqw.py?req=show&file=faq03.024.htp .
So I did a very ugly little hack on the pyrexc.py script to fix the
problem. (The hacked script just runs pyrexc as usual, but then
traverses each C file that pyrexc builds and applies the advice given
in the Python FAQ.
This is a brute-force and brittle approach. I expect it will break
against a release of Pyrex > 0.3.2. (I should have taken the time to
read Greg's compiler code and fix it there, but I've a temporary
shortage of Copious Free Time.)
Here's the revised script.
#!/usr/bin/env python
#
# Pyrex -- Main Program, Unix
#
import Pyrex.Compiler.Main
Pyrex.Compiler.Main.main(c_only = 1, use_listing_file = 0)
# ---- added by Graham Fawcett <faw...@uwindsor.ca>
# ---- fixes "Initializer not a constant" while building DLL on
MS-Windows
# ---- http://www.python.org/cgi-bin/faqw.py?req=show&file=faq03.024.htp
HEAD_INIT = 'PyObject_HEAD_INIT(0)'
TYPESTART = 'statichere PyTypeObject __pyx_type_'
GETATTR = 'PyObject_GenericGetAttr, /*tp_getattro*/'
SETATTR = 'PyObject_GenericSetAttr, /*tp_setattro*/'
ALLOC = 'PyType_GenericAlloc, /*tp_alloc*/'
FREE = '_PyObject_Del, /*tp_free*/'
TYPEINIT ='''__pyx_type_%(typename)s.ob_type = &PyType_Type;
__pyx_type_%(typename)s.tp_getattro = PyObject_GenericGetAttr;
/*tp_getattro*/
__pyx_type_%(typename)s.tp_setattro = PyObject_GenericSetAttr;
/*tp_setattro*/
__pyx_type_%(typename)s.tp_alloc = PyType_GenericAlloc; /*tp_alloc*/
__pyx_type_%(typename)s.tp_free = _PyObject_Del; /*tp_free*/
'''
def fix_c_file(pyx_filename):
filename = pyx_filename.split('.')[0] + '.c'
f = open(filename, 'r')
lines = f.readlines()
f.close()
lines = [line.rstrip() for line in lines]
types = []
f = open(filename, 'w')
for line in lines:
# strip the injurious assignments from the PyTypeObject def'n
if line.startswith(HEAD_INIT):
print >> f, 'PyObject_HEAD_INIT(NULL)'
elif line.startswith(TYPESTART):
typename = line[len(TYPESTART):].split(' ')[0]
types.append(typename)
print >> f, line
elif line.startswith(GETATTR):
print >> f, 'NULL,' + GETATTR.split(',')[1]
elif line.startswith(SETATTR):
print >> f, 'NULL,' + SETATTR.split(',')[1]
elif line.startswith(ALLOC):
print >> f, 'NULL,' + ALLOC.split(',')[1]
elif line.startswith(FREE):
print >> f, 'NULL,' + FREE.split(',')[1]
elif line.startswith('void init') and line.endswith('(void)
{'):
print >> f, line
# put the assignments in the module init function
for typename in types:
print >> f, TYPEINIT % locals()
else:
pass
print >> f, line
f.close()
import sys
if sys.platform == 'win32':
for pyx_filename in sys.argv[1:]:
fix_c_file(pyx_filename)
# ---- end of addition
Best wishes,
-- Graham
Greg Ewing
> I ran into a hitch when trying to build an extension class on
> my Win32 machine: the compiler complained with "initializer not a
> constant" errors. This is a known issue in Python; see the FAQ at
> http://www.python.org/cgi-bin/faqw.py?req=show&file=faq03.024.htp .
Thanks for pointing this problem out. I'll see if I can
do something about this in a future release.
--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
Graham Fawcett
> Graham Fawcett wrote:
>
> Thanks for pointing this problem out. I'll see if I can
> do something about this in a future release.
My pleasure, Greg.
I thought it might be helpful to summarize the installation of Pyrex
on Win32 for those who are interested in using Pyrex, but are
discouraged by the setup that may be required. I've put up some notes
at
http://cfl-x.uwindsor.ca/graham/pyrex/
that should make it a snap.
Thanks again for this great product!
-- Graham
David LeBlanc
GCC_BIN_DIR = 'd:/visstudio/vc98/bin'
PYREX_DIR = 'j:/python22/apps/Pyrex0.3.3'
PYTHON_VERSION = 'python22'
PYTHON_DIR = 'j:/' + PYTHON_VERSION
.
.
.
def compile(name):
print '\ncompile'
cmd('cl -I%s/include -c %s.c -o %s.o' % (PYTHON_DIR, name, name))
.
.
.
def wrap(name):
print '\nwrap'
cmd('link /DLL /SUBSYSTEM:console /DEF:%s.def /OUT:%s.pyd /LIBPATH:%s/libs
%s.lib %s.obj' %
(name, name, PYTHON_DIR, PYTHON_VERSION, name))
Obviously, you'll need to change the path definitions to suit your
particular installation.
I did not encounter the initializer problem described by Greg Fawcett. This
may differ if you do more then compile the supplied demos, which is all i've
done so far.
David LeBlanc
Seattle, WA USA
Matthias Baas
wrote:
>Making pyrexc use native (Visual C) compiler:
I think the easiest way to compile Pyrex modules (either on Windows,
Linux or anything else) is by using the distutils. For example, to
compile the file shrubbery.pyx you just need the following setup.py
script (which assumes you have "pyrexc" somewhere in your path):
#########################################
from distutils.core import setup, Extension
import os
os.system("pyrexc shrubbery.pyx")
setup(name="shrubbery",
ext_modules=[Extension("shrubbery", ["shrubbery.c"])])
#########################################
Now you can compile the module with "setup.py build" or "setup.py
install" and then the distutils take care of the build process.
The nice thing is, that you don't have to set any paths (unless you
need custom libraries/header files) and that it works on any platform.
However, I don't know if the distutils can also be used for other
compilers than Visual C++ on Windows.
- Matthias -
Alex Martelli
...
> However, I don't know if the distutils can also be used for other
> compilers than Visual C++ on Windows.
Yes, with some preliminary work -- see (single long URL, may be split):
http://www.python.org/dev/doc/devel/inst/tweak-flags.html#SECTION000620000000000000000
This covers Borland and GNU compilers (both cygwin and mingw32) and
summarily mentions the preliminary work you have to perform, too.
Alex
Graham Fawcett
> I think the easiest way to compile Pyrex modules (either on Windows,
> Linux or anything else) is by using the distutils. For example, to
> compile the file shrubbery.pyx you just need the following setup.py
> script (which assumes you have "pyrexc" somewhere in your path):
You're right, of course, Matthias. It didn't occur to me that I was
replacing distutils functionality with my little script. Pyrex the
language is so deceptively similar to Python -- it "feels" more like
.py-->.pyc compilation than "real" extension programming. I'll revise
the suggestions I posted on my Web site.
I suppose that one could add a distutils.command.pyrexc as a
subcommand of "build_ext" to handle the .pyx compilation. That way the
os.system() call in the build script could be avoided. I've never
written a distutils command, but perhaps I'll give that a try.
Regards,
-- Graham
Greg Ewing
> I suppose that one could add a distutils.command.pyrexc as a
> subcommand of "build_ext" to handle the .pyx compilation. That way the
> os.system() call in the build script could be avoided. I've never
> written a distutils command, but perhaps I'll give that a try.
I've been too busy fixing bugs to look into this
myself, but that definitely seems to be the way
to go. If anyone wants to have a go and contribute
the result, it'll be much appreciated!
Graham Fawcett
> I suppose that one could add a distutils.command.pyrexc as a
> subcommand of "build_ext" to handle the .pyx compilation. That way the
> os.system() call in the build script could be avoided. I've never
> written a distutils command, but perhaps I'll give that a try.
Okay, I wrote the distutils command. Actually I just added a
subpackage to Pyrex that you can import into your setup script:
from distutils.core import setup
from distutils.extension import Extension
import Pyrex.Distutils
setup(name='foo', ext_modules=[Extension("foo", ["foo.pyx"])])
Note that the Extension() specifies a .pyx file, not a .c file. You
can specify .c files as well, but of course Pyrex will ignore them.
Pyrex.Distuils (my subpackage) is a build_ext replacement -- it
import-hacks itself into the distutils.command package, so that it
gets a first shot at any build_ext calls. Once it's finished, it
returns control to the "real" build_ext.
I tested on Win32 only, with mingw32 and msvc compilers, but it should
run fine on other platforms.
I've put a modified distribution of Pyrex 0.3.3 at
http://cfl-x.uwindsor.ca/graham/pyrex/ if anyone wants to try it
out. Here are the two key source files:
###########################################
# Pyrex.Distutils.__init__.py
# July 2002, Graham Fawcett
#
# this hack was inspired by the way Thomas Heller got py2exe
# to appear as a distutil command
#
# we replace distutils.command.build_ext with our own version
# and keep the old one under the module name _build_ext,
# so that *our* build_ext can make use of it.
import sys
import distutils.command.build_ext
sys.modules['_build_ext'] = distutils.command.build_ext
import build_ext
sys.modules['distutils.command.build_ext']=build_ext
###########################################
# Pyrex.Distutils.build_ext.py
"""
Subclasses disutils.command.build_ext,
replacing it with a Pyrex version that compiles pyx->c
before calling the original build_ext command.
"""
# July 2002, Graham Fawcett
# Pyrex is (c) Greg Ewing.
import Pyrex.Compiler.Main
from Pyrex.Compiler.Errors import PyrexError
from distutils.dep_util import newer
import os
import sys
# in Distutils.__init__, the original build_ext
# is inserted into sys.modules as _build_ext
import _build_ext
class build_ext (_build_ext.build_ext):
description = "compile Pyrex scripts, then build C/C++ extensions
(compile/link to build directory)"
def run (self):
if not self.extensions:
return
# collect the names of the source (.pyx) files
pyx_sources = []
for e in self.extensions:
pyx_sources.extend(e.sources)
pyx_sources = [source for source in pyx_sources if
source.endswith('.pyx')]
for pyx in pyx_sources:
# should I raise an exception if it doesn't exist?
if os.path.exists(pyx):
source = pyx
target = source.replace('.pyx', '.c')
if newer(source, target) or self.force:
self.pyrex_compile(source)
# compiling with mingw32 gets an "initializer not
a constant" error
#
http://www.python.org/cgi-bin/faqw.py?req=show&file=faq03.024.htp
# doesn't appear to happen with MSVC
# so if we are compiling with mingw32, massage the
Pyrex-generated
# C files to compile properly
if self.compiler=='mingw32':
self.fix_c_file(target)
# correct the extension sourcefile names in setup script
# so that .pyx files are refered to as .c files
# then we can call the "real" build_ext on them.
for e in self.extensions:
e.sources = [src.replace('.pyx', '.c') for src in
e.sources]
# run the original build_ext.
_build_ext.build_ext.run(self)
def pyrex_compile(self, source):
try:
Pyrex.Compiler.Main.compile(source, c_only=1,
use_listing_file=0)
except PyrexError, e:
print e
sys.exit(1)
def fix_c_file(self, filename):
print 'fixing ' + filename
HEAD_INIT = 'PyObject_HEAD_INIT(0)'
TYPESTART = 'statichere PyTypeObject __pyx_type_'
GETATTR = 'PyObject_GenericGetAttr, /*tp_getattro*/'
SETATTR = 'PyObject_GenericSetAttr, /*tp_setattro*/'
ALLOC = 'PyType_GenericAlloc, /*tp_alloc*/'
FREE = '_PyObject_Del, /*tp_free*/'
TYPEINIT ='''__pyx_type_%(typename)s.ob_type = &PyType_Type;
__pyx_type_%(typename)s.tp_getattro = PyObject_GenericGetAttr;
/*tp_getattro*/
__pyx_type_%(typename)s.tp_setattro = PyObject_GenericSetAttr;
/*tp_setattro*/
__pyx_type_%(typename)s.tp_alloc = PyType_GenericAlloc;
/*tp_alloc*/
__pyx_type_%(typename)s.tp_free = _PyObject_Del; /*tp_free*/
'''
f = open(filename, 'r')
lines = f.readlines()
f.close()
lines = [line.rstrip() for line in lines]
types = []
f = open(filename, 'w')
for line in lines:
if line.startswith(HEAD_INIT):
print >> f, 'PyObject_HEAD_INIT(NULL)'
elif line.startswith(TYPESTART):
typename = line[len(TYPESTART):].split(' ')[0]
types.append(typename)
print >> f, line
elif line.startswith(GETATTR):
print >> f, 'NULL,' + GETATTR.split(',')[1]
elif line.startswith(SETATTR):
print >> f, 'NULL,' + SETATTR.split(',')[1]
elif line.startswith(ALLOC):
print >> f, 'NULL,' + ALLOC.split(',')[1]
elif line.startswith(FREE):
print >> f, 'NULL,' + FREE.split(',')[1]
elif line.startswith('void init') and
line.endswith('(void) {'):
print >> f, line
for typename in types:
print >> f, TYPEINIT % locals()
else:
pass
print >> f, line
f.close()
###########################################
This was fun to do! I hope that it will be useful to others. Having
read through the Distutils source, I have a new and profound respect
for its authors.
Regards,
-- Graham
Thomas Heller
>
> Okay, I wrote the distutils command. Actually I just added a
> subpackage to Pyrex that you can import into your setup script:
>
> from distutils.core import setup
> from distutils.extension import Extension
> import Pyrex.Distutils
>
> setup(name='foo', ext_modules=[Extension("foo", ["foo.pyx"])])
>
> Note that the Extension() specifies a .pyx file, not a .c file. You
> can specify .c files as well, but of course Pyrex will ignore them.
>
> Pyrex.Distuils (my subpackage) is a build_ext replacement -- it
> import-hacks itself into the distutils.command package, so that it
> gets a first shot at any build_ext calls. Once it's finished, it
> returns control to the "real" build_ext.
Some notes, just to spread some distutils knowledge:
1. The 'official' way to replace a standard distutils command
class is done in the following way: Derive your class, say
'my_build_ext', from the standard distutils command:
class my_build_ext(build_ext):
....
and then pass a cmdclass parameter to the setup command:
setup(...
cmdclass = {'build_ext': my_build_ext},
...)
2. Looking at distutils command.build_ext source, there is already
a method which 'precompiles' source files into .c files, it's the
swig_sources() method. Sound a little hackish, but probably we could
get away with just overriding this method in the my_build_ext class.
Maybe something like the swig_sources() command should be promoted
into an 'official' hook for precompilation?
Thomas
Thomas Heller
> a constant" error
> #
> http://www.python.org/cgi-bin/faqw.py?req=show&file=faq03.024.htp
> # doesn't appear to happen with MSVC
> # so if we are compiling with mingw32, massage the
> Pyrex-generated
> # C files to compile properly
>
Isn't this problem (initializer not a constant) solved by compiling
the file in C++ mode (with .cpp extension)?
Thomas
Graham Fawcett
> Graham Fawcett wrote:
>
> > I suppose that one could add a distutils.command.pyrexc as a
> > subcommand of "build_ext" to handle the .pyx compilation. That way the
> > os.system() call in the build script could be avoided. I've never
> > written a distutils command, but perhaps I'll give that a try.
>
>
> I've been too busy fixing bugs to look into this
> myself, but that definitely seems to be the way
> to go. If anyone wants to have a go and contribute
> the result, it'll be much appreciated!
I've posted a first-blush version to the list, under the subject
"Pyrex and Distuils: an enhanced build_ext command".
Cheers,
-- Graham
Graham Fawcett
>
> 1. The 'official' way to replace a standard distutils command
> class is done in the following way: Derive your class, say
> 'my_build_ext', from the standard distutils command:
>
> class my_build_ext(build_ext):
> ....
>
> and then pass a cmdclass parameter to the setup command:
> setup(...
> cmdclass = {'build_ext': my_build_ext},
> ...)
Ironically, Thomas, you inspired the approach that I took -- I
borrowed it from py2exe. ;-) But, all things being equal, "simple and
explicit" always trumps "import hack". Thanks, I'll take this into
account.
> 2. Looking at distutils command.build_ext source, there is already
> a method which 'precompiles' source files into .c files, it's the
> swig_sources() method. Sound a little hackish, but probably we could
> get away with just overriding this method in the my_build_ext class.
> Maybe something like the swig_sources() command should be promoted
> into an 'official' hook for precompilation?
When I subclassed build_ext, I just cut out the swig_sources() method,
assuming it was SWIG-specific. But I see the analogy and agree that
this would be a clean place to put the "pyrexc" code.
Thanks, Thomas,
-- Graham
Graham Fawcett
>
> class is done in the following way: Derive your class, say
> 'my_build_ext', from the standard distutils command:
>
> class my_build_ext(build_ext):
> ....
>
> and then pass a cmdclass parameter to the setup command:
> setup(...
> cmdclass = {'build_ext': my_build_ext},
> ...)
Ironically, Thomas, you inspired the approach that I took -- I
borrowed it from py2exe. ;-) But, all things being equal, "simple and
explicit" always trumps "import hack". Thanks, I'll take this into
account.
> 2. Looking at distutils command.build_ext source, there is already
> a method which 'precompiles' source files into .c files, it's the
> swig_sources() method. Sound a little hackish, but probably we could
> get away with just overriding this method in the my_build_ext class.
> Maybe something like the swig_sources() command should be promoted
> into an 'official' hook for precompilation?
When I subclassed build_ext, I just cut out the swig_sources() method,
Graham Fawcett
Yes -- at first I couldn't get c++ compilation to work, so I tried the
other approach suggested in the FAQ.
I just spent some time trying again, and I think I've got the c++
approach working. I still need to massage the Pyrex-generated C files,
but the code is much cleaner.
I also saw your suggestions re: adding command classes using the
canoncial approach, and the extension point at swig_sources(), and
have simplified my code accordingly.
Here are the revised files:
#######################################
#Pyrex.Distutils.__init__.py
from build_ext import build_ext
#######################################
#Pyrex.Distutils.build_ext.py
#
# Subclasses disutils.command.build_ext,
# replacing it with a Pyrex version that compiles pyx->c
# before calling the original build_ext command.
# July 2002, Graham Fawcett
# Pyrex is (c) Greg Ewing.
import distutils.command.build_ext
import Pyrex.Compiler.Main
from Pyrex.Compiler.Errors import PyrexError
from distutils.dep_util import newer
import os
import sys
class build_ext (distutils.command.build_ext.build_ext):
description = "compile Pyrex scripts, then build C/C++ extensions
(compile/link to build directory)"
def finalize_options (self):
distutils.command.build_ext.build_ext.finalize_options(self)
# compiling with mingw32 gets an "initializer not a constant"
error
# doesn't appear to happen with MSVC!
# so if we are compiling with mingw32,
# switch to C++ mode, to avoid the problem
if self.compiler == 'mingw32':
self.swig_cpp = 1
def swig_sources (self, sources):
if not self.extensions:
return
# collect the names of the source (.pyx) files
pyx_sources = []
pyx_sources = [source for source in sources if
source.endswith('.pyx')]
extension = self.swig_cpp and '.cpp' or '.c'
for pyx in pyx_sources:
# should I raise an exception if it doesn't exist?
if os.path.exists(pyx):
source = pyx
target = source.replace('.pyx', extension)
if newer(source, target) or self.force:
self.pyrex_compile(source)
if self.swig_cpp:
# rename .c to .cpp (Pyrex always builds .c
...)
if os.path.exists(target):
os.unlink(target)
os.rename(source.replace('.pyx', '.c'),
target)
# massage the cpp file
self.c_to_cpp(target)
return [src.replace('.pyx', extension) for src in pyx_sources]
def pyrex_compile(self, source):
try:
Pyrex.Compiler.Main.compile(source, c_only=1,
use_listing_file=0)
except PyrexError, e:
print e
sys.exit(1)
def c_to_cpp(self, filename):
"""touch up the Pyrex generated c/cpp files to meet
mingw32/distutils requirements."""
f = open(filename, 'r')
lines = [line for line in f.readlines() if not
line.startswith('staticforward PyTypeObject __pyx_type_')]
f.close()
f = open(filename, 'w')
lines.insert(1, 'extern "C" {\n')
lines.append('}\n')
f.write(''.join(lines))
f.close()
#######################################
# a sample setup.py
from distutils.core import setup
from distutils.extension import Extension
from Pyrex.Distutils import build_ext
setup(name='foo',
ext_modules=[ Extension("foo", ["foo.pyx"]) ],
cmdclass = {'build_ext': build_ext}
)
#######################################
Tested on Win32, msvc and mingw32.
I'm happy so far -- the revision is shorter, cleaner and more
explicit.
Comments are welcome!
Cheers,
-- Graham