How to build extensions on Windows?

[Kevin D. Smith]

I've written a simple Python extension for UNIX, but I need to get it
working on Windows now. I'm having some difficulties figuring out how
to do this. I've seen web pages that say that MS Visual Studio is
required, and other that say that's not true, that MinGW will work.
Then there is Mike Fletcher's web page
(http://www.vrplumber.com/programming/mstoolkit/) that describes in
detail how to build extensions, but most of the links to external
software are no longer valid. I think it's safe to say that I am
completely lost, as there appears to be no authoritative, up-to-date
description on how to make this work.

--
Kevin D. Smith

[Lawrence Oluyede]

Kevin D. Smith <Kevin...@sas.com> wrote:
> Then there is Mike Fletcher's web page
> (http://www.vrplumber.com/programming/mstoolkit/) that describes in
> detail how to build extensions, but most of the links to external
> software are no longer valid. I think it's safe to say that I am
> completely lost, as there appears to be no authoritative, up-to-date
> description on how to make this work.

I managed to set up a compilation toolchain in Windows following that
tutorial so what's your problem?

I installed MS .NET 1.1 and its SDK, the Platform SDK for Windows 2003
sever and the mstoolkit (you have to borrow it from somewhere because
it's not available anymore)
Then I hacked distutils and all worked well.

The only issue is to find the MS Toolkit 2003...

--
Lawrence - http://www.oluyede.org/blog
"Nothing is more dangerous than an idea
if it's the only one you have" - E. A. Chartier

[Sybren Stuvel]

Kevin D Smith enlightened us with:

> I've written a simple Python extension for UNIX, but I need to get
> it working on Windows now. I'm having some difficulties figuring
> out how to do this.

I had to do the same, and I didn't get much result. My solution:
install Cygwin, use the Python that comes with that, and use gcc just
like you're used to. Works like a charm, but the compiled extension is
incompatible with the regular Windows Pythons from python.org and
ActiveState.

Sybren
--
Sybren Stüvel
Stüvel IT - http://www.stuvel.eu/

[Kevin D. Smith]

On 2006-09-07 09:28:42 -0400, rhy...@myself.com (Lawrence Oluyede) said:

> Kevin D. Smith <Kevin...@sas.com> wrote:
>> Then there is Mike Fletcher's web page
>> (http://www.vrplumber.com/programming/mstoolkit/) that describes in
>> detail how to build extensions, but most of the links to external
>> software are no longer valid. I think it's safe to say that I am
>> completely lost, as there appears to be no authoritative, up-to-date
>> description on how to make this work.
>
> I managed to set up a compilation toolchain in Windows following that
> tutorial so what's your problem?
>
> I installed MS .NET 1.1 and its SDK, the Platform SDK for Windows 2003
> sever and the mstoolkit (you have to borrow it from somewhere because
> it's not available anymore)
> Then I hacked distutils and all worked well.
>
> The only issue is to find the MS Toolkit 2003...

So in other words, what you're saying is that the only issue I have
left is the exact issue that I described in my initial post that you
claimed isn't a problem... Great! I guess I'l get right down to work
compiling that extension now.

--
Kevin D. Smith

[Lawrence Oluyede]

What I mean is that you have to find a way to get the toolkit. I don't
think MS will sue you if you borrow the compiler from a friend or
"download" it. Otherwise you can try with MingW I guess...

[Jason]

I don't know about MinGW, but you can get the Microsoft compilers by
installing Visual C++ 2005 Express. I'm guessing the old toolkit is
deprecated. While you must register each Visual Studio Express module
that you download, I don't think the actual command-line tools are
encumbered.

Why not try it out and let us know how it goes?

(Visual Studio 2005 Express:
http://msdn.microsoft.com/vstudio/express/)

--Jason

[Filip Wasilewski]

There is an easy way to build Python extensions on Windows with MinGW
and it works fine for me. Just follow these steps:

1. Get MinGW gcc and/or g++, preferably via MinGW installer from [1].
You may have to restart your computer or manually edit PATH system
environment variable to include MinGW's bin directory (default is
c:\mingw\bin). Then check if it is there by typing `path` in the cmd
window.

2. Get pexports-0.42h.zip from [2] and extract pexports.exe file

3. Prepare MinGW compatible .a library file
pexports.exe c:\WINDOWS\system32\python24.dll > python24.def
c:\mingw\bin\dlltool.exe --dllname python24.dll --def python24.def
--output-lib libpython24.a

4. Place the new libpython24.a file in the Python's libs directory (but
not in the Lib dir), default is c:\python24\libs

5. Build your extension by executing your setup script with
`--compiler=mingw32` parameter.
python setup.py build --compiler=mingw32

Additionally you may wish to put a distutils.cfg file in the
c:\python\lib\distutils dir containing following entries:
[build]
compiler = mingw32

This will tell the distutils script to use MinGW compiler by default
when executing `python setup.py build` command.

best,
fw

[1] http://sourceforge.net/projects/mingw/
[2] http://starship.python.net/crew/kernr/mingw32/pexports-0.42h.zip

[Lawrence Oluyede]

Jason <tenax....@gmail.com> wrote:
> I don't know about MinGW, but you can get the Microsoft compilers by
> installing Visual C++ 2005 Express. I'm guessing the old toolkit is
> deprecated. While you must register each Visual Studio Express module
> that you download, I don't think the actual command-line tools are
> encumbered.
>
> Why not try it out and let us know how it goes?
>
> (Visual Studio 2005 Express:
> http://msdn.microsoft.com/vstudio/express/)

Because you can't use it to distribute extensions. Python is compiled
against the 2003 version and distutils will simply don't let you compile
with the 2005 version. If you manage to hack distutils to use the 2005
anyway it won't work for sure outside your home box.

[cas...@comcast.net]

> 1. Get MinGW gcc and/or g++, preferably via MinGW installer from [1].
> You may have to restart your computer or manually edit PATH system
> environment variable to include MinGW's bin directory (default is
> c:\mingw\bin). Then check if it is there by typing `path` in the cmd
> window.

1a. [Optional] Install MSYS, also from [1]. If your extension relies on
another C library, this may make it easier to compile that library.

>
> 2. Get pexports-0.42h.zip from [2] and extract pexports.exe file
>
> 3. Prepare MinGW compatible .a library file
> pexports.exe c:\WINDOWS\system32\python24.dll > python24.def
> c:\mingw\bin\dlltool.exe --dllname python24.dll --def python24.def
> --output-lib libpython24.a
>
> 4. Place the new libpython24.a file in the Python's libs directory (but
> not in the Lib dir), default is c:\python24\libs

I believe steps 2 through 4 are only required for Python 2.3 or
earlier. I have not needed to do those steps for Python 2.4 and 2.5.

[Jarek Zgoda]

Filip Wasilewski napisał(a):

> There is an easy way to build Python extensions on Windows with MinGW
> and it works fine for me. Just follow these steps:

It was brougt to my attention that mingw-compiled extensions for Python
2.4 use other malloc() that Python 2.4, is it true? Can this have any
impact on stability?

--
Jarek Zgoda
http://jpa.berlios.de

[Filip Wasilewski]

Jarek Zgoda wrote:
> Filip Wasilewski napisa³(a):

>
> > There is an easy way to build Python extensions on Windows with MinGW
> > and it works fine for me. Just follow these steps:
>
> It was brougt to my attention that mingw-compiled extensions for Python
> 2.4 use other malloc() that Python 2.4, is it true? Can this have any
> impact on stability?

I have inspected the dll dependencies for python24.dll (ActiveState
build, but I think this is also true for regular one) and my custom
extensions built with MinGW and both of them use malloc() from
msvcr71.dll. I think that as long as the extensions are linked against
proper version of msvcr (7.1) there should not be such surprises.

Other thing is that Python uses it's own memory allocator [1] and to
avoid problems with memory corruption one should not try to allocate
memory with PyMem_Malloc() and free it with free() and so forth.

cheers,
fw

[1] http://www.python.org/doc/2.4.3/api/memoryOverview.html

["Martin v. Löwis"]

Kevin D. Smith schrieb:

> Then there is Mike Fletcher's web page
> (http://www.vrplumber.com/programming/mstoolkit/) that describes in
> detail how to build extensions, but most of the links to external
> software are no longer valid. I think it's safe to say that I am
> completely lost, as there appears to be no authoritative, up-to-date
> description on how to make this work.

If all else fails, buy a copy of Visual Studio 2003. They are available
fairly cheap at Ebay.

Regards,
Martin

[Kevin D. Smith]

On 2006-09-07 13:20:13 -0400, "Jason" <tenax....@gmail.com> said:
> I don't know about MinGW, but you can get the Microsoft compilers by
> installing Visual C++ 2005 Express. I'm guessing the old toolkit is
> deprecated. While you must register each Visual Studio Express module
> that you download, I don't think the actual command-line tools are
> encumbered.
>
> Why not try it out and let us know how it goes?
>
> (Visual Studio 2005 Express:
> http://msdn.microsoft.com/vstudio/express/)

This almost worked (at least, it appears to). I got the module to
build and install using VS 2005. It works fine if I run python from
the directory where I built the extension. However, if you go to any
other directory, run python, and try to import the module, I get the
following error:

ImportError: dynamic module does not define init function (initsasSQL)

Why would it work from one directory, but not another?

--
Kevin D. Smith

[Fredrik Lundh]

Kevin D.Smith wrote:

> This almost worked (at least, it appears to). I got the module to
> build and install using VS 2005. It works fine if I run python from
> the directory where I built the extension. However, if you go to any
> other directory, run python, and try to import the module, I get the
> following error:
>
> ImportError: dynamic module does not define init function (initsasSQL)
>
> Why would it work from one directory, but not another?

do you perhaps have a different "sassql.dll" file in your Python path?

(Python's standard import mechanism looks for PYD, DLL, PY, PYW, and
PYC, in that order.)

</F>

[John Machin]

Any support for the radical notion of the error message giving the full
path of the file that it's complaining about? If so, I'll lob in an
enhancement request in the morning ...

Cheers,
John

[Fredrik Lundh]

John Machin wrote:

> Any support for the radical notion of the error message giving the full
> path of the file that it's complaining about? If so, I'll lob in an
> enhancement request in the morning ...

there's always "python -vv", of course.

</F>

["Martin v. Löwis"]

John Machin schrieb:

> Any support for the radical notion of the error message giving the full
> path of the file that it's complaining about? If so, I'll lob in an
> enhancement request in the morning ...

A patch would be appreciated much more so.

Regards,
Martin

[John Machin]

Thank you; I wasn't aware of that. Given that (as at 2.4.3) searching
the .chm docs for "interpreter", "command", and "option" don't yield
any pointers to the interpreter command line options, and given that
python -h gives no clue:
"""
-u : unbuffered binary stdout and stderr (also PYTHONUNBUFFERED=x)
see man page for details on internal buffering relating to
'-u'
-v : verbose (trace import statements) (also PYTHONVERBOSE=x)
-V : print the Python version number and exit
-W arg : warning control (arg is action:message:category:module:lineno)
"""

could you please explain "of course"? I'm not sure how the general
populace could have known, apart from vague recollections of some Unix
utilities using a -v, -vv, -vvv etc convention.

TIA,
John

[Fuzzyman]

For some value of 'cheap'. They seem to go for about £100 on ebay in
the UK. :-(

More interestingly, someone reported on Python-dev recently a speed
improvement of around 30% (from memory) by compiling with VC 8. I know
the grumble (almost certainly correctly) about Microsoft's 'odd'
interpretation of the C standards in VC 8, but it looks like there are
major benefits to switching...

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

> Regards,
> Martin

[John Machin]

Hi Martin, I do hope you don't regret opening Pandora's box :-)

Does the following look about right?
I was somewhat bemused by the limit of 200 bytes on the module name in
the error message-- I woild have thought about 20 was more appropriate.
Sorry but I can't test this (Windows box, don't have whatever version
of C compiler is necessary to compile Python).

Cheers,
John
8<---
--- Python/importdl.c.orig 2006-09-14 09:53:59.984375000 +1000
+++ Python/importdl.c 2006-09-14 09:55:53.625000000 +1000
@@ -44,8 +44,8 @@
return NULL;
if (p == NULL) {
PyErr_Format(PyExc_ImportError,
- "dynamic module does not define init function
(init%.200s)",
- shortname);
+ "dynamic module (%s) does not define init function
(init%.200s)",
+ pathname, shortname);
return NULL;
}
oldcontext = _Py_PackageContext;
8<---

["Martin v. Löwis"]

Fuzzyman schrieb:

> More interestingly, someone reported on Python-dev recently a speed
> improvement of around 30% (from memory) by compiling with VC 8. I know
> the grumble (almost certainly correctly) about Microsoft's 'odd'
> interpretation of the C standards in VC 8, but it looks like there are
> major benefits to switching...

You may or may not know that it is futile arguing about compiler
switching for released versions of Python, i.e. 2.3, 2.4, and 2.5.
Whether or not it might be a good idea: it can't be done, for
compatibility with prior releases.

Regards,
Martin

["Martin v. Löwis"]

John Machin schrieb:

> Hi Martin, I do hope you don't regret opening Pandora's box :-)
>
> Does the following look about right?

Technically, yes. I wonder whether it will generate unreadable error
messages, though (one would have to try).

> I was somewhat bemused by the limit of 200 bytes on the module name in
> the error message-- I woild have thought about 20 was more appropriate.
> Sorry but I can't test this (Windows box, don't have whatever version
> of C compiler is necessary to compile Python).

I'm not sure where this limit comes from, either. This would need to
be researched to find out whether it is just nonsensical, or, if
there is a rationale for it, whether it applies to the path name as
well.

Regards,
Martin

[Fuzzyman]

Of course, but Python development continues... Into the future..

Fuzzyma
http://www.voidspace.org.uk/python/index.shtml

>
> Regards,
> Martin

["Martin v. Löwis"]

Fuzzyman schrieb:

>> You may or may not know that it is futile arguing about compiler
>> switching for released versions of Python, i.e. 2.3, 2.4, and 2.5.
>> Whether or not it might be a good idea: it can't be done, for
>> compatibility with prior releases.
>
> Of course, but Python development continues... Into the future..

It's clear that Python can't stay as VS 2003 forever. I don't
know what Microsoft's product plans are, but I hope that we
could skip VS 2005 entirely, and move to VS 2007 for Python 2.6.
That, of course, assumes that there is a VS 2007 release
sufficiently before Python 2.6.

Regards,
Martin

[Lawrence Oluyede]

"Martin v. Löwis" <mar...@v.loewis.de> wrote:
> That, of course, assumes that there is a VS 2007 release
> sufficiently before Python 2.6.

I just spoke with an MVP for .NET (so nothing official obviously) and he
told me the next VS version will ship after Windows Vista for sure and
that (he mentioned the actual alpha status of the available components)
will be likely in the second half of the year or maybe later.

Anyway after the release of Vista we will all know something more I
guess.

HTH

[michael....@gmail.com]

Borland released a free version of their C++ compiler and IDE on 9/4,
coinciding with my need to move my GeoTrans extension from Linux to
Windows. I didn't need the IDE for the project, since my
GeoTransMethodsSetup.py script from Linux worked fine, running it with
the command line argument "--compiler=bcpp". Add paths to the include
directories and library directories.

The biggest headache was a bunch of nonsense linker error messages,
which turned out to be because I had made the mistake of installing the
compiler under "Program Files", and setup does not behave well with
spaces in the path name. As a quick work-around, I used the DOS 8.3
filename. Next time I will install Borland in a path with no spaces in
the name. So, I was able to use a state-of-the-art compiler, rather
than work with an obsolete version of some compiler relic.

from distutils.core import setup, Extension
GeoTransMethods = Extension('GeoTransMethods',
include_dirs = ['C:\python24\include'],
library_dirs = [
r"C:\PROGRA~1\Borland\BDS\4.0\lib",
r"C:\PROGRA~1\Borland\BDS\4.0\lib\release",
r"C:\PROGRA~1\Borland\BDS\4.0\lib\obj",
r"C:\PROGRA~1\Borland\BDS\4.0\lib\PSDK",
r"C:\PROGRA~1\Borland\BDS\4.0\lib\Indy9"],
sources = ["GeoTransMethods.c", "mgrs.c", "utm.c", "ups.c",
"tranmerc.c", "polarst.c"])

setup(name="GeoTransMethods", version="1.0",
ext_modules=[GeoTransMethods])

[michael....@gmail.com]

I neglected to mention another detail. There are a number of postings
around regarding earlier Borland compilers. There is a brief overview
at http://docs.python.org/inst/tweak-flags.html
It mentions the necessity of converting the object file format (COFF)
of python libraries built with Visual C++ to Borland's OMF object file
format. You need to download coff2omf and run it to make
Borland-linkable copies of python24.lib and any other VC++ built libs,
such as zlib:

coff2omf python24.lib python24_bcpp.lib

Distutils run with the bccp compiler option will look for _bccp
versions and use them before attempting to use the VC++ versions.