python setup.py build_ext --inplace fails with error: command 'gcc' failed with exit status 1

[Joaquin Abian]

I recently changed from winxp 32-bits to win 7 64-bits and I'm trying
to rebuild my toolchain for cython.
As a test, I'm trying to compile a simple cython extension.
Cython was installed from Cython-0.14.1.win-amd64-py2.6.‌exe (gohlke
repository)
Python is 2.6.6 64-bit (ActiveState).

The compiler is mingw-w64-bin_i686-mingw_20110430 (gcc 4.7.0 20110430
(experimental))
For convenience I renamed the executables in mingw\bin from, for
example, x86_64-w64-mingw32-gcc.exe to gcc.exe.
The mingw setup is working as I can produce executables from a minimal
helloword.c file

The python program is a simple 'hello word' and setup.py is:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("hello2", ["hello2.pyx"])]
)

I have a correct distutils.cfg file indicating mingw as the compiler.
I have also the needed libpyton.a file that I prepared following the
instructions in the Cython tutorial

Initially, when I tried to build the extension I got an error saying
than the parameter -mno-cygwin did not exist.
Then, I modified the mingw compiler class in distutils\cygwinccompiler
eliminating this parameter from the call to gcc.
The error disappeared but then I got this one I can not understand:

C:\Python26\programas\Cython>python setup.py build_ext --inplace
running build_ext
skipping 'hello2.c' Cython extension (up-to-date)
building 'hello2' extension
C:\mingw\bin\gcc.exe -mdll -O -Wall -IC:\Python26\include -IC:
\Python26\PC -c hello2.c -o build\temp.win-amd64-2.6\Release\hello2.o
writing build\temp.win-amd64-2.6\Release\hello2.def
C:\mingw\bin\gcc.exe -shared -s build\temp.win-amd64-2.6\Release
\hello2.o build\temp.win-amd64-2.6\Release\hello2.def -LC:
\Python26\libs -LC:\Python26\PCbuild\amd64 -lpython26 -lmsvcr90 -o C:
\Python26\programas\Cython\hello2.pyd
c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-
w64-mingw32/lib/../lib/libmingw32.a(lib64_libmingw32_a-
mingw_helpers.o): In function `_decode_pointer':
c:\bb\vista64-mingw32\mingw-x86-x86_64\build\build\mingw\obj/../../../
build/mingw/mingw-w64-crt/crt/mingw_helpers.c:20: multiple definition
of `_decode_pointer'
c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-
w64-mingw32/lib/../lib/libmsvcr90.a(dwpfs00301.o):(.text+0x0): first
defined here
c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-
w64-mingw32/lib/../lib/libmingw32.a(lib64_libmingw32_a-
mingw_helpers.o): In function `_encode_pointer':
c:\bb\vista64-mingw32\mingw-x86-x86_64\build\build\mingw\obj/../../../
build/mingw/mingw-w64-crt/crt/mingw_helpers.c:26: multiple definition
of `_encode_pointer'
c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-
w64-mingw32/lib/../lib/libmsvcr90.a(dwpfs00311.o):(.text+0x0): first
defined
here
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

Any hint is appreciated

Joaquin

[Ralf Schmitt]

Joaquin Abian <gatoy...@gmail.com> writes:

> mingw_helpers.o): In function `_encode_pointer':
> c:\bb\vista64-mingw32\mingw-x86-x86_64\build\build\mingw\obj/../../../
> build/mingw/mingw-w64-crt/crt/mingw_helpers.c:26: multiple definition
> of `_encode_pointer'
> c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-
> w64-mingw32/lib/../lib/libmsvcr90.a(dwpfs00311.o):(.text+0x0): first
> defined
> here
> collect2: ld returned 1 exit status
> error: command 'gcc' failed with exit status 1
>
> Any hint is appreciated

please don't download the .def files linked from the cython wiki. use
gendef[1] instead (i.e. your libpython26.a is currently broken).

I also had to adapt cygwinccompiler to *not* link with MSVCRT90. Our
tests did fail intermittently when linking with that, importing a mingw
compiled c extension did fail immediately when running under gdb.
However we did use tdm gcc 4.5.1, YMMV.

[1] http://sourceforge.net/apps/trac/mingw-w64/wiki/gendef

Hope that's enough to get you started,

Cheers,
- Ralf

[Joaquin Abian]

Thanks for your help.

I installed tdm64-gcc-4.5.2.exe for 64-bit compilation
I prepared a new .def file with 'gendef.exe python26.dll' and produced
a new libpython26.a, that I placed in the python26/libs folder.
I eliminated the link to MSVCRT90, as you indicated, by returning an
empty list from the get_msvcr() function in cygwinccompiler (although
I do not understand what the implications of this action are for the
extension...)

Then I ran python setup.py build_ext --inplace with no errors, and my
hello2.pyd file was produced!
However when I try to import it I get:

>>> from mython import hello2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: DLL load failed: No se encontr¾ el proceso especificado.

(in english: ImportError: DLL load failed: specified procedure could
not be found.)

Any hint?

Thanks in advance
Joaquin








On May 21, 9:16 pm, Ralf Schmitt <sch...@gmail.com> wrote:

[Ralf Schmitt]

Joaquin Abian <gatoy...@gmail.com> writes:

> Thanks for your help.
>
> I installed tdm64-gcc-4.5.2.exe for 64-bit compilation
> I prepared a new .def file with 'gendef.exe python26.dll' and produced
> a new libpython26.a, that I placed in the python26/libs folder.
> I eliminated the link to MSVCRT90, as you indicated, by returning an
> empty list from the get_msvcr() function in cygwinccompiler (although
> I do not understand what the implications of this action are for the
> extension...)
>
> Then I ran python setup.py build_ext --inplace with no errors, and my
> hello2.pyd file was produced!
> However when I try to import it I get:
>
>>>> from mython import hello2
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: DLL load failed: No se encontr¾ el proceso especificado.
>
> (in english: ImportError: DLL load failed: specified procedure could
> not be found.)
>
> Any hint?

yes: apply the patch from http://bugs.python.org/issue4709

(sorry, I did forget that one).

Cheers,
- Ralf

[Ralf Schmitt]

Joaquin Abian <gatoy...@gmail.com> writes:

> Thanks for your help.
>
> I installed tdm64-gcc-4.5.2.exe for 64-bit compilation
> I prepared a new .def file with 'gendef.exe python26.dll' and produced
> a new libpython26.a, that I placed in the python26/libs folder.
> I eliminated the link to MSVCRT90, as you indicated, by returning an
> empty list from the get_msvcr() function in cygwinccompiler (although
> I do not understand what the implications of this action are for the
> extension...)

Start gdb with with "gdb python", then start python inside gdb with "r",
then import your c extension with "import my_extension".
Doing that without linking with MSVCRT90 works flawlessly.
Doing that with the extension linked with MSVCRT90 results in a
segmentation fault. If you use extension linked with MSVCRT90 you will
get intermittent segfaults (at least we did get them). It's no fun.

Cheers,
- Ralf

[Joaquin Abian]

I understand from your answer that I should apply the patch for pyconfig.h and compile python26 from source.

Is this right?

Unfortunately, I am an average python user and I have not experience about compiling python in windows.

As an end-user I like cython because it is a very convenient, easy to learn tool that allows me to speed up code without needing to go deeper in C and compilation issues or to use more complicated tools such as Pyrex or Swing...  

So compiling python in order to use cython is probably too much for me.

There is an easier path I could follow? Has this pacht already included in some python version?

For me, it would be easier to update to python27 than to mess with compiling python.

Thanks again for your time

joaquin

2011/5/22 Ralf Schmitt <sch...@gmail.com>

[sch...@gmail.com]

Joaquin Abian <gatoy...@gmail.com> writes:

> I understand from your answer that I should apply the patch for pyconfig.h
> and compile python26 from source.
> Is this right?

no, just patch the file in your python installation directory. Or you
could compile with MS_WIN64 defined on the command line.

[Joaquin Abian]

Thanks, it worked!

Maybe this summary can be useful for others:

Step by step recipe to compile 64-bit cython extensions with python
2.6.6 with mingw compiler in win 7 64-bit

**Install mingw compiler**
1) Install tdm64-gcc-4.5.2.exe for 64-bit compilation

**Apply patch to python.h**
2) Modify python.h in C:\python26\include as indicated in
http://bugs.python.org/file12411/mingw-w64.patch

**Modify distutils**
3) Eliminate all the parameters -mno-cygwin fom the call to gcc in the
Mingw32CCompiler class in Python26\Lib\distutils\cygwinccompiler.py
4) In the same module, modify get_msvcr() to return an empty list
instead of ['msvcr90'] when msc_ver == '1500' .

**Produce the libpython.a file**
5) Obtain gendef.exe from mingw-w64-bin_x86_64-
mingw_20101003_sezero.zip
(gendef.exe is not available in the tmd64 distribution. Another
solution is to compile gendef from source...)
6) Copy python26.dll (located at C\windows\system32) to the user
directory (C:\Users\myname)
7) Produce the python26.def file with:
> gendef.exe C:\Users\myname\python26.dll
8) Move the python.def file produced (located in the folder from where
gendef was executed) to the user directory
9) Produce the libpython.a with:
> dlltool -v --dllname python26.dll --def C:\Users\myname
\python26.def --output-lib C:\Users\myname\libpython26.a
10) Move the created libpython26.a to C:\Python26\libs

**Produce your .pyd extension**
11) Create a test hello.pyx file and a setup.py file as indicated in
cython tutorial (http://docs.cython.org/src/quickstart/build.html)
12) Compile with
> python setup.py build_ext --inplace

Done!




On May 23, 11:27 am, sch...@gmail.com wrote:

[Robert Bradshaw]

On Sat, May 21, 2011 at 12:16 PM, Ralf Schmitt <sch...@gmail.com> wrote:
> Joaquin Abian <gatoy...@gmail.com> writes:
>
>> mingw_helpers.o): In function `_encode_pointer':
>> c:\bb\vista64-mingw32\mingw-x86-x86_64\build\build\mingw\obj/../../../
>> build/mingw/mingw-w64-crt/crt/mingw_helpers.c:26: multiple definition
>> of `_encode_pointer'
>> c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.7.0/../../../../x86_64-
>> w64-mingw32/lib/../lib/libmsvcr90.a(dwpfs00311.o):(.text+0x0): first
>> defined
>> here
>> collect2: ld returned 1 exit status
>> error: command 'gcc' failed with exit status 1
>>
>> Any hint is appreciated
>
> please don't download the .def files linked from the cython wiki. use
> gendef[1] instead (i.e. your libpython26.a is currently broken).

Could you update the wiki to point to this (with the relevant
instructions) instead? Thanks.

[Ana Vélez]

Hello,
Could you tell how did you apply the patch to python.h?
Thank you so much