AttributeError in runtests.py
Christoph
Hi,
I just downloaded and installed cython from github (449e3b58a9fe4826179b7208241ee991e7360a7b).
At http://wiki.cython.org/HackerGuide I found that the way to run the Cython tests is via runtests.py.
But for me this fails:
$ python runtests.py
Traceback (most recent call last):
File "runtests.py", line 148, in <module>
OPENMP_C_COMPILER_FLAGS = get_openmp_compiler_flags('c')
File "runtests.py", line 139, in get_openmp_compiler_flags
compiler_version = matcher(output).group(1)
AttributeError: 'NoneType' object has no attribute 'group'
I am on Mac OS X Lion (XCode 4.3).
Maybe it would be useful to add the info on how to run the Cython tests to the normal installation instructions?
Christoph
PS: Reporting bugs could be made simpler for new users. Starting at http://cython.org/ there's a link "Bug & Feature Tracker", which leads to http://trac.cython.org/cython_trac/. There it says that I should write an email to Robert Bradshaw to get a Trac account, but no contact email address is given. So in the end I decided to write to cython-users (which I guess is not the proper way to report bugs) because I didn't want to sign up for cython-devel and couldn't quickly set up a trac account myself.
Stefan Behnel
> I just downloaded and installed cython from github
> (449e3b58a9fe4826179b7208241ee991e7360a7b).
>
> At http://wiki.cython.org/HackerGuide I found that the way to run the
> Cython tests is via runtests.py.
>
> But for me this fails:
>
> $ python runtests.py
>
> Traceback (most recent call last):
>
> File "runtests.py", line 148, in <module>
>
> OPENMP_C_COMPILER_FLAGS = get_openmp_compiler_flags('c')
>
> File "runtests.py", line 139, in get_openmp_compiler_flags
>
> compiler_version = matcher(output).group(1)
>
> AttributeError: 'NoneType' object has no attribute 'group'
>
>
> I am on Mac OS X Lion (XCode 4.3).
Seems to be something wrong with the compiler setup code for OpenMP here.
It should at least give a better error message. Thanks for bringing it up.
> Maybe it would be useful to add the info on how to run the Cython tests to
> the normal installation instructions?
Tests are not often run by "ordinary users", but I agree that it makes
sense to do that, if only to allow users to check if their local setup is
properly working.
Want to make a start?
> PS: Reporting bugs could be made simpler for new users. Starting at
> http://cython.org/ there's a link "Bug & Feature Tracker", which leads to
> http://trac.cython.org/cython_trac/. There it says that I should write an
> email to Robert Bradshaw to get a Trac account, but no contact email
> address is given.
He rather regularly posts on both the users list and the developers list,
though, so his e-mail address isn't exactly a state secret.
> So in the end I decided to write to cython-users (which I
> guess is not the proper way to report bugs)
It's fine for user problems in general. Not all reported problems are bugs,
so it's better to have them start off on the users list even if they are.
> because I didn't want to sign
> up for cython-devel and couldn't quickly set up a trac account myself.
Understandable. However, we need to restrict the trac site to authenticated
users because trac and other wiki sites tend to receive tons of automated
spam, and the overhead of keeping that cleaned up is way larger than that
of manually adding a user from time to time.
Stefan
Christoph Deil
Tests are not often run by "ordinary users", but I agree that it makes
sense to do that, if only to allow users to check if their local setup is
properly working.
Stefan Behnel
> I just downloaded and installed cython from github
> (449e3b58a9fe4826179b7208241ee991e7360a7b).
>
> At http://wiki.cython.org/HackerGuide I found that the way to run the
> Cython tests is via runtests.py.
>
> But for me this fails:
>
> $ python runtests.py
> Traceback (most recent call last):
> File "runtests.py", line 148, in <module>
> OPENMP_C_COMPILER_FLAGS = get_openmp_compiler_flags('c')
> File "runtests.py", line 139, in get_openmp_compiler_flags
> compiler_version = matcher(output).group(1)
> AttributeError: 'NoneType' object has no attribute 'group'
>
> I am on Mac OS X Lion (XCode 4.3).
According to the pandas bug report you posted, you are using clang as C
compiler. The code above was written with gcc in mind. I pushed a quick fix
for the test runner that disables OpenMP detection for non-gccs. We should
eventually add support for them.
The reply by "wesm" in the bug report is right, though - it's not unlikely
that something has gone wrong when you updated parts of your system with
development versions. Maybe just some old binaries or header files still
lying around in the wrong places or being in the wrong order in the PATH or
include paths. Just because you can import NumPy doesn't mean that
everything else is able to find the right configuration.
Stefan
Christoph Deil
Christoph, 13.04.2012 02:38:I just downloaded and installed cython from github(449e3b58a9fe4826179b7208241ee991e7360a7b).At http://wiki.cython.org/HackerGuide I found that the way to run theCython tests is via runtests.py.But for me this fails:$ python runtests.pyTraceback (most recent call last):File "runtests.py", line 148, in <module>OPENMP_C_COMPILER_FLAGS = get_openmp_compiler_flags('c')File "runtests.py", line 139, in get_openmp_compiler_flagscompiler_version = matcher(output).group(1)AttributeError: 'NoneType' object has no attribute 'group'I am on Mac OS X Lion (XCode 4.3).
According to the pandas bug report you posted, you are using clang as C
compiler. The code above was written with gcc in mind. I pushed a quick fix
for the test runner that disables OpenMP detection for non-gccs. We should
eventually add support for them.
The reply by "wesm" in the bug report is right, though - it's not unlikely
that something has gone wrong when you updated parts of your system with
development versions. Maybe just some old binaries or header files still
lying around in the wrong places or being in the wrong order in the PATH or
include paths. Just because you can import NumPy doesn't mean that
everything else is able to find the right configuration.
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.7.3-py2.7-macosx-10.7-x86_64.egg/pandas/__init__.py", line 10, in <module>
import pandas._tseries as lib
File "numpy.pxd", line 177, in init pandas._tseries (pandas/src/tseries.c:119198)
ValueError: numpy.ndarray has the wrong size, try recompilingStefan Behnel
> On Apr 13, 2012, at 11:00 AM, Stefan Behnel wrote:
>
>> Christoph, 13.04.2012 02:38:
>>> I just downloaded and installed cython from github
>>> (449e3b58a9fe4826179b7208241ee991e7360a7b).
>>>
>>> At http://wiki.cython.org/HackerGuide I found that the way to run the
>>> Cython tests is via runtests.py.
>>>
>>> But for me this fails:
>>>
>>> $ python runtests.py
>>> Traceback (most recent call last):
>>> File "runtests.py", line 148, in <module>
>>> OPENMP_C_COMPILER_FLAGS = get_openmp_compiler_flags('c')
>>> File "runtests.py", line 139, in get_openmp_compiler_flags
>>> compiler_version = matcher(output).group(1)
>>> AttributeError: 'NoneType' object has no attribute 'group'
>>>
>>> I am on Mac OS X Lion (XCode 4.3).
>>
>> According to the pandas bug report you posted, you are using clang as C
>> compiler. The code above was written with gcc in mind. I pushed a quick fix
>> for the test runner that disables OpenMP detection for non-gccs. We should
>> eventually add support for them.
>
> Thanks Stefan, now I can run the tests.
> I can now see why you said that most normal users don't run them after installing cython, they take 30 min on my Macbook. :-)
Yes. It helps to disable all compiler optimisations when running the tests.
There's no use in optimising code that only runs once.
> There are 8 errors.
Good to know. And thanks for testing. We do not generally test with clang,
I don't even think it's installed on our build/test server. Looks like we
should add it there.
> Also clang emits many warnings during the cython build (300 warnings) and tests (8000 warnings) because cython generates pointer self-assignments like e.g. this:
>
> cython3.c:3382:14: warning: explicitly assigning a variable of type 'PyObject *' (aka 'struct _object *') to itself [-Wself-assign]
> __pyx_self = __pyx_self;
That's actually done to *avoid* a warning about an unused variable (or
rather function argument). Works in gcc at least.
We might be able to replace it by a "CYTHON_UNUSED" tag in the signature.
> There's one other less-frequent warning about unused functions (5 in the cython build, 400 in the cython tests):
>
> fused_def.cpp:29546:18: warning: unused function '__Pyx_Method_ClassMethod' [-Wunused-function]
> static PyObject* __Pyx_Method_ClassMethod(PyObject *method) {
Yep, I've seen those, too. Basically, this utility function is included in
the generated sources in more cases than it is really used. This should be
fixed.
> Similar warnings also appear for most Mac OS X Lion users installing a
> project using cython, because clang is the default compiler. This kind
> of makes it difficult to spot real build problems, so if you can improve
> cython to generate code that compiles more cleanly with clang, that
> would be great.
Any warning we can get rid of is an improvement.
> I have put full logs from the build, docs and tests here:
> http://dl.dropbox.com/u/4923986/bug_reports/cython-build.log
> http://dl.dropbox.com/u/4923986/bug_reports/cython-docs.log
> http://dl.dropbox.com/u/4923986/bug_reports/cython-tests.log
Thanks!
>> The reply by "wesm" in the bug report is right, though - it's not unlikely
>> that something has gone wrong when you updated parts of your system with
>> development versions. Maybe just some old binaries or header files still
>> lying around in the wrong places or being in the wrong order in the PATH or
>> include paths. Just because you can import NumPy doesn't mean that
>> everything else is able to find the right configuration.
>
> I'll try to find the old files then and maybe recompiling numpy.
>
> The ValueError I had (see https://github.com/pydata/pandas/issues/1040 ) trying to build pandas is generated by Cython, right?
> File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.7.3-py2.7-macosx-10.7-x86_64.egg/pandas/__init__.py", line 10, in <module>
> import pandas._tseries as lib
> File "numpy.pxd", line 177, in init pandas._tseries (pandas/src/tseries.c:119198)
> ValueError: numpy.ndarray has the wrong size, try recompiling
> First I tried recompiling pandas, which of course didn't help. Maybe it could say more explicitly: try recompiling numpy?
No, the important thing is that it finds the same version of NumPy and its
header files at compile time that will be used at runtime. That does not
seem to be the case here.
Stefan
Christoph Deil
On Apr 13, 2012, at 2:00 PM, Stefan Behnel wrote:
> Christoph Deil, 13.04.2012 13:28:
>>
>> Thanks Stefan, now I can run the tests.
>> I can now see why you said that most normal users don't run them after installing cython, they take 30 min on my Macbook. :-)
>
> Yes. It helps to disable all compiler optimisations when running the tests.
> There's no use in optimising code that only runs once.
How can I disable the compiler optimizations when running the tests?
Maybe you could set no optimization as the default for runtests.py then?
>
>
>> There are 8 errors.
>
> Good to know. And thanks for testing. We do not generally test with clang,
> I don't even think it's installed on our build/test server. Looks like we
> should add it there.
That would be very nice for all clang users (i.e. most Mac users).
Thank you!
>
>
>> Also clang emits many warnings during the cython build (300 warnings) and tests (8000 warnings) because cython generates pointer self-assignments like e.g. this:
>>
>> cython3.c:3382:14: warning: explicitly assigning a variable of type 'PyObject *' (aka 'struct _object *') to itself [-Wself-assign]
>> __pyx_self = __pyx_self;
>
> That's actually done to *avoid* a warning about an unused variable (or
> rather function argument). Works in gcc at least.
>
> We might be able to replace it by a "CYTHON_UNUSED" tag in the signature.
I usually omit the function argument name to get rid of these warnings, i.e.
f(int /*a*/, float /*b*/)
instead of
f(int a, float b).
This solution works with gcc and clang and the code is just as readable.
Of course I don't know what the CYTHON_UNUSED tag is, it might be a better solution, I just wanted to mention it.
>>> The reply by "wesm" in the bug report is right, though - it's not unlikely
>>> that something has gone wrong when you updated parts of your system with
>>> development versions. Maybe just some old binaries or header files still
>>> lying around in the wrong places or being in the wrong order in the PATH or
>>> include paths. Just because you can import NumPy doesn't mean that
>>> everything else is able to find the right configuration.
>>
>> I'll try to find the old files then and maybe recompiling numpy.
>>
>> The ValueError I had (see https://github.com/pydata/pandas/issues/1040 ) trying to build pandas is generated by Cython, right?
>> File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.7.3-py2.7-macosx-10.7-x86_64.egg/pandas/__init__.py", line 10, in <module>
>> import pandas._tseries as lib
>> File "numpy.pxd", line 177, in init pandas._tseries (pandas/src/tseries.c:119198)
>> ValueError: numpy.ndarray has the wrong size, try recompiling
>> First I tried recompiling pandas, which of course didn't help. Maybe it could say more explicitly: try recompiling numpy?
>
> No, the important thing is that it finds the same version of NumPy and its
> header files at compile time that will be used at runtime. That does not
> seem to be the case here.
>
> Stefan
After being unable to compile numpy master and making the tests run without segfault, I am now simply using numpy 1.6.1 from Macports, which solved my original problem. I just noticed that this introduced problems with other python packages using numpy ... working on a Mac with numpy / scipy is really a pain at the moment, partly because of the C compiler issues, but even more so because of the Fortran compiler situation.
Thanks for all your help!
Christoph
Stefan Behnel
> On Apr 13, 2012, at 2:00 PM, Stefan Behnel wrote:
>> Christoph Deil, 13.04.2012 13:28:
>>>
>>> Thanks Stefan, now I can run the tests.
>>> I can now see why you said that most normal users don't run them after installing cython, they take 30 min on my Macbook. :-)
>>
>> Yes. It helps to disable all compiler optimisations when running the tests.
>> There's no use in optimising code that only runs once.
>
> How can I disable the compiler optimizations when running the tests?
By passing appropriate CFLAGS, e.g. CFLAGS="-O0 -fno-strict-aliasing" for
Py2.x. (That's "minus Oh Zero", not a double-0, just in case ...)
> Maybe you could set no optimization as the default for runtests.py then?
Not really, because we don't know how to configure your compiler for your
environment. Normally, the CFLAGS are inherited automatically from the
build of your CPython installation, and that is usually (and rightfully)
built with optimisations enabled.
>>> Also clang emits many warnings during the cython build (300 warnings) and tests (8000 warnings) because cython generates pointer self-assignments like e.g. this:
>>>
>>> cython3.c:3382:14: warning: explicitly assigning a variable of type 'PyObject *' (aka 'struct _object *') to itself [-Wself-assign]
>>> __pyx_self = __pyx_self;
>>
>> That's actually done to *avoid* a warning about an unused variable (or
>> rather function argument). Works in gcc at least.
>>
>> We might be able to replace it by a "CYTHON_UNUSED" tag in the signature.
>
> I usually omit the function argument name to get rid of these warnings, i.e.
> f(int /*a*/, float /*b*/)
> instead of
> f(int a, float b).
>
> This solution works with gcc and clang and the code is just as readable.
The tricky bit is to change the signature exactly under the right
conditions, not what to change about it.
> Of course I don't know what the CYTHON_UNUSED tag is, it might be a better solution, I just wanted to mention it.
It's a compiler macro that says "I know that this argument is not being
used, don't warn me about it". We use it in many places, so, yes, it's the
right way to do it.
> working on a Mac with numpy / scipy is really a pain at the moment,
> partly because of the C compiler issues, but even more so because
> of the Fortran compiler situation.
Can't you just switch to gcc by setting CC=gcc ?
Stefan
Christoph Deil
Christoph Deil, 13.04.2012 15:53:On Apr 13, 2012, at 2:00 PM, Stefan Behnel wrote:Christoph Deil, 13.04.2012 13:28:Thanks Stefan, now I can run the tests.I can now see why you said that most normal users don't run them after installing cython, they take 30 min on my Macbook. :-)Yes. It helps to disable all compiler optimisations when running the tests.There's no use in optimising code that only runs once.How can I disable the compiler optimizations when running the tests?
By passing appropriate CFLAGS, e.g. CFLAGS="-O0 -fno-strict-aliasing" for
Py2.x. (That's "minus Oh Zero", not a double-0, just in case ...)
working on a Mac with numpy / scipy is really a pain at the moment,partly because of the C compiler issues, but even more so becauseof the Fortran compiler situation.
Can't you just switch to gcc by setting CC=gcc ?
Stefan
Stefan Behnel
> I tried this command:
> $ export CFLAGS="-O0 -fno-strict-aliasing"; time python runtests.py
>
> Now runtests.py took 23 minutes (approx. 30 min. before).
> I don't know if -O0 was actually used
Given the timings, it clearly was.
> how can I tell runtests.py to display the command (clang ...) it is executing?
That's a long standing bug in the test runner. It suppresses the gcc
commands when the refnanny module has been built before the tests. You can
disable the refnanny with "--no-refnanny". And that bug should eventually
be fixed as well ...
> Now with numpy 1.6.1 from Macports (instead of git master) I still see the same 8 test failures (pasted below).
> [...]
Ok, those are there because the OpenMP setup is not working on your side.
In fact, clang doesn't have OpenMP support at all. I'm not sure why my
quick fix didn't disable those tests on your side - it should have ...
Stefan