undeclared name not built in errors
seventhunders
python modules within cython. Moreover the behavior varies from
platform to platform. I have a cython module that compiles and runs
perfectly on windows xp, windows 7 64 bit, archlinux 64 and a bunch
of fedora versions but fails miserably on xubunu 9.10 (running in a
vmware virtual machine). The problem manifests in python import
declarations not quite working.
Here is the source in question:
---------------pygpumat.pyx-----------------------
from gpumat cimport *
cimport numpy as np
import numpy as np
import pylab
import sys
......
xords = np.arange(self.cptr.ord.start, xend,
self.cptr.ord.inc)
# print xords
garr = g.getarr()
# print g.shape()
# print garr
pylab.plot(xords, abs(garr))
.....
---------------------------------------------------------------------
I then run cython on it using something like this:
cython --cplus -I. -I/usr/lib/python2.6/dist-packages/numpy/core/
include -I/usr/include/python2.6 -I/usr/local/cuda/include
pygpumat.pyx
and I get this error:
Error converting Pyrex file to C:
------------------------------------------------------------
...
xords = np.arange(self.cptr.ord.start, xend,
self.cptr.ord.inc)
# print xords
garr = g.getarr()
# print g.shape()
# print garr
pylab.plot(xords, abs(garr))
^
------------------------------------------------------------
/home/matt/Projects/FlightTest/pygpumat.pyx:860:13: undeclared name
not builtin: pylab
My ubuntu setup uses gcc version 4.4.1 cython 0.13, python 2.6.4
matplotlib 0.99 numpy 1.3. pylab is part of matplotlib.
I note that you can also get weird errors on this platform if cimport
numpy as np is not in front of import numpy as np.
Am I doing something wrong here, or is this a bug? I do note that it
works most everywhere else I've ported it.
Oh one more thing. It really is trying to create a .cpp file and not
a .c file despite the Error converting Pyrex file to C, since
the .cpp file exists but the .c file does not.
Stefan Behnel
> I have a cython module that compiles and runs
> perfectly on windows xp, windows 7 64 bit, archlinux 64 and a bunch
> of fedora versions but fails miserably on xubunu 9.10 (running in a
> vmware virtual machine). The problem manifests in python import
> declarations not quite working.
>
> Here is the source in question:
> ---------------pygpumat.pyx-----------------------
>
> from gpumat cimport *
IMHO, "from ... (c)import *" has the same code smell in Cython code that it
has in Python.
From the snippet you posted, I can't see a reason why this should fail.
Could you provide a link to the complete code file(s) so that others can
try to reproduce this?
> Oh one more thing. It really is trying to create a .cpp file and not
> a .c file despite the Error converting Pyrex file to C, since
> the .cpp file exists but the .c file does not.
Yes, that message dates back to the old days when C++ code generation
wasn't a common thing in Pyrex, long before Cython even existed.
Stefan
Lisandro Dalcin
> seventhunders, 15.10.2010 10:25:
>>
>>
>> from gpumat cimport *
>
> IMHO, "from ... (c)import *" has the same code smell in Cython code that it
> has in Python.
>
And it smells so bad that Cython could emit warnings for the cimport
case, where a name clash occurs. What do you think?
--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169
Robert Bradshaw
Import vs. cimport is a known source of oddness, and we're looking into this.
> Am I doing something wrong here, or is this a bug? I do note that it
> works most everywhere else I've ported it.
I just tried
from gpumat cimport *
cimport numpy as np
import numpy as np
import pylab
import sys
def foo(xords, garr):
pylab.plot(xords, abs(garr))
and was not able to reproduce your error. Are you sure the Cython on
that machine is up to date?
- Robert
seventhunders
yesterday and cython -V
reports version 0.13. Unfortunately I can't release the entire code
since it's proprietary and to make matters
worse it only seems to occur for this particular setup. Could it be
something in python that causes this?
Maybe I should try to reinstall python.
I'm going to try to create a smaller example of this and see if I can
post my results.
On Oct 15, 11:30 am, Robert Bradshaw <rober...@math.washington.edu>
wrote:
Stefan Behnel
[keeping the e-mail history properly formatted helps in keeping things
readable]
seventhunders, 15.10.2010 19:10:
>>> This particular error seems to crop up when trying to import certain
>>> python modules within cython. Moreover the behavior varies from
>>> platform to platform. I have a cython module that compiles and runs
>>> perfectly on windows xp, windows 7 64 bit, archlinux 64 and a bunch
>>> of fedora versions but fails miserably on xubunu 9.10
>
> Unfortunately I can't release the entire code
> since it's proprietary and to make matters
> worse it only seems to occur for this particular setup.
In that case, why do you bother with running Cython on that system at all?
I'd be a lot more comfortable with shipping well tested C sources for the
build, instead of letting users run their own version of Cython on it,
which may or may not generate reproducible code.
> Could it be something in python that causes this?
Not impossible. I assume you want to imply that it's not the same Python
version on all installations?
Stefan
seventhunders
On Oct 15, 1:23 pm, Stefan Behnel <stefan...@behnel.de> wrote:
>
> In that case, why do you bother with running Cython on that system at all?
> I'd be a lot more comfortable with shipping well tested C sources for the
> build, instead of letting users run their own version of Cython on it,
> which may or may not generate reproducible code.
dependencies I'm trying to get an Nvidia cuda emulator (ocelot)
running
on some system. Ocelot is developed on ubuntu and so I thought I
would try to run it in a virtual machine since all my machines have
either too recent software
or too old software.
> > Could it be something in python that causes this?
>
> Not impossible. I assume you want to imply that it's not the same Python
> version on all installations?
running this stuff. There might even be a python 2.5 somewhere.
However I just went through a nightmare install of python2.7 + numpy +
matplotlib + cython0.13 + dependencies to see if that would fix it,
but unfortunately not.
All this stuff was compiled from source. So this begs the question,
what is different about this installation that causes cython to barf?
Since I'm compiling from source could
I be linking in a bad library? Apparently just a simple minded
example seems to work ok as well, so it must be some fairly complex
interaction.
Hopefully I'll have a chance to look at it again next week.
Lisandro Dalcin
What happens if you add a line with "global pylab" before the offending line?
Robert Bradshaw
I have no idea. The cythonization process should be very consistant
from platform to platform (much more so than the C compilation). Does
the following at least compile for you?
- Robert
----------------------------------------
seventhunders
right now. But I will try these suggestions tonight.
I did try a fairly simple pylab example, not too different from yours
and it did compile. Thus the failure is due to a more
complex interaction. What I probably need to do is to start a trial
and error process of deleting code until I can find a minimal
subset that still causes the failure. I'm going to try another linux
operating system here shortly and see if that works.
On Oct 16, 3:55 am, Robert Bradshaw <rober...@math.washington.edu>
wrote:
Robert Bradshaw
> I'm not at the machine that has the vmware ubuntu virtual machine
> right now. But I will try these suggestions tonight.
> I did try a fairly simple pylab example, not too different from yours
> and it did compile. Thus the failure is due to a more
> complex interaction. What I probably need to do is to start a trial
> and error process of deleting code until I can find a minimal
> subset that still causes the failure. I'm going to try another linux
> operating system here shortly and see if that works.
Yes, that error is very strange indeed. I know it sounds trivial, but
I would be sure to double-check that you're actually compiling the
code and using the software versions that you think you are. And
if/when you do find a bug, please let us know.
seventhunders
I installed my code on a more recent version of Ubuntu and at first
everything seems fine there.
Applying cython to my code (yes it is version 0.13) works and it
compiles with a few warnings, but otherwise seems to work.
However the test script I have been using that bypasses distutils
causes the same bug as the older version of ubuntu in my virtual
machine.
I believe that means that distutils must be invoking cython with
additional or different parameters, or perhaps in a different
environment.
Just as a reference, the distutils build script on the old ubuntu
fails with the same error, but the distutils on the new ubuntu works
just fine.
Here is the setup.py script I'm using:
-------------------------------------------------------------------------------------
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
DISTUTILS_DEBUG=True
# platform specific include directories and libraries
do32=False
if do32:
incdirs =['.','/usr/local/include/python2.5', '/usr/local/lib/
python2.5/site-packages/numpy/core/include' , '/usr/local/cuda/
include', '/usr/local/cudasdk/common/inc', ]
libdirs=['.', '/usr/local/cuda/lib', '/usr/local/cudasdk/lib', '/usr/
lib']
else:
incdirs =['.','/usr/include/python2.6', '/usr/lib64/python2.6/dist-
packages/numpy/core/include' , '/usr/local/cuda/include' ]
libdirs=['.', '/usr/local/cuda/lib64', '/usr/lib64']
# gpu accelerated matrix libraries
libs = ['gpumat', 'cuda', 'cublas', 'cudart', 'cufft' ]
ext_modules = [
Extension( "gpumat",
sources=["pygpumat.pyx"], # filename of
our Pyrex/Cython source
language="c++", # this causes Pyrex/
Cython to create C++ source
include_dirs = incdirs,
library_dirs = libdirs, # if needed
libraries=libs # ditto
# extra_compile_args=["-g"],
# extra_link_args=["-g"],
),
Extension("geo",
sources=["geocoords.pyx"],
language="c++",
include_dirs = incdirs,
library_dirs = libdirs, # if needed
libraries=libs # ditto
)
]
setup(
name ="pygpu", # name of package
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules,
)
-------------------------------------------
Using this running:
python setup.py build_ext --inplace
works just fine.
However if I invoke this:
include -I/usr/include/python2.6 -I/usr/local/cuda/include
pygpumat.pyx
it into bugcython.pyx and run:
include -I/usr/include/python2.6 -I/usr/local/cuda/include
it compiles to c++ as expected.
Do you know a way to display how distutils invokes cython?
On Oct 18, 4:08 pm, Robert Bradshaw <rober...@math.washington.edu>
wrote:
Robert Bradshaw
Distutils invokes Cython programatically, not via the command line.
It's all in Cython/Distutils/... What cython is your command line
"cython" getting? Also, I would make sure there's not an issue with
gpumat.pxd vs. pygpumat.pyx vs. naming the module gpumat.
- Robert
seventhunders
Well even the command line eventually invokes cython
programatically.
So presuming they are both using the same cython one would hope for
the same results.
For my experiment with the later version of ubuntu, there was only
one cython install, since
it's a newly installed system. That one works (as long as I don't
use the command line)
Back at the older ubuntu virtual machine. I made a couple of changes
and got cython to
compile to c++, but gcc fails on the compiled code. I implemented
your suggested change with
regards to a global declaration of pylab I also renamed gpumat.pxd to
pygpumat.pxd. This seemed to
help the distutils compilation on the older ubuntu somewhat.
For the global declaration fix, I now have this code snippet from
pygpumat.pyx
----------------------------------------
cpdef plot(Gcorr self, title, xlab) :
""" Plot the correlation values """
global pylab
cdef Gmat g = self.getmat()
cdef float xend = self.cptr.ord.start + (<float>
self.cptr.ord.N) * self.cptr.ord.inc
self.cptr.ord.inc)
# print xords
garr = g.getarr()
# print g.shape()
# print garr
pylab.plot(xords, abs(garr))
pylab.xlabel(xlab)
pylab.title(title)
------------------------------------------------
plot is a method in an extension class. I've never had to tread
module names as global variables before, but that seemed to get me
past those errors. I probably ought to take your simple example and
make it a method of an extension
class to see if the behavior can be duplicated in a simpler example.
However I got a bunch of compile errors in my compiled code.
Some of the first few errors are:
-------------------------------------------------------
pygpumat.cpp:899: error: ‘__pyx_f_8pygpumat_cempty’ declared as an
‘inline’ variable
pygpumat.cpp:900: error: ‘__pyx_f_8pygpumat_cmat’ declared as an
‘inline’ variable
pygpumat.cpp:901: error: ‘__pyx_f_8pygpumat_rempty’ declared as an
...
------------------------------------------
Here is a relevant section of pygpumat.cpp:
--------------------------
...
static CYTHON_INLINE struct __pyx_obj_8pygpumat_Gmat
*(*__pyx_f_8pygpumat_cempty)(int, int, int __pyx_skip_dispatch); /
*proto*/
static CYTHON_INLINE struct __pyx_obj_8pygpumat_Gmat
*(*__pyx_f_8pygpumat_cmat)(int, int, int __pyx_skip_dispatch); /
*proto*/
static CYTHON_INLINE struct __pyx_obj_8pygpumat_Gmat
*(*__pyx_f_8pygpumat_rempty)(int, int, int __pyx_skip_dispatch); /
*proto*/
...
---------------------------
These functions are declared inline in pygpumat.pyx
---------------------------------------------
cpdef inline Gmat cempty(int r, int c) :
""" create an empty complex matrix """
...
-----------------------------------------------
Also there are definite differences between the command line
invocation of cython and
the progammatic one.
On Oct 18, 5:46 pm, Robert Bradshaw <rober...@math.washington.edu>
wrote:
>
Robert Bradshaw
>
> Well even the command line eventually invokes cython
> programatically.
> So presuming they are both using the same cython one would hope for
> the same results.
Yes. It looks like you're passing a whole lot of options to the
distutils command that you're not even mentioning on the command line.
You shouldn't have to, unless you're assigning to it somewhere in the
body. (Note that "import pylab" is an assignment.)
> I probably ought to take your simple example and
> make it a method of an extension
> class to see if the behavior can be duplicated in a simpler example.
Yes, it's hard to even guess what the problem is without a
reproducible example.
I wasn't able to reproduce this, perhaps because I only have part of
the source?
> Also there are definite differences between the command line
> invocation of cython and
> the progammatic one.
You're passing them different options.
seventhunders
source that I can release, but
I've been stymied by a few issues. One is that I can get a whole
range of unexpected errors depending
on which code segment I remove, and then when I think I've isolated
it, my toy case appears to just work.
I'll keep around the virtual machine that exhibits the problem, and
perhaps I can get to it a bit later.
On Oct 19, 10:38 am, Robert Bradshaw <rober...@math.washington.edu>
wrote: