C++ Wrapper Confusion
SevenThunders
success in the past wrapping a C library, however I find the state of
the C++ documentation rather confusing. I believe there have been
some prior posts on this issue. My first question is what wrapping
syntax is actually suported in Cython 0.12.1? The online
documentation for version 0.12 has the 'older' syntax here,
http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html .
Some earlier posts suggest that this is deprecated and that we should
use the rather terse discussion here:
http://wiki.cython.org/WrappingCPlusPlus
Although the effort to provide C++ support is greatly appreciated,
there are a few things missing from this documentation. There is no
discussion about how to define the extension class and some issues are
missing, like how do you handle pass by reference. The syntax for
that seems inconsistent.
Right now I just want to know what's actually supported in my version
of Cython 0.12.1. I tried the new syntax and immediately got a syntax
error in this little code snippet:
cdef extern from "gpumat.h" :
ctypedef struct dim3
ctypedef enum MatrixType :
Realmatrix = 0
Complexmatrix=1
Cuppermatrix = 2
ctypedef struct gpumat
void fromgpu(gpumat gin, float *hval)
cdef cppclass gpumat:
int nrows
int ncols
The error was,
python setup.py build_ext --inplace
running build_ext
cythoning pygpumat.pyx to pygpumat.cpp
Error converting Pyrex file to C:
------------------------------------------------------------
...
Complexmatrix=1
Cuppermatrix = 2
ctypedef struct gpumat
void fromgpu(gpumat gin, float *hval)
cdef cppclass gpumat:
int nrows
^
------------------------------------------------------------
pygpumat.pyx:19:12: Syntax error in simple statement list
Robert Bradshaw
> I am trying to wrap an existing C++ class in cython. I've had some
> success in the past wrapping a C library, however I find the state of
> the C++ documentation rather confusing. I believe there have been
> some prior posts on this issue. My first question is what wrapping
> syntax is actually suported in Cython 0.12.1? The online
> documentation for version 0.12 has the 'older' syntax here,
> http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html .
>
> Some earlier posts suggest that this is deprecated and that we should
> use the rather terse discussion here:
> http://wiki.cython.org/WrappingCPlusPlus
The new syntax is not available in Cython 0.12.1, but will be in
Cython 0.13 which should be out soon (yes, soon has dragged on for a
month or two more than we would have liked, but I think the issues
uncovered by Sage have been resolved (Craig?) and as has been
discussed, if we need to postpone closures due to a refcounting bug,
so be it).
> Although the effort to provide C++ support is greatly appreciated,
> there are a few things missing from this documentation.
Though it has been getting better, there is still *lot* of things
missing from this documentation.
> There is no discussion about how to define the extension class
See http://docs.cython.org/src/tutorial/cdef_classes.html
> and some issues are missing, like how do you handle pass by
> reference.
There's no support for this in the old syntax (you have to fake it
with pointers) but in the new you should be able to declare reference
arguments just like in C++.
> The syntax for that seems inconsistent.
It's still in prototype phase, though ready to be used (though you
need the cython-devel branch). We'd welcome more examples of how it
could be improved.
That's because cppclass isn't supported in 0.12.1
- Robert
SevenThunders
there was stealth support
for the new syntax.
I downloaded a development version and attempted to use it to start
wrapping a somewhat complex C++ class (that is fortunately no longer
templated). I pointed my PYTHONPATH to the install directory and the
linux PATH variable to the underlying bin directory. I ran into some
issues, some of which I could overcome, but the final issue is
baffling to me.
First I found that cython doesn't seem to handle type declarations of
the form std::string. Or at least whenever I used it as a type
signature in a function call it complained of a syntax error.
I then found that it doesn't like the ~ character in the destructor
declaration.
Next, when wrapping my extension class, it's single data attribute
is the pointer to the underlying C++ class. But how do you
dereference that pointer in cython code? After all a lot of my
methods take (const gpumat &) as the type signature. Well it seems
you have to drop the const declaration and then if you erroneously try
to pass *gptr as an argument, it treats it as a star argument within
python syntax, which I suppose is a kind of unbracket or list
extraction operator (reasonable I suppose). So I think you have to
do something like self.gptr[0] as an argument.
I then found that cuda's dim3 struct type had to be declared
explicitly for some reason.
Finally, after doing all this I ran cython on this code:
-------------pygpumat.pyx---------------
import numpy as np
cimport numpy as np
import sys
cdef extern from "gpumat.h" :
ctypedef struct dim3:
int x, y, z
ctypedef enum MatrixType :
Realmatrix = 0
Complexmatrix=1
Cuppermatrix = 2
ctypedef struct gpumat
void fromgpu(gpumat &gin, float *hval)
cdef cppclass gpumat:
int nrows
int ncols
int ld
MatrixType mtype
float *val
dim3 block
dim3 grid
dim3 residual
gpumat()
gpumat(unsigned i, unsigned j)
gpumat(unsigned i, unsigned j, unsigned nld)
gpumat(unsigned i, unsigned j, unsigned nld, MatrixType mt)
gpumat(unsigned i, unsigned j, MatrixType mt)
gpumat(unsigned i, unsigned j, unsigned nld, MatrixType mt,
float *fptr)
gpumat(unsigned i, unsigned j, unsigned nld, float *fptr)
float *tohost()
void fromhost(float *hval)
# ~gpumat()
void cgauss()
void had(gpumat &rhs)
gpumat herm()
void write(char *fname)
# void write(std::string fname)
void append(char *fname)
# void append(std::string fname)
void read(char *fname)
# void read(std::string fname)
gpumat(char *fname)
# gpumat(std::string fname)
void conj()
void zset(float *z)
void rset(float a)
void zeros()
void setdiag(float *src)
unsigned iscomplex(void)
# end class gpumat
# std::ostream& operator<<(std::ostream& output, gpumat & p)
cdef class Gmat:
cdef gpumat *gptr # C++ pointer instance
def __cinit__(self, gpumat *gp) :
self.gptr = gp
def __dealloc__(self) :
del self.gptr
# obtain an ndarray copy
cpdef np.ndarray getarr(self) :
cdef np.ndarray nt
if (self.gp.iscomplex()) :
nt = np.empty(shape = (self.gp.nrows, self.gp.ncols),
dtype = np.complex64,order='Fortran')
else :
nt = np.empty(shape = (self.gp.nrows, self.gp.ncols),
dtype = np.float32,order='Fortran')
fromgpu(self.gp[0], nt.data)
return(nt)
def __str__(self) :
cdef np.ndarray nt = self.getarr()
return nt.__str__()
cpdef Gmat gcgauss(tuple rc) :
# cdef MatrixType mtype = Complexmatrix
cdef gpumat *gptr = new gpumat(rc[0],rc[1],Complexmatrix)
cdef Gmat g = Gmat(gptr)
g.cgauss()
return(g)
-----------------------------------------
Unfortunately I get the following cryptic python error (warning it's
long).
[matt@myhost cudamex]$ python setup.py build_ext --inplace
running build_ext
cythoning pygpumat.pyx to pygpumat.cpp
Traceback (most recent call last):
File "setup.py", line 17, in <module>
cmdclass = {'build_ext': build_ext}
File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/lib/python2.6/distutils/dist.py", line 975, in
run_commands
self.run_command(cmd)
File "/usr/lib/python2.6/distutils/dist.py", line 995, in
run_command
cmd_obj.run()
File "/usr/lib/python2.6/distutils/command/build_ext.py", line 340,
in run
self.build_extensions()
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Distutils/build_ext.py", line 72, in
build_extensions
ext.sources = self.cython_sources(ext.sources, ext)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Distutils/build_ext.py", line 191, in
cython_sources
full_module_name=module_name)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Main.py", line 718, in compile
return compile_single(source, options, full_module_name)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Main.py", line 665, in compile_single
return run_pipeline(source, options, full_module_name)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Main.py", line 563, in run_pipeline
err, enddata = context.run_pipeline(pipeline, source)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Main.py", line 224, in run_pipeline
data = phase(data)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/ParseTreeTransforms.py", line 965, in
__call__
return super(AnalyseDeclarationsTransform,
self).__call__(root)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Visitor.py", line 276, in __call__
return super(CythonTransform, self).__call__(node)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Visitor.py", line 259, in __call__
return self.visit(root)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Visitor.py", line 28, in visit
return handler_method(obj)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/ParseTreeTransforms.py", line 973, in
visit_ModuleNode
node.analyse_declarations(self.env_stack[-1])
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/ModuleNode.py", line 63, in
analyse_declarations
self.body.analyse_declarations(env)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Nodes.py", line 341, in
analyse_declarations
stat.analyse_declarations(env)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Nodes.py", line 341, in
analyse_declarations
stat.analyse_declarations(env)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Nodes.py", line 396, in
analyse_declarations
self.body.analyse_declarations(env)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Nodes.py", line 341, in
analyse_declarations
stat.analyse_declarations(env)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Nodes.py", line 1050, in
analyse_declarations
attr.analyse_declarations(scope)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Nodes.py", line 939, in
analyse_declarations
api = self.api)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Symtab.py", line 1564, in
declare_cfunction
self.check_base_default_constructor(pos)
File "/home/matt/Downloads/cython-devel-d9a3831ec027/cython-devel-
d9a3831ec027/Cython/Compiler/Symtab.py", line 1537, in
check_base_default_constructor
if len(entry.type.base_classes) == 0:
AttributeError: 'CStructOrUnionType' object has no attribute
'base_classes'
-------------------
Perhaps there is something wrong with how I'm installing or using
cython?
On Apr 7, 1:36 pm, Robert Bradshaw <rober...@math.washington.edu>
wrote:
> On Apr 7, 2010, at 10:06 AM, SevenThunders wrote:
>
> > I am trying to wrap an existing C++ class in cython. I've had some
> > success in the past wrapping a C library, however I find the state of
> > the C++ documentation rather confusing. I believe there have been
> > some prior posts on this issue. My first question is what wrapping
> > syntax is actually suported in Cython 0.12.1? The online
> > documentation for version 0.12 has the 'older' syntax here,
> >http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html.
>
> > Some earlier posts suggest that this is deprecated and that we should
> > use the rather terse discussion here:
> >http://wiki.cython.org/WrappingCPlusPlus
>
> The new syntax is not available in Cython 0.12.1, but will be in
> Cython 0.13 which should be out soon (yes, soon has dragged on for a
> month or two more than we would have liked, but I think the issues
> uncovered by Sage have been resolved (Craig?) and as has been
> discussed, if we need to postpone closures due to a refcounting bug,
> so be it).
>
> > Although the effort to provide C++ support is greatly appreciated,
> > there are a few things missing from this documentation.
>
> Though it has been getting better, there is still *lot* of things
> missing from this documentation.
>
> > There is no discussion about how to define the extension class
>
> Seehttp://docs.cython.org/src/tutorial/cdef_classes.html
Robert Bradshaw
> OK well that makes sense then. I erroneously got the impression that
> there was stealth support
> for the new syntax.
>
> I downloaded a development version and attempted to use it to start
> wrapping a somewhat complex C++ class (that is fortunately no longer
> templated). I pointed my PYTHONPATH to the install directory and the
> linux PATH variable to the underlying bin directory. I ran into some
> issues, some of which I could overcome, but the final issue is
> baffling to me.
>
> First I found that cython doesn't seem to handle type declarations of
> the form std::string. Or at least whenever I used it as a type
> signature in a function call it complained of a syntax error.
You can't use C++ :: to denote namespaces, as :: has a completely
meaning in Python. Instead, use
cdef extern from "string" namespace std:
cdef cppclass string
(On that note, one thing that we don't yet provide is any nice Python
<-> C++ string conversion.)
> I then found that it doesn't like the ~ character in the destructor
> declaration.
Use del (which is they Python syntax). Again, the ~ operator already
has a meaning in Python.
FYI, Some good examples can be found at http://hg.cython.org/cython-devel/file/tip/tests/run/cpp_templates.pyx#l1
(and elsewhere in that directory). I think it's easier to take the
perspective that you're writing Python that can call C++ than to try
and think about writing C++ with Python syntax.
> Next, when wrapping my extension class, it's single data attribute
> is the pointer to the underlying C++ class. But how do you
> dereference that pointer in cython code? After all a lot of my
> methods take (const gpumat &) as the type signature. Well it seems
> you have to drop the const declaration
const support is a long-standing defect of Cython (though lower
priority than some)
> and then if you erroneously try
> to pass *gptr as an argument, it treats it as a star argument within
> python syntax, which I suppose is a kind of unbracket or list
> extraction operator (reasonable I suppose).
Yep.
> So I think you have to do something like self.gptr[0] as an argument.
That's exactly right. You can also import the actual dereference
operator (as a function) from cython.operator, see the above link.
This is required in case the * and [] have been overridden in
incompatible ways.
> I then found that cuda's dim3 struct type had to be declared
> explicitly for some reason.
Were you using it (or its members?)
> Finally, after doing all this I ran cython on this code:
Looks like you found a bug, I've made a ticket. http://trac.cython.org/cython_trac/ticket/523
Can you boil this down to a smaller example?
The new C++ support is a huge step forward, but there are still a lot
of bugs to shake out, as it needs a lot of testing. Thanks so much for
the feedback and hopefully we can get it working for you. There's
always the "old way" of doing things, but that's both deprecated and
clumsy.
- Robert
SevenThunders
--------------badmat.pyx ---------------
import numpy as np
cimport numpy as np
import sys
cdef extern from "gpumat.h" :
ctypedef struct dim3:
int x, y, z
ctypedef enum MatrixType :
Realmatrix = 0
Complexmatrix=1
Cuppermatrix = 2
ctypedef struct gpumat
void fromgpu(gpumat &gin, float *hval)
cdef cppclass gpumat:
int nrows
int ncols
int ld
MatrixType mtype
float *val
dim3 block
dim3 grid
dim3 residual
gpumat()
-----------------------------------------
The default constructor line gpumat() causes cython to complain.
By the way I left the numpy import directives in the .pyx file, but
they will generate a ton of errors when you actually start trying to
compile your .cpp file. (Just delete the last gpumat()) line to see
it. I seem to recall a statement in the documentation about C linkage
not really working for the C++ wrapper, although actually they are
compile time errors.
I think that deficiency may cause more problems than first thought, if
that's what's causing this error.
I suppose I should supply a .h file, but I can't give out my
proprietary gpumat.h, which is rather complex and it has a bunch of
dependencies on Nvidia's cuda drivers and libraries. I'll try to
create a minimal .h file and then add it to the bug tracker entry.
I guess I have a fundamental question about whether I will be able to
use the cython/numpy integration in my C++ wrapper. If not then I'll
have to go back to C linkage. As it is my C++ code is just a wrapper
for a bunch of C calls into some CUDA kernels. I do have a bunch of C+
+ code that uses my matrix class, but I suppose I can just create a C
wrapper for them if worst comes to worst.
What would be nice is if there was a SWIG, or maybe SIP translator
that translated C++ not directly to python, but rather to cython and
to a cython extension class to be used in python. It would solve some
of the speed deficiencies of the latter efforts and would permit
modifications/enhancements to the extension class at the cython level,
preserving some of the speed improvements.
In theory such a facility could use C linkage at the cython level,
though ultimately C++ integration will be superior. In my case it
would be desirable since my class probably has a hundred or more
methods and it will annoying wrapping them into a cdef'd class by
hand.
On Apr 7, 4:51 pm, Robert Bradshaw <rober...@math.washington.edu>
wrote:
> On Apr 7, 2010, at 1:19 PM, SevenThunders wrote:
>
> > OK well that makes sense then. I erroneously got the impression that
> > there was stealth support
> > for the new syntax.
>
> > I downloaded a development version and attempted to use it to start
> > wrapping a somewhat complex C++ class (that is fortunately no longer
> > templated). I pointed my PYTHONPATH to the install directory and the
> > linux PATH variable to the underlying bin directory. I ran into some
> > issues, some of which I could overcome, but the final issue is
> > baffling to me.
>
> > First I found that cython doesn't seem to handle type declarations of
> > the form std::string. Or at least whenever I used it as a type
> > signature in a function call it complained of a syntax error.
>
> You can't use C++ :: to denote namespaces, as :: has a completely
> meaning in Python. Instead, use
>
> cdef extern from "string" namespace std:
> cdef cppclass string
>
> (On that note, one thing that we don't yet provide is any nice Python
> <-> C++ string conversion.)
>
> > I then found that it doesn't like the ~ character in the destructor
> > declaration.
>
> Use del (which is they Python syntax). Again, the ~ operator already
> has a meaning in Python.
>
> FYI, Some good examples can be found athttp://hg.cython.org/cython-devel/file/tip/tests/run/cpp_templates.py...
> (and elsewhere in that directory). I think it's easier to take the
> perspective that you're writing Python that can call C++ than to try
> and think about writing C++ with Python syntax.
>
> > Next, when wrapping my extension class, it's single data attribute
> > is the pointer to the underlying C++ class. But how do you
> > dereference that pointer in cython code? After all a lot of my
> > methods take (const gpumat &) as the type signature. Well it seems
> > you have to drop the const declaration
>
> const support is a long-standing defect of Cython (though lower
> priority than some)
>
> > and then if you erroneously try
> > to pass *gptr as an argument, it treats it as a star argument within
> > python syntax, which I suppose is a kind of unbracket or list
> > extraction operator (reasonable I suppose).
>
> Yep.
>
> > So I think you have to do something like self.gptr[0] as an argument.
>
> That's exactly right. You can also import the actual dereference
> operator (as a function) from cython.operator, see the above link.
> This is required in case the * and [] have been overridden in
> incompatible ways.
>
> > I then found that cuda's dim3 struct type had to be declared
> > explicitly for some reason.
>
> Were you using it (or its members?)
>
> > Finally, after doing all this I ran cython on this code:
>
> Looks like you found a bug, I've made a ticket.http://trac.cython.org/cython_trac/ticket/523
Robert Bradshaw
Hmm.... One issue might be that you declare gpumat to be a struct
first, then a cpp class. In fact, removing that (and putting the
fromgpu declaration later) seems to get things working fine. Of course
we should catch this rather than crash. There does still seem to be
issues with predeclarations.
> By the way I left the numpy import directives in the .pyx file, but
> they will generate a ton of errors when you actually start trying to
> compile your .cpp file. (Just delete the last gpumat()) line to see
> it. I seem to recall a statement in the documentation about C linkage
> not really working for the C++ wrapper, although actually they are
> compile time errors.
C++ and numpy should work fine together--perhaps there are name
conflicts between gpumat.h and the numpy header files?
> I think that deficiency may cause more problems than first thought, if
> that's what's causing this error.
>
> I suppose I should supply a .h file, but I can't give out my
> proprietary gpumat.h, which is rather complex and it has a bunch of
> dependencies on Nvidia's cuda drivers and libraries. I'll try to
> create a minimal .h file and then add it to the bug tracker entry.
That would be very helpful.
> I guess I have a fundamental question about whether I will be able to
> use the cython/numpy integration in my C++ wrapper. If not then I'll
> have to go back to C linkage. As it is my C++ code is just a wrapper
> for a bunch of C calls into some CUDA kernels. I do have a bunch of
> C+
> + code that uses my matrix class, but I suppose I can just create a C
> wrapper for them if worst comes to worst.
Yep. Also, you can of course only bother wrapping the parts that you
need/can't otherwise get to work in C++.
> What would be nice is if there was a SWIG, or maybe SIP translator
> that translated C++ not directly to python, but rather to cython and
> to a cython extension class to be used in python. It would solve some
> of the speed deficiencies of the latter efforts and would permit
> modifications/enhancements to the extension class at the cython level,
> preserving some of the speed improvements.
>
> In theory such a facility could use C linkage at the cython level,
> though ultimately C++ integration will be superior. In my case it
> would be desirable since my class probably has a hundred or more
> methods and it will annoying wrapping them into a cdef'd class by
> hand.
Yep, providing some kind of wrapper/generator has been discussed
before and would be very nice, though there are a lot of design issues
to tackle here. Of course often the C/C++ API is very unpythonic and
so the manual wrapping allows one to focus on providing a more
pleasant to use front end.
- Robert
SevenThunders
declarations, though I am far from a C++ expert, hacker mainly. For
example my existing gpumat class has cupper as a subclass. Several
methods return a cupper type inside gpumat, ie the QR decomposition
method qr(). Several methods in cupper return or require knowledge
of gpumat. The easy way around this is to forward declare cupper.
Is there another way to forward declare classes or C++ methods in
Cython? I know the :: notation won't fly.
As for my numpy problems, I think they occur because it can't find
the numpy include files. I probably need to add them to the setup.py
distutils mechanism. So now to figure out where my distro has put my
python installation files...
On Apr 7, 7:30 pm, Robert Bradshaw <rober...@math.washington.edu>
wrote:
Robert Bradshaw
> There is often no simple way to get around the need for forward
> declarations, though I am far from a C++ expert, hacker mainly. For
> example my existing gpumat class has cupper as a subclass. Several
> methods return a cupper type inside gpumat, ie the QR decomposition
> method qr(). Several methods in cupper return or require knowledge
> of gpumat. The easy way around this is to forward declare cupper.
Forward declarations should work, and they work for structs, but
something seems to be (sometimes?) broken for cpp classes. I'd fix
this if I had time, but I simply don't right now.
> Is there another way to forward declare classes or C++ methods in
> Cython? I know the :: notation won't fly.
>
> As for my numpy problems, I think they occur because it can't find
> the numpy include files. I probably need to add them to the setup.py
> distutils mechanism.
That would do it.
> So now to figure out where my distro has put my
> python installation files...
That's where the numpy.get_include() method comes in handy, use it
right in your setup.py.
- Robert
SevenThunders
confusion occurs between a forward declared class and it's
constructor.
Unfortunately I now have another show stopper bug that has prevented
me from moving forward with this.
http://groups.google.com/group/cython-users/browse_thread/thread/817a6a8add296d11
I filed a bug report here:
http://trac.cython.org/cython_trac/ticket/524
There seems to be a whole class of bugs similar to this. If I could
figure a work around I might be able to move forward. Perhaps it's a
syntax error on my part...
On Apr 8, 12:53 am, Robert Bradshaw <rober...@math.washington.edu>
wrote:
Robert Bradshaw
> Just from the nature of the bug it almost seems as though some
> confusion occurs between a forward declared class and it's
> constructor.
>
> Unfortunately I now have another show stopper bug that has prevented
> me from moving forward with this.
> http://groups.google.com/group/cython-users/browse_thread/thread/817a6a8add296d11
>
> I filed a bug report here:
> http://trac.cython.org/cython_trac/ticket/524
>
> There seems to be a whole class of bugs similar to this. If I could
> figure a work around I might be able to move forward. Perhaps it's a
> syntax error on my part...
Looks like
badmat(unsigned i, unsigned j, unsigned nld)
was used but not defined. We *should* be raising an error here.
> --
> To unsubscribe, reply using "remove me" as the subject.
Robert Bradshaw
> On Apr 8, 2010, at 10:38 AM, SevenThunders wrote:
>
>> Just from the nature of the bug it almost seems as though some
>> confusion occurs between a forward declared class and it's
>> constructor.
>>
>> Unfortunately I now have another show stopper bug that has prevented
>> me from moving forward with this.
>> http://groups.google.com/group/cython-users/browse_thread/thread/817a6a8add296d11
>>
>> I filed a bug report here:
>> http://trac.cython.org/cython_trac/ticket/524
>>
>> There seems to be a whole class of bugs similar to this. If I could
>> figure a work around I might be able to move forward. Perhaps
>> it's a
>> syntax error on my part...
>
> Looks like
> badmat(unsigned i, unsigned j, unsigned nld)
> was used but not defined. We *should* be raising an error here.
http://hg.cython.org/cython-devel/rev/78e6dc767358
Looks like error reporting for C++ wrapping isn't being tested at all :
(.
- Robert
SevenThunders
Unfortunately for me though my original .pyx file did not have that
error, but it still generates an error that I can not reproduce right
now with a simple test case. It might require my somewhat complex and
proprietary include file gpumat.h. The include file compiles under
gcc just fine.
Here is the .pyx file I'm using so far
-------------- gpumat.pyx ---------------------------------
import numpy as np
cimport numpy as np
import sys
cdef extern from "gpumat.h" :
ctypedef struct dim3:
int x, y, z
ctypedef enum MatrixType :
Realmatrix = 0
Complexmatrix=1
Cuppermatrix = 2
cdef cppclass gpumat:
int nrows
int ncols
int ld
MatrixType mtype
float *val
dim3 block
dim3 grid
dim3 residual
gpumat()
unsigned iscomplex()
# end class gpumat
# std::ostream& operator<<(std::ostream& output, gpumat & p)
void fromgpu(gpumat &gin, float *hval)
# Python wrapper for a float pointer
cdef class FloatPtr :
cdef float *ptr
cdef setf(FloatPtr self, float *fp) :
self.ptr = fp
cdef float *getf(FloatPtr self) :
return self.ptr
cdef class Gmat:
cdef gpumat *gptr # C++ pointer instance
def __cinit__(self, int r, int c, int ld, int mt, FloatPtr f):
if (f.ptr == NULL) :
self.gptr = new gpumat(r,c, ld, <MatrixType> mt)
else:
self.gptr = new gpumat(r,c,ld, <MatrixType> mt,f.ptr)
def __dealloc__(self) :
del self.gptr
# obtain an ndarray copy
cpdef np.ndarray getarr(self) :
cdef np.ndarray nt
if (self.gp.iscomplex()) :
nt = np.empty(shape = (self.gp.nrows, self.gp.ncols),
dtype = np.complex64,order='Fortran')
else :
nt = np.empty(shape = (self.gp.nrows, self.gp.ncols),
dtype = np.float32,order='Fortran')
fromgpu(self.gptr[0], <float *>nt.data)
return(nt)
def __str__(self) :
cdef np.ndarray nt = self.getarr()
return nt.__str__()
cpdef Gmat cgauss(int r, int c) :
# cdef MatrixType mtype = Complexmatrix
cdef FloatPtr f
f.setf(NULL)
cdef Gmat g = Gmat(r,c,r,Complexmatrix, f)
g.gptr[0].cgauss()
return(g)
----------------------------------------------------
It cython's the .pyx file just fine and fails compiling the .cpp file
as follows:
$ python setup.py build_ext --inplace
running build_ext
cythoning pygpumat.pyx to pygpumat.cpp
building 'gpumat' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-
D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-
size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC -I. -I/usr/
include/python2.6 -I/usr/lib64/python2.6/site-packages/numpy/core/
include -I/usr/local/cuda/include -I/usr/local/cudasdk/common/inc -I/
usr/include/python2.6 -c pygpumat.cpp -o build/temp.linux-x86_64-2.6/
pygpumat.o
In file included from gpumat.h:27,
from pygpumat.cpp:175:
mempool.h: In function ‘AType* msalloc(size_t)’:
mempool.h:76: warning: comparison between signed and unsigned integer
expressions
In file included from pygpumat.cpp:175:
gpumat.h: In member function ‘void gpumat::getcumem()’:
gpumat.h:232: warning: unused variable ‘status’
In file included from pygpumat.cpp:175:
gpumat.h: In member function ‘cufftHandle gpumat::fftplan()’:
gpumat.h:687: warning: unused variable ‘cfr’
pygpumat.cpp: In function ‘PyObject* __Pyx_Import(PyObject*,
PyObject*)’:
pygpumat.cpp:4346: error: expected unqualified-id before ‘=’ token
pygpumat.cpp:4352: error: expected primary-expression before ‘=’ token
pygpumat.cpp:4353: error: expected primary-expression before ‘)’ token
pygpumat.cpp:4369: error: expected primary-expression before ‘,’ token
pygpumat.cpp:4373: error: expected primary-expression before ‘)’ token
pygpumat.cpp:4373: error: expected primary-expression before ‘)’ token
pygpumat.cpp:4373: error: expected primary-expression before ‘)’ token
pygpumat.cpp:4373: error: expected primary-expression before ‘)’ token
/usr/lib64/python2.6/site-packages/numpy/core/include/numpy/
__multiarray_api.h: At global scope:
/usr/lib64/python2.6/site-packages/numpy/core/include/numpy/
__multiarray_api.h:968: warning: ‘int _import_array()’ defined but not
used
/usr/lib64/python2.6/site-packages/numpy/core/include/numpy/
__ufunc_api.h:182: warning: ‘int _import_umath()’ defined but not used
pygpumat.cpp:1536: warning: ‘int
__pyx_pf_5numpy_7ndarray___getbuffer__(PyObject*, Py_buffer*, int)’
defined but not used
pygpumat.cpp:2344: warning: ‘void
__pyx_pf_5numpy_7ndarray___releasebuffer__(PyObject*, Py_buffer*)’
defined but not used
error: command 'gcc' failed with exit status 1
The section of the .cpp file with the error looks like this:
------------------pygpumat.cpp -------------------------
...
static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list) {
PyObject *__import__ = 0; // line 4346, the first offending line
PyObject *empty_list = 0;
PyObject *module = 0;
PyObject *global_dict = 0;
PyObject *empty_dict = 0;
PyObject *list;
__import__ = __Pyx_GetAttrString(__pyx_b, "__import__"); // line
4352
if (!__import__)
goto bad;
.....
----------------------------------------
On Apr 8, 10:55 pm, Robert Bradshaw <rober...@math.washington.edu>
wrote:
> On Apr 8, 2010, at 7:18 PM, Robert Bradshaw wrote:
>
>
>
> > On Apr 8, 2010, at 10:38 AM, SevenThunders wrote:
>
> >> Just from the nature of the bug it almost seems as though some
> >> confusion occurs between a forward declared class and it's
> >> constructor.
>
> >> Unfortunately I now have another show stopper bug that has prevented
> >> me from moving forward with this.
> >>http://groups.google.com/group/cython-users/browse_thread/thread/817a...
Lisandro Dalcin
> pygpumat.cpp: In function ‘PyObject* __Pyx_Import(PyObject*,
> PyObject*)’:
> pygpumat.cpp:4346: error: expected unqualified-id before ‘=’ token
> pygpumat.cpp:4352: error: expected primary-expression before ‘=’ token
>
>
> ------------------pygpumat.cpp -------------------------
> ...
> static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list) {
> PyObject *__import__ = 0; // line 4346, the first offending line
> PyObject *empty_list = 0;
Could you try to apply the patch below to Cython sources and give a
try? We should not use __import__ for a local variable name :-)
diff -r b442fb71c1de Cython/Compiler/ExprNodes.py
--- a/Cython/Compiler/ExprNodes.py Sat Mar 20 18:07:43 2010 +0100
+++ b/Cython/Compiler/ExprNodes.py Fri Apr 09 22:55:08 2010 -0300
@@ -6287,14 +6287,14 @@
""",
impl = """
static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list) {
- PyObject *__import__ = 0;
+ PyObject *py__import__ = 0;
PyObject *empty_list = 0;
PyObject *module = 0;
PyObject *global_dict = 0;
PyObject *empty_dict = 0;
PyObject *list;
- __import__ = __Pyx_GetAttrString(%(BUILTINS)s, "__import__");
- if (!__import__)
+ py__import__ = __Pyx_GetAttrString(%(BUILTINS)s, "__import__");
+ if (!py__import__)
goto bad;
if (from_list)
list = from_list;
@@ -6310,11 +6310,11 @@
empty_dict = PyDict_New();
if (!empty_dict)
goto bad;
- module = PyObject_CallFunctionObjArgs(__import__,
+ module = PyObject_CallFunctionObjArgs(py__import__,
name, global_dict, empty_dict, list, NULL);
bad:
Py_XDECREF(empty_list);
- Py_XDECREF(__import__);
+ Py_XDECREF(py__import__);
Py_XDECREF(empty_dict);
return module;
}
--
Lisandro Dalcin
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
SevenThunders
compiled. The patch appears to comment out a bunch of code, if I did
it correctly. The bad news is that my code segfaults. It segfaults
prior to doing anything interesting like running a kernel on my gpu
device. Rather it appears to fail when I attempt to assign a NULL to
a float pointer. In particular for the class that wraps the float
pointer, defined in cython like this:
---------------------------------
# Python wrapper for a float pointer
cdef class FloatPtr :
cdef float *ptr
cdef setf(FloatPtr self, float *fp) :
self.ptr = fp
cdef float *getf(FloatPtr self) :
return self.ptr
It segfaults when I make this cython call:
------------------------
cdef FloatPtr f
f.setf(NULL)
-----------------------------
Did I do something wrong here?
On Apr 9, 9:56 pm, Lisandro Dalcin <dalc...@gmail.com> wrote:
Robert Bradshaw
> The good news is that the patch allows the cythoned code to be
> compiled. The patch appears to comment out a bunch of code, if I did
> it correctly.
Excellent.
> The bad news is that my code segfaults. It segfaults
> prior to doing anything interesting like running a kernel on my gpu
> device. Rather it appears to fail when I attempt to assign a NULL to
> a float pointer. In particular for the class that wraps the float
> pointer, defined in cython like this:
> ---------------------------------
> # Python wrapper for a float pointer
> cdef class FloatPtr :
> cdef float *ptr
> cdef setf(FloatPtr self, float *fp) :
> self.ptr = fp
> cdef float *getf(FloatPtr self) :
> return self.ptr
> -----------------------------------
>
> It segfaults when I make this cython call:
> ------------------------
> cdef FloatPtr f
> f.setf(NULL)
>
> -----------------------------
> Did I do something wrong here?
When you do that, "f" here is None. You need to write
cdef FloatPtr f = FloatPtr()
to actually create the FloatPtr object to store the flaot*. All Python
objects are allocated in the heap (in other words, no magic
constructors and deconstructors are called when declaring things).
- Robert
Lisandro Dalcin
> On Apr 12, 2010, at 1:27 PM, SevenThunders wrote:
>
>> The good news is that the patch allows the cythoned code to be
>> compiled. The patch appears to comment out a bunch of code, if I did
>> it correctly.
>
> Excellent.
>
FYI, http://hg.cython.org/cython-devel/rev/fb26a68ec463
SevenThunders
is a very impressive tool IMHO. My simple wrapper start is actually
working Hoorah!
I just have a couple of comments with regards to some pitfalls I ran
into in case it may help someone else.
Prior to carefully reading Robert's fix to my attempt to wrap a
pointer I decided that my problem was a lack of a __cinit__
initializer in FloatPtr(). So I did this:
------------------------
# Python wrapper for a float pointer
cdef class FloatPtr :
cdef float *ptr
def __cinit__(self) :
print "Initializing pointer to NULL"
self.ptr = NULL
cdef setf(FloatPtr self, float *fp) :
print "Setting pointer to %ld" % <long int>fp
self.ptr = fp
cdef float *getf(FloatPtr self) :
return self.ptr
-----------------------
This compiled and ran for a ways without segfaulting, but in fact I
was not initializing my pointer to null
when I did this.
---------------------------
cdef FloatPtr f = FloatPtr
--------------------------
It failed to have the () in the declaration.
One issue that was driving me crazy was the need to write code that
looks like this
self.gptr[0].nrows or self.gptr[0].method(arg1, arg2) etc. It would
be nice to have some pythonic syntactic sugar to fix this.
My suggestion is to create an automated way to wrap pointers in a fake
Python class, similar in functionality to the FloatPtr class above,
but with some syntactic sugar. In particular I don't want to write
gptr[0].nrows, I want to write gptr.nrows, which would be possible
if gptr were a 'class'. For the actual C/C++ code that cython
generates one could strip away the fake class and just generate gptr-
>nrows. For a Python extension one could wrap it an actual Python
class as I had to wrap FloatPtr. (I guess depending on whether you
have a cdef or a cpdef.) To generate this type perhaps one could
declare it like Ptr<float *> fptr or Ptr[float *] fptr, or
something more Pythonic, Ptr or something like it being a keyword in
cython.
This would come in handy for all the other times one has to wrap C++
or C pointers in a python class within Cython and it would certainly
be convenient if I can wrap the class pointers inside such a
construct. Thinking further ahead, if one could then extend the
functionality of the wrapper in cython..... hmmm, it's a micro step
towards automating some of the more basic aspects of the wrapper
python class.
On Apr 12, 5:01 pm, Robert Bradshaw <rober...@math.washington.edu>
wrote:
Lisandro Dalcin
>
> My suggestion is to create an automated way to wrap pointers in a fake
> Python class, similar in functionality to the FloatPtr class above,
> but with some syntactic sugar.
Do you have any syntax in mind?
> In particular I don't want to write
> gptr[0].nrows, I want to write gptr.nrows, which would be possible
> if gptr were a 'class'.
cdef XXX *gptr = self.gptr
gptr.rows ...
gptr.cols ....
Just a single extra line (though you would need it in every [c][p]def method)...
PS: This works because Cython converts gptr.nrows in Cython code to
gptr->rows in the generated C code.
SevenThunders
On Apr 12, 5:04 pm, Lisandro Dalcin <dalc...@gmail.com> wrote:
> On 12 April 2010 18:01, Robert Bradshaw <rober...@math.washington.edu> wrote:
>
> > On Apr 12, 2010, at 1:27 PM, SevenThunders wrote:
>
> >> The good news is that the patch allows the cythoned code to be
> >> compiled. The patch appears to comment out a bunch of code, if I did
> >> it correctly.
>
> > Excellent.
>
> FYI,http://hg.cython.org/cython-devel/rev/fb26a68ec463
>
Was this link supposed to have the corrected ExprNodes.py? I
downloaded a later snapshot from mercurial and got the same error,
and the error persists even when using the
raw file copy from the link you provided. I was trying to install it
on another computer.
I'll have to get my hands on my patched version on another machine and
see what the differences are.
Robert Bradshaw
You should be able to simply write self.gptr.nrows, etc. right now.
> My suggestion is to create an automated way to wrap pointers in a fake
> Python class, similar in functionality to the FloatPtr class above,
> but with some syntactic sugar. In particular I don't want to write
> gptr[0].nrows, I want to write gptr.nrows, which would be possible
> if gptr were a 'class'. For the actual C/C++ code that cython
> generates one could strip away the fake class and just generate gptr-
>> nrows. For a Python extension one could wrap it an actual Python
> class as I had to wrap FloatPtr. (I guess depending on whether you
> have a cdef or a cpdef.) To generate this type perhaps one could
> declare it like Ptr<float *> fptr or Ptr[float *] fptr, or
> something more Pythonic, Ptr or something like it being a keyword in
> cython.
>
> This would come in handy for all the other times one has to wrap C++
> or C pointers in a python class within Cython and it would certainly
> be convenient if I can wrap the class pointers inside such a
> construct. Thinking further ahead, if one could then extend the
> functionality of the wrapper in cython..... hmmm, it's a micro step
> towards automating some of the more basic aspects of the wrapper
> python class.
Yes, there's a huge amount that can be done towards automatically
wrapping stuff. In terms of pointers in general, one of the questions
is that of memory management. Usually people use NumPy when they want
a Python-level float*, which is probably why no one's done it yet.
- Robert
SevenThunders
CUDA. The GPU memory has pointers that are
'wrapped' as float pointers on the host CPU. Thus those float
pointers can not be used in numpy or
by anything else except for the underlying gpu device.
I was talking about a general capability, an automated way to wrap
pointers of any kind. Probably C++ class pointers would be the
primary use.
If I can already use self.gptr.nrows as is, without doing
self.gptr[0].nrows, then I suppose the syntactic sugar
issue is already addressed, assuming that that interpretation will
remain in future versions of cython.
However I can imagine a lot of utility for a generic pointer wrapper
depending on how far you want to take it.
Wouldn't it be nice at the Python level if you could index them,
compare them, use them for iterators, set them, resize them, and
at the cython level extend them. If you automatically wrapped a class
pointer inside one of these and then gave the programmer the ability
to extend them, it might be a bit easier to write the extension
class. One could then incrementally add capabilities as cython
progresses. For example a nice initial
capability would be to automatically wrap declared constructors and
provide a delete method to be called when the reference count goes to
zero
as the current the current __dealloc__ attribute does.
If I understand what you are saying, when someone declares
cdef Foo *Fptr
Fptr.attr is equivalent to Fptr->attr in the generated C++ code. If
we want access to Fptr from python we have to wrap it ourselves of
course.
If we had a way of automatically generating a minimal wrapper template
for a given pointer I think that would be pretty slick, especially if
we
had to pass pointers around to other functions in Python. I'm not
sure what the most consistent syntax for this would be. Just thinking
out loud here:
cpdef Foo *Fptr
Does this even compile right now if Foo is a declared C++ class? What
sort of object does this become in Python? My suggestion is to
autocreate a cython class
for pointers in general, that can be further extended somehow. So
maybe something like
cpdef Ptr[Foo] Fptr(arg1, arg2, arg3) or
cpdef Ptr Fptr = Foo(arg1, arg2, arg3) or
cpdef Ptr[Foo] Fptr = Foo(arg1, arg2, arg3) or
cpdef Fptr = Ptr(Foo(arg1,arg2,arg3))
The cpdef would permit the class to be accessed the same way Fptr is
within Cython, but for Python creates a skeleton class with some
minimal functionality (at first). The latter
approach is intriguing because it suggests a generic container class
whose initial argument is the pointer/reference to be wrapped. To
support something like Fptr.method1(),
if method1 is actually a method of the underlying C++ class is a
matter of adding that attribute for the existing autocreated python
class. For cython of course one would always expand
Fptr.method1 to Fptr->method1.
To extend it one would use something like
cpdef Ptr[Foo] :
cdef int attr1
def method1(self) :
....
Again the idea here is that as soon as one declares a Ptr[Foo]:
'class' some minimal functionality is already generated at the Python
level. Say a default constructor and destructor, increment and
decrement? maybe even a bracket operator if Foo is a fundamental type
and so forth. Perhaps the attributes exposed would be required to
have the same signature as the C++ code. Perhaps one automatically
links to that code just by declaring the signature. The current
constraint for a lot of that is that one can not pass arbitrary C++
constructs and types around in Python. But if you automatically
wrapped pointers for use in Python the problem kind of goes away.
You should be able to pass anything once it's been auto-wrapped.
The class name in Python space for the above example would be Foo.
The Ptr[Foo] syntax or something like it, is just suggestive of some
kind of templating operation for creating generic code. The
autowrapped pointer idea might also be useful for situations wherein
you can not use python functionality because it's essentially C/C++
code in cython. One construct I often have to use for debugging is
something like this
cdef np.ndarray arr
arr = .... stuff
print (<object> arr).shape
A more pythonic and unifying approach would be to do
print Ptr(arr).shape
if the magical pointer container class could be created.
I'm just throwing out a few ideas for thought here.
On Apr 13, 2:52 pm, Robert Bradshaw <rober...@math.washington.edu>
wrote:
SevenThunders
groups seemed to eat my prior response.
The float pointer I'm wrapping is what's called a 'device pointer' in
CUDA. It has no meaning outside of the
GPU device that it was created for.
If I understanding your meaning correctly if I declare
cdef Foo *fptr
for a cppclass Foo, I can then freely use ptr.attr in lieu of ptr-
>attr or ptr[0].attr? That would solve the syntax sugar issue.
What I was alluding too earler was an idea for automating minimal
functionality in some kind of pointer wrapper. What if we want to
carry around a Foo pointer in Python even if it's
to just pass into another function? I'm assuming that we can't do
cpdef Foo *fptr
since *ptr is not Python syntax or more precisely pointer to Foo is
not a defined python type. What if cython supported a generic pointer
class Ptr with some 'magic' syntax. Say something like
cpdef *fptr = Foo(a1,a2,a3)
where a1, a2, a3 are constructor arguments. In this case the pointer
nature of fptr tells Cython we want to create an inherited class Foo
that extends some base pointer wrapper class (say Ptr).
Maybe we indicate this by
cpdef *fptr = Ptr(Foo(a1,a2,a3)). In the latter syntax Ptr would
return a class Foo that inherits a base pointer class with some
minimal functionality.
For cython code we'd want fptr.attr to expand to fptr->attr in C++ or
C.
In Python we'd want to use fptr.attr with minimum fuss. Some extra
syntax could support extending the base vanilla Ptr class. Perhaps at
a minimum it generates a default __cinit__ and
__dealloc__ using a default constructor and destructor which could be
overridden. We'd therefore want to be able to write something like:
def cppclass Foo(Ptr) :
cdef int method1(float a1, char *a2, int a3)
def void method2(float a1, char * a2)
The declaration of method1 without a body should generate a
corresponding call to the C++ method. The declaration of method 2
should also generate a connection to the C++ method method2. It would
presume that char * has already been wrapped with the default pointer
wrapper so that it can still be called from Python. The syntax might
need some altering a bit for consistency, especially if you were going
to automatically generate code.
Just some thoughts.
On Apr 13, 2:52 pm, Robert Bradshaw <rober...@math.washington.edu>
wrote:
Robert Bradshaw
> My code wraps an interface to some GPU accelerated kernels that use
> CUDA. The GPU memory has pointers that are
> 'wrapped' as float pointers on the host CPU. Thus those float
> pointers can not be used in numpy or
> by anything else except for the underlying gpu device.
>
> I was talking about a general capability, an automated way to wrap
> pointers of any kind. Probably C++ class pointers would be the
> primary use.
I think you're conflating two different ideas here--the fact that C/C+
+ uses pointers for both arrays and indirection is (to me at least)
more of implementation detail, though, of course, this being C the
boundaries are quite fuzzy.
I also think it would be useful to have a native array type, something
that behaves like a Python list but is backed by a float* or int* or
whatever. Ideally, this could be used to eliminate the need for users
to manually use malloc/free in 90% of the use cases. (For your use
case, of course, you would be setting the GPU pointer manually.)
Memory ownership can get messy. We already have NumPy and http://docs.python.org/c-api/cobject.html
, and of course bytes/str for char*, but I think there's a case to
be made for something in-between. In this case I'm not sure how much
use it would be to "extend" such an object--it would mostly be used
implicitly when converting to/from Python. We also try to avoid adding
a lot of magic syntax, as that is one of the things that makes C++ so
hard to learn (and read!) compared to, say, Python. Type parameters
for cdef classes would also go a long way towards a solution to this
issue.
The other main idea I see here is auto-generation of C++ class
wrapping, which is orthogonal to the fact that they're "pointer"
types. Some or all the methods could be available from Python and
Cython, initializers and deallocators could be set up nicely, etc. How
much boxing/unboxing and indirection could be eliminated depends on
how subclassing would work (C++, Cython, and pure Python subclasses).
Wrapper-generators are a very interesting and large topic, and could
almost be done independently of the Cython compiler itself, at least
as a first pass. (Imagine Boost.Cython or SWIG generating .pyx or .pxd
files, though perhaps we would want to do things differently.)
- Robert
SevenThunders
>
> I think you're conflating two different ideas here--the fact that C/C+
> + uses pointers for both arrays and indirection is (to me at least)
> more of implementation detail, though, of course, this being C the
> boundaries are quite fuzzy.
I am conflating two ideas here. But of course as you point out so
does C++.
>
> I also think it would be useful to have a native array type, something
> that behaves like a Python list but is backed by a float* or int* or
> whatever. Ideally, this could be used to eliminate the need for users
> to manually use malloc/free in 90% of the use cases. (For your use
> case, of course, you would be setting the GPU pointer manually.)
> Memory ownership can get messy. We already have NumPy andhttp://docs.python.org/c-api/cobject.html
> , and of course bytes/str for char*, but I think there's a case to
> be made for something in-between. In this case I'm not sure how much
> use it would be to "extend" such an object--it would mostly be used
> implicitly when converting to/from Python.
I agree that's where you want to be, but I was thinking of a kind of
spiral development technique for getting there, in the form of some
kind of minimal auto-wrap functionality. The first obvious case is to
just auto-wrap the pointer type. If one already has a PyCobject type,
then perhaps that's the infrastructure to use.
If you just provide minimal functionality however, then you want the
user to be able to fill in the functionality as needed.
We also try to avoid adding
> a lot of magic syntax, as that is one of the things that makes C++ so
> hard to learn (and read!) compared to, say, Python. Type parameters
> for cdef classes would also go a long way towards a solution to this
> issue.
It certainly does. C++ get's difficult because of an overly strict
compiler, a set of incomprehensible and arcane rules, a flawed
generic programming infrastructure <templates> and an attempt to throw
everything including the kitchen sink into the language. If you are
going wrap this language you can't avoid a little ugliness.
>
> The other main idea I see here is auto-generation of C++ class
> wrapping, which is orthogonal to the fact that they're "pointer"
> types. Some or all the methods could be available from Python and
> Cython, initializers and deallocators could be set up nicely, etc. How
> much boxing/unboxing and indirection could be eliminated depends on
> how subclassing would work (C++, Cython, and pure Python subclasses).
> Wrapper-generators are a very interesting and large topic, and could
> almost be done independently of the Cython compiler itself, at least
> as a first pass. (Imagine Boost.Cython or SWIG generating .pyx or .pxd
> files, though perhaps we would want to do things differently.)
>
Actually that would be pretty powerful. Doesn't Python already have a
couple of interesting metacompiler libraries? I'm not sure if these
are really meta compilers ala Meta II but perhaps some standard
templates using these tools, that generated cython code would do the
trick.
http://jinja.pocoo.org/2/documentation/intro
http://www.cheetahtemplate.org/
These are some of the ones used in PyCuda, but I'm not really
familiar with them.
Lisandro Dalcin
>
>
> On Apr 12, 5:04 pm, Lisandro Dalcin <dalc...@gmail.com> wrote:
>> On 12 April 2010 18:01, Robert Bradshaw <rober...@math.washington.edu> wrote:
>>
>> > On Apr 12, 2010, at 1:27 PM, SevenThunders wrote:
>>
>> >> The good news is that the patch allows the cythoned code to be
>> >> compiled. The patch appears to comment out a bunch of code, if I did
>> >> it correctly.
>>
>> > Excellent.
>>
>> FYI,http://hg.cython.org/cython-devel/rev/fb26a68ec463
>>
>
> Was this link supposed to have the corrected ExprNodes.py? I
> downloaded a later snapshot from mercurial and got the same error,
> and the error persists even when using the
> raw file copy from the link you provided. I was trying to install it
> on another computer.
>
Sorry, at this point I do not know the status of this... Does a
checkout of cython-devel repo works or not for your?
--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169
SevenThunders
forgot to touch my source code and it was picking up the old .cpp
file. It seems to be working now. Thanks again for getting a patch in
so quickly.
If given some time on Monday I might try to see if it will run under
windows as well, and I plan to start wrapping some more complex stuff
from my source, including some operator overloads and so forth.
On Apr 14, 11:41 pm, Lisandro Dalcin <dalc...@gmail.com> wrote:
Subscription settings: http://groups.google.com/group/cython-users/subscribe?hl=en