I am attempting to wrap the Mesh classes in [http://libmesh.sourceforge.net/ libMesh] with Cython. These classes define different iterators to sequentially treat different elements of the Mesh and these iterators are returned by accessor methods. I am unable to write an iteration loop in Cython because I cannot initialize any of these iterators on the stack.
The actual definitions in libMesh are a morass of "Templated forwarding ctor" scattered across a half-dozen headers, but it seems to distill to this:
== accessor.h ==
{{{
#!cpp
class A
{
public:
A(int i);
When I attempt to run test_accessor.py, g++ fails with:
{{{
#!cpp
/Users/guyer/Documents/research/FiPy/fipy/node_iterator/accessor.cpp: In function ‘PyObject* __pyx_pf_8accessor_3PyB_2do_something_with_my_A(__pyx_obj_8accessor_PyB*)’:
/Users/guyer/Documents/research/FiPy/fipy/node_iterator/accessor.cpp:542: error: no matching function for call to ‘A::A()’
/Users/guyer/Documents/research/FiPy/fipy/node_iterator/accessor.h:4: note: candidates are: A::A(int)
/Users/guyer/Documents/research/FiPy/fipy/node_iterator/accessor.h:2: note: A::A(const A&)
}}}
libMesh itself calls these accessor methods to stack allocate and initialize these iterators all the time, but I see the difference is that Cython generates:
If I change the Cython to
{{{
#!python
cdef A a = self.thisptr.my_A()
}}}
I get the somewhat more informative
{{{
accessor.pyx:14:15: C++ class must have a no-arg constructor to be stack allocated
}}}
but that doesn't address the fact that the the equivalent initialization is legal C++.
More significantly, in the actual code, where `node_iterator` is declared as a subclass of `Mesh`
{{{
#!python
cdef Mesh.node_iterator it = self.thisptr.active_nodes_begin()
}}}
results in
{{{
libMeshMesh.pyx:117:13: 'Mesh' is not a cimported module
On Mon, Oct 22, 2012 at 9:18 AM, Jon Guyer <gu...@nist.gov> wrote:
> I am attempting to wrap the Mesh classes in [http://libmesh.sourceforge.net/ > libMesh] with Cython. These classes define different iterators to
> sequentially treat different elements of the Mesh and these iterators are
> returned by accessor methods. I am unable to write an iteration loop in
> Cython because I cannot initialize any of these iterators on the stack.
> The actual definitions in libMesh are a morass of "Templated forwarding
> ctor" scattered across a half-dozen headers, but it seems to distill to
> this:
[snip]
> If I change the Cython to
> {{{
> #!python
> cdef A a = self.thisptr.my_A()
> }}}
> I get the somewhat more informative
> {{{
> accessor.pyx:14:15: C++ class must have a no-arg constructor to be stack
> allocated
> }}}
Which is the error we should be throwing.
> but that doesn't address the fact that the the equivalent initialization is legal C++.
The problem with the one-line solution is that it requires careful
control over the scope in which variables are declared vs.
initialized. Every bracket in C++ defines a new scope, which is not
the case in Python/Cython, hence the restriction (e.g. variables
assigned in a loop body can be used outside of it). Though it's simple
to make it work by manually tweaking the C in this case, the generic
case is much more complicated (and potentially ends up leaking the
scope constraints to the Python level).
Of course, this is a bug that should be fixed, but in the meantime
you'll have to allocate this as a pointer (which is what we'd have to
do under the hood to support this).
BTW, in Cython 0.17 you can iterate over C++ iterators using "for x in
myInstance: ..." which is much cleaner.
> More significantly, in the actual code, where `node_iterator` is declared as
> a subclass of `Mesh`
> {{{
> #!python
> cdef Mesh.node_iterator it = self.thisptr.active_nodes_begin()
> }}}
> results in
> {{{
> libMeshMesh.pyx:117:13: 'Mesh' is not a cimported module
> }}}
On Oct 22, 2012, at 9:32 PM, Robert Bradshaw wrote:
> On Mon, Oct 22, 2012 at 9:18 AM, Jon Guyer <gu...@nist.gov> wrote:
>> I am unable to write an iteration loop in
>> Cython because I cannot initialize any of these iterators on the stack.
>> but that doesn't address the fact that the the equivalent initialization is legal C++.
> The problem with the one-line solution is that it requires careful
> control over the scope in which variables are declared vs.
> initialized. Every bracket in C++ defines a new scope, which is not
> the case in Python/Cython, hence the restriction (e.g. variables
> assigned in a loop body can be used outside of it). Though it's simple
> to make it work by manually tweaking the C in this case, the generic
> case is much more complicated (and potentially ends up leaking the
> scope constraints to the Python level).
Thanks very much for your thorough explanation. I had not appreciated the broader issue you were tangling with. I guess my inclination would be to generate legal C++ code, limiting the scope of the variables from what Python would otherwise allow. This is a hybrid language after all and some constraints are inevitable. You see the implications of these choices much more clearly than I do, though.
> Of course, this is a bug that should be fixed, but in the meantime
> you'll have to allocate this as a pointer (which is what we'd have to
> do under the hood to support this).
OK, I had tried that, but wasn't getting it right. libMesh only returns its iterators by value and I was having trouble figuring out how to get a pointer out of that. I had tried to do things like
{{{
cdef A *a_ptr = &self.thisptr.my_A()
}}}
and
{{{
cdef A *a_ptr deref(a_ptr) = self.thisptr.my_A()
}}}
neither of which I really expected to work, and which I knew were dangerous constructs in C++ if they did.
I now see that
{{{
cdef A *a_ptr = new A(self.thisptr.my_A())
}}}
seems to do the trick.
Unfortunately, I'm then stymied by the "'Mesh' is not a cimported module" issue below when I try to do it for real.
> BTW, in Cython 0.17 you can iterate over C++ iterators using "for x in
> myInstance: ..." which is much cleaner.
I agree that that's a much nicer, more Pythonic way to do it. Unfortunately, the libMesh API doesn't support it.
I initially tried
{{{
for it in self.thisptr.active_nodes_begin():
}}}
but that raises
{{{
libMeshMesh.pyx:121:18: missing begin() on node_iterator
}}}
I then realized that you said myInstance and not myIterator, but the problem is that a libMesh Mesh has multiple iterators over both its nodes and its elements:
A Mesh has no explicit begin() and end() and they would be completely ambiguous if it did. I realize that the semantics might not be quite right, but for this use case, I'd really like to be able to write:
{{{
for it in self.thisptr.active_nodes_begin():
}}}
>> More significantly, in the actual code, where `node_iterator` is declared as
>> a subclass of `Mesh`
>> {{{
>> #!python
>> cdef Mesh.node_iterator it = self.thisptr.active_nodes_begin()
>> }}}
>> results in
>> {{{
>> libMeshMesh.pyx:117:13: 'Mesh' is not a cimported module
>> }}}
On Tue, Oct 23, 2012 at 5:33 AM, Jonathan Guyer <gu...@nist.gov> wrote:
> On Oct 22, 2012, at 9:32 PM, Robert Bradshaw wrote:
>> On Mon, Oct 22, 2012 at 9:18 AM, Jon Guyer <gu...@nist.gov> wrote:
>>> I am unable to write an iteration loop in
>>> Cython because I cannot initialize any of these iterators on the stack.
>>> but that doesn't address the fact that the the equivalent initialization is legal C++.
>> The problem with the one-line solution is that it requires careful
>> control over the scope in which variables are declared vs.
>> initialized. Every bracket in C++ defines a new scope, which is not
>> the case in Python/Cython, hence the restriction (e.g. variables
>> assigned in a loop body can be used outside of it). Though it's simple
>> to make it work by manually tweaking the C in this case, the generic
>> case is much more complicated (and potentially ends up leaking the
>> scope constraints to the Python level).
> Thanks very much for your thorough explanation. I had not appreciated the broader issue you were tangling with. I guess my inclination would be to generate legal C++ code, limiting the scope of the variables from what Python would otherwise allow. This is a hybrid language after all and some constraints are inevitable. You see the implications of these choices much more clearly than I do, though.
We certainly shouldn't be generating invalid C++ code. As for limiting
the scope, we try to follow Python semantics as close as possible, and
it would likely be confusing for the scope to ill-defined based on the
underlying implementation (which a user only has indirect control
over).
Likely what we'll eventually do is fake stack allocation.
>> Of course, this is a bug that should be fixed, but in the meantime
>> you'll have to allocate this as a pointer (which is what we'd have to
>> do under the hood to support this).
> OK, I had tried that, but wasn't getting it right. libMesh only returns its iterators by value and I was having trouble figuring out how to get a pointer out of that. I had tried to do things like
> {{{
> cdef A *a_ptr = &self.thisptr.my_A()
> }}}
> and
> {{{
> cdef A *a_ptr
> deref(a_ptr) = self.thisptr.my_A()
> }}}
> neither of which I really expected to work, and which I knew were dangerous constructs in C++ if they did.
> I now see that
> {{{
> cdef A *a_ptr = new A(self.thisptr.my_A())
> }}}
> seems to do the trick.
> Unfortunately, I'm then stymied by the "'Mesh' is not a cimported module" issue below when I try to do it for real.
Perhaps a more complete example would help with understanding this error.
>> BTW, in Cython 0.17 you can iterate over C++ iterators using "for x in
>> myInstance: ..." which is much cleaner.
> I agree that that's a much nicer, more Pythonic way to do it. Unfortunately, the libMesh API doesn't support it.
> I initially tried
> {{{
> for it in self.thisptr.active_nodes_begin():
> }}}
> but that raises
> {{{
> libMeshMesh.pyx:121:18: missing begin() on node_iterator
> }}}
> I then realized that you said myInstance and not myIterator, but the problem is that a libMesh Mesh has multiple iterators over both its nodes and its elements:
> A Mesh has no explicit begin() and end() and they would be completely ambiguous if it did. I realize that the semantics might not be quite right, but for this use case, I'd really like to be able to write:
> {{{
> for it in self.thisptr.active_nodes_begin():
> }}}
Yeah, but then it won't have the "end" to know where to stop. Perhaps
we could resurrect the for...from syntax for this:
for it from self.thisptr.active_nodes_begin() <= it <
self.thisptr.active_noces_end():
...
Another solution is to create a simple wrapper class ActiveNodes whose
end() and begin() method invoke active_nodes_begin, so you could write
>>> More significantly, in the actual code, where `node_iterator` is declared as
>>> a subclass of `Mesh`
>>> {{{
>>> #!python
>>> cdef Mesh.node_iterator it = self.thisptr.active_nodes_begin()
>>> }}}
>>> results in
>>> {{{
>>> libMeshMesh.pyx:117:13: 'Mesh' is not a cimported module
>>> }}}
Oh, that does look like a bug. All our tests of nested classes use
template classes (e.g. vector) so I guess this regression was never
detected. I sense a bugfix release coming up...
{{{
#!python
cdef class PyMesh:
cdef Mesh *thisptr
def __cinit__(self, unsigned int dim=1):
self.thisptr = new Mesh(dim)
def print_nodes(self):
cdef Node *it
for it in ActiveNodes(self.thisptr):
pass
}}}
and g++ raises
{{{
/Users/guyer/Documents/research/FiPy/fipy/fipy/meshes/libMesh/libMeshMesh.c pp: In function ‘PyObject* __pyx_pf_11libMeshMesh_6PyMesh_8print_nodes(__pyx_obj_11libMeshMesh_PyMesh* )’:
/Users/guyer/Documents/research/FiPy/fipy/fipy/meshes/libMesh/libMeshMesh.c pp:2060: error: no matching function for call to ‘libMesh::MeshBase::node_iterator::node_iterator()’
/Users/guyer/Downloads/libmesh-0.7.3.2/libmesh/include/mesh/mesh_base.h:881 : note: candidates are: libMesh::MeshBase::node_iterator::node_iterator(const libMesh::MeshBase::node_iterator&)
}}}
I'm back where I started because node_iterator doesn't have a no-arg constructor.
> libMesh::node_iterator __pyx_t_1 = __pyx_t_2->begin();
}}}
> Oh, that does look like a bug. All our tests of nested classes use
> template classes (e.g. vector) so I guess this regression was never
> detected. I sense a bugfix release coming up...
>> Oh, that does look like a bug. All our tests of nested classes use
>> template classes (e.g. vector) so I guess this regression was never
>> detected. I sense a bugfix release coming up...
>>> Oh, that does look like a bug. All our tests of nested classes use
>>> template classes (e.g. vector) so I guess this regression was never
>>> detected. I sense a bugfix release coming up...
In the interim, I was able to work around the issue of not being able to refer to a nested class by cdef'ing `node_iterator` outside the cdef for `Mesh`:
