Extension types / factory patterns for Python3

Phil Osteen

unread,

Jun 14, 2021, 1:20:18 AM6/14/21

to cython-users

I am not sure if the following is flawed but this pattern has worked for Python2 but is now breaking when changing to Python3 compilation.

# foo.h

``` c

#ifndef FOO_H

#define FOO_H

namespace foo {

class Foo {

public:

Foo() {};

~Foo() {};

};

}

#endif

```

# mwe_test.pxd

``` python

cdef extern from "foo.h" namespace "foo":

cdef cppclass Foo:

Foo() except +

cdef class CyFoo:

cdef Foo *foo_cpp

@staticmethod

cdef create(Foo *f)

```

# mwe_test.pyx

``` python

import cython

from mwe_test cimport *

cdef class CyFoo:

@staticmethod

cdef create(Foo *f):

cdef CyFoo f_wrap = CyFoo()

f_wrap.foo_cpp = f

return f_wrap

```

When running within a python2 environment, no errors in generating `cxx` files.

``` sh

cython --version # Cython version 0.29.21

cython --cplus -2 --output-file /tmp/mwe/mwe_test.cxx -I /tmp/mwe/src/mwe /tmp/mwe/src/mwe/mwe_test.pyx

```
When running the same thing from python3, I get an error:

``` sh

cython --version # Cython version 0.29.23

cython --cplus -3 --output-file /tmp/mwe/mwe_test.cxx -I /tmp/mwe/src/mwe /tmp/mwe/src/mwe/mwe_test.pyx

# Error compiling Cython file:

# src/mwe/mwe_test.pxd:8:16: C method 'create' is declared but not defined

```

Thanks for any tips

D Woods

unread,

Jun 14, 2021, 1:29:26 PM6/14/21

to cython-users

It seems to be working for me on 0.29.23 with both Python 2 and Python 3. The only things to note are:

1) that I'm running Cython from the same directory as the source rather than providing the file path.

2) I had to fix the indentation of `Foo() except +`

So I'm a bit puzzled.

The staticmethod `create` pattern is well-established and I think in the docs, so it should work. It's often useful for when you need C arguments (such as pointers)

Phil Osteen

unread,

Jun 15, 2021, 2:07:10 AM6/15/21

to cython-users

That got me wondering if my configuration is somehow broken; here's a slightly expanded example using Docker that should be fully reproducible. The slight expansion includes another error that also only manifested in python3.

The file structure is:

├── Dockerfile-py2

├── Dockerfile-py3

└── mwe

└── src

├── foo.h

└── mwe

├── __init__.py

├── mwe_test.pxd

└── mwe_test.pyx

And the associated commands to test are:

```

docker build -t py2_cython . -f Dockerfile-py2

docker build -t py3_cython . -f Dockerfile-py3

docker run py2_cython # works

docker run py3_cython # fails
```

Dockerfile-py2:

```

FROM python:2.7

RUN pip install Cython==0.29.23

WORKDIR /mwe

COPY mwe /mwe

CMD ["/bin/bash", "-c", "cython --cplus -2 --output-file mwe_test.cxx -I /mwe/src/mwe/ /mwe/src/mwe/mwe_test.pyx"]

```

Dockerfile-py3:

```

FROM python:3.6

RUN pip install Cython==0.29.23

WORKDIR /mwe

COPY mwe /mwe

CMD ["/bin/bash", "-c", "cython --cplus -3 --output-file mwe_test.cxx -I /mwe/src/mwe/ /mwe/src/mwe/mwe_test.pyx"]

```

foo.h:

```

#ifndef FOO_H

#define FOO_H

namespace foo {

class Foo {

public:

Foo() {f_ = 0;}

Foo(int f) {f_ = f;}

~Foo() {}

int f_;

};

}

#endif

```

__init__.py

```

class Foo:

def __init__(self):

self.f = 42

```

mwe_test.pxd:

```

cdef extern from "foo.h" namespace "foo":

cdef cppclass Foo:

Foo() except +

int f_

cdef class CyFoo:

cdef Foo *foo_cpp

@staticmethod

cdef create(Foo *f)

```

mwe_test.pyx:

```

import cython

from mwe_test cimport *

from mwe import Foo as Foo_py

fpy=Foo_py(42) # works fine

cdef class CyFoo:

def to_py(self):

return Foo_py(self.foo_cpp.f_) # fails

@staticmethod

cdef create(Foo *f):

cdef CyFoo f_wrap = CyFoo()

f_wrap.foo_cpp = f

return f_wrap

```

D Woods

unread,

Jun 15, 2021, 2:36:53 AM6/15/21

to cython-users

Don't have time to look into this fully right now (I'll try to at some point later if it isn't solved...) but:

> from mwe_test cimport *

really shouldn't be necessary. This happens automatically whether the files have the same name (i.e. `mwe_test.pyx` matches `mwe_test.pxd` and so automatically has the contents cimported). I know Python 2 and 3 differ about whether they need absolute or relative paths in imports so this could be a source of your problem.

Stefan Behnel

unread,

Jun 15, 2021, 2:40:45 AM6/15/21

to cython...@googlegroups.com

D Woods schrieb am 15.06.21 um 08:33:

> Don't have time to look into this fully right now (I'll try to at some
> point later if it isn't solved...) but:
>
>> from mwe_test cimport *
>
> really shouldn't be necessary.

That's an understatement. Basically, this line means that you are
(c)importing from your own module. I can't think of a use case where that
would be helpful, be it in Python or Cython.

I guess it would help if Cython raised a warning about this.

Stefan

Phil Osteen

unread,

Jun 16, 2021, 1:05:49 AM6/16/21

to cython-users

I had a feeling this would be something minor on my end; commenting that line resolved the errors.

I mistakenly thought this line was akin to a C `#include` directive (and therefore required by source files)