How to let multiple Cython modules share a single global(static) C/C++ variable

[Shashi Da]

For example. Assume there's a source file A.c which defines a static variable int a and two functions set_a and get_a to set and get its value.
Then there are two Cython pyx files calls these functions. After compiling, both pyd files have a independent variable a, and the values are not shared between modules.

My actual problem is similar but more complex. I have a static std::set pool defined in a cpp file. This pool is used to save memory by storing same value only once,and refer them by pointers. To increase the speed, accessing the pool is implemented by c++, and multiple Cython modules may access the poll directly or indirectly via C++ code. Similar to the previous problem, after compiling, each module have a independent pool, and this is obviously not what I want.

I think it can be solved if I can compile multiple pyx files into a single pyd. But I don't know how to do it. I tried include multiple pyx into a single pyx, but there are some weird compiling problems related to cimport.

Compiling the CPP file into a single DLL and link it to pyd files dynamicly may help, but I have no idea how to achieve this, too. Further more, this method may be harder to be cross-platform.

[Robert Bradshaw]

The standard solution would be to put this pool and its accessors into its own C library and link all of the modules against that same library.

Alternatively, define cdef functions in one of the modules that wrap set_a and get_a, and cimport them in all other modules. Yes, this'd be a layer of indirection, but it's unlikely to be measurable performancewise. If profiling shows that there is measurable overhead here, define a get_a_ptr in one of the modules, cimport it into the rest, and call set_a_ptr(primary_module.get_a_ptr()) when loading all the others. 

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shashi Da]

I have managed to build a shared dll to solve this problem.
The setup file is like

from setuptools import setup
from setuptools.extension import Extension
from distutils import ccompiler
from Cython.Build import cythonize

cython_extensions = [
Extension("m1", ["m1.pyx"], libraries=['a'], language='c++'),
Extension("m2", ["m2.pyx"], libraries=['a'], language='c++'),
]

extensions = cythonize(cython_extensions)

cc = ccompiler.new_compiler()
objs = cc.compile(["a.cpp"], extra_preargs=['-O2'])
cc.link_shared_lib(objs, "a", output_dir='.', extra_preargs=['/DLL'])

setup(
name="TestDll",
ext_modules=extensions
)

It cost me a bunch of time to figure out that I have to add extra_preargs=['/DLL'] when linking, otherwise, a linking error will happen. Is it a bug of distutils?

Besides the setup.py, the headers should be modified so that every functions and classes are exported by __declspec(dllexport).

Any better ideas?

[Shashi Da]

Using a single C library is what I'm doing,and the problem is how to dynamically link the C library. If statically linked, each extension module will have a independent pool itself. The problem exists for cdef way, too. So the problem is not how to code it, but how to build it.

[Robert Bradshaw]

On Wed, Apr 6, 2016 at 10:46 PM, Shashi Da <zjuni...@gmail.com> wrote:

Using a single C library is what I'm doing,and the problem is how to dynamically link the C library. If statically linked, each extension module will have a independent pool itself.

Is static linking the default in windows? 

 

The problem exists for cdef way, too. So the problem is not how to code it, but how to build it.

No, cimport of a cdef method defined in another module will import that module and grab a pointer to the method(s) at runtime. 

[Shashi Da]

Sorry, I confused your cdef method with cdef extern. I have tried wrapping and it works.

I have written all features by pure Python beforehead, but found it too slow, and cost too much memory. So I implemented some of the Python classes by C++, and wrapped them by Cython, while keep the Python interface unchanged. Each of the classes has a .h, a .cpp, a .pyd and a .pyx files. The C++ interface and Python interface are both declared in the pyd file. The pyx files are compiled together with its corresponding cpp file.

However, to improve the speed, some C++ methods are also called directly from other cpp files and pyx files. When compiling, the linker complains about unresolved symbols, so I added the corresponding cpp file to sources for each of the pyx files. As a result, one cpp file is compiled multiple times and each compiled extension have a independent pool.

Writing cdef wrappers means extra work, and won't solve the problems caused by calling C++ methods from another cpp files, so I think it would be better for me to just compile all hand-written cpp files to a dynamic library and link it against all pyx files.

I didn't know how to generate a dynamic library by distutils on Windows and found few results when googling. Is there a better/standard way to build shared library by setuptools?

[Robert Bradshaw]

On Thu, Apr 7, 2016 at 1:38 AM, Shashi Da <zjuni...@gmail.com> wrote:

Sorry, I confused your cdef method with cdef extern. I have tried wrapping and it works.

I have written all features by pure Python beforehead, but found it too slow, and cost too much memory. So I implemented some of the Python classes by C++, and wrapped them by Cython, while keep the Python interface unchanged. Each of the classes has a .h, a .cpp, a .pyd and a .pyx files. The C++ interface and Python interface are both declared in the pyd file. The pyx files are compiled together with its corresponding cpp file.

However, to improve the speed, some C++ methods are also called directly from other cpp files and pyx files. When compiling, the linker complains about unresolved symbols, so I added the corresponding cpp file to sources for each of the pyx files. As a result, one cpp file is compiled multiple times and each compiled extension have a independent pool.

The get/set_a_ptr method would allow you to share a cache (but still have the code compiled (statically) into each module and be callable from C++ too). 

Writing cdef wrappers means extra work, and won't solve the problems caused by calling C++ methods from another cpp files, so I think it would be better for me to just compile all hand-written cpp files to a dynamic library and link it against all pyx files.

I agree a shared, dynamically linked library is the right solution to this problem. 

 

I didn't know how to generate a dynamic library by distutils on Windows and found few results when googling. Is there a better/standard way to build shared library by setuptools?

You're asking about Windows, which I literally haven't used in years. There are lots of Windows users who also use Cython and/or distutils; hopefully one of them can speak up and help you. 

[Shashi Da]

Thank you for your help.

[Chris Barker]

On Thu, Apr 7, 2016 at 8:44 AM, Robert Bradshaw <robe...@gmail.com> wrote:

I agree a shared, dynamically linked library is the right solution to this problem. 

 

I didn't know how to generate a dynamic library by distutils on Windows and found few results when googling. Is there a better/standard way to build shared library by setuptools?

we're doing this for a big project for somewhat different reasons: we have a big pile of C++ that is quite interdependent, and have wrapped parts of it with Cython in a number of modules. I don't think we have anything static, but as each module needs access to various parts of the C++, we built a shared lib out of it, so each cyton module could link against it.

but we did not find a way for setuptools to build the dll (or so on *nix) -- we simply build that extenally.

here is our build script -- pretty ugly and complex, but maybe you can tease out what you need.

https://github.com/NOAA-ORR-ERD/PyGnome/blob/master/py_gnome/setup.py

BTW, on OS-X (Not sure about Linux), the dll was not neccesary. rather, we could build all the C== code into one cython extension, and then as long as that extension was imported first, the other extensions had access to to all the code in there. WE couldn't figure out how to get this to work on Windows, though. Thus the dll.

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Chris Barker]

On Thu, Apr 7, 2016 at 1:33 PM, Chris Barker <chris....@noaa.gov> wrote:

but we did not find a way for setuptools to build the dll (or so on *nix) -- we simply build that extenally.

I was wrong -- we were bulding it extenally, but are now having it be built by distutils

 

here is our build script -- pretty ugly and complex, but maybe you can tease out what you need.

https://github.com/NOAA-ORR-ERD/PyGnome/blob/master/py_gnome/setup.py

take a look at the elif platform == "win32":stanza around line 358.

so we build the C++ code into an extension, pasing flags so that a *.lib is also built. 

and the rest of the extensions get the *.lib added by adding it to extra_objects.

But I'm not entirely sure if those are statically or dynamically linked -- i.e. does each extension get it's own copy -- which would not work for you.

-CHB

[Shashi Da]

I tried and it works!

As far as I can see, a pyd, which is actually a dll, is built from all the cpp files and basic_types_ext.pyx. The generated lib file is just an import library and all working code are contained in basic_types_ext.pyd. On windows, basic_types_ext do not need to be imported. When importing other modules, with the help of the import library, basic_types_ext.pyd will be loaded as a dll. I'm not sure if it should be imported before any other extension module on Linux/Mac OSX.

Since I do not have that much external libraries, this method has fewer platform-dependent code than using distutils.ccompiler.

在 2016年4月8日星期五 UTC+8上午6:34:42，Chris Barker写道：

[Chris Barker]

On Thu, Apr 7, 2016 at 8:05 PM, Shashi Da <zjuni...@gmail.com> wrote:

I tried and it works!

good news.

 

As far as I can see, a pyd, which is actually a dll, is built from all the cpp files and basic_types_ext.pyx.

exactly -- so basic_types_ext has a small amount of python extension code, and all of the C++ lib code. but yes, it's called a "pyd" so it's clear it's a "proper" python extension, but it is "just" a dll.

 

On windows, basic_types_ext do not need to be imported. When importing other modules, with the help of the import library, basic_types_ext.pyd will be loaded as a dll.

right -- on windows, basic_types.pyd (the one with all the C++ code) is linked to all teh other extensions by passing the .lib file to extra_objects.

now that I think about it, we could probably do that with *nix, too, to be more consitent. THis all evolved over the years...

I'm not sure if it should be imported before any other extension module on Linux/Mac OSX.

Since I do not have that much external libraries, this method has fewer platform-dependent code than using distutils.ccompiler.

yup -- seems a pretty good solution to your problem.

Note: it would be nice to write this up as a recipe  and put it in the Cython Wiki...

It's not a terribly uncommon need.

Also note: we've only done with with py2 and the MSVS2008 compiler -- not sure if it all works the same with VS2015, which is used by py3.5+

-CHB

[Shashi Da]

I'm using Python 3.5 and VS2015. The generated lib file's name is module.cp35-win_amd64.lib, with platform informations in the filename. Everything else is just as expected.

[Chris Barker]

On Fri, Apr 8, 2016 at 9:14 AM, Shashi Da <zjuni...@gmail.com> wrote:

I'm using Python 3.5 and VS2015. The generated lib file's name is module.cp35-win_amd64.lib, with platform informations in the filename. Everything else is just as expected.

awesome -- great to know.

-CHB

 

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.