How to let multiple Cython modules share a single global (static) C variable of a static library

23 views
Skip to first unread message

Edward Sr

unread,
Apr 10, 2022, 1:15:27 PMApr 10
to cython-users
Hi,

I know that there's already a similar thread here but my desired solution is different, more strict.

I have a C code, and I'm wrapping some internal functions using Cython.
Now since they are internal functions, I MUST work with static libraries instead of shared (Linux environment).
In the internal C code, there's a global (extern) static variable, that is modified and accessed from multiple different C files.

My problem is that, since Cython libraries are shared, the value of this variable differs depending on which Cython module I'm using... which affects the result of the functions I'm wrapping.
I don't access the get/set of this variable or the variable itself directly from the Cython code.

Assuming that I can't change the C code (at least not refactoring, like moving functions from place to place or changing the variable to not be global...), how can I resolve this issue via Cython?!!
I also don't want to put all modules into one file, because the project is not small at all, and has multiple hierarchies.

I will write a song in binary for whoever helps me solve this!! :D
Thanks in advance.

Stefan Behnel

unread,
Apr 20, 2022, 9:04:38 AMApr 20
to cython...@googlegroups.com
Hi,

Edward Sr schrieb am 10.04.22 um 19:09:
> I know that there's already a similar thread here
> <https://groups.google.com/g/cython-users/c/GAAPYb2X304/m/FCPHN9CpAwAJ> but
> my desired solution is different, more strict.

Thanks for searching, and for providing the reference.


> I have a C code, and I'm wrapping some internal functions using Cython.
> Now since they are internal functions, I MUST work with static libraries
> instead of shared (Linux environment).
> In the internal C code, there's a global (extern) static variable, that is
> modified and accessed from multiple different C files.

That's unfortunate but not so uncommon. You are right that this means you
have to make sure the library is only there once. Or, that at least those
parts of the library that depend on that global variable are only used from
one library instance.


> My problem is that, since Cython libraries are shared, the value of this
> variable differs depending on which Cython module I'm using... which
> affects the result of the functions I'm wrapping.
> I don't access the get/set of this variable or the variable itself directly
> from the Cython code.
>
> Assuming that I can't change the C code (at least not refactoring, like
> moving functions from place to place or changing the variable to not be
> global...), how can I resolve this issue via Cython?!!
> I also don't want to put all modules into one file, because the project is
> not small at all, and has multiple hierarchies.

What I would try is to have one low-level Cython extension module that
wraps the C library with a C interface (of cdef functions and maybe some
basic extension types), exposed in the corresponding lowlevelmodule.pxd
file (with the same name as the module). Then, other modules no longer need
to link against the library but can cimport the low-level extension module
and call into the C library through that. Internally, this cimport
mechanism uses CPython's PyCapsule mechanism for exposing C-APIs from
extension modules.

You can still mix this approach with other statically linked extension
modules, as long as those do not need to share state with the low-level
wrapper module (which owns the state of the global variable). I'm actually
doing that in lxml's etree and objectify modules (rather by accident),
which both link against libxml2, but all configurable functionality resides
in lxml.etree – objectify only uses library functions that operate
independently. So, this can work, and allows for trading a somewhat larger
package size (more copies of the library) for speed (direct library calls,
e.g. for short-running functions).


> I will write a song in binary for whoever helps me solve this!! :D

Let's hear more from that. :)

Stefan

Edward Sr

unread,
Apr 24, 2022, 3:25:38 PMApr 24
to cython...@googlegroups.com
This is exactly what I eventually did, and yes, of course, it worked.
Now you assured me that this is the best solution for now and no other, better, alternatives. :)
Thanks for your reply and for the additional details.


> I will write a song in binary for whoever helps me solve this!! :D

Let's hear more from that. :)


I'll make a poem, let it be short :) 
01010010 01101111 01110011 01100101 01110011 00100000 01100001 01110010 01100101 00100000 01110010 01100101 01100100 00001010 01010110 01101001 01101111 01101100 01100101 01110100 01110011 00100000 01100001 01110010 01100101 00100000 01100010 01101100 01110101 01100101 00001010 01000011 01111001 01110100 01101000 01101111 01101110 00100000 01100011 01101111 01101101 01101101 01110101 01101110 01101001 01110100 01111001 00100000 01101001 01110011 00100000 01100111 01101111 01101111 01100100 00001010 01000010 01110101 01110100 00100000 01110100 01101000 01100101 00100000 01100010 01100101 01110011 01110100 00100000 01101111 01101110 01100101 00100000 01101001 01110011 00100000 01111001 01101111 01110101 00001010
 
Stefan

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/c04f319e-70b4-b165-6b29-4b8c7f67c472%40behnel.de.
Reply all
Reply to author
Forward
0 new messages