Reference counting and native memory management practices / toolsets

[jean-mich...@csiro.au]

Hi,

I'd be interested to hear what you are using to systematically manage reference counting if you need to do so using python cffi (or other interop) for handling unmanaged memory resources. A couple of years ago I was scouting for readily available Python packages but to my surprise did not locate an obvious candidate (notably one without unwanted side baggage).

I've since built the refcount package which I've published primarily  to reuse across my projects but is hopefully of use for others. Suggestions, contributions and feedback welcome. 

Cheers.

[Armin Rigo]

Hi, 

A quick note: this package is relying on the internal reference count of CPython, by having __del__ being called promptly; and so it is likely to not work very well with PyPy or other non-cpython implementations of Python (seem to leak or at least consume far more memory).

Armin

--
-- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
---
You received this message because you are subscribed to the Google Groups "python-cffi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-cffi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/09f5d88b-e4ee-45d5-9a58-a03c73f018cbn%40googlegroups.com.

[jean-mich...@csiro.au]

Thank you (belatedly) Armin for pointing this out. I am not a PyPy user but will see if I can understand its GC behavior, if I get this right. A leak would be a no-no of course; delayed disposal would not shock me, if similar in behavior to .NET, R, etc.

[Armin Rigo]

Hi Jean-Michel,

On Mon, 12 Apr 2021 at 01:56, jean-mich...@csiro.au
<jean-mich...@csiro.au> wrote:
> Thank you (belatedly) Armin for pointing this out. I am not a PyPy user but will see if I can understand its GC behavior, if I get this right. A leak would be a no-no of course; delayed disposal would not shock me, if similar in behavior to .NET, R, etc.

This is a class of problems that isn't really a leak, but has all the
appearances of one. The problem is that a GC like PyPy doesn't know
that each of the small proxy objects holds a potentially large amount
of memory behind the scenes. So PyPy won't bother running its own GC
very often. For example, if your program consumes 100 times more
memory in behind-the-scene objects than the total amount of Python
objects created (which can easily occur in some situations), then the
memory usage can grow 100 times above normal. It's a similar reason
for why it seems, on CPython, that you don't have to close files
explicitly, but on PyPy you do---or you run into EMFILE as soon as
your GC heap grows a bit larger and the GC collects less often. This
is not specific to PyPy; all non-reference-counting GCs suffer from it
(including Java and .NET).

One solution is to design the APIs so that they require explicit
finalization, with proxy objects that have got an explicit reference
count and explicit "incref" and "decref" methods. For some cases, you
can add a syntactic layer on top (e.g. context managers).

Another solution is to use some semi-internal API to tell the GC that
there are more resources behind its managed objects. In PyPy it is
``__pypy__.add_memory_pressure()``. See also
https://docs.microsoft.com/en-us/dotnet/api/system.gc.addmemorypressure
for the motivation from C#.


A bientôt,

Armin.

[jm]

Bonjour Armin and group,

First, thank you Armin for your last reply - I thought I had done that.

On  the side of work projects I've spruced up a few things (documentation, testing, CI, pypi and conda-forge releases) for the refcount package. I am using it for Python bindings for various C/C++ environmental simulation and forecasting (e.g. this module for an example).

I'd like to explore if this is possible to "advertise" it a bit more. I am battled scarred enough to know some active communication is required to raise awareness. I'd take advice on how to go about it, starting from recipients from this list.

I'll next look at setting up unit tests for behaviors on Pypy (none of my work project use it yet, so out of scope till now). I've co-authored DynamicInterop in the .NET ecosystem and am decently aware (if needing a refresher) of the "with" context manager and related "Disposable" pattern or equivalents.

Cheers,

J-M