Puzzling behaviour of Py_IncRef

[Tony Flury]

I am writing a C extension module for an AVL tree, and I am trying to
ensure reference counting is done correctly. I was having a problem with
the reference counting so I worked up this little POC of the problem,
and I hope someone can explain this.

Extension function :

static PyObject *_Node_test_ref_count(PyObject *self)
{
    printf("\nIncrementing ref count for self - just for the hell
of it\n");
    printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
    Py_INCREF(self);
    printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
    fflush(stdout);
    return self;
}

As you can see this function purely increments the reference count of
the instance.

/Note: I understand normally this would be the wrong this to do, but
this is a POC of the issue, not live code. In the live code I am
attaching a 2nd nodes to each other, and the live code therefore
increments the ref-count for both objects - so even if the Python code
deletes it's reference the reference count for the instance should still
be 1 in order to ensure it doesn't get garbage collected./

This function is exposed as the test_ref method.

This is the test case :

    def test_000_009_test_ref_count(self):
        node = _Node("Hello")
        self.assertEqual(sys.getrefcount(node), 2)
        node.test_ref()
        self.assertEqual(sys.getrefcount(node), 3)

The output of this test case is :

test_000_009_test_ref_count (__main__.TestNode) ...
Incrementing ref count for self - just for the hell of it

 before self has a ref count of 2

 after self has a ref count of 3
FAIL

======================================================================
FAIL: test_000_009_test_ref_count (__main__.TestNode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/tony/Development/python/orderedtree/tests/test_orderedtree.py",
line 62, in test_000_009_test_ref_count
    self.assertEqual(sys.getrefcount(node), 3)
AssertionError: 2 != 3

So I understand why the first assert will be true - when the
sys.getrefcount() function is called the ref count is incremented
temporarily (as a borrowed reference), so there are now two references -
the 'node' variable, and the borrowed reference in the function call.

We then call the 'test_ref' method, and again that call causes a
borrowed reference (hence the ref count being 2 initially within the
method). The 'test_ref' method increments the reference of the instance
- as you can see from the output we now have a ref count of 3 - (that
count is the 'node' variable in the test case, the borrowed reference
due to the method call, and the artificial increment from the 'ref_test'
method).

When the 'ref_test' method exits I would expect the ref count of the
instance to now be 2 (one for the 'node' variable, and one as a result
of the artificial increment increment').

I would therefore expect the 2nd assertEqual in the test case to
succeed. - in this case the borrowed reference within sys.getfrefcount()
should cause the count to be 3.

As you see though that 2nd assertEqual fails - suggesting that the
refcount of 'node' is actually only 1 when the 'test_ref' method exits.

Can someone explain why the 'test_ref' method fails to change the
refcount of the 'node' instance.

[Chris Angelico]

On Wed, Jan 19, 2022 at 10:00 PM Tony Flury via Python-list
<pytho...@python.org> wrote:
> Extension function :
>
> static PyObject *_Node_test_ref_count(PyObject *self)
> {
> printf("\nIncrementing ref count for self - just for the hell
> of it\n");
> printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
> Py_INCREF(self);
> printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
> fflush(stdout);

At this point, the refcount has indeed been increased.

> return self;
> }

And then you say "my return value is this object".

The normal thing to do is to add a reference to whatever you're
returning. For instance, Py_RETURN_NONE will incref None and then
return it.

So you're incrementing the refcount, then returning it without
incrementing the refcount. Your code is actually equivalent to "return
self".

In order to actually leak a reference, you'd need to incref it twice.

ChrisA

[Tony Flury]

Chris - I am still puzzled - does  doing 'return self' automatically
decrement the ref count of the object ?, and why is that the desired
behaviour ? Effectively it results in a decrement of two, since at the
exit of the function the ref count is only 1 (as witnessed by the
subsequent call to assertEqual).

(I am not suggesting that it should be changed - I understand that would
be a breaking change !).

You say I am returning it without incrementing, but I am explicitly
incrementing it before the return.

[Chris Angelico]

Imagine you have a function which creates a brand new object. What
should its reference count be just before you return it? What should
the refcount be after it's been given to the caller? There has to be a
reference at all times, but only one, so that it can be properly
garbage collected; the act of returning it from a function has to
steal away a reference.

> (I am not suggesting that it should be changed - I understand that would
> be a breaking change !).
>
> You say I am returning it without incrementing, but I am explicitly
> incrementing it before the return.
>

Yes, you are incrementing it - but only once. The act of returning
something that you already have a reference to has to increment the
reference count and then return the thing. Picture this Python code:

_THING = object()
def get_thing():
return _THING

When you call get_thing(), it needs to give you back a new reference
to that object. Written in C, that would have to increment the
refcount, then return the thing. (If you disassemble that code in
CPython, what you'll see is that it does a LOAD_GLOBAL, which does the
reference count incrementing, and then it returns what's on the top of
the stack. In C, you simply return the pointer, so you manually
increment the refcount first.)

In order to increment the reference count and keep it higher, you need
to (1) increment the reference count, then (2) return the object,
which consists of incrementing the reference count and returning the
pointer.

ChrisA

[Barry Scott]

> On 19 Jan 2022, at 10:57, Tony Flury via Python-list <pytho...@python.org> wrote:
>
> I am writing a C extension module for an AVL tree, and I am trying to ensure reference counting is done correctly. I was having a problem with the reference counting so I worked up this little POC of the problem, and I hope someone can explain this.

The ref counting in the C API not easy to get right. Sometimes you must inc some times you must not inc.
There are a lot of libraries that take that problem away from you.

For example I maintain PyCXX that is a C++ library that allows your drive the C API without needed to worry about ref counting.
There are lots of other libraries as well that aim to do the same thing.

http://cxx.sourceforge.net/ <http://cxx.sourceforge.net/>

Barry

>
> Extension function :
>
> static PyObject *_Node_test_ref_count(PyObject *self)
> {
> printf("\nIncrementing ref count for self - just for the hell
> of it\n");
> printf("\n before self has a ref count of %ld\n", Py_REFCNT(self));
> Py_INCREF(self);
> printf("\n after self has a ref count of %ld\n", Py_REFCNT(self));
> fflush(stdout);

> return self;
> }
>
> As you can see this function purely increments the reference count of the instance.
>
> /Note: I understand normally this would be the wrong this to do, but this is a POC of the issue, not live code. In the live code I am attaching a 2nd nodes to each other, and the live code therefore increments the ref-count for both objects - so even if the Python code deletes it's reference the reference count for the instance should still be 1 in order to ensure it doesn't get garbage collected./
>
> This function is exposed as the test_ref method.
>
> This is the test case :
>
> def test_000_009_test_ref_count(self):
> node = _Node("Hello")
> self.assertEqual(sys.getrefcount(node), 2)
> node.test_ref()
> self.assertEqual(sys.getrefcount(node), 3)
>
> The output of this test case is :
>
> test_000_009_test_ref_count (__main__.TestNode) ...

> Incrementing ref count for self - just for the hell of it
>

> before self has a ref count of 2
>
> after self has a ref count of 3
> FAIL
>
> ======================================================================
> FAIL: test_000_009_test_ref_count (__main__.TestNode)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/home/tony/Development/python/orderedtree/tests/test_orderedtree.py", line 62, in test_000_009_test_ref_count
> self.assertEqual(sys.getrefcount(node), 3)
> AssertionError: 2 != 3
>
> So I understand why the first assert will be true - when the sys.getrefcount() function is called the ref count is incremented temporarily (as a borrowed reference), so there are now two references - the 'node' variable, and the borrowed reference in the function call.
>
> We then call the 'test_ref' method, and again that call causes a borrowed reference (hence the ref count being 2 initially within the method). The 'test_ref' method increments the reference of the instance - as you can see from the output we now have a ref count of 3 - (that count is the 'node' variable in the test case, the borrowed reference due to the method call, and the artificial increment from the 'ref_test' method).
>
> When the 'ref_test' method exits I would expect the ref count of the instance to now be 2 (one for the 'node' variable, and one as a result of the artificial increment increment').
>
> I would therefore expect the 2nd assertEqual in the test case to succeed. - in this case the borrowed reference within sys.getfrefcount() should cause the count to be 3.
>
> As you see though that 2nd assertEqual fails - suggesting that the refcount of 'node' is actually only 1 when the 'test_ref' method exits.
>
> Can someone explain why the 'test_ref' method fails to change the refcount of the 'node' instance.
>

> --
> https://mail.python.org/mailman/listinfo/python-list

[Greg Ewing]

On 20/01/22 12:09 am, Chris Angelico wrote:
> At this point, the refcount has indeed been increased.
>
>> return self;
>> }
>
> And then you say "my return value is this object".
>

> So you're incrementing the refcount, then returning it without
> incrementing the refcount. Your code is actually equivalent to "return
> self".

Chris, you're not making any sense. This is C code, so there's no
way that "return x" can change the reference count of x.

> The normal thing to do is to add a reference to whatever you're
> returning. For instance, Py_RETURN_NONE will incref None and then
> return it.
>

The OP understands that this is not a normal thing to do. He's
trying to deliberately leak a reference for the purpose of diagnosing
a problem.

It would be interesting to see what the actual refcount is after
calling this function.

--
Greg

[Chris Angelico]

On Fri, 21 Jan 2022 at 10:10, Greg Ewing <greg....@canterbury.ac.nz> wrote:
>
> On 20/01/22 12:09 am, Chris Angelico wrote:
> > At this point, the refcount has indeed been increased.
> >
> >> return self;
> >> }
> >
> > And then you say "my return value is this object".
> >
> > So you're incrementing the refcount, then returning it without
> > incrementing the refcount. Your code is actually equivalent to "return
> > self".
>
> Chris, you're not making any sense. This is C code, so there's no
> way that "return x" can change the reference count of x.

Yeah, I wasn't clear there. It was equivalent to *the Python code*
"return self". My apologies.

> > The normal thing to do is to add a reference to whatever you're
> > returning. For instance, Py_RETURN_NONE will incref None and then
> > return it.
> >
>
> The OP understands that this is not a normal thing to do. He's
> trying to deliberately leak a reference for the purpose of diagnosing
> a problem.
>
> It would be interesting to see what the actual refcount is after
> calling this function.
>

Yes, and that's why I was saying it would need a *second* incref.

ChrisA

[Tony Flury]

On 20/01/2022 23:12, Chris Angelico wrote:
> On Fri, 21 Jan 2022 at 10:10, Greg Ewing <greg....@canterbury.ac.nz> wrote:

>> On 20/01/22 12:09 am, Chris Angelico wrote:
>>> At this point, the refcount has indeed been increased.
>>>
>>>> return self;
>>>> }
>>> And then you say "my return value is this object".
>>>
>>> So you're incrementing the refcount, then returning it without
>>> incrementing the refcount. Your code is actually equivalent to "return
>>> self".
>> Chris, you're not making any sense. This is C code, so there's no
>> way that "return x" can change the reference count of x.

> Yeah, I wasn't clear there. It was equivalent to *the Python code*
> "return self". My apologies.
>

>> > The normal thing to do is to add a reference to whatever you're
>> > returning. For instance, Py_RETURN_NONE will incref None and then
>> > return it.
>> >
>>
>> The OP understands that this is not a normal thing to do. He's
>> trying to deliberately leak a reference for the purpose of diagnosing
>> a problem.
>>
>> It would be interesting to see what the actual refcount is after
>> calling this function.

After calling this without a double increment in the function the ref
count is still only 1 - which means that the 'return self' effectively
does a double decrement. My original message includes the Python code
which calls this 'leaky' function and you can see that despite the
'leaky POC' doing an increment ref count drops back to one after the return.

You are right this is not a normal thing to do, I am trying to
understand the behaviour so my library does the correct thing in all
cases - for example - imagine you have two nodes in a tree :

A --- > B

And your Python code has a named reference to A, and B also maintains a
reference to A as it's parent.

In this case I would expect A to have a reference count of 2 (counted as
3 through sys.getrefcount() - one for the named reference in the Python
code - and one for the link from B back to A; I would also expect B to
have a reference count here of 1 (just the reference from A - assuming
nothing else referenced B).

My original code was incrementing the ref counts of A and B and then
returning A. within the Python test code A had a refcount of 1 (and not
the expected 2), but the refcount from B was correct as far as I could tell.

> Yes, and that's why I was saying it would need a *second* incref.
>
> ChrisA

Thank you to all of you for trying to help - I accept that the only way
to make the code work is to do a 2nd increment.

I don't understand why doing a 'return self' would result in a double
decrement - that seems utterly bizzare behaviour - it obviously works,
but why.



--
Anthony Flury
email : anthon...@btinternet.com

[Barry]

> On 25 Jan 2022, at 14:50, Tony Flury via Python-list <pytho...@python.org> wrote:
>
> 

The return self in C will not change the ref count.

I would suggest setting a break point in your code and stepping out of the function and seeing that python’s code does to the ref count.

Barry

>
>
>
> --
> Anthony Flury
> email : anthon...@btinternet.com
>

> --
> https://mail.python.org/mailman/listinfo/python-list
>

[Tony Flury]

Barry,

something odd is going on because the Python code isn't doing anything
that would cause the reference count to go from 3 inside the C function
to 1 once the method call is complete.

As far as I know the only things that impact the reference counts are :

* Increments due to assigning a new name or adding it to a container.
* Increment due to passing the object to a function (since that binds
a new name)
* Decrements due to deletion of a name
* Decrement due to going out of scope
* Decrement due to being removed from a container.

None of those things are happening in the python code.

As posted in the original message - immediately before the call to the C
function/method sys.getrefcount reports the count to be 2 (meaning it is
actually a 1).

Inside the C function the ref count is incremented and the Py_REFCNT
macro reports the count as 3 inside the C function as expected (1 for
the name in the Python code, 1 for the argument as passed to the C
function, and 1 for the increment), so outside the function one would
expect the ref count to now be 2 (since the reference caused by calling
the function is then reversed).

However - Immediately outside the C function and back in the Python code
sys.getrefcount reports the count to be 2 again - meaning it is now
really 1. So that means that the refcount has been decremented twice
in-between the return of the C function and the execution of the
immediate next python statement. I understand one of those decrements -
the parameter's ref count is incremented on the way in so the same
object is decremented on the way out (so that calls don't leak
references) but I don't understand where the second decrement is coming
from.

Again there is nothing in the Python code that would cause that
decrement - the decrement behavior is in the Python runtime.

[MRAB]

The function returns a result, an object.

The calling code is discarding the result, so it's being DECREFed.

For example:

def foo():
return Node()

returns a new node, so its refcount is 1.

Calling 'foo' as statement:

foo()

discards the result; the result is DECREFed back to 0 and garbage collected.

If you wanted your C function to return None, you'd have:

Py_INCREF(Py_NONE);
return Py_None;

or, more succinctly:

Py_RETURN_NONE;

But you're returning the object itself, and you're INCREFing it first,
which is what you need to do anyway.

The 'extra' DECREF is coming from the result (i.e. self) being discarded.

If it wasn't DECREFed, a function could create a new object (refcount ==
1) and return it, and if the function was being called as a statement,
it would be discarded with the refcount still == 1, leading to a memory
leak.

[Greg Ewing]

The convention for refcounting in CPython is that a function
takes borrowed references as arguments and returns a new
reference.

The 'self' argument passed in is a borrowed reference. If you
want to return it, you need to create a new reference by
increfing it.

So what you have written is just the correct way to write
a do-nothing function that returns its argument, i.e. the
equivalent of the Python function

def f(self):
return self

As others have said, if you want to leak a reference, you
need an extra incref besides the one that's a normal part
of the return convention.

--
Greg

[Tony Flury]

So according to that I should increment twice if and only if the calling
code is using the result - which you can't tell in the C code - which is
very odd behaviour.

There is clearly something very deep here that I am simply not
understanding.

[Chris Angelico]

No, the return value from your C function will *always* have a
reference taken. Whether the return value is "used" or just dropped,
there's always going to be one ref used by the returning itself.

The standard way to return a value is always to incref it, then return
the pointer. That is exactly equivalent to Python saying "return
<thing>".

Incrementing twice is ONLY because you want to leak a reference.

ChrisA

[Tony Flury]

Chris,

You keep saying I am leaking a reference - my original code (not the POC
in the email) wasn't intending to leak a reference, it was incrementing
the reference count in order to accurately count references, from other
objects and i needed to double increment there so that the reference
count remained correct outside of the C code.

I did try to be clear - my intention was never to leak a reference (I
have been writing s/w long enough to know leaks are bad) - my POC code
in the original message was the only code which deliberately leaked a
reference in order to simply illustrate the problem.

I do appreciate the help you have tried to give - so thank you.

[Barry]

> On 25 Jan 2022, at 23:50, Tony Flury <tony....@btinternet.com> wrote:
>
> 
>
>
>> On 25/01/2022 22:28, Barry wrote:
>>

> Increments due to assigning a new name or adding it to a container.

> Increment due to passing the object to a function (since that binds a new name)

> Decrements due to deletion of a name

> Decrement due to going out of scope

> Decrement due to being removed from a container.
> None of those things are happening in the python code.
>

Run python and your code under a debugger and check the ref count of the object as you step through the code.

Don’t just step through your code but also step through the C python code.
That will allow you to see how this works at a low level.
Setting a watch point on the ref count will allow you run the code and just break as the ref count changes.

That is what I do when a see odd c api behaviour.

Barry

> As posted in the original message - immediately before the call to the C function/method sys.getrefcount reports the count to be 2 (meaning it is actually a 1).
>
> Inside the C function the ref count is incremented and the Py_REFCNT macro reports the count as 3 inside the C function as expected (1 for the name in the Python code, 1 for the argument as passed to the C function, and 1 for the increment), so outside the function one would expect the ref count to now be 2 (since the reference caused by calling the function is then reversed).
>
> However - Immediately outside the C function and back in the Python code sys.getrefcount reports the count to be 2 again - meaning it is now really 1. So that means that the refcount has been decremented twice in-between the return of the C function and the execution of the immediate next python statement. I understand one of those decrements - the parameter's ref count is incremented on the way in so the same object is decremented on the way out (so that calls don't leak references) but I don't understand where the second decrement is coming from.
>
> Again there is nothing in the Python code that would cause that decrement - the decrement behavior is in the Python runtime.
>
>>>
>>>

>>> --
>>> Anthony Flury
>>> email : anthon...@btinternet.com
>>>

>>> --
>>> https://mail.python.org/mailman/listinfo/python-list

[Tony Flury]

On 26/01/2022 22:41, Barry wrote:
>
>
> Run python and your code under a debugger and check the ref count of
> the object as you step through the code.
>
> Don’t just step through your code but also step through the C python code.
> That will allow you to see how this works at a low level.
> Setting a watch point on the ref count will allow you run the code and
> just break as the ref count changes.
>
> That is what I do when a see odd c api behaviour.
>
> Barry

Thanks - I have tried a few times on a few projects to run a debugger in
mixed language mode and never had any success.

I will have to try again.

[Barry Scott]

> On 27 Jan 2022, at 07:46, Tony Flury <tony....@btinternet.com> wrote:
>
>
> On 26/01/2022 22:41, Barry wrote:
>>
>>
>> Run python and your code under a debugger and check the ref count of the object as you step through the code.
>>
>> Don’t just step through your code but also step through the C python code.
>> That will allow you to see how this works at a low level.
>> Setting a watch point on the ref count will allow you run the code and just break as the ref count changes.
>>
>> That is what I do when a see odd c api behaviour.
>>
>> Barry
>
>
> Thanks - I have tried a few times on a few projects to run a debugger in mixed language mode and never had any success.
>
> I will have to try again.

You mean debugging Python and C/C++? In this case your python code is a simple test script and it's C that you care about.
Should not be difficult. I tend to use linux as my lead debug platform as its the easiest to work with. But Windows and macOS
also have very good debuggers.

Barry