[Python-Dev] Adding a scarier warning to object.__del__?

[Yonatan Zunger]

Hey everyone,

I just encountered yet another reason to beware of __del__: when it's called during interpreter shutdown, for reasons which are kind of obvious in retrospect, if it calls notify() on a threading.Condition, the waiting thread may or may not ever actually receive it, and so if it does that and then tries to join() the thread the interpreter may hang in a hard-to-debug way.

This isn't something that can reasonably be fixed, and (like in most cases involving __del__) there's a very simple fix of using weakref.finalize instead. My question for the dev list: How would people feel about changing the documentation for the method to more bluntly warn people against using it, and refer them to weakref.finalize and/or atexit.register as an alternative? The text already has an undertone of "lasciate ogni speranza, voi ch'entrate" but it may be helpful to be more explicit to avoid people getting caught in unexpected pitfalls.

Yonatan

[Andrew Svetlov]

__del__ is very useful not for interpreter shutdown but as a regular
destructor in object's lifecycle.

Action on the shutdown is another beast.
Personally, I prefer to do all finalization works by explicit calls instead.

> _______________________________________________
> Python-Dev mailing list -- pytho...@python.org
> To unsubscribe send an email to python-d...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/AAZQBWD6PHC4PVNCCPX4A2745SS7B3LS/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Thanks,
Andrew Svetlov
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/57CDW4NIYEQ3JEVX2JVCJDA5TXTC5MBR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Gregory P. Smith]

On Wed, Jan 1, 2020 at 6:40 AM Andrew Svetlov <andrew....@gmail.com> wrote:

__del__ is very useful not for interpreter shutdown but as a regular
destructor in object's lifecycle.

The reason we should warn people against ever implementing __del__ is that people rarely actually understand object lifecycle.  Its presence guarantees delayed garbage collection and that its code will code execute in a context way outside of normal control flow that all other methods are invoked from.

-gps

[Andrew Svetlov]

If the warning text tells about the delayed execution -- I'm +1.

-1 for something like "just don't use __del__, it is dangerous".

[Yonatan Zunger]

Oh, I'm absolutely thinking about clarity. Something like:

----

This method is called when the instance is about to be destroyed. Because it may be called by either the ordinary Python execution flow or by the garbage collector, it has very unusual semantics, and must be treated with great care. In almost all cases, you should instead use either weakref.finalize (if you want logic to be called when an object is no longer in use) or atexit.register (if you want to clean up before the interpreter exits).

The precise semantics of when __del__ is called on an object are implementation-dependent. For example:

* It might be invoked during the normal interpreter flow at a moment like function return, if the reference count for an object has dropped to zero;

* It might be invoked by the garbage collector, which happens during the normal interpreter flow but is called from an arbitrary thread;

* It might be invoked during interpreter shutdown, after various global variables (including modules!) have already been deleted, or other threads' interpreters have already stopped.

* It may not be invoked at all; it is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.

In particular, this means that, unless your program is structured so that you can deterministically control when an object will be destroyed, implementations of __del__():

* Must not assume that any variables outside self exist, including modules or global variables;

* Must not attempt to acquire locks or otherwise block, since they may (e.g.) be invoked inside a thread which already holds those resources;

* Must not rely on being able to signal other threads and wait for their responses (even thread joins), since those threads' interpreters may have exited;

* Must not cause failures (e.g. leaving external devices in an invalid state) if they are never invoked.

Note that del x doesn’t directly call x.__del__() — the former decrements the reference count for x by one, and the latter is only called when x’s reference count reaches zero. Depending on the implementation, it is possible for a reference cycle to prevent the reference count of an object from going to zero. (e.g., in CPython, a common cause of reference cycles is when an exception is caught and stored in a local variable; the exception contains a reference to the traceback, which in turn references the locals of all frames caught in the traceback.) In this case, the cycle will be later detected and deleted by the cyclic garbage collector. 

If a base class has a __del__() method, the derived class’s __del__() method, if any, must explicitly call it to ensure proper deletion of the base class part of the instance.

It is possible (though not recommended!) for the __del__() method to postpone destruction of the instance by creating a new reference to it. This is called object resurrection. It is implementation-dependent whether __del__() is called a second time when a resurrected object is about to be destroyed; the current CPython implementation only calls it once.

[Armin Rigo]

Hi,

On Thu, 2 Jan 2020 at 03:59, Yonatan Zunger <zun...@humu.com> wrote:
> It is possible (though not recommended!) for the __del__() method to postpone destruction of the instance by creating a new reference to it. This is called object resurrection. It is implementation-dependent whether __del__() is called a second time when a resurrected object is about to be destroyed; the current CPython implementation only calls it once.

"...in most cases."


Armin Rigo

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/LHY2EM7PMSSUAMORBOHIUZAYKHILAMJO/

[Andrew Svetlov]

I would say that the "recommended" weakref.finalize() shares very many
limitations of __del__(), that's why hard to buy it.
atexit.register() is not a common thing, the recommendation of using
atexit for file descriptor closing *in general* looks weird, while it
can be a good solution in some particular case.

On Thu, Jan 2, 2020 at 1:05 PM Armin Rigo <armin...@gmail.com> wrote:
>
> Hi,
>
> On Thu, 2 Jan 2020 at 03:59, Yonatan Zunger <zun...@humu.com> wrote:
> > It is possible (though not recommended!) for the __del__() method to postpone destruction of the instance by creating a new reference to it. This is called object resurrection. It is implementation-dependent whether __del__() is called a second time when a resurrected object is about to be destroyed; the current CPython implementation only calls it once.
>
> "...in most cases."
>
>
> Armin Rigo

--
Thanks,
Andrew Svetlov

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/VY2PX376FVNKR3KGFM557KR4OMRH22VL/

[Yonatan Zunger]

weakref.finalize always executes during the ordinary Python flow, IIUC -- e.g., it happens before the interpreter is stopping. I guess it does still have the "arbitrary thread" limitation -- which is a good point, I may have some code bugs of my own to fix. But it's a huge difference in terms of predictability of the interpreter state, and in particular both the issues around deleted modules and the ones around threading no longer working don't apply to it.

atexit is a pretty corner-case thing -- I agree that it shouldn't be the common thing at all, and the recommendation should be specifically for those use cases where it makes sense. FD's are a classic thing to close with weakref.finalize.

[Petr Viktorin]

Could there be a clearer separation between the language semantics and
current CPython behavior?

On 2020-01-02 03:53, Yonatan Zunger wrote:
> Oh, I'm absolutely thinking about clarity. Something like:
>
> ----
>
> This method is called when the instance is about to be destroyed.
> Because it may be called by either the ordinary Python execution flow or
> by the garbage collector, it has very unusual semantics, and must be
> treated with great care.

> The precise semantics of when __del__ is called on an object are
> implementation-dependent. For example:
> * It might be invoked during the normal interpreter flow at a moment
> like function return, if the reference count for an object has dropped
> to zero;
> * It might be invoked by the garbage collector, which happens during the
> normal interpreter flow but is called from an arbitrary thread;
> * It might be invoked during interpreter shutdown, after various global
> variables (including modules!) have already been deleted, or other
> threads' interpreters have already stopped.

> * It may not be invoked at all; it is /not guaranteed/ that __del__()

> methods are called for objects that still exist when the interpreter exits.

Note that similar caveats apply for alternatives to __del__:
weakref.finalize and atexit.register.

Doing any of the following in an implementation of __del__() may easily
lead to bugs, nondeterministic behavior, and inconsistent behavior
across different implementations or versions of Python:

> * Assuming that any variables outside self exist, including
> modules or global variables;
> * Attempting to acquire locks or otherwise block, since they may

> (e.g.) be invoked inside a thread which already holds those resources;

> * Relying on being able to signal other threads and wait for their

> responses (even thread joins), since those threads' interpreters may
> have exited;

> * Causing failures (e.g. leaving external devices in an invalid
> state) if __del__ is /never/ invoked.

In particular, del x does not call x.__del__() directly: the call may be
delayed or not happen at all.

In CPython, del x decrements the reference count for x by one, and
x.__del__() is only called either when x’s reference count reaches zero,
or by the cyclic garbage collector at an unspecified time (e.g. if a
reference cycle prevents the reference count from going to zero).

> If a base class has a __del__() method, the derived class’s __del__()
> method, if any, must explicitly call it to ensure proper deletion of the
> base class part of the instance.
>
> It is possible (though not recommended!) for the __del__() method to
> postpone destruction of the instance by creating a new reference to it.
> This is called object resurrection. It is implementation-dependent
> whether __del__() is called a second time when a resurrected object is

> about to be destroyed.
The current CPython implementation usually calls __del__ only once for
each object, but this is not guaranteed in all cases.

> <mailto:pytho...@python.org>

> > To unsubscribe send an email to

> python-d...@python.org <mailto:python-d...@python.org>

> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/pytho...@python.org/message/AAZQBWD6PHC4PVNCCPX4A2745SS7B3LS/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Thanks,
> Andrew Svetlov
> _______________________________________________
> Python-Dev mailing list -- pytho...@python.org

> <mailto:pytho...@python.org>

> To unsubscribe send an email to python-d...@python.org

> <mailto:python-d...@python.org>

> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/pytho...@python.org/message/57CDW4NIYEQ3JEVX2JVCJDA5TXTC5MBR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>

> _______________________________________________
> Python-Dev mailing list -- pytho...@python.org
> To unsubscribe send an email to python-d...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/

> Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/NKWOXME256DT53PRLU3IMWORT2DZTRHO/

> Code of Conduct: http://python.org/psf/codeofconduct/
>

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/7OKNWI32HPTFRKJUZFL5EKSV4MDU4Q5D/

[Jeff Allen]

On 02/01/2020 02:53, Yonatan Zunger wrote:

Oh, I'm absolutely thinking about clarity. ...

Could any revision also be clear what is *required of Python the language* vs. what is a CPython implementation detail? I always appreciate this care. There is good practice here and elsewhere in the existing documentation, but drift is easy for those steeped in CPython implementation. In the present case, it's a matter of avoiding an explicit requirement for a reference-counting approach to lifecycle management. Here the places in the proposal a change could achieve that.

The precise semantics of when __del__ is called on an object are implementation-dependent. For example:

* It might be invoked during the normal interpreter flow at a moment like function return, ...

We should continue here "... immediately the object is no nonger referenced;"

(It might not be called immediately, but that's implied by your implementation-dependent "might be".)

Note that del x doesn’t directly call x.__del__() — the former decrements the reference count for x by one, and the latter is only called when x’s reference count reaches zero. Depending on the implementation, it is possible for a reference cycle to prevent the reference count of an object from going to zero. (e.g., in CPython, a common cause of reference cycles is when an exception is caught and stored in a local variable; the exception contains a reference to the traceback, which in turn references the locals of all frames caught in the traceback.) In this case, the cycle will be later detected and deleted by the cyclic garbage collector.

I realise that most of this paragraph is existing text rearranged, and currently it fails to make the distinction I'm looking for in the "note" part. But it is clear in the next paragraph. I think it better to say, closer to the current text:

"""Note:: ``del x`` does not call ``x.__del__()`` directly. After ``del x``, variable ``x`` is undefined (or unbound). If the object it referenced is now no longer referenced at all,  that object's ``__del__()`` might be called, immediately or later, subject to the caveats already given.

*CPython implementation detail:* ``del x`` decrements the reference count of the object by one, and if that makes it zero, ``x.__del__()`` will be called immediately. It is possible for a reference cycle to prevent the reference count of any object in it from going to zero. A common cause of reference cycles is when an exception is caught and stored in a local variable; the exception contains a reference to the traceback, which in turn references the locals of all frames caught in the traceback. In this case, the cycle will be detected later and its objects deleted by the cyclic garbage collector."""

If a base class has a __del__() method, the derived class’s __del__() method, if any, must explicitly call it to ensure proper deletion of the base class part of the instance.

Possibly this thought belings with the "implementations of __del__(): * Must ..." paragraph.

But also, while I think there is scope for a better guidance, this is getting a bit long. Should there be a "HOW TO write a __del__ method (and how to avoid it)" to contain the advisory points being made? In-lining advice here, on how to survive the infernal circles of __del__, dilutes the scariness of the warning not to enter at all.

---

Jeff Allen

[Yonatan Zunger]

Jeff, I like all of your proposals. It does get a bit long, but I think it's not so long that it feels out-of-place in the data model docs, which is where you expect weird stuff to live. (I mean, look how long the discussion of metaclasses is -- and I certainly wouldn't want to eliminate that!) 

_______________________________________________

Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/A32QEUP4R6XTY5LQW56LKWJ3XBUZCHOR/

[Inada Naoki]

I don't want to recommend atexit or weakref because they have their pitfall.

I think the context manager should be considered at first when people want
to use __del__.

> _______________________________________________
> Python-Dev mailing list -- pytho...@python.org
> To unsubscribe send an email to python-d...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/

> Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/AAZQBWD6PHC4PVNCCPX4A2745SS7B3LS/

> Code of Conduct: http://python.org/psf/codeofconduct/

--
Inada Naoki <songof...@gmail.com>

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/RSG6DHNG7ZTJCNYSAMVOFFQSD7HFSGKE/

[Andrew Svetlov]

On Sat, Jan 4, 2020 at 5:43 AM Inada Naoki <songof...@gmail.com> wrote:
>
> I don't want to recommend atexit or weakref because they have their pitfall.
>
> I think the context manager should be considered at first when people want
> to use __del__.
>

I support this proposal :)

> --
> Inada Naoki <songof...@gmail.com>
> _______________________________________________
> Python-Dev mailing list -- pytho...@python.org
> To unsubscribe send an email to python-d...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/RSG6DHNG7ZTJCNYSAMVOFFQSD7HFSGKE/
> Code of Conduct: http://python.org/psf/codeofconduct/

--
Thanks,
Andrew Svetlov

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/YA6574IBBBIMAJESCFKJXLISFQXXQ475/

[Antoine Pitrou]

On Wed, 1 Jan 2020 11:51:41 -0800

"Gregory P. Smith" <gr...@krypto.org> wrote:

> On Wed, Jan 1, 2020 at 6:40 AM Andrew Svetlov <andrew....@gmail.com>
> wrote:
>
> > __del__ is very useful not for interpreter shutdown but as a regular
> > destructor in object's lifecycle.
> >
>
> The reason we should warn people against ever implementing __del__ is that
> people rarely actually understand object lifecycle. Its presence
> guarantees delayed garbage collection and that its code will code execute
> in a context way outside of normal control flow that all other methods are
> invoked from.

For the latter (executes in an arbitrary context): agreed.
For the former (guarantees delayed garbage collection): what do you
mean?

Regardless, +1 for pointing to `weakref.finalize` as a better
alternative for things that may survive until interpreter shutdown.
Most people don't know about this kind of detail, and it can bite hard
when working on complex systems.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/3KP3YPMR2HX33QTBV7D5TBHIJBPL6FNW/

[Greg Ewing]

On 5/01/20 11:41 am, Antoine Pitrou wrote:

> +1 for pointing to `weakref.finalize` as a better
> alternative for things that may survive until interpreter shutdown.

I'm not sure that weakref.finalize is much better. It can still be
called at an arbitrary time from an arbitrary thread with the rest
of the system in an unknown state, so it needs to be written very
defensively to cope with that.

The main thing I can see weakref.finalize having over __del__ is
that it doesn't give you access to the destroyed object, so you
can't accidentally rely on some part of it that's no longer valid.
But that's only a subset of the ways it can get you into trouble.

I don't think a blanket warning against using __del__ would be
very helpful, because many of its pitfalls also apply to the
suggested alternatives. This is an area where there's no
substitute for an in-depth understanding of the issues.

--
Greg

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/DHH6QWTHSD4TCQ2V2BEMHYG6K36KRSHY/

[Nick Coghlan]

On Fri., 3 Jan. 2020, 3:52 am Yonatan Zunger, <zun...@humu.com> wrote:

weakref.finalize always executes during the ordinary Python flow, IIUC -- e.g., it happens before the interpreter is stopping. I guess it does still have the "arbitrary thread" limitation -- which is a good point, I may have some code bugs of my own to fix. But it's a huge difference in terms of predictability of the interpreter state, and in particular both the issues around deleted modules and the ones around threading no longer working don't apply to it.

atexit is a pretty corner-case thing -- I agree that it shouldn't be the common thing at all, and the recommendation should be specifically for those use cases where it makes sense.

And in the context of this particular discussion, has the limitation that atexit handlers run *after* the interpreter waits for non-daemon threads to stop.

Cheers,

Nick.

[Antoine Pitrou]

On Sun, 05 Jan 2020 12:37:24 +1300
Greg Ewing <greg....@canterbury.ac.nz> wrote:
> On 5/01/20 11:41 am, Antoine Pitrou wrote:
>
> > +1 for pointing to `weakref.finalize` as a better
> > alternative for things that may survive until interpreter shutdown.
>
> I'm not sure that weakref.finalize is much better. It can still be
> called at an arbitrary time from an arbitrary thread with the rest
> of the system in an unknown state, so it needs to be written very
> defensively to cope with that.

I didn't say otherwise. The main thing going for `weakref.finalize` is
that it is called at the beginning of the interpreter shutdown while
__del__ may be called much later in the process.

> I don't think a blanket warning against using __del__ would be
> very helpful, because many of its pitfalls also apply to the
> suggested alternatives. This is an area where there's no
> substitute for an in-depth understanding of the issues.

I didn't suggest a blanket warning either. I do believe there should
be a discussion, somewhere in the docs, of the various options
available for finalization and their respective pitfalls.

However, many (most?) Python developers know about __del__, while few
know about `weakref.finalize`, so it makes sense to add a pointer to
the latter in the docs for the former.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/YEMIGOLDY4KZYG3KU2UEXWMS3G6G7BC4/