Design thought for callbacks

[Cem Karan]

Hi all, I'm working on a project that will involve the use of callbacks, and I want to bounce an idea I had off of everyone to make sure I'm not developing a bad idea. Note that this is for python 3.4 code; I don't need to worry about any version of python earlier than that.

In order to inform users that certain bits of state have changed, I require them to register a callback with my code. The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive. Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead. My question is simple; is this a good design? If not, why not? Are there any potential 'gotchas' I should be worried about?

Thanks,
Cem Karan

[Frank Millman]

"Cem Karan" <cfka...@gmail.com> wrote in message
news:33677AE8-B2FA-49F9...@gmail.com...

I tried something similar a while ago, and I did find a gotcha.

The problem lies in this phrase - "if they are no longer alive, they are

automatically removed from the WeakSet, preventing me from accidentally
calling them when they are dead."

I found that the reference was not removed immediately, but was waiting to
be garbage collected. During that window, I could call the callback, which
resulted in an error.

There may have been a simple workaround. Perhaps someone else can comment.

Frank Millman

[Chris Angelico]

On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan <cfka...@gmail.com> wrote:
> In order to inform users that certain bits of state have changed, I require them to register a callback with my code. The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive. Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead. My question is simple; is this a good design? If not, why not? Are there any potential 'gotchas' I should be worried about?
>

No, it's not. I would advise using strong references - if the callback
is a closure, for instance, you need to hang onto it, because there
are unlikely to be any other references to it. If I register a
callback with you, I expect it to be called; I expect, in fact, that
that *will* keep my object alive.

ChrisA

[Cem Karan]

OK, so it would violate the principle of least surprise for you. Interesting. Is this a general pattern in python? That is, callbacks are owned by what they are registered with?

In the end, I want to make a library that offers as few surprises to the user as possible, and no matter how I think about callbacks, they are surprising to me. If callbacks are strongly-held, then calling 'del foo' on a callable object may not make it go away, which can lead to weird and nasty situations. Weakly-held callbacks mean that I (as the programmer), know that objects will go away after the next garbage collection (see Frank's earlier message), so I don't get 'dead' callbacks coming back from the grave to haunt me.

So, what's the consensus on the list, strongly-held callbacks, or weakly-held ones?

Thanks,
Cem Karan

[Chris Angelico]

On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfka...@gmail.com> wrote:
> OK, so it would violate the principle of least surprise for you. Interesting. Is this a general pattern in python? That is, callbacks are owned by what they are registered with?
>
> In the end, I want to make a library that offers as few surprises to the user as possible, and no matter how I think about callbacks, they are surprising to me. If callbacks are strongly-held, then calling 'del foo' on a callable object may not make it go away, which can lead to weird and nasty situations. Weakly-held callbacks mean that I (as the programmer), know that objects will go away after the next garbage collection (see Frank's earlier message), so I don't get 'dead' callbacks coming back from the grave to haunt me.
>
> So, what's the consensus on the list, strongly-held callbacks, or weakly-held ones?

I don't know about Python specifically, but it's certainly a general
pattern in other languages. They most definitely are owned, and it's
the only model that makes sense when you use closures (which won't
have any other references anywhere).

If you're expecting 'del foo' to destroy the object, then you have a
bigger problem than callbacks, because that's simply not how Python
works. You can't _ever_ assume that deleting something from your local
namespace will destroy the object, because there can always be more
references. So maybe you need a more clear way of saying "I'm done
with this, get rid of it".

ChrisA

[Cem Karan]

On Feb 21, 2015, at 12:41 AM, Frank Millman <fr...@chagford.com> wrote:

>
> "Cem Karan" <cfka...@gmail.com> wrote in message
> news:33677AE8-B2FA-49F9...@gmail.com...
>> Hi all, I'm working on a project that will involve the use of callbacks,
>> and I want to bounce an idea I had off of everyone to make sure I'm not
>> developing a bad idea. Note that this is for python 3.4 code; I don't
>> need to worry about any version of python earlier than that.
>>

>> In order to inform users that certain bits of state have changed, I
>> require them to register a callback with my code. The problem is that
>> when I store these callbacks, it naturally creates a strong reference to
>> the objects, which means that if they are deleted without unregistering
>> themselves first, my code will keep the callbacks alive. Since this could
>> lead to really weird and nasty situations, I would like to store all the
>> callbacks in a WeakSet
>> (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That
>> way, my code isn't the reason why the objects are kept alive, and if they
>> are no longer alive, they are automatically removed from the WeakSet,
>> preventing me from accidentally calling them when they are dead. My
>> question is simple; is this a good design? If not, why not?
>> Are there any potential 'gotchas' I should be worried about?
>>
>

> I tried something similar a while ago, and I did find a gotcha.
>

> The problem lies in this phrase - "if they are no longer alive, they are

> automatically removed from the WeakSet, preventing me from accidentally
> calling them when they are dead."
>

> I found that the reference was not removed immediately, but was waiting to
> be garbage collected. During that window, I could call the callback, which
> resulted in an error.
>
> There may have been a simple workaround. Perhaps someone else can comment.

THAT would be one heck of a gotcha! Must have been fun debugging that one!

Cem Karan

[Mark Lawrence]

> Frank Millman
>

https://docs.python.org/3/library/gc.html has a collect function. That
seems like a simple workaround, but whether or not it classifies as a
good solution I'll leave to others, I'm not qualified to say.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[Cem Karan]

On Feb 21, 2015, at 8:15 AM, Chris Angelico <ros...@gmail.com> wrote:

> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfka...@gmail.com> wrote:
>> OK, so it would violate the principle of least surprise for you. Interesting. Is this a general pattern in python? That is, callbacks are owned by what they are registered with?
>>
>> In the end, I want to make a library that offers as few surprises to the user as possible, and no matter how I think about callbacks, they are surprising to me. If callbacks are strongly-held, then calling 'del foo' on a callable object may not make it go away, which can lead to weird and nasty situations. Weakly-held callbacks mean that I (as the programmer), know that objects will go away after the next garbage collection (see Frank's earlier message), so I don't get 'dead' callbacks coming back from the grave to haunt me.
>>
>> So, what's the consensus on the list, strongly-held callbacks, or weakly-held ones?
>
> I don't know about Python specifically, but it's certainly a general
> pattern in other languages. They most definitely are owned, and it's
> the only model that makes sense when you use closures (which won't
> have any other references anywhere).

I agree about closures; its the only way they could work. When I was originally thinking about the library, I was trying to include all types of callbacks, including closures and callable objects. The callable objects may pass themselves, or one of their methods to the library, or may do something really weird.

Although I just realized that closures may cause another problem. In my code, I expect that many different callbacks can be registered for the same event. Unregistering means you request to be unregistered for the event. How do you do that with a closure? Aren't they anonymous?

> If you're expecting 'del foo' to destroy the object, then you have a
> bigger problem than callbacks, because that's simply not how Python
> works. You can't _ever_ assume that deleting something from your local
> namespace will destroy the object, because there can always be more
> references. So maybe you need a more clear way of saying "I'm done
> with this, get rid of it".

Agreed about 'del', and I don't assume that the object goes away at the point. The problem is debugging and determining WHY your object is still around. I know a combination of logging and gc.get_referrers() will probably help you figure out why something is still around, but I'm trying to avoid that headache.

I guess the real problem is how this creates cycles in the call graph. User code effectively owns the library code, which via callbacks owns the user code. I have no idea what the best point the cycle is to break it, and not surprise someone down the road. The only idea I have is to redesign the library a little, and make anything that accepts a callback actually be a subclass of collections.abc.Container, or even collections.abc.MutableSet. That makes it very obvious that the object owns the callback, and that you will need to remove your object to unregister it. The only problem is how to handle closures; since they are anonymous, how do you decide which one to remove?

Thanks,
Cem Karan

[Cem Karan]

Unfortunately, depending on how many objects you have in your object graph, it can slow your code down a fair amount. I think Frank is right about how a WeakSet might be a bad idea in this case. You really need to know if an object is alive or dead, and not some indeterminate state.

Thanks,
Cem Karan

[Devin Jeanpierre]

On Fri, Feb 20, 2015 at 9:42 PM, Chris Angelico <ros...@gmail.com> wrote:
> No, it's not. I would advise using strong references - if the callback
> is a closure, for instance, you need to hang onto it, because there
> are unlikely to be any other references to it. If I register a
> callback with you, I expect it to be called; I expect, in fact, that
> that *will* keep my object alive.

For that matter, if the callback is a method, you need to hang onto
it, because method wrappers are generated on demand, so the method
would be removed from the valid callbacks instantly.

Weak references for callbacks are broken.

-- Devin

[Chris Angelico]

On Sun, Feb 22, 2015 at 1:07 AM, Cem Karan <cfka...@gmail.com> wrote:
> I agree about closures; its the only way they could work. When I was originally thinking about the library, I was trying to include all types of callbacks, including closures and callable objects. The callable objects may pass themselves, or one of their methods to the library, or may do something really weird.
>
> Although I just realized that closures may cause another problem. In my code, I expect that many different callbacks can be registered for the same event. Unregistering means you request to be unregistered for the event. How do you do that with a closure? Aren't they anonymous?
>

They're objects, same as any other, so the caller can hang onto a
reference and then say "now remove this one". Simple example:

callbacks = []
def register_callback(f): callbacks.append(f)
def unregister_callback(f): callbacks.remove(f)
def do_callbacks():
for f in callbacks:
f()

def make_callback(i):
def inner():
print("Callback! %d"%i)
register_callback(inner)
return inner

make_callback(5)
remove_me = make_callback(6)
make_callback(7)
unregister_callback(remove_me)
do_callbacks()

The other option is for your callback registration to return some kind
of identifier, which can later be used to unregister the callback.
This is a good way of avoiding reference cycles (the ID could be a
simple integer - maybe the length of the list prior to the new
callback being appended, and then the unregistration process is simply
"callbacks[id] = None", and you skip the Nones when iterating), and
even allows you to register the exact same function more than once,
for what that's worth.

When I do GUI programming, this is usually how things work. For
instance, I use GTK2 (though usually with Pike rather than Python),
and I can connect a signal to a callback function. Any given signal
could have multiple callbacks attached to it, so it's similar to your
case. I frequently depend on the GTK engine retaining a reference to
my function (and thus to any data it requires), as I tend not to hang
onto any inner objects that don't need retention. Once the parent
object is destroyed, all its callbacks get dereferenced. Consider this
simplified form:

def popup_window():
w = Window()
# Add layout, info, whatever it takes
btn = Button("Close")
w.add(btn) # actually it'd be added to a layout
btn.signal_connect("clicked", lambda *args: w.destroy())

The GUI back end will hang onto a reference to the window, because
it's currently on screen; to the button, because it's attached to the
window; and to my function, because it's connected to a button signal.
Then when you click the button, the window gets destroyed, which
destroys the button, which unregisters all its callbacks. At that
point, there are no refs to the function, so it can get disposed of.
That button function was the last external reference to the window,
and now that it's not on screen, its Python object can also be
disposed of, as can the button inside. So it'll all clean up fairly
nicely; as long as the callback gets explicitly deregistered, that's
the end of everything.

ChrisA

[Cem Karan]

On Feb 21, 2015, at 9:36 AM, Chris Angelico <ros...@gmail.com> wrote:

> On Sun, Feb 22, 2015 at 1:07 AM, Cem Karan <cfka...@gmail.com> wrote:
>> I agree about closures; its the only way they could work. When I was originally thinking about the library, I was trying to include all types of callbacks, including closures and callable objects. The callable objects may pass themselves, or one of their methods to the library, or may do something really weird.
>>
>> Although I just realized that closures may cause another problem. In my code, I expect that many different callbacks can be registered for the same event. Unregistering means you request to be unregistered for the event. How do you do that with a closure? Aren't they anonymous?
>>
>
> They're objects, same as any other, so the caller can hang onto a
> reference and then say "now remove this one". Simple example:
>
> callbacks = []
> def register_callback(f): callbacks.append(f)
> def unregister_callback(f): callbacks.remove(f)
> def do_callbacks():
> for f in callbacks:
> f()
>
> def make_callback(i):
> def inner():
> print("Callback! %d"%i)
> register_callback(inner)
> return inner
>
> make_callback(5)
> remove_me = make_callback(6)
> make_callback(7)
> unregister_callback(remove_me)
> do_callbacks()

Yeah, that's pretty much what I thought you'd have to do, which kind of defeats the purpose of closures (fire-and-forget things). BUT it does answer my question, so no complaints about it!

So, either you keep a reference to your own closure, which means that the library doesn't really need to, or the library keeps hold of it for you, in which case you don't have a reasonable way of removing it.

> The other option is for your callback registration to return some kind
> of identifier, which can later be used to unregister the callback.
> This is a good way of avoiding reference cycles (the ID could be a
> simple integer - maybe the length of the list prior to the new
> callback being appended, and then the unregistration process is simply
> "callbacks[id] = None", and you skip the Nones when iterating), and
> even allows you to register the exact same function more than once,
> for what that's worth.

That would work. In the cases where someone might register & unregister many callbacks, you might use UUIDs as keys instead (avoids the ABA problem).

OK, so if I'm reading your code correctly, you're breaking the cycle in your object graph by making the GUI the owner of the callback, correct? No other chunk of code has a reference to the callback, correct?

Thanks,
Cem Karan

[Chris Angelico]

On Sun, Feb 22, 2015 at 2:45 AM, Cem Karan <cfka...@gmail.com> wrote:
> OK, so if I'm reading your code correctly, you're breaking the cycle in your object graph by making the GUI the owner of the callback, correct? No other chunk of code has a reference to the callback, correct?

Correct. The GUI engine ultimately owns everything. Of course, this is
a very simple case (imagine a little notification popup; you don't
care about it, you don't need to know when it's been closed, the only
event on it is "hit Close to destroy the window"), and most usage
would have other complications, but it's not uncommon for me to build
a GUI program that leaves everything owned by the GUI engine.
Everything is done through callbacks. Destroy a window, clean up its
callbacks. The main window will have an "on-deletion" callback that
terminates the program, perhaps. It's pretty straight-forward.

ChrisA

[Marko Rauhamaa]

Chris Angelico <ros...@gmail.com>:

> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan <cfka...@gmail.com> wrote:

>> In order to inform users that certain bits of state have changed, I
>> require them to register a callback with my code. The problem is that
>> when I store these callbacks, it naturally creates a strong reference
>> to the objects, which means that if they are deleted without
>> unregistering themselves first, my code will keep the callbacks
>> alive. Since this could lead to really weird and nasty situations,

>> [...]

>
> No, it's not. I would advise using strong references - if the callback
> is a closure, for instance, you need to hang onto it, because there
> are unlikely to be any other references to it. If I register a
> callback with you, I expect it to be called; I expect, in fact, that
> that *will* keep my object alive.

I use callbacks all the time but haven't had any problems with strong
references.

I am careful to move my objects to a zombie state after they're done so
they can absorb any potential loose callbacks that are lingering in the
system.


Marko

[Steven D'Aprano]

Frank Millman wrote:

> I tried something similar a while ago, and I did find a gotcha.
>
> The problem lies in this phrase - "if they are no longer alive, they are
> automatically removed from the WeakSet, preventing me from accidentally
> calling them when they are dead."
>
> I found that the reference was not removed immediately, but was waiting to
> be garbage collected. During that window, I could call the callback, which
> resulted in an error.

I don't understand how this could possibly work. (Or fail to work, as the
case may be.)

If the callback has been garbage collected, then you cannot call it, because
you don't have any references to it and so cannot refer to it in any way.

If the callback has *not* been garbage collected, then you can safely call
it. You have a reference to the callback, therefore it exists. (If Python
ever garbage collects an object that still has references to it, that would
be a critical bug, and you would likely get some sort of seg fault).

The only thing I can think of is you have a situation where your callback
refers to another object, B, via a weak reference. Once all the regular
strong references to the callback and B are gone, theoretically you could
have a race condition where the callback is waiting to be garbage collected
but B has already been garbage collected. If, in that window, you call the
callback, *and* if the callback fails to correctly check that the weak
reference to B still exists, then you could get a Python exception.

The solution is simple: anytime you have a weak reference, you must always
check it before you use it.

Other than that, I cannot see how calling a function which has *not* yet
been garbage collected can fail, just because the only reference still
existing is a weak reference.


--
Steven

[Marko Rauhamaa]

Steven D'Aprano <steve+comp....@pearwood.info>:

> Other than that, I cannot see how calling a function which has *not*
> yet been garbage collected can fail, just because the only reference
> still existing is a weak reference.

Maybe the logic of the receiving object isn't prepared for the callback
anymore after an intervening event.

The problem then, of course, is in the logic and not in the callbacks.


Marko

[Steven D'Aprano]

Cem Karan wrote:

>
> On Feb 21, 2015, at 8:15 AM, Chris Angelico <ros...@gmail.com> wrote:
>
>> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfka...@gmail.com> wrote:
>>> OK, so it would violate the principle of least surprise for you.
>>> Interesting. Is this a general pattern in python? That is, callbacks
>>> are owned by what they are registered with?
>>>
>>> In the end, I want to make a library that offers as few surprises to the
>>> user as possible, and no matter how I think about callbacks, they are
>>> surprising to me. If callbacks are strongly-held, then calling 'del
>>> foo' on a callable object may not make it go away, which can lead to
>>> weird and nasty situations.

How?

The whole point of callbacks is that you hand over responsibility to another
piece of code, and then forget about your callback. The library will call
it, when and if necessary, and when the library no longer needs your
callback, it is free to throw it away. (If I wish the callback to survive
beyond the lifetime of your library's use of it, I have to keep a reference
to the function.)

>>> Weakly-held callbacks mean that I (as the
>>> programmer), know that objects will go away after the next garbage
>>> collection (see Frank's earlier message), so I don't get 'dead'
>>> callbacks coming back from the grave to haunt me.

I'm afraid this makes no sense to me. Can you explain, or better still
demonstrate, a scenario where "dead callbacks rise from the grave", so to
speak?

>>> So, what's the consensus on the list, strongly-held callbacks, or
>>> weakly-held ones?
>>
>> I don't know about Python specifically, but it's certainly a general
>> pattern in other languages. They most definitely are owned, and it's
>> the only model that makes sense when you use closures (which won't
>> have any other references anywhere).
>
> I agree about closures; its the only way they could work.

*scratches head* There's nothing special about closures. You can assign them
to a name like any other object.

def make_closure():
x = 23
def closure():
return x + 1
return closure

func = make_closure()

Now you can register func as a callback, and de-register it when your done:

register(func)
unregister(func)


Of course, if you thrown away your reference to func, you have no (easy) way
of de-registering it. That's no different to any other object which is
registered by identity. (Registering functions by name is a bad idea, since
multiple functions can have the same name.)

As an alternative, your callback registration function might return a ticket
for the function:

ticket = register(func)
del func
unregister(ticket)

but that strikes me as over-kill. And of course, the simplest ticket is to
return the function itself :-)

> When I was
> originally thinking about the library, I was trying to include all types
> of callbacks, including closures and callable objects. The callable
> objects may pass themselves, or one of their methods to the library, or
> may do something really weird.

I don't think they can do anything too weird. They have to pass a callable
object. Your library just calls that object. You shouldn't need to care
whether it is a function, a method, a type, a callable instance, or
something else. You just call it, and when you're done calling it forever,
you just throw it away.

> Although I just realized that closures may cause another problem. In my
> code, I expect that many different callbacks can be registered for the
> same event. Unregistering means you request to be unregistered for the
> event. How do you do that with a closure? Aren't they anonymous?

Not unless you create them using lambda. Using the make_closure function
above:


py> func = make_closure()
py> func.__name__
'closure'

Of course, if you call make_closure twice, both functions will have the same
internal name. You can set the function __name__ and __qualname__ to fix
that. This is how the functools.wraps decorator works.

But that's a red herring. Don't register functions by name! Not all callable
objects have names, and those that do, you may have multiple *distinct*
callbacks with the same name.

There are two reasonable approaches: unregister by identity, or by returning
a ticket which uniquely identifies the callback. The user is responsible
for keeping track of their own ticket. If I lose it, I can't unregister my
callback any more. So sad, sucks to be me.


The simplest possible identity-based scheme would be something like this:


# don't hate me for using a global variable
CALLBACKS = []

def register(func):
if func not in CALLBACKS:
CALLBACKS.append(func)

def unregister(func):
try:
CALLBACKS.remove(func)
except ValueError:
pass


That's probably a bit too simple, since it won't behave as expected with
bound methods. The problem is that bound methods are generated on the fly,
so this won't work:

register(instance.spam)
# later
unregister(instance.spam) # a different instance!

I would have to do this:

bound_method = instance.spam
register(bound_method)
unregister(bound_method)


But a more sophisticated unregister function should work:

# Untested
def unregister(func):
for i, f in enumerate(CALLBACKS):
if (f is func) or (isinstance(f, types.MethodType)
and f.__wrapped__ is func):
del CALLBACKS[i]
return


The simplest possible ticket-based system is probably something like this:

CALLBACKS = {}
NEXT_TICKET = 1

def register(func):
global NEXT_TICKET
ticket = NEXT_TICKET
NEXT_TICKET += 1
callbacks[ticket] = func
return ticket

def unregister(ticket):
if ticket in CALLBACKS:
del CALLBACKS[ticket]

>> If you're expecting 'del foo' to destroy the object, then you have a
>> bigger problem than callbacks, because that's simply not how Python
>> works. You can't _ever_ assume that deleting something from your local
>> namespace will destroy the object, because there can always be more
>> references. So maybe you need a more clear way of saying "I'm done
>> with this, get rid of it".
>
> Agreed about 'del', and I don't assume that the object goes away at the
> point. The problem is debugging and determining WHY your object is still
> around. I know a combination of logging and gc.get_referrers() will
> probably help you figure out why something is still around, but I'm trying
> to avoid that headache.

Why do you care? Surely all your library should care about is whether or not
they have a reference to the callback.If they do, they should call it (when
appropriate). If they don't, they aren't responsible for it.

> I guess the real problem is how this creates cycles in the call graph.
> User code effectively owns the library code, which via callbacks owns the
> user code. I have no idea what the best point the cycle is to break it,
> and not surprise someone down the road. The only idea I have is to
> redesign the library a little, and make anything that accepts a callback
> actually be a subclass of collections.abc.Container, or even
> collections.abc.MutableSet. That makes it very obvious that the object
> owns the callback, and that you will need to remove your object to
> unregister it.

My brain hurts from the complexity of your solution. What is the actual
problem you are trying to solve? I would like to see an example of an
actual failure before trying to solve a purely theoretical failure mode.

If I register something as a callback, I expect that callback will stay
alive for as long as the callbacks are needed. If I might want to
unregister it, then I have to keep a reference to the function, otherwise
how will I know what I am unregistering?

# this makes no sense and cannot work
register(func)
del func
# later
unregister(func)

So if I do that, (1) it won't work; (2) I'll probably get an exception; (3)
the solution is "don't do that"; and (4) solving this problem is not YOUR
responsibility.

When your code is done with the callbacks, you can just remove them, no
questions asked. If I still have a reference to the callback, that
reference will still be valid no matter what you do. If I don't have a
reference to it, presumably that's because I don't need it any more. I
can't access the callback anyway.

> The only problem is how to handle closures; since they are
> anonymous, how do you decide which one to remove?

You identify them by identity, or by a ticket, the same as for any other
object.



--
Steven

[Grant Edwards]

On 2015-02-21, Cem Karan <cfka...@gmail.com> wrote:
>
> On Feb 21, 2015, at 12:42 AM, Chris Angelico <ros...@gmail.com> wrote:
>
>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan <cfka...@gmail.com> wrote:
>>> In order to inform users that certain bits of state have changed, I require them to register a callback with my code. The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive. Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead. My question is simple; is this a good design? If not, why not? Are there any potential 'gotchas' I should be worried about?
>>>
>>
>> No, it's not. I would advise using strong references - if the callback
>> is a closure, for instance, you need to hang onto it, because there
>> are unlikely to be any other references to it. If I register a
>> callback with you, I expect it to be called; I expect, in fact, that
>> that *will* keep my object alive.
>
> OK, so it would violate the principle of least surprise for you.

And me as well. I would expect to be able to pass a closure as a
callback and not have to keep a reference to it. Perhaps that just a
leftover from working with other languages (javascript, scheme, etc.).
It doesn't matter if it's a string, a float, a callback, a graphic or
whatever: if I pass your function/library an object, I expect _you_ to
keep track of it until you're done with it.

> Interesting. Is this a general pattern in python? That is,
> callbacks are owned by what they are registered with?

I'm not sure what you mean by "owned" or why it matters that it's a
callback: it's an object that was passed to you: you need to hold onto
a reference to it until you're done with it, and the polite thing to
do is to delete references to it when you're done with it.

> So, what's the consensus on the list, strongly-held callbacks, or
> weakly-held ones?

--
Grant

[Marko Rauhamaa]

Grant Edwards <inv...@invalid.invalid>:

> the polite thing to do is to delete references to it when you're done
> with it.

I disagree with that recommendation. You should do the natural thing and
not care who holds references to who.


Marko

[Steven D'Aprano]

I don't understand this. What is "the natural thing" if not to delete
references to an object when you are done with it? Normally you just let
things go out of scope, but if that won't happen, you have to take active
steps, such as calling del or setting the reference to None.


--
Steven

[Chris Angelico]

I think the disagreement here is over the interpretation of "done with
it". If you drop all references to a connected socket object, Python
can rightly assume that you're done with the file and want to close
it; but what if you drop all references to a listening socket that's
been configured to call a function whenever someone connects?

def client_connected(sock):
sock.send("Hello!\r\n")
# whatever

listener = socket(23, on_accept=client_connected)

What should happen if that main socket isn't bound to a name? In my
opinion, the fact that it's configured for callback mode should mean
that it's kept alive. But it's also understandable to want to treat it
as "done", that it can be disposed of. It seems weird to me that you
should have to have a name somewhere that you'll never use, though.

ChrisA

[Steven D'Aprano]

But you are using it. You might not be using it by name, but you are using
it via the callback function. What did you expect, that Python should read
your mind and somehow intuit that you still care about this socket
listener, but not some other socket listener that you are done with?

If you have a servant that follows you around everywhere, throwing objects
away when you stop using them, then naturally if you stop using something
it will be thrown away. What did you expect?

You don't have to bind the listener to a name. Any reference will do. You
can dump it in a bucket:

bucket_of_stuff = []
bucket_of_stuff.append(some_function(a, b, c))
bucket_of_stuff.append(make_web_server())
bucket_of_stuff.append(socket(23, on_accept=client_connected))


So long as your bucket is still alive, the garbage collector won't collect
it or its contents.

Hypothetically we could have a system in place where you instruct the
garbage collector to not collect an object, then drop all references to it:

gc.never_collect(socket(23, on_accept=client_connected))

but that's a potential memory leak, because now you have no way of telling
the GC to collect it again once you've finished listening. If you never
finish listening, or at least not until your application shuts down, it
doesn't count as a leak. But in general, if listeners might come and go,
but you have no way for them to be garbage collected once they are done,
then that's a leak.

What's the easiest way for the GC to flag an object as "never collect this"?
It can keep a reference to it. Keeping a list of things you want to be kept
alive is simple, easy and obvious:

# hypothetically inside gc.py
_ALIVE = []
def never_collect(obj):
_ALIVE.append(obj)


but that's so trivial that it's not worth putting it in the gc module.
Besides, that means any other code could reach into the gc and remove your
listener from the "keep alive list" without your knowledge or permission.

It's simpler and safer to just keep it alive yourself:

alive = []
alive.append(socket(...))

but of course that's just my bucket of stuff under a different name.

Using the idiom "keep objects alive by keeping a reference to them" (e.g.
bind them to a name, or stick them in a list which you keep) makes things
much simpler. You can trivially flag objects as "collect" or "don't
collect" as needed, without having to import the gc module or memorise some
obscure API or worry about implementation details ("if I flag an object
as 'never collect' *twice*, do I have to unflag it twice to undo it?"). You
just use the regular interface to the garbage collector: the existence of a
reference, any reference, keeps an object alive.




--
Steven

[Chris Angelico]

On Sun, Feb 22, 2015 at 3:38 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> But you are using it. You might not be using it by name, but you are using
> it via the callback function. What did you expect, that Python should read
> your mind and somehow intuit that you still care about this socket
> listener, but not some other socket listener that you are done with?
>

> You don't have to bind the listener to a name. Any reference will do. You
> can dump it in a bucket:
>
> bucket_of_stuff = []
> bucket_of_stuff.append(some_function(a, b, c))
> bucket_of_stuff.append(make_web_server())
> bucket_of_stuff.append(socket(23, on_accept=client_connected))

Sure, and whether it's a name or a list-element reference doesn't
matter: it seems wrong to have to stash a thing in a bucket in order
to keep its callbacks alive. I expect the callbacks _themselves_ to
keep it alive. But I can understand the opposite POV.

ChrisA

[Frank Millman]

"Steven D'Aprano" <steve+comp....@pearwood.info> wrote in message
news:54e8af1b$0$12976$c3e8da3$5496...@news.astraweb.com...

You are right. I tried to reproduce the problem and I can't.

Before describing what I think was happening, I want to clarify something.

Most of this thread uses the word 'callback' in the sense of an
'asynchronous' scenario - the caller wants something to happen some time in
the future, and then forget about it, but it is important that it does
actually happen eventually.

That is not what I was doing, and it is not what I thought the OP was asking
for.

"In order to inform users that certain bits of state have changed, I require
them to register a callback with my code."

This sounds to me like a pub/sub scenario. When a 'listener' object comes
into existence it is passed a reference to a 'controller' object that holds
state. It wants to be informed when the state changes, so it registers a
callback function with the controller. When the controller detects a change
in state, it calls all the callback functions, thereby notifying each
listener. When the listener goes out of scope, it is important that it
deregisters with the controller.

Now back to my scenario. You are right that so long as the controller
maintains a reference to the callback function, the listener cannot be
garbage collected, and therefore the callback will always succeed.

As far as I can remember, I had a situation where the listener used the
information to pass the information to a gui on a client. When the listener
was no longer required, a close() fiunction was called which cleaned up and
closed connections. When the callback was called after the listener was
closed, whatever it was trying to do failed (I forget the details).

Hope this makes sense.

Frank

[Ian Kelly]

Count me in the weak-ref crowd. It may be a nuisance to keep a
reference around on the object registering the callback, but it's
preferable to the alternative of messing around with disposables in
order to ensure that the callback gets cleaned up and doesn't create a
memory leak. I would also rather have my code fail by losing a
callback reference, which should be relatively easy to spot and
diagnose, than to have said memory leak go unnoticed.

[Marko Rauhamaa]

Steven D'Aprano <steve+comp....@pearwood.info>:

Obviously, if you are hoarding objects in a collection, you're going to
land in trouble.

What I mean, though, is that you shouldn't think you need to create
object destructors where you routinely set all members to None.


Marko

[Chris Angelico]

On Sun, Feb 22, 2015 at 6:52 PM, Marko Rauhamaa <ma...@pacujo.net> wrote:
> What I mean, though, is that you shouldn't think you need to create
> object destructors where you routinely set all members to None.

Sure, not *routinely*. It'd be a special case where it's not
specifically a destructor, and its job is to break a reference cycle.
For instance, you might have a close() method that clears out a bunch
of references, which will then allow everything to get cleaned up
promptly. Or (a very common case for me) a callback saying "remote end
is gone" (eg on a socket) might wipe out the callbacks, thus removing
their refloops.

ChrisA

[Marko Rauhamaa]

Chris Angelico <ros...@gmail.com>:

> Or (a very common case for me) a callback saying "remote end is gone"
> (eg on a socket) might wipe out the callbacks, thus removing their
> refloops.

Refloops are not to be worried about, let alone removed.


Marko

[Chris Angelico]

Why? They force the use of the much slower cycle-detecting GC, rather
than the quick and efficient CPython refcounter. I don't know how
other Pythons work, but mark-and-sweep has its own costs, and I don't
know of any system that's both prompt and able to detect refloops.
Helping it along means your program doesn't waste memory. Why such a
blanket statement?

ChrisA

[Marko Rauhamaa]

Chris Angelico <ros...@gmail.com>:

> On Sun, Feb 22, 2015 at 7:34 PM, Marko Rauhamaa <ma...@pacujo.net> wrote:
>> Refloops are not to be worried about, let alone removed.
>
> Why?

Because the whole point of GC-languages is that you should stop worrying
about memory. Trying to mastermind and micromanage GC in the application
is, pardon my French, an antipattern.

> They force the use of the much slower cycle-detecting GC, rather than
> the quick and efficient CPython refcounter.

Java's Hotspot doesn't bother with refcounters but is much faster than
Python. CPython's refcounters are a historical accident that a Python
application developer shouldn't even be aware of.

> I don't know how other Pythons work, but mark-and-sweep has its own
> costs, and I don't know of any system that's both prompt and able to
> detect refloops.

It's exceedingly difficult (and pointless) to detect cycles in your
object structures. Python is going to have to do a GC occasionally
anyway. Yes, your worst-case response times are going to suffer, but
that's the cost of doing business.

> Helping it along means your program doesn't waste memory. Why such a
> blanket statement?

Because worrying Python programmers with evil spirits (reference loops)
leads to awkward coding practices and takes away one of the main
advantages of Python as a high-level programming language.


Marko

[Gregory Ewing]

Frank Millman wrote:
> "In order to inform users that certain bits of state have changed, I require
> them to register a callback with my code."
>
> This sounds to me like a pub/sub scenario. When a 'listener' object comes
> into existence it is passed a reference to a 'controller' object that holds
> state. It wants to be informed when the state changes, so it registers a
> callback function with the controller.

Perhaps instead of registering a callback function, you
should be registering the listener object together with
a method name.

You can then keep a weak reference to the listener object,
since if it is no longer referenced elsewhere, it presumably
no longer needs to be notified of anything.

--
Greg

[Chris Angelico]

On Sun, Feb 22, 2015 at 8:14 PM, Marko Rauhamaa <ma...@pacujo.net> wrote:
>> Helping it along means your program doesn't waste memory. Why such a
>> blanket statement?
>
> Because worrying Python programmers with evil spirits (reference loops)
> leads to awkward coding practices and takes away one of the main
> advantages of Python as a high-level programming language.

Right, and I suppose that, by extension, we should assume that the
Python interpreter can optimize this?

def fib(x):
if x<2: return x
return fib(x-2)+fib(x-1)

Just because a computer can, in theory, recognize that this is a pure
function, doesn't mean that we can and should depend on that. If you
want this to be optimized, you either fix your algorithm or explicitly
memoize the function - you don't assume that Python can do it for you.

Even when you write in a high level language, you need to understand
how computers work.

ChrisA

[Steven D'Aprano]

Why? Do you expect that the Python garbage collector special cases callbacks
to keep them alive even when there are no references to them? How would it
distinguish a callback from some other function?

If I stuff a function in a list:

[len]

would you expect the presence of the function to keep the list alive when
there are no references to the list?

Apart from "But I really, really, REALLY want a magical pony that feeds
itself and never poops and just disappears when I don't want it around!"
wishful-thinking, which I *totally* get, I don't see how you think this is
even possible. Maybe I'm missing something, but it seems to me that what
you're wishing for is impossible.

Perhaps if we had a special "callback" type which was treated as a special
case by the garbage collector. But that opens up a big can of worms: how do
you close/delete objects which are kept alive by the presence of callbacks
if you don't have a reference to either the object or the callback?



--
Steven

[Chris Angelico]

On Sun, Feb 22, 2015 at 9:32 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> Why? Do you expect that the Python garbage collector special cases callbacks
> to keep them alive even when there are no references to them? How would it
> distinguish a callback from some other function?

No no no. It's the other way around. _Something_ has to be doing those
callbacks, and it's that _something_ that should be keeping them
alive. The fact that it's a registered callback should itself *be* a
reference (and not a weak reference), and should keep it alive.

ChrisA

[Laura Creighton]

somebody, I got confused with the indent level wrote:

>> They force the use of the much slower cycle-detecting GC, rather than
>> the quick and efficient CPython refcounter.

Somebody has misunderstood something here. When it comes to efficient
garbage collectors, refcounting is a turtle. The CPython one is no
exception. Ref counting, however, is fairly easy to write. But when
the PyPy project first replaced its refcounting gc with its very first
and therefore not very efficient at all nursery gc ... that was the very
first time when a bunch of python programs ran faster on pypy than on
CPython. This was before pypy had a JIT.

And today the pypy channel is full of people who want to link their
C extension into some Python code running on PyPy, and who find that
their C extension slows things down. There are lots of reasons for
this, but one of the most common problems is 'this C extension is
faking refcounting. All of this is wasted effort for PyPy and
usually makes the thing unJITable as well.' Many of these people
rewrite their C extension as pure Python and find that then, with
PyPy, they get the speed improvements they were looking for.

So: two points.

One reason you might not want to rely on ref counting, because you expect
your code to run under PyPy one day.

and

If you are interested in manipulating garbage collection -- especially if
this is for your own pleasure and enjoyment, a worthy goal in my books --
you could do a lot worse than write your own gc in RPython for PyPy.
The gc code is not mixed in with all of the other VM stuff, so a gc is
small, and you don't have to worry about clobbering anything else while
you are working. So it is great for experimenting, which was the whole
point. Hacking gcs is fun! :)

Laura

[Cem Karan]

How do you handle returning information? E.g., the user types in a number and expects that to update the internal state of your code somewhere.

Thanks,
Cem Karan

[Cem Karan]

So, if I were designing a library for you, you would be willing to have a 'zombie' attribute on your callback, correct? This would allow the library to query its callbacks to ensure that only 'live' callbacks are called. How would you handle closures?

Thanks,
Cem Karan

[Marko Rauhamaa]

Cem Karan <cfka...@gmail.com>:

Sorry, don't understand the question.


Marko

[Cem Karan]

This was PRECISELY the situation I was thinking about. My hope was to make the callback mechanism slightly less surprising by allowing the user to track them, releasing them when they aren't needed without having to figure out where the callbacks were registered. However, it appears I'm making things more surprising rather than less.

Thanks,
Cem Karan

[Chris Angelico]

On Sun, Feb 22, 2015 at 11:07 PM, Cem Karan <cfka...@gmail.com> wrote:
>> Correct. The GUI engine ultimately owns everything. Of course, this is
>> a very simple case (imagine a little notification popup; you don't
>> care about it, you don't need to know when it's been closed, the only
>> event on it is "hit Close to destroy the window"), and most usage
>> would have other complications, but it's not uncommon for me to build
>> a GUI program that leaves everything owned by the GUI engine.
>> Everything is done through callbacks. Destroy a window, clean up its
>> callbacks. The main window will have an "on-deletion" callback that
>> terminates the program, perhaps. It's pretty straight-forward.
>
> How do you handle returning information? E.g., the user types in a number and expects that to update the internal state of your code somewhere.

Not sure what you mean by "returning". If the user types in a number
in a GUI widget, that would trigger some kind of on-change event, and
either the new text would be a parameter to the callback function, or
the callback could query the widget. In the latter case, I'd probably
have the callback as a closure, and thus able to reference the object.

ChrisA

[Marko Rauhamaa]

Cem Karan <cfka...@gmail.com>:

When dealing with callbacks, my advice is to create your objects as
explicit finite state machines. Don't try to encode the object state
implicitly or indirectly. Rather, give each and every state a symbolic
name and log the state transitions for troubleshooting.

Your callbacks should then consider what to do in each state. There are
different ways to express this in Python, but it always boils down to a
state/transition matrix.

Callbacks sometimes cannot be canceled after they have been committed to
and have been shipped to the event pipeline. Then, the receiving object
must brace itself for the impending spurious callback.


Marko

[Laura Creighton]

In a message of Sun, 22 Feb 2015 07:16:14 -0500, Cem Karan writes:

>This was PRECISELY the situation I was thinking about. My hope was
>to make the callback mechanism slightly less surprising by allowing
>the user to track them, releasing them when they aren't needed
>without having to figure out where the callbacks were registered.
>However, it appears I'm making things more surprising rather than
>less.

You may be able to accomplish your goal by using a Queue with a
producer/consumer model.
see: http://stackoverflow.com/questions/9968592/turn-functions-with-a-callback-into-python-generators

especially the bottom of that.

I haven't run the code, but it looks mostly reasonable, except that
you do not want to rely on the Queue maxsize being 1 here, and
indeed, I almost always want a bigger Queue in any case. Use
Queue.task_done if blocking the producer features in your design.

The problem that you are up against is that callbacks are inherantly
confusing, even to programmers who are learning about them for the
first time. They don't fit people's internal model of 'how code works'.
There isn't a whole lot one can do about that except to
try to make the magic do as little as possible, so that more of the
code works 'the way people expect'.

Laura

[Cem Karan]

On Feb 21, 2015, at 12:27 PM, Steven D'Aprano <steve+comp....@pearwood.info> wrote:

> Cem Karan wrote:
>
>>
>> On Feb 21, 2015, at 8:15 AM, Chris Angelico <ros...@gmail.com> wrote:

>>
>>> On Sun, Feb 22, 2015 at 12:13 AM, Cem Karan <cfka...@gmail.com> wrote:
>>>> OK, so it would violate the principle of least surprise for you.

>>>> Interesting. Is this a general pattern in python? That is, callbacks
>>>> are owned by what they are registered with?
>>>>

>>>> In the end, I want to make a library that offers as few surprises to the
>>>> user as possible, and no matter how I think about callbacks, they are
>>>> surprising to me. If callbacks are strongly-held, then calling 'del
>>>> foo' on a callable object may not make it go away, which can lead to
>>>> weird and nasty situations.
>
> How?
>
> The whole point of callbacks is that you hand over responsibility to another
> piece of code, and then forget about your callback. The library will call
> it, when and if necessary, and when the library no longer needs your
> callback, it is free to throw it away. (If I wish the callback to survive
> beyond the lifetime of your library's use of it, I have to keep a reference
> to the function.)

Marko mentioned it earlier; if you think you've gotten rid of all references to some chunk of code, and it is still alive afterwards, that can be surprising.

>>>> Weakly-held callbacks mean that I (as the
>>>> programmer), know that objects will go away after the next garbage
>>>> collection (see Frank's earlier message), so I don't get 'dead'
>>>> callbacks coming back from the grave to haunt me.
>
> I'm afraid this makes no sense to me. Can you explain, or better still
> demonstrate, a scenario where "dead callbacks rise from the grave", so to
> speak?

"""
#! /usr/bin/env python

class Callback_object(object):
def __init__(self, msg):
self._msg = msg
def callback(self, stuff):
print("From {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
def __init__(self):
self._callbacks = list()
def register_callback(self, callback):
self._callbacks.append(callback)
def execute_callbacks(self):
for thing in self._callbacks:
thing('Surprise!')

if __name__ == "__main__":
foo = Callback_object("Evil Zombie")
lib = Fake_library()
lib.register_callback(foo.callback)

# Way later, after the user forgot all about the callback above
foo = Callback_object("Your Significant Other")
lib.register_callback(foo.callback)

# And finally getting around to running all those callbacks.
lib.execute_callbacks()
"""

Output:
From Evil Zombie: Surprise!
From Your Significant Other: Surprise!

In this case, the user made an error (just as Marko said in his earlier message), and forgot about the callback he registered with the library. The callback isn't really rising from the dead; as you say, either its been garbage collected, or it hasn't been. However, you may not be ready for a callback to be called at that moment in time, which means you're surprised by unexpected behavior.

>>>> So, what's the consensus on the list, strongly-held callbacks, or
>>>> weakly-held ones?
>>>

>>> I don't know about Python specifically, but it's certainly a general
>>> pattern in other languages. They most definitely are owned, and it's
>>> the only model that makes sense when you use closures (which won't
>>> have any other references anywhere).
>>
>> I agree about closures; its the only way they could work.
>
> *scratches head* There's nothing special about closures. You can assign them
> to a name like any other object.
>
> def make_closure():
> x = 23
> def closure():
> return x + 1
> return closure
>
> func = make_closure()
>
> Now you can register func as a callback, and de-register it when your done:
>
> register(func)
> unregister(func)
>
>
> Of course, if you thrown away your reference to func, you have no (easy) way
> of de-registering it. That's no different to any other object which is
> registered by identity. (Registering functions by name is a bad idea, since
> multiple functions can have the same name.)
>
> As an alternative, your callback registration function might return a ticket
> for the function:
>
> ticket = register(func)
> del func
> unregister(ticket)
>
> but that strikes me as over-kill. And of course, the simplest ticket is to
> return the function itself :-)

Agreed on all points; closures are just ordinary objects. The only difference (in my opinion) is that they are 'fire and forget'; if you are registering or tracking them then you've kind of defeated the purpose. THAT is what I meant about how you handle closures.

>
>> When I was
>> originally thinking about the library, I was trying to include all types
>> of callbacks, including closures and callable objects. The callable
>> objects may pass themselves, or one of their methods to the library, or
>> may do something really weird.
>
> I don't think they can do anything too weird. They have to pass a callable
> object. Your library just calls that object. You shouldn't need to care
> whether it is a function, a method, a type, a callable instance, or
> something else. You just call it, and when you're done calling it forever,
> you just throw it away.

That doesn't quite solve the problem, but it comes close. The headache (as shown in my earlier code) is that you think you've gotten rid of something before it is called, but it turns out you haven't. I'm starting to think that there isn't a solution to this other than telling the programmer "Don't do that".

>> Although I just realized that closures may cause another problem. In my
>> code, I expect that many different callbacks can be registered for the
>> same event. Unregistering means you request to be unregistered for the
>> event. How do you do that with a closure? Aren't they anonymous?
>
> Not unless you create them using lambda. Using the make_closure function
> above:
>
>
> py> func = make_closure()
> py> func.__name__
> 'closure'
>
> Of course, if you call make_closure twice, both functions will have the same
> internal name. You can set the function __name__ and __qualname__ to fix
> that. This is how the functools.wraps decorator works.
>
> But that's a red herring. Don't register functions by name! Not all callable
> objects have names, and those that do, you may have multiple *distinct*
> callbacks with the same name.
>
> There are two reasonable approaches: unregister by identity, or by returning
> a ticket which uniquely identifies the callback. The user is responsible
> for keeping track of their own ticket. If I lose it, I can't unregister my
> callback any more. So sad, sucks to be me.
>
>
> The simplest possible identity-based scheme would be something like this:
>
>
> # don't hate me for using a global variable
> CALLBACKS = []
>
> def register(func):
> if func not in CALLBACKS:
> CALLBACKS.append(func)
>
> def unregister(func):
> try:
> CALLBACKS.remove(func)
> except ValueError:
> pass
>
>
> That's probably a bit too simple, since it won't behave as expected with
> bound methods. The problem is that bound methods are generated on the fly,
> so this won't work:
>
> register(instance.spam)
> # later
> unregister(instance.spam) # a different instance!
>
> I would have to do this:
>
> bound_method = instance.spam
> register(bound_method)
> unregister(bound_method)
>
>
> But a more sophisticated unregister function should work:
>
> # Untested
> def unregister(func):
> for i, f in enumerate(CALLBACKS):
> if (f is func) or (isinstance(f, types.MethodType)
> and f.__wrapped__ is func):
> del CALLBACKS[i]
> return

Are you sure about that? I just tested out the following code, and it appears to work correctly:

"""
#! /usr/bin/env python

class Callback_object(object):
def __init__(self, msg):
self._msg = msg
def callback(self, stuff):
print("From {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
def __init__(self):
self._callbacks = set()
def register_callback(self, callback):
self._callbacks.add(callback)
def unregister_callback(self, callback):
self._callbacks.discard(callback)
def execute_callbacks(self):
for thing in self._callbacks:
thing('Surprise!')

if __name__ == "__main__":
foo = Callback_object("Evil Zombie")
lib = Fake_library()
lib.register_callback(foo.callback)
lib.unregister_callback(foo.callback)
lib.execute_callbacks()
"""

I'll admit though, I don't know if it worked because I got lucky, or if python guarantees it works...

> The simplest possible ticket-based system is probably something like this:
>
> CALLBACKS = {}
> NEXT_TICKET = 1
>
> def register(func):
> global NEXT_TICKET
> ticket = NEXT_TICKET
> NEXT_TICKET += 1
> callbacks[ticket] = func
> return ticket
>
> def unregister(ticket):
> if ticket in CALLBACKS:
> del CALLBACKS[ticket]
>

I'd probably go with something similar to this, except that I'd use UUIDs for the tickets. I know me and my users, and somewhere along the line I'd use a ticket to unregister from the wrong callback dictionary!

>>> If you're expecting 'del foo' to destroy the object, then you have a
>>> bigger problem than callbacks, because that's simply not how Python
>>> works. You can't _ever_ assume that deleting something from your local
>>> namespace will destroy the object, because there can always be more
>>> references. So maybe you need a more clear way of saying "I'm done
>>> with this, get rid of it".
>>
>> Agreed about 'del', and I don't assume that the object goes away at the
>> point. The problem is debugging and determining WHY your object is still
>> around. I know a combination of logging and gc.get_referrers() will
>> probably help you figure out why something is still around, but I'm trying
>> to avoid that headache.
>
> Why do you care? Surely all your library should care about is whether or not
> they have a reference to the callback.If they do, they should call it (when
> appropriate). If they don't, they aren't responsible for it.

I care because I care about anyone using my code. Telling them 'tough, its your problem' doesn't get you many friends. Making a library that performs as expected and where it is easy to debug what went wrong makes everyone (including me!) happy.

>> I guess the real problem is how this creates cycles in the call graph.
>> User code effectively owns the library code, which via callbacks owns the
>> user code. I have no idea what the best point the cycle is to break it,
>> and not surprise someone down the road. The only idea I have is to
>> redesign the library a little, and make anything that accepts a callback
>> actually be a subclass of collections.abc.Container, or even
>> collections.abc.MutableSet. That makes it very obvious that the object
>> owns the callback, and that you will need to remove your object to
>> unregister it.
>
> My brain hurts from the complexity of your solution. What is the actual
> problem you are trying to solve? I would like to see an example of an
> actual failure before trying to solve a purely theoretical failure mode.

If I'm making your head hurt, then my solution is a bad solution. The whole reason I started this discussion was to figure out if an alternative method would make more sense to my potential endusers (fellow programmers). If strongly-held callbacks cause fewer headaches, then that is what I'll go with.

Thanks,
Cem Karan

[Cem Karan]

On Feb 21, 2015, at 3:57 PM, Grant Edwards <inv...@invalid.invalid> wrote:

> On 2015-02-21, Cem Karan <cfka...@gmail.com> wrote:

>>
>> On Feb 21, 2015, at 12:42 AM, Chris Angelico <ros...@gmail.com> wrote:
>>
>>> On Sat, Feb 21, 2015 at 1:44 PM, Cem Karan <cfka...@gmail.com> wrote:

>>>> In order to inform users that certain bits of state have changed, I require them to register a callback with my code. The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive. Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet). That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead. My question is simple; is this a good design? If not, why not? Are there any potential 'gotchas' I should be worried about?
>>>>
>>>

>>> No, it's not. I would advise using strong references - if the callback
>>> is a closure, for instance, you need to hang onto it, because there
>>> are unlikely to be any other references to it. If I register a
>>> callback with you, I expect it to be called; I expect, in fact, that
>>> that *will* keep my object alive.
>>

>> OK, so it would violate the principle of least surprise for you.
>

> And me as well. I would expect to be able to pass a closure as a
> callback and not have to keep a reference to it. Perhaps that just a
> leftover from working with other languages (javascript, scheme, etc.).
> It doesn't matter if it's a string, a float, a callback, a graphic or
> whatever: if I pass your function/library an object, I expect _you_ to

> keep track of it until you're done with it.

>
>> Interesting. Is this a general pattern in python? That is,
>> callbacks are owned by what they are registered with?
>

> I'm not sure what you mean by "owned" or why it matters that it's a
> callback: it's an object that was passed to you: you need to hold onto

> a reference to it until you're done with it, and the polite thing to

> do is to delete references to it when you're done with it.

I tend to structure my code as a tree or DAG of objects. The owner refers to the owned object, but the owned object has no reference to its owner. With callbacks, you get cycles, where the owned owns the owner. As a result, if you forget where your object has been registered, it may be kept alive when you aren't expecting it. My hope was that with WeakSets I could continue to preserve the DAG or tree while still having the benefits of callbacks. However, it looks like that is too surprising to most people.

Thanks,
Cem Karan

[Cem Karan]

You were saying that you move your objects into a zombie state. I assumed that you meant you marked them in some manner (e.g., setting 'is_zombie' to True), so that anything that has a strong reference to the object knows the object is not supposed to be used anymore. That way, regardless of where or how many times you've registered your object for callbacks, the library can do something like the following (banged out in my mail application, may have typos):

"""
_CALLBACKS = []

def execute_callbacks():
global _CALLBACKS
_CALLBACKS = [x for x in _CALLBACKS if not x.is_zombie]
for x in _CALLBACKS:
x()
"""

That will lazily unregister callbacks that are in the zombie state, which will eventually lead to their collection by the garbage collector. It won't work for anything that you don't have a reference for (lambdas, etc.), but it should work in a lot of cases.

Is this what you meant?

Thanks,
Cem Karan

[Steven D'Aprano]

That's much more reasonable than what you said earlier:

it seems wrong to have to stash a thing in a bucket in order

to keep its callbacks alive. I expect the callbacks themselves to
keep it alive.


So yes. If I bind a callback to a button, say, or a listener, then the
button (or listener) keeps the callback alive, *not* the callback keeping
the button or listener alive.

But if there are no references to the button, or the listener, then it will
be garbage-collected, which will free the references to the callback and
allow it to be garbage-collected as well (if there are no further
references to it).



--
Steven

[Marko Rauhamaa]

Cem Karan <cfka...@gmail.com>:

> You were saying that you move your objects into a zombie state. I
> assumed that you meant you marked them in some manner (e.g., setting
> 'is_zombie' to True),

Yes, but even better:

self.set_state(ZOMBIE)

> so that anything that has a strong reference to the object knows the
> object is not supposed to be used anymore.

The other way round: the zombie object knows to ignore callbacks sent
its way. It's not the responsibility of the sender to mind the
receiver's internal state.

I nowadays tend to implement states as inner classes. Here's how I've
implemented the zombie state of one class:

class Delivery...:
def __init__(...):
...
class ZOMBIE(STATE):
def handle_connected(self):
pass
def handle_eof(self):
pass
def handle_response(self, code, response):
pass
def handle_io_error(self, errcode):
pass
def zombifie(self):
assert False
def transaction_timeout(self):
assert False


Marko

[Cem Karan]

We're thinking of the same thing. I try to structure what little GUI code I write using the MVP pattern (http://en.wikipedia.org/wiki/Model-view-presenter), so I have these hub and spoke patterns. But you're right, if you have a partially evaluated callback that has the presenter as one of the parameters, that would do it for a GUI. I was thinking more of a DAG of objects, but now that I think about it, callbacks wouldn't make sense in that case.

Thanks,
Cem Karan

[Steven D'Aprano]

Marko Rauhamaa wrote:

> Chris Angelico <ros...@gmail.com>:
>
>> On Sun, Feb 22, 2015 at 7:34 PM, Marko Rauhamaa <ma...@pacujo.net> wrote:
>>> Refloops are not to be worried about, let alone removed.
>>
>> Why?
>
> Because the whole point of GC-languages is that you should stop worrying
> about memory. Trying to mastermind and micromanage GC in the application
> is, pardon my French, an antipattern.

While it would be nice to be able to stop worrying about memory, try to
calculate 1000**1000**1000 and see how that works for you.

Garbage collection enables us to *mostly* automate the allocation and
deallocation of memory. If doesn't mean we can forget about it. GC is an
abstraction that frees us from most of the grunt work of allocating memory,
but it doesn't mean that there is never any need to think about memory. GC
is a leaky abstraction. Depending on the implementation, it may cause
distracting and annoying pauses in your application and/or resource leaks.
Even if there are no pauses, GC still carries a performance penalty. Good
programmers need to be aware of the limitations of their tools, and be
prepared to code accordingly.

When writing programs for educational purposes, we should try to code in the
simplest and most elegant way with no thought given to annoying practical
matters. At least at first. But when writing programs for actual use, we
should write for the implication we have, not the one we wish we had.

>> They force the use of the much slower cycle-detecting GC, rather than
>> the quick and efficient CPython refcounter.
>
> Java's Hotspot doesn't bother with refcounters but is much faster than
> Python. CPython's refcounters are a historical accident that a Python
> application developer shouldn't even be aware of.

I don't know about Java's Hotspot, but I do know that CPython's ref counting
garbage collector has at least one advantage over the GC used by Jython and
IronPython: unlike them, open files are closed as soon as they are no
longer in use. Code like this may run out of operating system file handles
in Jython:

i = 0
while True:
f = open('/tmp/x%d' % i)
i += 1

while CPython will just keep going. I suppose it will *eventually* run out
of some resource, but probably not file handles.

Oh, a bit of trivia: Apple is abandoning their garbage collector and going
back to a reference counter:

https://developer.apple.com/news/?id=02202015a

Word on Reddit is that Apple is concerned about performance and battery
life.

P.S. A reminder that reference counting *is* a form of garbage collection.

>> I don't know how other Pythons work, but mark-and-sweep has its own
>> costs, and I don't know of any system that's both prompt and able to
>> detect refloops.
>
> It's exceedingly difficult (and pointless) to detect cycles in your
> object structures. Python is going to have to do a GC occasionally
> anyway. Yes, your worst-case response times are going to suffer, but
> that's the cost of doing business.

In *general*, you're right. Who wants to spend all their time worrying about
cycles when the GC can do it for you? But if cycles are rare, and in known
parts of your code where it is simple to break them when you're done,
there's no disadvantage to doing so. Leave the GC for the hard cases.

It's like explicitly closing a file, either with file.close() or a context
manager. When using CPython, it doesn't really matter whether you close the
file or not, since the ref counter will normally close it automatically as
soon as the file goes out of scope. But it is cheap and easy to do so, so
why not do it? Then, when it otherwise would matter, say you are running
under Jython, it doesn't because you've closed the file.

>> Helping it along means your program doesn't waste memory. Why such a
>> blanket statement?
>
> Because worrying Python programmers with evil spirits (reference loops)
> leads to awkward coding practices and takes away one of the main
> advantages of Python as a high-level programming language.

I think you exaggerate a tad. We're not trying to scare beginners, we're a
group of moderately experienced coders discussing "best practice" (or at
least "reasonable practice") when using callbacks.


--
Steven

[Chris Angelico]

On Mon, Feb 23, 2015 at 12:45 AM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
>> No no no. It's the other way around. _Something_ has to be doing those
>> callbacks, and it's that _something_ that should be keeping them
>> alive. The fact that it's a registered callback should itself *be* a
>> reference (and not a weak reference), and should keep it alive.
>
> That's much more reasonable than what you said earlier:
>
> it seems wrong to have to stash a thing in a bucket in order
> to keep its callbacks alive. I expect the callbacks themselves to
> keep it alive.
>
>
> So yes. If I bind a callback to a button, say, or a listener, then the
> button (or listener) keeps the callback alive, *not* the callback keeping
> the button or listener alive.

I meant the same thing, but my terminology was poor. Yes, that's
correct; it's not any sort of magic about it being a callback, but
more that the one you register it with becomes the owner of something.
Hence, no weak references.

ChrisA

[Marko Rauhamaa]

Steven D'Aprano <steve+comp....@pearwood.info>:

> I don't know about Java's Hotspot, but I do know that CPython's ref counting
> garbage collector has at least one advantage over the GC used by Jython and
> IronPython: unlike them, open files are closed as soon as they are no
> longer in use.

You can't depend on that kind of behavior. Dangling resources may or may
not be cleaned up, ever.

> Oh, a bit of trivia: Apple is abandoning their garbage collector and going
> back to a reference counter:
>
> https://developer.apple.com/news/?id=02202015a
>
> Word on Reddit is that Apple is concerned about performance and battery
> life.

That truly is a bit OT here.

> It's like explicitly closing a file, either with file.close() or a context
> manager.

Both methods are explicit. Closing files and other resources are not
directly related to GC.

Here's the thing: GC relieves your from dynamic memory management. You
are still on your own when it comes to other resources.

> We're not trying to scare beginners, we're a group of moderately
> experienced coders discussing "best practice" (or at least "reasonable
> practice") when using callbacks.

Who mentioned beginners? I'm abiding by the same best practices I'm
advocating.


Marko

[Cem Karan]

I think what you're suggesting is that library users register a Queue instead of a callback, correct? The problem is that I'll then have a strong reference to the Queue, which means I'll be pumping events into it after the user code has gone away. I was hoping to solve the problem of forgotten registrations in the library.

Thanks,
Cem Karan

[Cem Karan]

Nononono, I'm NOT encoding anything implicitly! As Frank mentioned earlier, this is more of a pub/sub problem. E.g., 'USB dongle has gotten plugged in', or 'key has been pressed'. The user code needs to decide what to do next, the library code provides a nice, clean interface to some potentially weird hardware.

Thanks,
Cem Karan

[Cem Karan]

I see what you're saying, but I don't think it gains us too much. If I store an object and an unbound method of the object, or if I store the bound method directly, I suspect it will yield approximately the same results.

Thanks,
Cem Karan

[Ethan Furman]

On 02/22/2015 05:13 AM, Cem Karan wrote:

> Output:
> From Evil Zombie: Surprise!
> From Your Significant Other: Surprise!
>
> In this case, the user made an error (just as Marko said in his earlier message),
> and forgot about the callback he registered with the library. The callback isn't
> really rising from the dead; as you say, either its been garbage collected, or it
> hasn't been. However, you may not be ready for a callback to be called at that
> moment in time, which means you're surprised by unexpected behavior.

But the unexpected behavior is not a problem with Python, nor with your library -- it's a bug in the fellow-programmer's
code, and you can't (or at least shouldn't) try to prevent those kinds of bugs from manifesting -- they'll just get
bitten somewhere else by the same bug.

--
~Ethan~

[Cem Karan]

I agree with you, but until a relatively new programmer has gotten used to what callbacks are and what they imply, I want to make things easy. For example, if the API subclasses collections.abc.MutableSet, and the documentation states that you can only add callbacks to this particular type of set, then a new programmer will naturally decide that either a) they need to dispose of the set, and if that isn't possible, then b) they need to delete their callback from the set. It won't occur to them that their live object will just magically 'go away'; its a member of a set!

My goal is to make things as pythonic (whatever that means in this case) and obvious as possible. Ideally, a novice can more or less guess what will happen with my API without really having to read the documentation on it.

Thanks,
Cem Karan

[Marko Rauhamaa]

Cem Karan <cfka...@gmail.com>:

> My goal is to make things as pythonic (whatever that means in this
> case) and obvious as possible. Ideally, a novice can more or less
> guess what will happen with my API without really having to read the
> documentation on it.

If you try to shield your user from the complexities of asynchronous
programming, you will only cause confusion. You will definitely need to
document all nooks and crannies of the semantics of the callback API and
your user will have to pay attention to every detail of your spec.

Your user, whether novice or an expert, will thank you for your
unambiguous specification even if it is complicated.


Marko

[Cem Karan]

Documentation is a given; it MUST be there. That said, documenting something, but still making it surprising, is a bad idea. For example, several people have been strongly against using a WeakSet to hold callbacks because they expect a library to hold onto callbacks. If I chose not to do that, and used a WeakSet, then even if I documented it, it would still end up surprising people (and from the sound of it, more people would be surprised than not).

Thanks,
Cem Karan

[Laura Creighton]

In a message of Sun, 22 Feb 2015 17:09:01 -0500, Cem Karan writes:

>Documentation is a given; it MUST be there. That said, documenting
>something, but still making it surprising, is a bad idea. For
>example, several people have been strongly against using a WeakSet to
>hold callbacks because they expect a library to hold onto callbacks.
>If I chose not to do that, and used a WeakSet, then even if I
>documented it, it would still end up surprising people (and from the
>sound of it, more people would be surprised than not).

>Thanks, Cem Karan

No matter what you do, alas, will surprise the hell out of people
because callbacks do not behave as people expect. Among people who
have used callbacks, what you are polling is 'what are people
familiar with', and it seems for the people around here, now,
WeakSets are not what they are familiar with.

But that is not so surprising. How many people use WeakSets for
_anything_? I've never used them, aside from 'ooh! cool shiny
new language feature! Let's kick it around the park!' That people
aren't familiar with WeakSets doesn't mean all that much.

The question I have is does this architecture make things harder,
easier or about the same to debug? To write tests for? to do Test
Driven Design with?

Laura

[Chris Angelico]

On Mon, Feb 23, 2015 at 9:29 AM, Laura Creighton <l...@openend.se> wrote:
> But that is not so surprising. How many people use WeakSets for
> _anything_? I've never used them, aside from 'ooh! cool shiny
> new language feature! Let's kick it around the park!' That people
> aren't familiar with WeakSets doesn't mean all that much.

I haven't used weak *sets*, but I've used weak *mappings* on occasion.
It's certainly not a common thing, but they have their uses. I have a
MUD which must guarantee that there be no more than one instance of
any given room (identified by a string that looks like a Unix path),
but which will, if it can, flush rooms out of memory when nothing
refers to them. So it has a mapping from the path strings to the
instances, but with weak refs for the instances; if anything else is
referring to that instance (eg a player character in the room), it'll
hang around, and any time anyone else needs that room, they'll get the
same instance back from the mapping; but any time the garbage
collector notices that a room can be disposed of, it will be.

Definitely not common though.

ChrisA

[Ian Kelly]

Well, it ties the weak ref to the lifetime of the object owning the
callback rather than to the lifetime of the potentially unreferenced
callback itself. I'm not fond of the scheme though because it forces
the callback to be a method, and I'd prefer not to make that
assumption.

Also, I just noticed that Python 3.4 adds a weakref.WeakMethod class
that solves the problem for the bound method case. That still leaves
the closure and lambda cases, but here's a thought: add an optional
argument to the callback registration method that specifies what
object to tie the weak ref to. Something like:

class Listenable:
def __init__(self):
self._callbacks = weakref.WeakKeyDictionary()

def listen(self, callback, owner=None):
if owner is None:
if isinstance(callback, types.MethodType):
owner = weakref.WeakMethod(callback)
else:
owner = callback
self._callbacks.setdefault(owner, []).append(callback)

def do_callbacks(self, message):
for callbacks in self._callbacks.values():
for callback in callbacks:
callback(message)

[Cem Karan]

On Feb 22, 2015, at 5:29 PM, Laura Creighton <l...@openend.se> wrote:

> In a message of Sun, 22 Feb 2015 17:09:01 -0500, Cem Karan writes:
>
>> Documentation is a given; it MUST be there. That said, documenting
>> something, but still making it surprising, is a bad idea. For
>> example, several people have been strongly against using a WeakSet to
>> hold callbacks because they expect a library to hold onto callbacks.
>> If I chose not to do that, and used a WeakSet, then even if I
>> documented it, it would still end up surprising people (and from the
>> sound of it, more people would be surprised than not).
>
>> Thanks, Cem Karan
>
> No matter what you do, alas, will surprise the hell out of people
> because callbacks do not behave as people expect. Among people who
> have used callbacks, what you are polling is 'what are people
> familiar with', and it seems for the people around here, now,
> WeakSets are not what they are familiar with.

And that's fine. I know that regardless of what I do, some people are going to be surprised. I'm trying to develop APIs that reduce that surprise as far as possible. That means I can spend more time coding and less time answering questions... :)

> But that is not so surprising. How many people use WeakSets for
> _anything_? I've never used them, aside from 'ooh! cool shiny
> new language feature! Let's kick it around the park!' That people
> aren't familiar with WeakSets doesn't mean all that much.

Actually, I use them when building caches of stuff, and I use weak references when I have trees of stuff so the child nodes know of, but don't hold onto, their parents. But I agree with you, there aren't a huge number of use-cases.

> The question I have is does this architecture make things harder,
> easier or about the same to debug? To write tests for? to do Test
> Driven Design with?

Good questions! That was why I was asking about 'gotchas' with WeakSets originally. Honestly, the only way to know for sure would be to write two APIs for doing similar things, and then see how people react to them. The problem is, how do you set up such a study so it is statistically valid?

Cem

[Frank Millman]

"Cem Karan" <cfka...@gmail.com> wrote in message
news:A3C11A70-5846-4915...@gmail.com...

>
>
> Good questions! That was why I was asking about 'gotchas' with WeakSets
> originally. Honestly, the only way to know for sure would be to write two
> APIs for doing similar things, and then see how people react to them. The
> problem is, how do you set up such a study so it is statistically valid?
>

Just in case you missed Steven's comment on my 'gotcha', and my reply, it is
worth repeating that what I reported as a gotcha was not what it seemed.

If you set up the callback as a weakref, and the listening object goes out
of scope, it will wait to be garbage collected. However, as far as I can
tell, the weakref is removed at the same time as the object is gc'd, so
there is no 'window' where the weakref exists but the object it is
referencing does not exist.

My problem was that I had performed a cleanup operation on the listening
object before letting it go out of scope, and it was no longer in a valid
state to deal with the callback, resulting in an error. If you do not have
that situation, your original idea may well work.

Frank

[Steven D'Aprano]

Cem Karan wrote:

>
> On Feb 21, 2015, at 12:27 PM, Steven D'Aprano
> <steve+comp....@pearwood.info> wrote:

>> The simplest possible identity-based scheme would be something like this:
>>
>>
>> # don't hate me for using a global variable
>> CALLBACKS = []
>>
>> def register(func):
>> if func not in CALLBACKS:
>> CALLBACKS.append(func)
>>
>> def unregister(func):
>> try:
>> CALLBACKS.remove(func)
>> except ValueError:
>> pass

Oops! That's not identity-based, that's *equality* based.

Both the `in` operator and the list `remove` method implicitly perform
equality checks, not identity checks. Which means that they will work with
methods, since method equality compares against the underlying function,
which is the same:


py> class Spam(object):
... def method(self):
... pass
...
py> s = Spam()
py> a = s.method
py> b = s.method
py> a is b
False
py> a == b
True
py> a.__func__ is b.__func__
True


So, when I say this:

>> That's probably a bit too simple, since it won't behave as expected with
>> bound methods. The problem is that bound methods are generated on the
>> fly, so this won't work:

I was mistaken.

> Are you sure about that? I just tested out the following code, and it
> appears to work correctly:

You are correct.



--
Steven

[Gregory Ewing]

Cem Karan wrote:
> I tend to structure my code as a tree or DAG of objects. The owner refers to
> the owned object, but the owned object has no reference to its owner. With
> callbacks, you get cycles, where the owned owns the owner.

This is why I suggested registering a listener object
plus a method name instead of a callback. It avoids that
reference cycle, because there is no long-lived callback
object keeping a reference to the listener.

--
Greg

[rand...@fastmail.us]

How does that help? Everywhere you would have had a reference to the
"callback object", you now have a reference to the listener object.
You're just shuffling deck chairs around: if B shouldn't reference A
because A owns B, then removing C from the B->C->A reference chain does
nothing to fix this.

[Gregory Ewing]

Cem Karan wrote:
> On Feb 22, 2015, at 5:15 AM, Gregory Ewing <greg....@canterbury.ac.nz>
> wrote:
>
>> Perhaps instead of registering a callback function, you should be
>> registering the listener object together with a method name.
>

> I see what you're saying, but I don't think it gains us too much. If I store
> an object and an unbound method of the object, or if I store the bound method
> directly, I suspect it will yield approximately the same results.

It would be weird and unpythonic to have to register both
an object and an unbound method, and if you use a bound
method you can't keep a weak reference to it.

--
Greg

[Cem Karan]

Thank you Frank, I did read Steve's comment to your reply earlier, but what you said in your original reply made sense to me. I don't have control over user code. That means that if someone wants to write code such that they perform some kind of cleanup and are no longer able to handle the callback, they are free to do so. While I can't prevent this from happening, I can make it as obvious as possible in my code that before you perform any cleanup, you also need to unregister from the library. That is my main goal in developing pythonic/obvious methods of registering callbacks.

Thanks,
Cem Karan

[Cem Karan]

I'm combining two messages into one,

Greg, random832 said what I was thinking earlier, that you've only increased the diameter of your cycle without actually fixing it. Can you give a code example where your method breaks the cycle entirely?

Thanks,
Cem Karan

[Fabio Zadrozny]

Hi Cem,

I didn't read the whole long thread, but I thought I'd point you to what I'm using in PyVmMonitor (http://www.pyvmmonitor.com/) -- which may already cover your use-case.

Take a look at the callback.py at https://github.com/fabioz/pyvmmonitor-core/blob/master/pyvmmonitor_core/callback.py

And its related test (where you can see how to use it): https://github.com/fabioz/pyvmmonitor-core/blob/master/_pyvmmonitor_core_tests/test_callback.py (note that it falls back to a strong reference on simple functions -- i.e.: usually top-level methods or methods created inside a scope -- but otherwise uses weak references).

Best Regards,

Fabio

On Sat, Feb 21, 2015 at 12:44 AM, Cem Karan <cfka...@gmail.com> wrote:

Hi all, I'm working on a project that will involve the use of callbacks, and I want to bounce an idea I had off of everyone to make sure I'm not developing a bad idea.  Note that this is for python 3.4 code; I don't need to worry about any version of python earlier than that.

In order to inform users that certain bits of state have changed, I require them to register a callback with my code.  The problem is that when I store these callbacks, it naturally creates a strong reference to the objects, which means that if they are deleted without unregistering themselves first, my code will keep the callbacks alive.  Since this could lead to really weird and nasty situations, I would like to store all the callbacks in a WeakSet (https://docs.python.org/3/library/weakref.html#weakref.WeakSet).  That way, my code isn't the reason why the objects are kept alive, and if they are no longer alive, they are automatically removed from the WeakSet, preventing me from accidentally calling them when they are dead.  My question is simple; is this a good design?  If not, why not?  Are there any potential 'gotchas' I should be worried about?

Thanks,
Cem Karan
--
https://mail.python.org/mailman/listinfo/python-list

[Gregory Ewing]

rand...@fastmail.us wrote:
> On Tue, Feb 24, 2015, at 00:20, Gregory Ewing wrote:
>
>>This is why I suggested registering a listener object
>>plus a method name instead of a callback. It avoids that
>>reference cycle, because there is no long-lived callback
>>object keeping a reference to the listener.
>
> How does that help? Everywhere you would have had a reference to the
> "callback object", you now have a reference to the listener object.

The point is that the library can keep a weak reference
to the listener object, whereas it can't reliably keep
a weak reference to a bound method.

--
Greg

[Cem Karan]

On Feb 24, 2015, at 8:23 AM, Fabio Zadrozny <fab...@gmail.com> wrote:

> Hi Cem,
>
> I didn't read the whole long thread, but I thought I'd point you to what I'm using in PyVmMonitor (http://www.pyvmmonitor.com/) -- which may already cover your use-case.
>
> Take a look at the callback.py at https://github.com/fabioz/pyvmmonitor-core/blob/master/pyvmmonitor_core/callback.py
>
> And its related test (where you can see how to use it): https://github.com/fabioz/pyvmmonitor-core/blob/master/_pyvmmonitor_core_tests/test_callback.py (note that it falls back to a strong reference on simple functions -- i.e.: usually top-level methods or methods created inside a scope -- but otherwise uses weak references).

That looks like a better version of what I was thinking about originally. However, various people on the list have convinced me to stick with strong references everywhere. I'm working out a possible API right now, once I have some code that I can use to illustrate what I'm thinking to everyone, I'll post it to the list.

Thank you for showing me your code though, it is clever!

Thanks,
Cem Karan

[Cem Karan]

I think I see what you're talking about now. Does WeakMethod (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve this problem? Note that I can force my users to use the latest stable version of python at all times, so WeakMethod IS available to me.

Thanks,
Cem Karan

[Gregory Ewing]

Cem Karan wrote:
> I think I see what you're talking about now. Does WeakMethod
> (https://docs.python.org/3/library/weakref.html#weakref.WeakMethod) solve
> this problem?

Yes, that looks like it would work.

--
Greg

[Cem Karan]

Cool!

Thanks,
Cem Karan

[Ian Kelly]

On Feb 26, 2015 4:00 AM, "Cem Karan" <cfka...@gmail.com> wrote:
>
>
> On Feb 26, 2015, at 12:36 AM, Gregory Ewing <greg....@canterbury.ac.nz> wrote:
>

> Cool!

Sometimes I wonder whether anybody reads my posts. I suggested a solution involving WeakMethod four days ago that additionally extends the concept to non-method callbacks (requiring a small amount of extra effort from the client in those cases, but I think that is unavoidable. There is no way that the framework can determine the appropriate lifetime for a closure-based callback.)

[Ethan Furman]

On 02/26/2015 11:54 AM, Ian Kelly wrote:

> Sometimes I wonder whether anybody reads my posts.

It's entirely possible the OP wasn't ready to understand your solution four days ago, but two days later the OP was.

--
~Ethan~

[Fabio Zadrozny]

​Hi Cem,

Well, I decided to elaborate a bit on the use-case I have and how I use it (on a higher level): http://pydev.blogspot.com.br/2015/02/design-for-client-side-applications-in.html

So, you can see if it may be worth for you or not (I agree that sometimes you should keep strong references, but for my use-cases, weak references usually work better -- with the only exception being closures, which is handled different anyways but with the gotcha of having to manually unregister it).

Best Regards,

Fabio​

[Cem Karan]

Thank you Ethan, that was precisely my problem.

Thanks,
Cem Karan

[Cem Karan]

As I mentioned in an earlier post, I've been quite busy at home, and expect to be for a few days to come, so I apologize both for being so late posting, and for not posting my own API plans.

Your blog post has given me quite a bit to think about, thank you! Do you mind if I work up an API similar to yours? I'm planning on using a different license (not LGPL), which is why I ask.

Thanks,
Cem Karan

[Cem Karan]

On Feb 26, 2015, at 2:54 PM, Ian Kelly <ian.g...@gmail.com> wrote:

> On Feb 26, 2015 4:00 AM, "Cem Karan" <cfka...@gmail.com> wrote:
> >
> >
> > On Feb 26, 2015, at 12:36 AM, Gregory Ewing <greg....@canterbury.ac.nz> wrote:
> >

> > Cool!

>
> Sometimes I wonder whether anybody reads my posts. I suggested a solution involving WeakMethod four days ago that additionally extends the concept to non-method callbacks (requiring a small amount of extra effort from the client in those cases, but I think that is unavoidable. There is no way that the framework can determine the appropriate lifetime for a closure-based callback.)

I apologize about taking so long to reply to everyone's posts, but I've been busy at home.

Ian, it took me a while to do some research to understand WHY what you were suggesting was important; you're right about storing the object as well as the method/function separately, but I think that WeakMethod might solve that completely, correct? Are there any cases where WeakMethod wouldn't work?

Thanks,
Cem Karan

[Ian Kelly]

On Mon, Mar 2, 2015 at 4:04 AM, Cem Karan <cfka...@gmail.com> wrote:
> On Feb 26, 2015, at 2:54 PM, Ian Kelly <ian.g...@gmail.com> wrote:
>> On Feb 26, 2015 4:00 AM, "Cem Karan" <cfka...@gmail.com> wrote:
>> >
>> >
>> > On Feb 26, 2015, at 12:36 AM, Gregory Ewing <greg....@canterbury.ac.nz> wrote:
>> >

>> > Cool!
>>
>> Sometimes I wonder whether anybody reads my posts. I suggested a solution involving WeakMethod four days ago that additionally extends the concept to non-method callbacks (requiring a small amount of extra effort from the client in those cases, but I think that is unavoidable. There is no way that the framework can determine the appropriate lifetime for a closure-based callback.)
>
> I apologize about taking so long to reply to everyone's posts, but I've been busy at home.
>
> Ian, it took me a while to do some research to understand WHY what you were suggesting was important; you're right about storing the object as well as the method/function separately, but I think that WeakMethod might solve that completely, correct? Are there any cases where WeakMethod wouldn't work?

WeakMethod only works for bound method objects. If you pass it a
non-method function, you'll get a TypeError:

>>> from weakref import WeakMethod
>>> WeakMethod(lambda: None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.4/weakref.py", line 49, in __new__
.format(type(meth))) from None
TypeError: argument should be a bound method, not <class 'function'>

This check uses duck typing, so you could perhaps write a method-like
class with __self__ and __func__ attributes and pass that to the
WeakMethod constructor in the case of non-methods. There's a bigger
issue with this however, which is that WeakMethod works by keeping
weak references to *both* the object and the function, meaning that as
soon as the function has no other references, the WeakMethod expires
even if the object is still alive. This isn't a problem for methods
because it's the transience of the method object, not the underlying
function, that WeakMethod seeks to work around. But it doesn't by
itself do anything to solve the problem of closures or lambdas that
may themselves be transient.

Revisiting the implementation I suggested previously, I want to make a
correction. This would be better solved with a WeakValueDictionary:

class Listenable:
def __init__(self):
self._callbacks = weakref.WeakValueDictionary()

def listen(self, callback, owner=None):
if owner is None:
if isinstance(callback, types.MethodType):
owner = weakref.WeakMethod(callback)
else:
owner = callback

# TODO: Should anything happen if the callback is already in the dict?
self._callbacks[callback] = owner

def do_callbacks(self, message):
for callback in self._callbacks.keys():
callback(message)

This approach has two benefits over the previous one: it simplifies
the callback management a bit, and it avoids making the assumption
that the owner is hashable (it assumes instead that the callback is
hashable, but I think that's reasonable).

[Cem Karan]

Hi all, I apologize for taking so long to reply, but neither my work schedule nor the weather have been kind in the past week. That said, I've been thinking long and hard about what everyone has said, and have decided that it would be useful to write a wrap-up email that attempts to encapsulate everything everyone has said, as a record of sorts if nothing else. As such, this email is fairly long and involved.

=======================
Analysis of the problem
=======================

My original question was 'what is the least surprising/most pythonic way to write a callback API?' Through reading what everyone has said, I realized that I wasn't being specific enough, simply because callback APIs can be quite different. At the very least, the following questions need to be answered:

1) When a callback is registered, does it replace the prior callback?
2) If more than one callback can be registered, is there an ordering to them?
3) Can a callback be registered more than once?
4) When and how are callbacks deregistered?
5) Who is responsible for maintaining a strong reference to the callback?

As far as I know, there isn't a standard method to indicate to the caller that one callback replaces another one except via well-written documentation. My personal feeling is that callbacks that replace other callbacks should be properties of the library. By implementing a setter, getter, and deleter for each callback, the library makes it obvious that there is one and only one callback active at a time. The only difficulty is making sure the user knows that the library retains the callback, but this is a documentation problem.

I realized that ordering could be a problem when I read through the documentation to asyncio.call_soon(). It promises that callbacks will be called in the order in which they were registered. However, there are cases where the order doesn't matter. Registration in both of these cases is fairly simple; the former appends the callback to a list, while the latter adds it to a set. The list or set can be a property of the library, and registration is simply a matter of either inserting or adding. But this brings up point 3; if a callback can be registered at most once and ordering matters, then we need something that is both a sequence and a set. Subclassing either (or both) collections.abc.MutableSequence or collections.abc.MutableSet will lead to confusion due to unexpected violations of PEP 3119 (https://www.python.org/dev/peps/pep-3119/). Once again, the only option appears to be careful documentation.

Registration is only half the problem. The other half is determining when a callback should be unregistered. Some callbacks are one-shots and are automatically unregistered as soon as they are called. Others will be called each time an event occurs until they are explicitly unregistered from the library. Which happens is another design choice that needs to be carefully documented.

Finally, we come to the part that started my original question; who retains the callback. I had originally asked everyone if it would be surprising to store callbacks as weak references. The idea was that unless someone else maintained a strong reference to the callback, it would be garbage collected, which would save users from 'surprising' results such as the following:

"""
#! /usr/bin/env python

class Callback_object(object):
def __init__(self, msg):
self._msg = msg
def callback(self, stuff):
print("From {0!s}: {1!s}".format(self._msg, stuff))

class Fake_library(object):
def __init__(self):
self._callbacks = list()
def register_callback(self, callback):
self._callbacks.append(callback)
def execute_callbacks(self):
for thing in self._callbacks:
thing('Surprise!')

if __name__ == "__main__":
cbo = Callback_object("Evil Zombie")
lib = Fake_library()
lib.register_callback(cbo.callback)

# Way later, after the user forgot all about the callback above
cbo = Callback_object("Your Significant Other")
lib.register_callback(cbo.callback)

# And finally getting around to running all those callbacks.
lib.execute_callbacks()
"""

However, as others pointed out using a weak reference could actually increase confusion rather than decrease it. The problem is that if there is a reference cycle elsewhere in the code, it is possible that the zombie object is still alive when it is supposed to be dead. This will likely be difficult to debug. In addition, different types of callables have different requirements in order to correctly store weak references to them. Both Ian Kelly and Fabio Zadrozny provided solutions to this, with Fabio providing a link to his code at http://pydev.blogspot.com.br/2015/02/design-for-client-side-applications-in.html.

====================================
Solution to my problem in particular
====================================

After considering all the comments above, I've decided to do the following for my API:

- All callbacks will be strongly retained (no weakrefs).
- Callbacks will be stored in a list, and the list will be exposed as a read-only property of the library. This will let users reorder callbacks as necessary, add them multiple times in a row, etc. I'm also hoping that by making it a list, it becomes obvious that the callback is strongly retained.
- Finally, callbacks are not one-shots. This just happens to make sense for my code, but others may find other methods make more sense.


Thanks again to everyone for providing so many comments on my question, and I apologize again for taking so long to wrap things up.

Thanks,
Cem Karan

[John Pote]

Hi everyone.
I recently had the problem of converting from an integer to its
representation as a list of binary bits, each bit being an integer 1 or
0, and vice versa. E.G.
0x53
becomes
[ 0, 1, 0, 1, 0, 0, 1, 1 ]

This I wanted to do for integers of many tens, if not hundreds, of bits.
Python very nicely expands integers to any size required, great feature.

Just wondered if there was a neat way of doing this without resorting to
a bit bashing loop.

Looking forward to some interesting answers,
John

[Ben Finney]

John Pote <john...@o2.co.uk> writes:

> I recently had the problem of converting from an integer to its
> representation as a list of binary bits, each bit being an integer 1
> or 0, and vice versa.

Is this a homework assignment?

> E.G.
> 0x53
> becomes
> [ 0, 1, 0, 1, 0, 0, 1, 1 ]

>>> foo = 4567

>>> foo
4567
>>> "{foo:d}".format(foo=foo)
'4567'
>>> "{foo:b}".format(foo=foo)
'1000111010111'

>>> foo_binary_text = "{foo:b}".format(foo=foo)
>>> foo_binary_digits = list(foo_binary_text)
>>> foo_binary_digits
['1', '0', '0', '0', '1', '1', '1', '0', '1', '0', '1', '1', '1']

> Just wondered if there was a neat way of doing this without resorting
> to a bit bashing loop.

Python's string formatting and sequence types are quite powerful.

--
\ “As far as the laws of mathematics refer to reality, they are |
`\ not certain, and as far as they are certain, they do not refer |
_o__) to reality.” —Albert Einstein, 1983 |
Ben Finney

[MRAB]

On 2015-05-21 23:20, John Pote wrote:
> Hi everyone.

> I recently had the problem of converting from an integer to its
> representation as a list of binary bits, each bit being an integer 1 or

> 0, and vice versa. E.G.

> 0x53
> becomes
> [ 0, 1, 0, 1, 0, 0, 1, 1 ]
>

> This I wanted to do for integers of many tens, if not hundreds, of bits.
> Python very nicely expands integers to any size required, great feature.
>

> Just wondered if there was a neat way of doing this without resorting to
> a bit bashing loop.
>

> Looking forward to some interesting answers,
> John
>
>

I don't know how efficient you want it to be, but:

>>> number = 0x53
>>> bin(number)
'0b1010011'
>>> bin(number)[2 : ]
'1010011'
>>> list(map(int, bin(number)[2 : ]))
[1, 0, 1, 0, 0, 1, 1]

[Ian Kelly]

On Thu, May 21, 2015 at 4:31 PM, Ben Finney <ben+p...@benfinney.id.au> wrote:
>>>> "{foo:d}".format(foo=foo)
> '4567'
>>>> "{foo:b}".format(foo=foo)
> '1000111010111'

Which since there's nothing else in the format string can be simplified to:

>>> format(foo, "b")
'1000111010111'

[John Pote]

Thanks for the replies. Interesting that the offered solutions involve
converting to a binary text string and then the individual chars back to
ints. I had thought this would be a route to solve this problem but it
seemed a bit 'heavy' hence I thought it worthwhile asking the question.

My solution to converting a list of 1s and 0s back to an int is
listLen = len( binList )
n = sum( [ binList[i]*( 2**(listLen-1 - i) ) for i in
range(listLen)] )

In response to Ben Finney's question, I haven't done homework for 40
years! Genuine problem, I had decided that the clearest way to write the
algorithm I was working on was to use lists of 1s and 0s rather than
normal ints.

Thanks for the help,
John