hard references instead of weak ones

69 views
Skip to first unread message

Mustafa Atik

unread,
Aug 25, 2015, 8:54:18 AM8/25/15
to pypubsub
hi,

is it possible to work with hard references?

oliver

unread,
Aug 25, 2015, 10:22:02 AM8/25/15
to PyPubSub
Can you give an example of what you mean (some code that shows what you tried and didn't work as you expected)? 

On Tue, Aug 25, 2015 at 8:54 AM, Mustafa Atik <mua...@gmail.com> wrote:
hi,

is it possible to work with hard references?

--
You received this message because you are subscribed to the Google Groups "pypubsub" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pypubsub+u...@googlegroups.com.
To post to this group, send email to pypu...@googlegroups.com.
Visit this group at http://groups.google.com/group/pypubsub.
For more options, visit https://groups.google.com/d/optout.



--
Oliver
Author of these Open Source: PyPubSub, Lua-iCxx, iof
Regular contributor on StackOverflow

Mustafa Atik

unread,
Aug 25, 2015, 4:08:23 PM8/25/15
to pypubsub
In the following example, handler2 does not receive any invocation due to, I guess, weak referencing. That is, when init finishes, handler2() will be garbage collected. Am I wrong about this?

from pubsub import pub


def handler1():
        print("handler1 is running")
pub.subscribe(handler1, "someEvent")


def init(config):
    # doing some other works here
    def handler2():
        print("handler2 is running")
    pub.subscribe(handler2, "someEvent")
init({})
pub.sendMessage("someEvent")

oliver

unread,
Aug 25, 2015, 5:12:30 PM8/25/15
to PyPubSub
You are correct. Weak referencing is essential otherwise you will find you have to manually manage lifecycle of your objects that were listeners, just because they were registered as listeners, gets tedious. The pro is that when a listener is no longer used by the app, pubsub automatically forgets it. The con is that when a listener is no longer used by the app, pubsub automatically forgets it :)

The problem is that there is no way of knowing when the last strong reference will be destroyed so pubsub can't guard against your case. However, it can notify you: this is why there are debug utility classes, look at http://pubsub.sourceforge.net/usage/usage_advanced_debug.html?highlight=notify. During development I have one of those active, and can easily tell when a listener is removed. When your listener is not being called, there are only two reasons: it hasn't been registered, it was unsubscribed at unexpected time, or the topic was never emitted. The notifyier instance allows you to check for these.

class MonitorListenerSubscription(INotificationHanlder):
    def __init__(self, callback):
        self._weak_cb = getWeakRef(callback)  # use the class defined in pubsub
    def deadListener(self, weak_listener_wrapper):
        print('something has died')
    def notifyUnsubscribe(self, pubListener, topicObj):
        print('listener has been unsubd: ', pubListener, topicObj)
    def notifyDeadListener(self, pubListener, topicObj):
        print('listener has died: ', pubListener, topicObj)
    def notifySend(self, stage, topicObj, pubListener=None):
        if pubListener is self._weak_cb():
            print('listener', self._weak_cb(), 'was sent message of topic', topicObj)

(not tested; you might need the IgnoreNotificatiosnMixin so you don't need to implement the other methods). With this, any time one of your listeners is not being calls when you think it should, create an instance of the above for your handler, register it, and look at stdout (or just put "pass" instead of print and put a breakpoint and use debugger so you can get call stack at moment it is being unsubd, destroyed, or messaged. 

Oliver

Mustafa Atik

unread,
Aug 26, 2015, 1:52:14 AM8/26/15
to pypubsub
Thanks

Jérôme Laheurte

unread,
Aug 26, 2015, 6:44:14 AM8/26/15
to pypu...@googlegroups.com

> Le 25 août 2015 à 23:12, oliver <oliver.s...@gmail.com> a écrit :
>
> You are correct. Weak referencing is essential otherwise you will find you have to manually manage lifecycle of your objects that were listeners, just because they were registered as listeners, gets tedious. The pro is that when a listener is no longer used by the app, pubsub automatically forgets it. The con is that when a listener is no longer used by the app, pubsub automatically forgets it :)

Somehow related to this, I regularly find myself having to maintain a cache of references to a locally-defined callback in order to do curryfication. Dummy example:

def foo(self):
self._cache = set()
for index, obj in enumerate(objs):
self._bind(index, obj)

def _bind(self, index, obj):
def cb():
self.objChangedAtIndex(index)
obj.subscribe(cb, ‘changed’)
self._cache.add(cb) # Or else it gets dereferenced

def objChangedAtIndex(self, index):
pass

Of course it’s a PITA to maintain the cache if I don’t want my callbacks to live forever. What I’d like to be able to do is « obj.subscribe(self._objChangedAtIndex, moreArgs=(index,)) » or something. I couldn’t find anything in the documentation regarding this use case, did I miss something ?

Cheers
Jérôme

signature.asc

oliver

unread,
Aug 26, 2015, 8:58:01 AM8/26/15
to PyPubSub
You didn't miss anything. You would have to pass curried args by name, just like you have to pass named arguments when sending a message (if only to support currying non-contiguous positional parameters). Subscribe() currently takes a listener callable and topic name, so the API should be extendable with a **kwargs, allowing you to do this: 

def listen(arg1, arg2, arg3):
    pass
obj.subscribe(listen, 'topic', arg1=1, arg3=3)
pub.sendMessage('topic', arg2='a')  # calls listen(1, 'a', 3)
pub.sendMessage('topic', arg2='b')  # calls listen(1, 'b', 3)
pub.sendMessage('topic', arg2='c')  # calls listen(1, 'c', 3)

Would that work? The objects in the dict would have to be strong referenced which is not great (I would prefer to stick with the policy that when an object is no longer used outside of pubsub, it can be destroyed). So you'd have to be careful what you bind, although at most the "artificial" lifetime due to strong referencing by pubsub would be that of the listener, perhaps not too bad. 

Jérôme Laheurte

unread,
Aug 26, 2015, 9:25:13 AM8/26/15
to pypu...@googlegroups.com

Le 26 août 2015 à 14:58, oliver <oliver.s...@gmail.com> a écrit :


You didn't miss anything. You would have to pass curried args by name, just like you have to pass named arguments when sending a message (if only to support currying non-contiguous positional parameters). Subscribe() currently takes a listener callable and topic name, so the API should be extendable with a **kwargs, allowing you to do this: 

def listen(arg1, arg2, arg3):
    pass
obj.subscribe(listen, 'topic', arg1=1, arg3=3)
pub.sendMessage('topic', arg2='a')  # calls listen(1, 'a', 3)
pub.sendMessage('topic', arg2='b')  # calls listen(1, 'b', 3)
pub.sendMessage('topic', arg2='c')  # calls listen(1, 'c', 3)

Would that work? The objects in the dict would have to be strong referenced which is not great (I would prefer to stick with the policy that when an object is no longer used outside of pubsub, it can be destroyed). So you'd have to be careful what you bind, although at most the "artificial" lifetime due to strong referencing by pubsub would be that of the listener, perhaps not too bad. 

That would be perfect, at least for my use case. I typically need this to « forward » messages from objects to their container, adding context information, so unsubscribing is done manually when the object is removed from the container, making the strong ref not worrying at all.

Cheers
Jérôme

signature.asc

Oliver

unread,
Dec 23, 2015, 1:08:26 PM12/23/15
to pypubsub
Hi Jerome (and anyone else interested in this feature -- please feel free to chip in), I'm working to implement curried args, and need some input on your use case. Say you have the following:  

def listener(a, b, c):
    print(a, b, c)

pub.subscribe(listener, 'topic')
pub.sendMessage('topic', b=2, c=3)
pub.sendMessage('topic', a=1, b=2, c=3)

and the topic defines "a" as optional with default value 0. Then you would expect the first send to print "0 2 3" and the second one to print "1 2 3". But what would you expect to see printed if you instead subscribed listener with a curried arg that overlaps the optional topic arg: 

# "a" is "curried" arg, specific to this listener:
pub.subscribe(listener, 'topic', a=4)  

What is the most intuitive contract: 
  1. user expects "listener will always get a=2, regardless of what sender sends": in this case both prints are "4 2 3": curried value always overrides sendMessage
  2. user expects "listener will only get this curried value if the user does not specify "a" in the sendMessage": then first sendMessage prints "4 2 3", but second prints "1 2 3"; curried value only overrides default
  3. user expects "listener will always get the topic message value, if there is one": then first sendMessage prints "0 2 3", second prints "1 2 3", and the curried value is never actually used if topic defines it; curred value only used if not defined by topic
Case 2 seems rather unlikely, but is still valid. Not sure what is more intuitive, case 1 or case 3. Any opinion? I was thinking of using a "specially named kwarg" in sendMessage, say, "_when_" as in

pub.subscribe(listener, 'topic', a=4, _when_=UseCurry.always)       # case 1
pub.subscribe(listener, 'topic', a=4, _when_=UseCurry.sendDefault)  # case 2
pub.subscribe(listener, 'topic', a=4, _when_=UseCurry.notInTopic)   # case 3

Let me know your thoughts ASAP. Thanks, 
Oliver


On Wednesday, 26 August 2015 09:25:13 UTC-4, Jérôme Laheurte wrote:

Jérôme Laheurte

unread,
Dec 23, 2015, 1:26:06 PM12/23/15
to PyPubSub

Le 23 déc. 2015 à 19:08, Oliver <Oliver.S...@gmail.com> a écrit :

Hi Jerome (and anyone else interested in this feature -- please feel free to chip in), I'm working to implement curried args, and need some input on your use case. Say you have the following:  

def listener(a, b, c):
    print(a, b, c)

pub.subscribe(listener, 'topic')
pub.sendMessage('topic', b=2, c=3)
pub.sendMessage('topic', a=1, b=2, c=3)

and the topic defines "a" as optional with default value 0. Then you would expect the first send to print "0 2 3" and the second one to print "1 2 3". But what would you expect to see printed if you instead subscribed listener with a curried arg that overlaps the optional topic arg: 

# "a" is "curried" arg, specific to this listener:
pub.subscribe(listener, 'topic', a=4)  

What is the most intuitive contract: 
  1. user expects "listener will always get a=2, regardless of what sender sends": in this case both prints are "4 2 3": curried value always overrides sendMessage
  2. user expects "listener will only get this curried value if the user does not specify "a" in the sendMessage": then first sendMessage prints "4 2 3", but second prints "1 2 3"; curried value only overrides default
  3. user expects "listener will always get the topic message value, if there is one": then first sendMessage prints "0 2 3", second prints "1 2 3", and the curried value is never actually used if topic defines it; curred value only used if not defined by topic
4. None of the above :)

IMO the listener should exactly behave as if the curried argument disappeared from its signature, so this would raise a TypeError.

Best regards
Jérôme Laheurte

signature.asc

Oliver

unread,
Dec 23, 2015, 1:30:43 PM12/23/15
to pypubsub
Case 2 seems rather unlikely, but is still valid. Not sure what is more intuitive, case 1 or case 3. Any opinion? I was thinking of using a "specially named kwarg" in sendMessage, say, "_when_" as in

pub.subscribe(listener, 'topic', a=4, _when_=UseCurry.always)       # case 1
pub.subscribe(listener, 'topic', a=4, _when_=UseCurry.sendDefault)  # case 2
pub.subscribe(listener, 'topic', a=4, _when_=UseCurry.notInTopic)   # case 3

Let me know your thoughts ASAP. Thanks, 
Oliver

Actually case 2 is too much work. Cases 1 and 3 are easy, so I will just have a boolean: _overrideSent_=True: 

pub.subscribe(listener, 'topic', a=4) # case 1
pub.subscribe(listener, 'topic', a=4, _overrideSent_=False)  # case 3
Oliver 

Oliver

unread,
Dec 23, 2015, 1:38:04 PM12/23/15
to pypubsub


On Wednesday, 23 December 2015 13:26:06 UTC-5, Jérôme Laheurte wrote:
Makes sense too. So this would be the default.  

Jérôme Laheurte

unread,
Dec 23, 2015, 1:41:59 PM12/23/15
to PyPubSub

Le 23 déc. 2015 à 19:38, Oliver <Oliver.S...@gmail.com> a écrit :

IMO the listener should exactly behave as if the curried argument disappeared from its signature, so this would raise a TypeError.

Best regards
Jérôme Laheurte


Makes sense too. So this would be the default.  


Cool. Thanks for working on this, I’ll be able to axe a lot of my own code :)

Best regards
Jérôme Laheurte

signature.asc

Oliver

unread,
Dec 30, 2015, 12:58:32 AM12/30/15
to pypubsub
Jerome, 
Code committed. You can now write 

    pub.subscribe(listener, topic, **curriedArgs)

Let me know if it all works for you. 

On Wednesday, 23 December 2015 13:41:59 UTC-5, Jérôme Laheurte wrote:

Jérôme Laheurte

unread,
Dec 30, 2015, 5:35:38 AM12/30/15
to pypu...@googlegroups.com

Le 30 déc. 2015 à 06:58, Oliver <Oliver.S...@gmail.com> a écrit :

Jerome, 
Code committed. You can now write 

    pub.subscribe(listener, topic, **curriedArgs)

Let me know if it all works for you. 

I’ll try this today :) There are two typos, here is a fix. Also, the mail notifier from the buildbot seemed to have a problem with my SMTP, I fixed the configuration, we’ll see if it works on the next commit.

Index: src/pubsub/pub.py
===================================================================
--- src/pubsub/pub.py (révision 361)
+++ src/pubsub/pub.py (copie de travail)
@@ -161,7 +161,7 @@
     """Return true only if listener can subscribe to messages of given topic.
     If curriedArgNames can be a list of parameters of the given listener, that 
     should be assumed curried (i.e. actual listener signature is signature of 
-    given listener minus curried args).""
+    given listener minus curried args)."""
     return _topicMgr.getTopic(topicName).isValid(listener, curriedArgNames=curriedArgNames)
 
 
Index: tox.ini
===================================================================
--- tox.ini (révision 361)
+++ tox.ini (copie de travail)
@@ -5,7 +5,7 @@
 
 [tox]
 envlist = py26, py27, py32, py33, py34
-toxworkdir=/.tox/pypubsub
+toxworkdir=~/.tox/pypubsub
 
 [testenv]
 usedevelop = True

signature.asc

Jérôme Laheurte

unread,
Dec 30, 2015, 6:01:52 AM12/30/15
to pypu...@googlegroups.com
Le 30 déc. 2015 à 06:58, Oliver <Oliver.S...@gmail.com> a écrit :

Jerome, 
Code committed. You can now write 

    pub.subscribe(listener, topic, **curriedArgs)

Let me know if it all works for you. 

Just tested on my most annoying use case, works like a charm!

Jérôme Laheurte

signature.asc

Andy Bulka

unread,
Sep 20, 2022, 12:06:26 AM9/20/22
to pypubsub
So back to the initial question about handlers falling out of scope and being removed from the pub/sub system - I don't really want to get notified about that happening - I know its happening.  I need a solution?

I want to be able to have my handlers live in e.g. an init() or main() method and have them remain responsive to events.  For example, 

def listener_chose_dir(arg):
    print('listener_chose_dir', arg)

def main(page: Page):
    # I need the listener_chose_dir function defined HERE, in order to access 'page' variable etc. 
    # But if I define it here, the pub sub doesn't work.

    pub.subscribe(listener_chose_dir, 'chose_dir')

flet.app(target=main)


Oliver

unread,
Sep 20, 2022, 12:52:46 AM9/20/22
to pypubsub
You need to keep your listeners "alive" beyond the scope of main(), by keeping a reference to them. There are many ways to do this depending on the details of your design but here are 2: 

  • define a class and instantiate in main() but keep it in root of module (so it outlives main()): 
class MyListeners:
  def __init__(self, ...):  ...
  def listener_choose_dir(arg): ...

my_listeners = None

def main(page: Page): 
    global my_listeners 
    my_listeners   = MyListeners(page, ...)
    pub.subscribe(listener_chose_dir, 'chose_dir')

flet.app(target=main)
  •  use decorator 
    def main_listener_gen(page: Page, ...):
        # create a Python closure so vars in gen can be used later:
        def listener_choose_dir(arg):
             ... use page in here ...
        return  listener_choose_dir

    listener_chose_dir = None

    def main(page: Page):
        global 
listener_chose_dir 
        listener_chose_dir =  main_listener_gen(page, ...)
        pub.subscribe(listener_chose_dir, 'chose_dir')


Both use the same technique of defining an object that lives outside of main() but the object gets created by main() so data from main() can be used in the listener and the listener will not die after main() returns

  • You could also make main() a method of a class, if your flet.app() supports it: 
    class Main:
      def __init__(self, ...):  ...
      def __call__(self, page: Page): self.page = page # for flet.app()
      def listener_choose_dir(arg): ... use self.page ...

    main = Main()

    flet.app(target=main)


  • And of course you could define page at the module level: 
    page = None

    def listener_chose_dir(arg):
        ... use page ...

    def main(_page: Page):
        global page = _page
        pub.subscribe(listener_chose_dir, 'chose_dir')


HTH, 
Oliver

Reply all
Reply to author
Forward
0 new messages