Let's talk about observable members for Atom 1.0

[Chris Colbert]

Now that all of the typed containers (tuple, list, dict, set) have been implemented in the 1.0.0-dev branch, it's time to talk about observable members.

I suggest reading the following thread to get caught up on the topic:

https://groups.google.com/forum/#!topic/enaml/6r82qlFLWdk

The current state of affairs on the 1.0.0-dev branch is roughly as follows:

• Member validation and storage is fully implement and is faster and more efficient than the previous version.
• The signaling system is fully implement and is also faster and more efficient.
• Static notifications are gone; all signal handlers are registered on an Atom instance. 
• All of the typed containers (tuple, list, dict, set) are fully implemented in C++.

The last major thing we need to decide on is if/how to expose member change notification.

I've previously mentioned that I thought observers (particularly static decorator observers) in the 0.x series of Atom were abused and used to implement wonky control flow. My opinion on that has not changed. I don't think I'll be convinced to support a static observer system in 1.0. Instance observers (the .observe method), however, are up for discussion. I'm looking to strike a balance in the following areas:

• Efficiency - members which are not observed should not incur any cost in time or space when changing their value
• Control - the class author should be able to control which attributes can be observed by users of the class
• Ease of use - it should not require a bunch of boiler plate code to make a member observable
• Flexibility - it should be possible for multiple members to share the same notifiers
• Implementation - the implementation should ride on top of the existing signaling system

The points have led me to the following proposals/ideas, most of which are related to Matthieu Dartiailh's idea presented in the other thread (thanks Matthieu!).

Proposal #1:

A member is marked as observable as part of it's metadata:

class Foo(Atom):
    a = Int()
    b = Float(42.0, observable=True)

The C++ code would create an internal Signal for each member which is marked as observable. Users would call obj.observe('b', callback) to observe any changes. With respect to the design goals:

• Efficiency - the C++ code only needs to check if the signal pointer is null before notifying (basically free)
• Control - the author has complete control over what is observable
• Ease of use -  pretty simple
• Flexibility - each member has an independent set of callbacks == not very flexible 
• Implementation - uses signals internally

Proposal #2:

A member is marked as observable by providing a Signal as part of it's constructor:

class Foo(Atom):
    a = Int()
    b = Float(42.0, notify=Signal())

This is very similar to proposal #1 except that it allows for more flexibility using the following pattern:

class Foo(Atom):
    a = Int()
    changed = Signal()
    b = Float(42.0, notify=changed)
    c = Str(notify=changed)

This would allow the user to use a single callback to listen to both 'b' and 'c'. However, that leads to a weird question: "do I use obj.changed.connect(...), obj.observe('b'), or obj.observe('c')?" In this case, they would all be equivalent.

Proposal #3:

This is a modification to #2 which eliminates the ambiguity of how things are observed. For this case, the .observe() method is removed entirely, and all connections must be performed explicitly through signals.

class Foo(Atom):
    a = Int()
    a_changed = Signal()
    b = Float(42.0, notify=a_changed)

In this case, the class author has complete control over what signal is emitted when a member changes, and the user has a consistent interface of how they attach observers: they always connect to a signal.

The downside to all of the above approaches is that they do not give the author *full* control over how the signals are emitted. If a member value changes, the C++ code will emit the signal immediately. One can imagine cases where the author would like to batch/collapse signal notifications, or defer the notification to a later time. The above proposals would not allow for that. This leads me to the last proposal:

Proposal #4:

Give the author complete control. This approach would keep signals completely separate from members. That is, the C++ code would not emit a signal automatically on member change.

It would be the responsibility of the class author to emit a signal when a value is changed, typically by wrapping the member as a python property.

This would be more boiler plate, but would allow for the most control.

class Foo(Atom):

    b_changed = Signal()

    _b = Float(42.0)

    @property
    def b(self):
        return self._b

    @b.setter
    def b(self,value):
        if value != self._b:
            self._b = value
            # maybe defer
            self.b_changed(value)

These are my ideas so far. They are not very concrete. Feel free to propose any suggestions, alternatives, or ideas you may have.

Chris

[Matthieu Dartiailh]

Hi, Chris

I agree with you that suggestion one is too rigid, and to me suggestion 4 does indeed implies too much boiler code (and on top of that I can't see how Enaml will connect to that but that is another issue). Furthermore, I am not a big fan of having for every member I want to be able to observe to create a private attribute and a property, I tend to find that the code loses in readability when doing so. By the way would you envisage adding a nice way to replace static observers (such as a post-setattr method) (no big hopes here)?

I favor proposition 2 and 3.
One point that is not clear for me is the form handlers should have. I guess that we could ask user to use *args, **kwargs, and provide them a single dictionary most of the time (see further).
I think proposition 3 is cleaner because that way people won't 'forget' that they are listening to a signal not just to a member (if two members use the same signal, people might be surprised to get notifications for one when they ask to 'observe' the other. I might be interesting to have a way to query the signal connected to a member as it would allow to use Str('', changed=Signal()) even with this kind of connections.
When it comes to control on how notifications are emitted would it be possible to have some kind of context on the Atom object saying don't dispatch for this signal or queue notifications and dispatch them when leaving the context ? Such a thing could be done by checking bit flags in the emit method (I think) and deciding from it what to do then. The collapsing of the notifications in the case of the queue would depend on the signature we choose for notifications (they could get a tuple summarizing all the changes that happened, hence the most of the time previously). More general methods to avoid dispatching anything or batching everything could be possible too. I guess it will make the base Atom class a bit more complex but I really think it would be worth it.

Best,

Matthieu

[jelle feringa]

I'd gravitate to proposal #2.

However, I think that "notifier=Signal()" would be a good default value.

In my experience, not sending signals would be the non-default case.

Finally, what about "notifier=[Signal(), changed ]" in a case where you'd like to create more observers.

Would open up interesting possibilities?

Thinking about this, is there a place for something like a ConditionalSignal()?

Often, I deal with validating multiple signals [ does the opengl viewer exist? is the object shown in the viewer? if the object has changed, then update it in the viewer ] in my method that respond to the signal.

A ConditionalSignal would be a more adequate solution.

A bit tangential, but trying to give you an idea why I think being able to bind to more than a single Signal might have a place.

-jelle

[Chris Colbert]

At this point, i'm not too concerned about the question about the argument spec of the signal handlers. I think it may be simpler to pin down the semantics of notification first, and then talk about what should emitted with the notification.

A single static observer (basically a post setattr handler) would be relative cheap to implement (a single null pointer check), so I wouldn't be entirely opposed to implementing something like that. It would make the metaclass logic a bit more complex, but nothing too serious or expensive. Enaml uses those hooks quite a bit, and I have to think of an alternative which doesn't require boiler-plate properties, or expensive descriptors implemented in Python.

Any signal that you assign to a member would be queryable through the Member object, which can already be retrieved via `obj.get_member('name')`.

There are certainly use cases for deferring and batch signal notifications. Qt signals take an optional argument to QObject::connect() which can control whether the signal is emitted now or later, and whether duplicate signal connections are allowed. Atom already de-duplicates connections, but deferring a notification requires an event loop; not something I'm certain Atom should concern itself with, but also not fully convinced of that. Deferring notification until the end of a context is an interesting idea and I'd like to hear more about some use cases you have in mind for that.

I don't want to pursue an approach which will incur substantial space or time overhead for the direct-connected case, just to support the corner cases which could otherwise be implemented with a bit more boiler-plate on the class-author side.

[Chris Colbert]

My responses are inline.

On Sunday, October 19, 2014 1:47:57 PM UTC-4, jelle feringa wrote:

I'd gravitate to proposal #2.

However, I think that "notifier=Signal()" would be a good default value.

In my experience, not sending signals would be the non-default case.

I tend to disagree with this. I consider an observable member to be part of a class' public api (if it's private, you know when it changes and therefore don't need observers), and having a smaller public api footprint leads to better designed and more maintainable code. Having everything be observable by default encourages people to use observers for control flow, when a different abstraction would be more appropriate.

Furthermore, there is cost to setting an attribute on an observable member. If an object has observers for some members, it will be an O(log(n)) lookup to determine whether observers exist for a given member. I would prefer the class author to make the decision "This is part of my public API, and there will likely be users observing it." when declaring  a member. That way, modifications to all of your other attributes which don't need to be observed do not pay any cost.

Finally, what about "notifier=[Signal(), changed ]" in a case where you'd like to create more observers.

Would open up interesting possibilities?

What is 'changed' in this context? Could you elaborate a bit more on this example?

Thinking about this, is there a place for something like a ConditionalSignal()?

Often, I deal with validating multiple signals [ does the opengl viewer exist? is the object shown in the viewer? if the object has changed, then update it in the viewer ] in my method that respond to the signal.

A ConditionalSignal would be a more adequate solution.

I'm also not sure I'm following you here. Would you mind posting some pseudo-code?

[Matthieu Dartiailh]

Hi,

A post setattr handler would indeed be perfect for me (the only case missing being the one of containers but I will adapt). By the way it may be interesting to have a way to generate a signal for the current state of a member, for example after several in place notifications have been made to a container the user could fire it rather than going through the process of copy and re-assign.

I like the idea of Atom not having an event loop because I fear that the contrary could lead to awful situations where the Atom event loop and the Qt one (used by Enaml) do several things in the wrong order and the users get completely lost. The idea of the context just popped in answer to your worry of the user lacking control about how events are handled, but here is the beginning of a use case :
- in my application I need to transfer data across processes for monitoring purposes. What I currently do is that the part of the code generating new data emits a signal connected to the part responsible for the inter-process communication which sends the infos over a pipe. If a user knows he is going to generate a lot of data very quickly he might want to batch the infos to send a single message over the pipe.
- at the other end, the infos are used to update a GUI so getting a large packet rather than a ton of small ones would allow for better efficiency for the GUI. And in the same idea if the GUI is relying on a small number of signals to update but that these signals describe lets say 50 members and that all of them needs to be updated it would be nice to batch the message sent to the GUI rather than send 50.
In both cases a context looks like a nice way to go as the only things needed is to enclose the signal generating code in it. Does my use cases make sense ?

It seems to me that by using bit flags and a switch statement in C the direct connected case could still quite direct even if other way of dispatching exist, but I can be wrong.

Regards

Matthieu

--
You received this message because you are subscribed to the Google Groups "Enaml" group.
To unsubscribe from this group and stop receiving emails from it, send an email to enaml+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthieu Dartiailh]

Hi, Chris

I know you are fairly busy working on Phosphor right now but I would be happy to do some work to move Atom 1.0.0 forward in the meantime. If you can give me some directions about what remains to be done I could try to make some progress.
However if it would require too much of your time because some points are yet to be decided I will understand and wait.

Best regards
Matthieu

[Brad Buran]

I am happy to help with some code to get us to Atom 1.0 as well. In general, I favor option 2 or 3. Perhaps a decision should be made on which option most closely follows the Zen of Python (i.e., how complicated is it to explain)?

However, I'm not clear on how Enaml will work with Atom 1.0. How will Enaml know to update a GUI widget (or Container)? Will there be an opportunity to indicate a signal which indicates the Container needs to be updated? 

Brad