Notification Architecture

[Stephen Hansen]

H'lo.

On the 0.12 milestone (and in the 0.12 thread down below somewhere), it talks of "Notification Architecture" being a possible goal, and I see quite a few tickets all related to the same things-- people wanting more or less messages, on more or less subjects. And different information being sent.

There's a linked proposal that seems to talk of a different /kind/ of notification-- transactional events passing about the internals-- but I don't see how that's related directly.

I ask because the basic email-notification has never worked for my needs; I wrote my own module a few years ago when Trac was a much simpler thing, and now that I'm migrating our environment back to a vastly prettier Trac, I'm probably going to implement my own system again. Unless someone else has already written half of one and it may end up in 0.12 or someone has a better idea of how to implement it :)

I have in mind a modular system where users can easily (in Preferences, or whatever) "subscribe" to certain notifications. They can filter these in a flexible but mostly simple way, and then declare where they want them to go.

Global options don't work for me; and neither do the proposals made in the t.e.o tickets to add various checkboxes turning on and off certain states. I need something per-user configurable online, and capable of filtering (in at least the simplest way) and determining to receive notices based on factors besides owner/reporter/cc. 

There'd be default subscriptions that mimic what basically happens now, but more fine-grained control if they need it.

For example, I could create a subscription that was basically: "realm=ticket, category=created, condition=all, when=severity >= critical", and declare this should be emailed to an alert address that normally doesn't get trac-emails. A manager wants to receive notices when any ticket is changed in trac, but doesn't want to receive his own changes.. so he'd subscribe to "realm=ticket, category=changed, condition=others". The project managers, installers, and even a customer want to get a message to any issue that has the "Install-Rome 4.1 Upgrade" milestone, so they'd "realm=ticket, category=created|changed, condition=all, when=milestone = Install-Rome 4.1 Upgrade.". They don't want to have to add themselves to the CC's for them all.

My idea for implementing this at present involves around objects implementing two interfaces; ISubscription and IDelivery (Names temporary), then a tangible object that is used to interface to these systems. 

There would be lots of ISubscription components; each one registers a list of realms it can handle subscriptions for, and a list of categories in those realms it knows how to check for. The default TicketSubscription object would report that it can handle subscriptions for the "ticket" realm, and categories (off the top of my head), ("created", "comment added", "propchanged", "status changed") or whatever. The WikiSubscription would do the same, but for its own realm. Plug-Ins could enhance these. They may send messages themselves, and so add new realms; or they may add a category to an existing realm that indicates they are able to determine if someone should hear a notice based on details the default system doesn't have. Say one of the time-tracking modules could send notices of time-events to HR who are cracking down on overtime. Or whatever.

The Subscription components are basically responsible for providing a list of people who are interested in a given event. A Plug-In might add categories for messages it sends itself, or just add the ability to make the determination in a different way that someone is interested in a category that comes with the base systesm.

Then there is the IDelivery component, that can take a notice and a subscription and actually send it out. The EmailDelivery component does basically what NotifyEmail does; but an IrcDelivery or SMSDelivery is possible to. I haven't decided if it can handle formatting itself or just transmission, with an IMessageFormatter out there. 

The connection between these 'plug-able' components is the NotificationSystem (or whatever). 

One would do something like:

   notifier = NotificationSystem(self.env, ticket)

   notifier.notify("comment added", commentingUser, message, "Optional short message doing a quick summary(the IrcDelivery sends this instead!)")

The NotificationSystem then checks to see if it has anything registered that can handle notifications from the ticket realm; if not? It discards the message. If so, it then checks if any of those know how to determine if someone is subscribed to the "changed" category. TicketSubscription does, so it passes the details to it.

TicketSubscription scans over the "subscription" table, determines Joe wants to get a message anytime someone messes with his component, Bob wants to get a message anytime someone fiddles with his issues(but does not want to receive a message if he is the fiddler), and Daniel wants to stay on top of any tickets with a high priority-- which this is.

So it ends up yielding ( "email", "joe", "j...@example.com"), ( "email", "bob", " b...@example.com"), ( "im", "daniel", "aim:thebossman")

NotificationSystem then checks to see if it knows any IDelivery components that know how to handle "email" destinations, and it finds one so it sends it details on the event and Joe and Bob's addresses. Then it discovers InstantMessageDelivery knows how to handle "im" destinations, and so on and so forth.

Once the basic system is in place, I planned on writing subscription handlers for all the types of objects that send NotifyEmail right now, and then it should be able to just readily replace them.

So! Ahem.

Is anyone considering/planning a flexible notification system in particular for 0.12 (or 0.13, or...) Has work been done already? If no, and no, I'm going to end up doing the above just to satisfy our own needs. Unless the idea floating in my head for /how/ ends up not working at all. Ahem. Is such a system desirable in core-Trac? If not I'll just keep it when I'm done :)

--Stephen

[Alec Thomas]

I don't have the time to comment on everything in this mail, but overall
I think it's a good approach. From your description, this could be
entirely written as a plugin using the various I*ChangeListener
interfaces already in Trac. The question of whether it goes into core
Trac (replacing the existing system) can be deferred until it's complete
as it shouldn't (at first blush) require any patching of Trac core.

You'd also probably have to implement a backwards compatibility layer on
top of this system. eg. if "always_notify_owner" is set, an
ITicketChangeListener implementation adds a subscription for that user.

You should also definitely talk to Emmanuel Blot (eblot). He wrote the
original notification system and I believe he has been considering
extending it.

--
Evolution: Taking care of those too stupid to take care of themselves.

[Stephen Hansen]

I don't have the time to comment on everything in this mail, but overall
I think it's a good approach. From your description, this could be
entirely written as a plugin using the various I*ChangeListener
interfaces already in Trac. The question of whether it goes into core
Trac (replacing the existing system) can be deferred until it's complete
as it shouldn't (at first blush) require any patching of Trac core.

Yea, I tried to make it as plug-iny as possible. Hm. If I just totally disable

all the notification options in the trac.ini and use my own, then I'd probably 

need no core changes at all. Hadn't thought of that-- figured I'd have to replace 

the NotifyEmail calls. Cool, cool.

I figured the decision's be deferred anyways until I produced a patch and 

someone decided it didn't suck ;-) But I was seeking thoughts early, so if I

can satisfy my needs and larger interests of The Greater Good at the same 

time, might as well.

 

You'd also probably have to implement a backwards compatibility layer on
top of this system. eg. if "always_notify_owner" is set, an
ITicketChangeListener implementation adds a subscription for that user.

Point; a LegacySubscription component can just always return 

('email', 'username', 'f...@bar.com') if that option is on as if they had such

a subscription. And the other always_notify_* options. And a

CarbonCopySubscription component which tosses out everyone CC'd on a

ticket as if they had a subscription.

You should also definitely talk to Emmanuel Blot (eblot). He wrote the
original notification system and I believe he has been considering
extending it.

 Noted, thanks :)

--Stephen

[Endre Bakka]

> I ask because the basic email-notification has never worked for my
> needs; I

You're not alone...

> I'm
> probably going to implement my own system again.

Oh, this makes me very very happy. It is on my todo-list as well, but
given my workload, it's more of a oh-I-wish-list than a todo-list.

> I have in mind a modular system where users can easily (in Preferences,
> or
> whatever) "subscribe" to certain notifications. They can filter these in
> a
> flexible but mostly simple way, and then declare where they want them to
> go.
>
> Global options don't work for me; and neither do the proposals made in
> the
> t.e.o tickets to add various checkboxes turning on and off certain
> states. I
> need something per-user configurable online, and capable of filtering
> (in at
> least the simplest way) and determining to receive notices based on
> factors
> besides owner/reporter/cc.

Interesting idea, and way better than what we have today for sure. I have
given the notifcation system some thought as well ("various checkboxes
turning on and off certain states" might be me), so I wanted to chip in
with my two cents.

My original thought was as follows (short version):
- Each ticket has a notification section where you can specify which
events you want to be notified for that particular ticket (e.g. when a
comment is added, when status changes etc).
- In the preferences, each user can set default rules that are
automatically applied when a ticket is created (e.g. when owner, notify on
all changes, when reporter, only notify on comments, status and
milestone). But the user can still change the type of notification he
wants on each and every ticket. I guess these rules could be similar to
the filters you were thinking of.

I love the filters you propose - that's way more flexible and powerful
than what I had envisioned, but I also miss the possibility of adjusting
the notification settings for each and every ticket. Of course, adding
notification for a single ticket is just a matter of creating a
subscription with "ticketid=433", but if you want to change the filter for
a ticket that already is caught it starts getting cumbersome.

Have you given this aspect any thought?

> So it ends up yielding ( "email", "joe", "j...@example.com"),
> ( "email", "bob", "b...@example.com"), ( "im", "daniel", "aim:thebossman")
>
> NotificationSystem then checks to see if it knows any IDelivery
> components
> that know how to handle "email" destinations, and it finds one so it
> sends
> it details on the event and Joe and Bob's addresses. Then it discovers
> InstantMessageDelivery knows how to handle "im" destinations, and so on
> and
> so forth.

Oh, that's neat. I really like that :-)

> idea floating in my head for /how/ ends up not working at all. Ahem. Is
> such
> a system desirable in core-Trac? If not I'll just keep it when I'm done
> :)

As Alec suggested, a plug-in is the way to go. It will be available as
soon as it is finished (don't have to wait x months for the 0.12 release),
and you don't have to worry whether it will be accepted into the core or
not. You might discover that you need some new extension points, but these
should be fairly easy to get into the core.

- Endre

[Emmanuel Blot]

> You should also definitely talk to Emmanuel Blot (eblot). He wrote the
> original notification system and I believe he has been considering
> extending it.

Actually, I did not wrote it: I mostly fixed some bugs in the
notification format so that Trac complies with RFCs and it works with
most MUAs (email clients).

Notification subsystem needs a major rewrite, and the last time "we"
talk about it (many months ago) IChangeListener interface was to be
used to replace the current notification architecture.

There was already to many changes scheduled for 0.11, so it has been
decided to postpone notification changes to the next release (0.12).

However, before starting any work on the notification architecture, we
need to go through the very long list of notification enhancement
tickets (and suggestions about the various roles of
notification-related options and fields). This is required to define
which features should be left in the Trac core, which API(s) should be
offered to plugins, etc.

The notification option set also needs to be cleaned up and simplify.
Another big change is to move system-wide settings to user settings.
Notification in 0.12 is likely to break compatibility with previous
release of Trac (both from an API perspective and from a configuration
perspective).

Cheers,
Manu

[Stephen Hansen]

Interesting idea, and way better than what we have today for sure. I have
given the notifcation system some thought as well ("various checkboxes
turning on and off certain states" might be me), so I wanted to chip in
with my two cents.

The problem with per-ticket rules and lots of check boxes is no one is gonna be happy. There's no less then two tickets on t.e.o I saw which referenced such a preference, and where one person wants a lot of options, another wants only a few. I'm not even sure what I want! *laugh* Which is part of the reason why I was going for a pluggable architecture in and of itself. 

My original thought was as follows (short version):

(snip per-ticket notifications and default choices)

I love the filters you propose - that's way more flexible and powerful
than what I had envisioned, but I also miss the possibility of adjusting
the notification settings for each and every ticket. Of course, adding
notification for a single ticket is just a matter of creating a
subscription with "ticketid=433", but if you want to change the filter for
a ticket that already is caught it starts getting cumbersome.

Have you given this aspect any thought?

I have actually, and have such a thing planned.. but not for the core Announcer plug-in (so far that's the name I've come up with). The basic system will have the user-editable subscription system letting people set rules; "mine", "all, with priority > 3" or "others, with milestone matches Release-*" and such. Everytime a ticket is changed, the TicketSubscription component (an implementation of IAnnouncerSubscription) would scan the list of rules it has cached and see if that ticket matches any of them. The rules are stored for the user, not the ticket-- but you're right. If someone wants to make a per ticket rule, they could easily make "all, with id = 433".

But that's beyond cumbersome, you're right. :)

Instead, I have in mind a WatchedTicketSubscription plug-in that can be added. It's separate from the core announcement architecture, and would in theory add a 'Watch This' button to the ticket UI-- however one goes about doing that, I don't know yet :) It might have various options you can select (comment added, status changed, etc).

How it stores these choices and determines that you want to watch Ticket #4 has nothing to do with how the filters above work; the system itself doesn't require subscriptions to be stored or processed in any particular way. The rules above were just one implementation of determining subscriptions, a general purpose subscription system; a specific desire could just be a plug-in on that core... an IAnnouncementSubscription implementation could decide that you have a subscription based on the phase of the moon if it wants :)

The idea is one plug-in component implements the rule-based filter system for general preferences; one mimics current behavior to allow the system to completely replace the core notifications if desired while letting someone turn off certain features and refine with the rules; one which adds a 'watched' tag (with varying levels of specifity) to wiki pages and tickets. One does something I hadn't thought of. All look the same to the AnnouncementSystem master; all just report back subscriptions to an event.
 

> idea floating in my head for /how/ ends up not working at all. Ahem. Is
> such
> a system desirable in core-Trac? If not I'll just keep it when I'm done
> :)

As Alec suggested, a plug-in is the way to go. It will be available as
soon as it is finished (don't have to wait x months for the 0.12 release),
and you don't have to worry whether it will be accepted into the core or
not. You might discover that you need some new extension points, but these
should be fairly easy to get into the core.

Yeah, Plug-in it'll be. If I end up doing this, I expect it to be done on weeks if not a month or so-- it being a requirement to make the system work the way my company needs it to is motivational.

My only real concern is it works for us; beyond that I'm pleased as punch if it ends up being useful for others. I'm still going through the core code and peering, and through the various notification tickets on t.e.o to see if anyone has thought of something I hadn't considered.

(Now to see if the Boss will give me comp-days for work-related weekend work; making trac not spam him if he's not interested in something is surely day-job-work! :))

Thanks for the feedback/ideas.

--Stephen