Re: Gaia e-mail app wants fancier support from notifications API/Gaia notifications UI; what do we do?

[Andrew Sutherland]

Pinging for more responses to this; we need to align the UX wireframes
for v1.2 with whatever is likely to be reality. Thanks!

On 07/11/2013 03:39 AM, Andrew Sutherland wrote:
> Currently, notifications as implemented in Gaia have the following
> characteristics:
> - we turn on the screen if it is off so the user can see the
> notifications
> - we do a toaster notification
> - we set a status bar notification
> - we play a ringtone unless the notification volume level is set to 0
> - we vibrate if enabled
> - the UI representation of the notification includes the timestamp of
> when the notification was generated
> - the title of the notification is text/plain that is displayed in
> bold with no supported markup
> - the body of the notification is text/plain that has no special
> display characteristics and with no supported markup
>
>
> In Gaia v1.2 the e-mail app is going to be doing periodic
> synchronization and generating notifications. The following concerns
> flow from the current UX mockups by :robmac which you can possibly
> find at https://mozilla.box.com/s/idsnmmwatjxs26nc3zox or if that URL
> is no more, you can find it by going to
> https://mozilla.box.com/applications and following the path "email >
> interaction > koi1.2" and picking the PDF that has "1.2" encoded in it
> somehow and doesn't say 'signatures' in it.
>
> 1) We do not want to turn on the screen for e-mail notifications if
> the screen is not already on. We expect to generate a lot of these
> notifications, so it seems like we'd be killing the device's battery
> and setting the user's pocket on fire for no good reason. We're
> theorizing turning on the screen might make sense for other apps
> still, so this is an additional bit of information we would want to
> convey to the implementation. Right now it seems like we'd have to
> overload the tag implementation to encode such hints to the Gaia
> System app.
>
> 2) This bit depends on how gaia implements replacement via tag, but we
> might also want to be able to express meta-data that allows us to
> effectively show/update a notification without triggering the
> toaster/ringtone/vibration fireworks show.
>
> 3) We want to be able to provide our own time-stamp associated with
> the notification, something not supported by the WHATWG API but
> supported by Chrome's notification extension API at
> http://developer.chrome.com/extensions/notifications.html. If you got
> the new e-mail yesterday but we only fetched it now, it'd be arguably
> nicer if we could use the timestamp of the message rather than when
> the notification was generated.
>
> 4) A single text/plain body for the notification is insufficient for
> us to best delineate the different salient bits of info about new
> mails which could include all of: message author, message subject,
> message body snippet, account on which this message was retrieved.
> This could be improved by either supporting a weak form of mark-up or
> supporting additional structured information to be included in the
> notification which is amenable to a standardized presentation that
> would also work for e-mail. For example, the aforementioned Chrome
> extension API at
> http://developer.chrome.com/extensions/notifications.html takes an
> optional 'items' attribute which is an array of objects with 'title'
> and 'message' properties which might be sufficient for both our
> ungrouped (1 message per notification) and grouped (> 1 message per
> notification) needs.
>
>
> The set of options that jump out at me are:
> - your notifications are too fancy; we don't even have screen real
> estate for all that!
> - evil hacks by overloading the 'tag' field
> - pursue adding more stuff to the WHATWG/W3C API and be very sorry
> about not pursuing it in the past
> - pursue standardizing the Chrome API even though it doesn't appear to
> be targeted as a web API because their use cases are very similar to
> our use-cases (although B2G v1.0/v1.1 devices don't really have the
> screen real estate for the buttons.)
>
> Thanks,
> Andrew
>
> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi

[Jonas Sicking]

Another alternative is to put the user in charge here. And let the
user on a per-application basis configure things like:

* Can this application create notifications at all
* Do those notifications turn on the screen
* Do those notifications show up on the lock screen

We could then preconfigure those settings such that the email app by
default do not turn on the screen.

The experience on iOS appears to be that notifications is the mobile
version of popups, and that it's good if the user can control how a
given applications notifications are displayed.

However if we don't want to do that (or don't have time to), I'm fine
with adding a moz-prefixed property to the Notification constructor
dictionary which allows an application to say that the screen should
not be turned on.

> 2) This bit depends on how gaia implements replacement via tag, but we might
> also want to be able to express meta-data that allows us to effectively
> show/update a notification without triggering the toaster/ringtone/vibration
> fireworks show.

The idea from the spec is that if you use a replacement, then a new
toaster/whatnot is *not* triggered.

If you do want a new toaster/whatnot, then delete the old message and
create a new one.

Note that the spec still is lacking the ability to get references to
notifications created by a previous run of the app, in order to remove
those notifications. Anne is working on defining a function to do
that.

> 3) We want to be able to provide our own time-stamp associated with the
> notification, something not supported by the WHATWG API but supported by
> Chrome's notification extension API at
> http://developer.chrome.com/extensions/notifications.html. If you got the
> new e-mail yesterday but we only fetched it now, it'd be arguably nicer if
> we could use the timestamp of the message rather than when the notification
> was generated.

I think iOS uses a similar feature when displaying calendar
notifications. They use it to display the time of when an upcoming
event starts. This way the notification doesn't just contain the text
"Event X starts in 10 minutes", but rather instead contains the text
"Event X" and next to it is a dynamically updating timestamp.

Sounds like this setup would work even for timestamps in the past.

Anne, this is something that I think we should examine adding to the
standard given how many notification systems now have seen the need
for it (iOS, Chrome, us).

> 4) A single text/plain body for the notification is insufficient for us to
> best delineate the different salient bits of info about new mails which
> could include all of: message author, message subject, message body snippet,
> account on which this message was retrieved. This could be improved by
> either supporting a weak form of mark-up or supporting additional structured
> information to be included in the notification which is amenable to a
> standardized presentation that would also work for e-mail. For example, the
> aforementioned Chrome extension API at
> http://developer.chrome.com/extensions/notifications.html takes an optional
> 'items' attribute which is an array of objects with 'title' and 'message'
> properties which might be sufficient for both our ungrouped (1 message per
> notification) and grouped (> 1 message per notification) needs.

This one I'm much more worried about. Getting into too fancy content
in the notification means that we're quickly going to run into
problems with realestate I think.

Looking at the UX drafts, I wonder if we can get that behavior by
being smarter in gaia. I.e. could gaia have smarts such that once we
get enough notifications from the same app, we group them together.
And if there's enough of them we display the title and text on the
same line and cut off the text as needed.

Maybe if we add a "groupable" flag to the notification the app can be
in control over if grouping for the notification makes sense or not?

Another option would be to make the API support explicit grouping. So
that you can create a notification group which carries some metadata.
Then add a set of opened notifications to that group. Maybe by giving
the group a tag, and then being able to set a group-tag on each
message.

> The set of options that jump out at me are:
> - your notifications are too fancy; we don't even have screen real estate
> for all that!
> - evil hacks by overloading the 'tag' field
> - pursue adding more stuff to the WHATWG/W3C API and be very sorry about not
> pursuing it in the past
> - pursue standardizing the Chrome API even though it doesn't appear to be
> targeted as a web API because their use cases are very similar to our
> use-cases (although B2G v1.0/v1.1 devices don't really have the screen real
> estate for the buttons.)

Generally speaking we should always aim for that everything that we
add ends up in the spec. If for no other reason because generally we
should assume that if we have a need, that others have similar needs.

However I'm not sure how sorry we should be about not pursuing this in
the past. We shouldn't wildly guess at what needs app authors will
have. A while ago we did talk through "future notification needs" with
the UX team and many of the requirements above did not come up there.
However we did discuss some other ones (such as progress bars), which
we started working on. Though arguably we could have worked harder on
getting those features added to the spec.

Either way, maybe a good next step here, if we need to make progress
faster, is to set up a meeting and discuss what immediate needs we
have for use cases that the current API doesn't solve.

/ Jonas

[Jonas Sicking]

There are two other things that I'm surprised that you are not mentioning.

One I sort of mentioned above. A way for an application to get a list
of the currently active notifications that it has opened, so that it
can remove them or replace them. Or register click handlers on them.

Another thing is the ability to use system messages to wake up an app
when a notification is clicked. I guess right now we just open
whatever URL was clicked, but I would have thought that having more
control over that would be desired?

/ Jonas

[Andrew Sutherland]

On 07/16/2013 06:24 PM, Jonas Sicking wrote:
> Another alternative is to put the user in charge here. And let the
> user on a per-application basis configure things like:
>
> * Can this application create notifications at all
> * Do those notifications turn on the screen
> * Do those notifications show up on the lock screen
>
> We could then preconfigure those settings such that the email app by
> default do not turn on the screen.

I like the idea of letting the user define the maximum capability
envelope the app can have, but I think we still want the app to be able
to decide to use less than its maximum privileges which still implies
some extra fields in a dictionary.

For example, there is a requested e-mail feature for what amounts to
priority notifications on a per-contact basis. I think it would be
reasonable for the e-mail app to generate notifications with a higher
priority for this case.

Of course, we could still make this more of a user-decision thing.
Notifications could take an explicit priority class so that a relatively
easy-to-configure settings UI could let the user say what they want out
of high/medium/low priority notifications
(screen/lock/sounds/vibration/visual toaster). e-mail by default would
generate low priority notifications, SMS would generate medium, priority
e-mail notifications would be high.

If we wanted to facilitate super-fancy replacement home and lock screens
with more extensive configuration UIs, the notification API could allow
for including an extra bag of meta-data.

> The idea from the spec is that if you use a replacement, then a new
> toaster/whatnot is *not* triggered.
>
> If you do want a new toaster/whatnot, then delete the old message and
> create a new one.

Right; we'll need to clean up the Gaia implementation further once the
not-landed patch lands then.

> This one I'm much more worried about. Getting into too fancy content
> in the notification means that we're quickly going to run into
> problems with realestate I think.

As long as whatever is showing the notifications is smart enough to add
some max-height directives / set flex ratios accordingly, I'm not sure
it's that overwhelming a problem. Apps that want to get fancy would
need to rein themselves in when they notice they're running on a device
with a very small screen size or suffer the effects of failing to do so.

> Looking at the UX drafts, I wonder if we can get that behavior by
> being smarter in gaia. I.e. could gaia have smarts such that once we
> get enough notifications from the same app, we group them together.
> And if there's enough of them we display the title and text on the
> same line and cut off the text as needed.
>
> Maybe if we add a "groupable" flag to the notification the app can be
> in control over if grouping for the notification makes sense or not?
>
> Another option would be to make the API support explicit grouping. So
> that you can create a notification group which carries some metadata.
> Then add a set of opened notifications to that group. Maybe by giving
> the group a tag, and then being able to set a group-tag on each
> message.

I think I would prefer just running with the Google Chrome notification
strategy here since it would provide the e-mail app with maximum control
and would enable things like accepting/rejecting a calendar invite
direct from a notification, but I'll table that in the interest of
helping flush out the grouping and potential problems.


I think the information automated grouping would need would be:
- A group tag for conveying which notifications want to be grouped together
- Maybe some title text to display as a header to the group like
"asuth...@asutherland.org" to indicate my account. Without this, the
grouping logic could just default to the app name. This is more of a UX
decision than anything...
- A plural-form string dictionary that can express something like "N new
messages". Without this, the grouping logic would need to fall back to
"N notifications" or something equally generic.

I like your idea of creating an explicit group because then there's no
problems where you have different notifications providing different
strings but claiming to be part of the same group. I don't think we
would want to pre-declare these groups in the manifest because of our
desire to be able to group notifications on a per-account basis for the
e-mail app. However, it wouldn't be crazy if the group did have the
ability to name a notification type that was pre-declared in the
manifest so that the super-fancy settings UI could know about the
different classes of notifications e-mail might generate.

Other information that could be provided on a per-notification basis:
- Arbitrary numeric value encoding display priority for display in the
list. This would allow the e-mail app to weight some messages as more
important than others.
- Maybe assign a weight to the notification. So rather than actually
generating N notifications for N messages, we could generate 5
notifications, and just assign a weight of N-5 to the last one (which
would then have the lowest priority.) This would also allow summaries
like "5 messages in dev-gaia", "900 messages in dev-bikeshed".

Andrew

[Andrew Sutherland]

On 07/17/2013 01:02 AM, Jonas Sicking wrote:
> One I sort of mentioned above. A way for an application to get a list
> of the currently active notifications that it has opened, so that it
> can remove them or replace them. Or register click handlers on them.
>
> Another thing is the ability to use system messages to wake up an app
> when a notification is clicked. I guess right now we just open
> whatever URL was clicked, but I would have thought that having more
> control over that would be desired?

Yeah, those are very good points too. My immediate concern was over
getting the UX wireframes updated (for selfish reasons related to
wanting to avoid inaccurate moztrap test cases and the resulting bugmail :).

There's a mismatch between the somewhat desktop-centric nature of the
notification API and the reality of our resource-limited devices. We
really want something like the mozAlarms API where all notifications
come to us via system messages. Having onclick/onshow/onerror/onclose
exist at all seems like an invitation to some combination of memory
leaks and apps that only work as long as they are never closed with
outstanding messages.

Perhaps we could do the following:

- Get that API to expose the list of currently active notifications
which is a win for everyone

- Define semantics that if you want persistent notifications then you
use the system message mechanism (where your entrypoint is defined in
the manifest, like for alarms), that we disable
onclick/onshow/onerror/onclose (the setters throw if you use them), and
instead you get all those notifications via system messages. Like
"notification-click", "notification-show", "notification-error",
"notification-close". Only "notification-click" actually will wake up
the app, and maybe the others will actually just get discarded if the
app is not active.

- If you don't use our system message thing for persistent
notifications, you get the straight-up normal W3C notification API, but
all your notifications get closed when your app gets closed.

- Maybe add a 'data' field to notifications since otherwise the tag
field might get very overloaded for this purpose or have to be used to
collect data out of IndexedDB/a cookie/localStorage if tag's replacement
features are required.


Andrew

[Thinker K.F. Li]

Do you mean to have an optional string parameter that can carry hidden
data that is invisible by the user? So, (e-mail) Apps can piggyback a
bunch of meta-data/or hints to system app?

--
Sinker
--
天教懶漫帶疏狂

[Anne van Kesteren]

On Tue, Jul 16, 2013 at 3:24 PM, Jonas Sicking <jo...@sicking.cc> wrote:
> Anne, this is something that I think we should examine adding to the
> standard given how many notification systems now have seen the need
> for it (iOS, Chrome, us).

Okay. So Notifications have an associated time stamp, which is the
time they are created, unless specified otherwise through a timeStamp
dictionary member? I'll add that.


--
http://annevankesteren.nl/

[Andrew Sutherland]

It's often ambiguous to me whether to capitalize it as 'timestamp' or
'timeStamp'. The Google Chrome API uses 'eventTime' which I think
neatly avoids that confusion and also seems slightly more clear about
what the timestamp is for.

Andrew

[Anne van Kesteren]

On Wed, Jul 17, 2013 at 12:06 PM, Andrew Sutherland
<asuth...@asutherland.org> wrote:
> It's often ambiguous to me whether to capitalize it as 'timestamp' or
> 'timeStamp'. The Google Chrome API uses 'eventTime' which I think neatly
> avoids that confusion and also seems slightly more clear about what the
> timestamp is for.

Event has timeStamp, but Jonas indicated we want to differentiate from
that as not all notifications will have this. So yeah, we'll go with
eventTime. It'll also use a Date object.


--
http://annevankesteren.nl/

[Mounir Lamouri]

On 16/07/13 15:24, Jonas Sicking wrote:
> I think iOS uses a similar feature when displaying calendar
> notifications. They use it to display the time of when an upcoming
> event starts. This way the notification doesn't just contain the text
> "Event X starts in 10 minutes", but rather instead contains the text
> "Event X" and next to it is a dynamically updating timestamp.
>
> Sounds like this setup would work even for timestamps in the past.
>

> Anne, this is something that I think we should examine adding to the
> standard given how many notification systems now have seen the need
> for it (iOS, Chrome, us).

I think we should discuss this is the appropriate mailing list before
adding it to the specs, just for a matter of courtesy.

--
Mounir

[Mounir Lamouri]

On 17/07/13 00:05, Andrew Sutherland wrote:
> - Define semantics that if you want persistent notifications then you
> use the system message mechanism (where your entrypoint is defined in
> the manifest, like for alarms), that we disable
> onclick/onshow/onerror/onclose (the setters throw if you use them), and
> instead you get all those notifications via system messages. Like
> "notification-click", "notification-show", "notification-error",
> "notification-close". Only "notification-click" actually will wake up
> the app, and maybe the others will actually just get discarded if the
> app is not active.

Then, only "notification-show" should be a system message. FWIW, I think
we should fix that using event pages if it can wait until we have those
implemented.

> - If you don't use our system message thing for persistent
> notifications, you get the straight-up normal W3C notification API, but
> all your notifications get closed when your app gets closed.

I am not a big fan of the idea of having side effects depending on the
page listening to an event or not. This might be quite unintuitive. We
could as well simply discard a notification when clicked and if the app
is closed and there is no event page listening to the event, the
notification would be discarded without the app opening. This could be
considered as a bug in the app instead of us trying to be smarter than
we should.

> - Maybe add a 'data' field to notifications since otherwise the tag
> field might get very overloaded for this purpose or have to be used to
> collect data out of IndexedDB/a cookie/localStorage if tag's replacement
> features are required.

I am not sure I understand. What would be a valid use case for that?

--
Mounir

[Andrew Sutherland]

On 07/18/2013 02:25 PM, Mounir Lamouri wrote:
> Then, only "notification-show" should be a system message. FWIW, I think
> we should fix that using event pages if it can wait until we have those
> implemented.

(Note: I'm assuming you mean 'notification-click'. 'notification-show'
is for when the notification is actually displayed to the user in case
there is a backlog of messages to notify about.)

Event pages sounds like the right solution. Is that something that can
happen for v1.2? It seems like a pretty high probability that the
e-mail app will either get reaped or be compelled to close itself...

>> - Maybe add a 'data' field to notifications since otherwise the tag
>> field might get very overloaded for this purpose or have to be used to
>> collect data out of IndexedDB/a cookie/localStorage if tag's replacement
>> features are required.
> I am not sure I understand. What would be a valid use case for that?

E-mail's planned notification behaviour goes like this when you click on it:
- Showing 1 message, bring up that message
- Showing multiple messages for an account (we want them grouped per
account), bring up the inbox for that account.

In the case where the app has been closed and so any 'onclick' handlers
are dead, we need to be able to extract information from the
notification to tell us what we should do. I think the intent is that
we could use the 'tag' field for this purpose. However, the 'tag' field
is also currently specified to be the only field that can be used to
update a notification without causing all the fireworks to go off.
Notification instances are also immutable, so we can't change the tag on
an existing notification to re-purpose it.

So let's say we have a notification showing 1 message and then we decide
that it now should be talking about multiple messages but we don't want
fireworks. If I was encoding the message id in the notification's
original tag, it's now a lie when I've updated the presentation. So my
primary options are to either just create a handle for a tag like
'account1-notification1' that I persist locally in a database, try and
encode the information in some of the presentation values (maybe append
"?mydata=blah" to the icon URL?), or run some inference logic based on
storing a high water mark in the database that would allow me to infer
what I probably was displaying.

If we have a data field that is separate from the tag field, then that
problem goes away. data could be a DOMString for simplicity, or it
could be a serialized structured clone with bounded storage size.


I think there's a complicating factor here in that there are really 2
types of notifications:
- Persistent notifications that bring you up-to-date with what has
happened since some date in time via summary.
- Ephemeral growl-style notifications that appear and then are gone. An
example would be an IRC client showing you every message that
name-checks you.

Since e-mail can be so distracting, is highly asynchronous, and
frequently can be stuff you don't really care about, the e-mail app
wants more of the persistent notification for summary purposes case.

The notification API spec seems like it was designed more with the
ephemeral growl-notifications use-case in mind. The Chrome
notifications API seems more targeted at the persistent notifications case.

Andrew

[Vivien Nicolas]

On 17/07/2013 07:02, Jonas Sicking wrote:
> On Tue, Jul 16, 2013 at 3:24 PM, Jonas Sicking <jo...@sicking.cc> wrote:
>
>

> Another thing is the ability to use system messages to wake up an app
> when a notification is clicked. I guess right now we just open
> whatever URL was clicked, but I would have thought that having more
> control over that would be desired?

I think what you described here is what has been done for bug 818148.