TCP networking protocol
Evan Schoenberg
Growl and will be continuing to improve networking in general.
What TCP-based networking protocol? That's where this email and you,
dear reader, come in.
1. The protocol should be relatively easy for both sender and receiver.
2. It should not be platform-dependent.
3. We need to have some form of authentication, preferably with
encryption of notifications so that there are no concerns with sending
potentially sensitive data over the wire. That encryption, keeping
with (1) and (2), should be straightforward for a wide variety of
platforms to use.
Peter and I discussed the protocol a bit off-list already.
I initially suggested an XML-based protocol, as XML is easy to read
(for a human) and straightforward to write and parse in most languages
given the wide variety of XML libraries available. This would look
something like:
<notification>
<title>Something happened!</title>
<description>This thing happened, and I thought you should know.</
description>
<image>base64-encoded-image</image>
</notification>
However, he pointed out that the majority of the data in a
notification is the image data, and raw bits and XML don't mix,
necessitating base64 encoding or the like. Also, the full document
must be parsed for it to make sense. If, instead, we could indicate
the number of bytes to follow were to be interpreted literally, that
would be a computational savings. (Peter, please clarify if I
misunderstood any of that)
His counter-suggestion was something similar to MIME headers, like this:
Action: Notification
Title: Something happened!
Description: This thing happened, and I thought you should know.
Image: 103908
<103908 bytes of image data>
Any thoughts or preferences for one approach versus the other?
Also, I have no experience in the production of a secure protocol. We
just need to be able to say, "Is the same password (or lack of
password) being specified on both ends?" with that password being the
decryptor for the message, I think. Could someone with more knowledge
than that expand on those thoughts with what might be an appropriate
authentication system?
Cheers,
Evan
Gary Kramlich
> Growl and will be continuing to improve networking in general.
> What TCP-based networking protocol? That's where this email and you,
> dear reader, come in.
> 1. The protocol should be relatively easy for both sender and receiver.=
> 2. It should not be platform-dependent.
> 3. We need to have some form of authentication, preferably with
> encryption of notifications so that there are no concerns with sending
> potentially sensitive data over the wire. That encryption, keeping wit=
h
> (1) and (2), should be straightforward for a wide variety of platforms
> to use.
>=20
> Peter and I discussed the protocol a bit off-list already.
I assume you mean one central server and many clients? If so, I assume
the clients are pushing? If it's distributive, are you planning on
pushing or pulling?
> I initially suggested an XML-based protocol, as XML is easy to read (fo=
r
> a human) and straightforward to write and parse in most languages given=
> the wide variety of XML libraries available. This would look something=
> like:
> <notification>
> <title>Something happened!</title>
> <description>This thing happened, and I thought you should
> know.</description>
> <image>base64-encoded-image</image>
> </notification>
>=20
> However, he pointed out that the majority of the data in a notification=
> is the image data, and raw bits and XML don't mix, necessitating base64=
> encoding or the like. Also, the full document must be parsed for it to
> make sense. If, instead, we could indicate the number of bytes to
> follow were to be interpreted literally, that would be a computational
> savings. (Peter, please clarify if I misunderstood any of that)
This could probably be easily rectified by something like xmlrpc over htt=
ps.
> His counter-suggestion was something similar to MIME headers, like this=
:
>=20
> Action: Notification
> Title: Something happened!
> Description: This thing happened, and I thought you should know.
> Image: 103908
> <103908 bytes of image data>
I assume the text would be a constant encoding, something like utf-8 or
would be specified. Also, is this supposed to be like http headers?
Also, what happens if a description has an embed newline in.
> Any thoughts or preferences for one approach versus the other?
I'd say go with an xmlrpc approach passing a url to the image's in the
main notification chunk. Thus if this is being pushed (see above about
the push/pull question) to a client that doesn't have a graphics
environment, there is no need for it to grab an image it can't use.
> Also, I have no experience in the production of a secure protocol. We
> just need to be able to say, "Is the same password (or lack of password=
)
> being specified on both ends?" with that password being the decryptor
> for the message, I think. Could someone with more knowledge than that
> expand on those thoughts with what might be an appropriate
> authentication system?
Developing secure protocols is no easy task, which is why I'd suggest
xmlrpc over https, but of course, then you need ssl certs. You could
use shared key encryption, but again, you need a way to pass the key.
You could also use an existing crypter using a password as the salt, but
without any throttling, brute force attacks could be possible.
> Cheers,
> Evan
--=20
Gary Kramlich <gr...@reaperworld.com>
Peter Hosey
> Also, the full [XML] document must be parsed for it to make sense.
Not necessarily; it depends on the parser. expat, I think, can just
suck XML straight off a file-descriptor (e.g., socket) without having
to suck it all in at once. NSXMLParser, on the other hand, requires an
entire document at once.
> (Peter, please clarify if I misunderstood any of that)
Aside from the above, you nailed it.
> Any thoughts or preferences for one approach versus the other?
I would point out that we should not worry too much about details of
various protocol suggestions. I think we should pick an overall scheme
(XML, MIME, _____, _____, …) and refine from there.
> Also, I have no experience in the production of a secure protocol.
> We just need to be able to say, "Is the same password (or lack of
> password) being specified on both ends?" with that password being
> the decryptor for the message, I think. Could someone with more
> knowledge than that expand on those thoughts with what might be an
> appropriate authentication system?
Not that I have any experience in this, either, but we do want to not
send passwords over the wire. We should use some sort of hash.
Currently, I'd recommend one of the SHA family, such as SHA-1 or
SHA-256.
As of OpenSSL 0.9.7l (included with Mac OS X 10.5.3), the openssl(1)
command-line front end doesn't support SHA-256. Python does, however.
I have no idea what hashing options are available on Windows.
Peter Hosey
> I assume you mean one central server and many clients?
That's our current system, and I think all thought we've given to a
new protocol has been in the same direction. It's certainly a simple
system.
That said, I, for one, am open to alternatives.
> If it's distributive, are you planning on pushing or pulling?
I dislike polling behavior, so I say that nodes in such a system
should push notifications to other nodes.
> This could probably be easily rectified by something like xmlrpc
> over https.
Indeed, doesn't TLS solve the encryption problem neatly?
Not that I've ever used TLS at an API level.
> I assume the text would be a constant encoding, something like utf-8
> or would be specified.
Yeah, I was envisioning UTF-8. Header names would be ASCII.
> Also, is this supposed to be like http headers?
Yup. Very similar to HTTP, actually, as the example below will show.
> Also, what happens if a description has an embed newline in.
I considered this. We can provide a Description-Length header, as an
alternative to Description:
---
NOTIFY Download succeeded
Protocol-Version: 1.0
Application-Name: GrowlSafari
Title: Download succeeded
Description-Length: 44
Image-Length: 1048576
File: Growl-1.2.dmg
Destination: ~/Downloads
<image bytes>
---
We may also want to provide a header that specifies the order of such
payloads. In this case, it would be “Payloads: Description Image”.
For every name specified in the value of that header, the length
header would be “<name>-Length”.
Chris Peel
Has anyone considered JSON? It's a much lighter protocol than XML is
(in bytes transmitted at least) and is very widely used already.
Secondly, how about simply sending a URL to the image to use rather
than trying to send the image itself? This means the end client can
use O/S native ways of retrieving the image and there's no concern
over big/little-endian formatting, etc.
Peter Hosey
> Has anyone considered JSON? It's a much lighter protocol than XML
> is (in bytes transmitted at least) and is very widely used already.
How available are JSON parsers?
I know that on the Mac, there's a JSON framework that anybody can use
(BSD-licensed, just like Growl). Are ready-made and easy-to-use JSON
parsers also available for Windows and GTK?
Also, how does JSON handle binary data (such as images)? Would we have
to rely on something like Base64 or, worse, URL-encoding?
> Secondly, how about simply sending a URL to the image to use rather
> than trying to send the image itself?
Attractive for bandwidth reasons, in cases where the recipient doesn't
need the image (notification turned off, non-visual display selected,
etc.), but does it really matter, especially over a local network?
Even through an intertube, you can send up to a few hundred K in a
reasonable amount of time.
There's added complexity here: The sender must now implement a server
of its own, in addition to connecting to a notification server, and
the server must connect back to the sender, rather than passively
waiting for everything to come to it.
Is it worth it?
Vinay Venkatesh
up with a common protocol so that "growl servers" can talk to
libnotifiy on linux and vice versa. We had discussed doing it in
XML. While it requires base64 encoding, etc for binary data, it's
easy enough to do that it's not terribly difficult to implement.
As far as passwords go, it could just as easily be Public/Private Key
encrypted, where a key is specified for a host after the initial
registration with the server.
/VInay
Peter Hosey
> I've been working with Christian Hammond on coming up with a common
> protocol so that "growl servers" can talk to libnotifiy on linux and
> vice versa.
Sounds relevant to this thread. Would you mind inviting him to this
list?
> We had discussed doing it in XML. While it requires base64
> encoding, etc for binary data, it's easy enough to do that it's not
> terribly difficult to implement.
My experience with XML is that it usually involves one of:
1. Writing handlers for a stream-based parser, which is a pain in the
ass
2. Slurping an entire document in at once, then processing it with
XPath or something
3. Using a higher-level library that that does one of those for you
Based on this knowledge, I am not a fan.
For a MIME-style format, you could write your own parser and it'd
*still* be simpler code than #1 and lower-overhead than #2. For JSON,
you'd have to do #2, but it'd still be lower overhead than XML.
If there's any real advantage to XML, I'd love to hear it, but I
haven't heard it yet.
> As far as passwords go, it could just as easily be Public/Private
> Key encrypted, …
Interesting. Advantages/disadvantages over plain hashing?
> … where a key is specified for a host after the initial registration
> with the server.
What do you mean?
Gary Kramlich
<snip>
>> As far as passwords go, it could just as easily be Public/Private
>> Key encrypted, …
>>
>
> Interesting. Advantages/disadvantages over plain hashing?
>
I assume by plain hashing you mean just the password? If so, is there
nothing that growl could notify that might be considered "private" and
thus should be encrypted as well?
--
Gary Kramlich
Ofri
1. Both sides generate a private/public key pairs.
2. Both sides exchange the public keys.
3. Both sides ask the user to provide a password, which should be
identical on both machines.
4. Each side encrypts the hash of the password using its private key
and sends it to the other side. The receiving side decrypts the hash
and compares it with the hash of the password it holds. If both hashes
match, the other side is trusted and its public key is saved.
5. All future communication is encrypted using the private/public keys.
Ofri
Ofri
side's public key. I don't think it matter that much (won't be
surprised if I'm wrong) but it just seems more correct.
Ofri
briandunnington
than XML, and parsers should be available for every platform. i have
actually been thinking about this topic for quite awhile and will add
my thoughts, as well as a few questions. i uploaded a simple
description of what a JSON-based protocol might look like to the Files
section of this group if anyone wants to take a look at it (http://
groups.google.com/group/growl-development/web/
growl_protocol_proposal_json.txt?hl=en). some of the key points are:
- image data can be passed in several formats, including using a URL
(http://site/image.jpg) or raw data (similar to the data:// urls
supported by browsers). that should allow enough flexibility for
different scenarios (desktop, network notifications, web-based, etc)
- allow the inclusion of custom attributes in addition to the
standard ones. (ex: x-myAttribute). this would work similar to how
HTTP headers allow you to add custom headers. these custom attributes
could be used by the display notification or in the callback (see
below)
- callbacks are specified in one of several formats, similar to image
data. they can be a URL or a special identifier that indicates to
local window/app that sent the notification. parameters can be passed
back with the callback, including custom attributes as specified
above.
i also have a few questions for the group. you said this should be TCP
based, but are you really just trying to figure out the data format? i
mean, could the same data format also be sent over UDP if desired (and
the Growl app supported it)? i know that several people have expressed
interest in the ability to broadcast notifications, and the ability to
'fire-and-forget' without requiring the client to acknowledge receipt
is very hand in many cases. i think that a standard data format is the
important piece here.
also, when you say you want a 'secure protocol', do you actually mean
you want the notifications encrypted, or do you simply want to verify
that the same password was used on both ends (similar to how it works
now). if it is the later, then using the password as a hash is the
easiest way to go. the password is never sent, and the authorization
check is easy. of course, this does NOT protect sensitive information
as it goes over the wire from being *read* (only from being modified).
if you want that as well, then a more complicated scheme needs to be
implemented.
in a thread in the Growl uses group, i think someone said that they
envisioned this protocol eventually replacing both the exisiting TCP
and UDP protocols - is that still the goal? if so, does that mean this
protocol will be used for network notifications, or for local
(desktop) type notifications as well? if the goal is to eventually be
cross platform, i would suggest that even local desktop apps use the
protocol to communicate - then there is only one type of message to
send whether your app is running locally or on the network or on the
internet. (in my Growl for Windows implementation, i used the UDP
protocol for the desktop notifications as well and it worked good,
minus the support for images and callback notifications that are not
part of that protocol).
briandunnington
are working on a Notifications API as well:
http://code.google.com/p/gears/wiki/NotificationAPI
this has not been released in any version of Gears yet and the actual
API details are not published, but i thought it might apply to this
discussion. not sure if we want to work with them to define the
standard or not, but just wanted to include anyone who was working on
something similar.
Peter Hosey
> also wanted to point out that the Gears team (formerly Google Gears)
> are working on a Notifications API as well:
>
> http://code.google.com/p/gears/wiki/NotificationAPI
Hm. Interesting. We probably should reach out to them and see if
they'd like to get on board, too.
Peter Hosey
> i would second the JSON approach - it is nice and easy, less verbose
> than XML, and parsers should be available for every platform.
I agree that they should be, but are they? :)
- Is there a JSON parser for Windows?
- Is there a JSON parser for Linux?
(Browsers and WebKit don't count.)
> i uploaded a simple description of what a JSON-based protocol might
> look like to the Files section of this group if anyone wants to take
> a look at it (http://groups.google.com/group/growl-development/web/growl_protocol_proposal_json.txt?hl=en
> ).
That URL is 404. We actually need to go to the Files area to view it:
http://groups.google.com/group/growl-development/files
Thanks; it looks like a good example. I'd use MIME types instead of
having these “IconFormat.Jpg” constants, but as I said, we can debate
details later.
> - image data can be passed in several formats, including using a URL
> (http://site/image.jpg) or raw data (similar to the data:// urls
> supported by browsers).
Supporting data: URLs themselves would relieve the need to specify the
type separately. For an HTTP URL, the machine retrieving the image
would probably use a HEAD request first to find out the type (or just
blindly GET it and close the connection if it's not a usable type). A
data: URL specifies the type itself.
Still incurs base64 overhead (encoding/decoding time, plus extra
bandwidth), though.
> - callbacks are specified in one of several formats, similar to
> image data.
Good point. I'm really not sure how to handle callbacks.
In the case of TCP, a two-way connection that lasts until the
notification fades out or is clicked could work. (On the other hand,
if we use a single connection for *everything*, from registration
onward, click feedback could become more problematic. Am I wrong in
that?)
I'm not sure what to do about UDP. Maintaining a queue of
notifications waiting for click feedback is just asking for SYN-flood-
type problems.
> you said this should be TCP based, but are you really just trying to
> figure out the data format?
At this time, yes.
> i mean, could the same data format also be sent over UDP if desired
> (and the Growl app supported it)?
Sure. (See below.)
> i think that a standard data format is the important piece here.
Agreed.
> also, when you say you want a 'secure protocol', do you actually
> mean you want the notifications encrypted, or do you simply want to
> verify that the same password was used on both ends (similar to how
> it works now).
I don't think we've decided that yet.
> in a thread in the Growl [discuss] group, i think someone said that
> they envisioned this protocol eventually replacing both the
> exisiting TCP and UDP protocols - is that still the goal?
That was me, and yes, that's still what I want.
> if so, does that mean this protocol will be used for network
> notifications,
Yup.
> or for local (desktop) type notifications as well? if the goal is to
> eventually be cross platform, i would suggest that even local
> desktop apps use the protocol to communicate - then there is only
> one type of message to send whether your app is running locally or
> on the network or on the internet. (in my Growl for Windows
> implementation, i used the UDP protocol for the desktop
> notifications as well and it worked good, minus the support for
> images and callback notifications that are not part of that protocol).
Good idea. Eating our own dog food, as the saying goes.
I'm not sure that we'll do that, as I think it's more work for the
framework—NSConnection is just *so easy*, and we already have it
working. Nonetheless, using the new network protocol for both is a
sound idea, and you may be able to convince me that we should do it.
briandunnington
On Aug 4, 8:16 pm, Peter Hosey <p...@growl.info> wrote:
> On Aug 04, 2008, at 10:56:43, briandunnington wrote:
>
> I agree that they should be, but are they? :)
>
> - Is there a JSON parser for Windows?
> - Is there a JSON parser for Linux?
>
> (Browsers and WebKit don't count.)
in. i am not a linux guy either, but i would bet money that there is a
JSON parser available.
>
> Thanks; it looks like a good example. I'd use MIME types instead of
> having these “IconFormat.Jpg” constants, but as I said, we can debate
> details later.
this point.
>
> Good point. I'm really not sure how to handle callbacks.
app (local or remote) should be a key goal, so i definitely think this
area should get some serious thought.
>
> In the case of TCP, a two-way connection that lasts until the
> notification fades out or is clicked could work. (On the other hand,
> if we use a single connection for *everything*, from registration
> onward, click feedback could become more problematic. Am I wrong in
> that?)
>
> I'm not sure what to do about UDP. Maintaining a queue of
> notifications waiting for click feedback is just asking for SYN-flood-
> type problems.
notifcations are received and that is it (growl can forget about the
sending application at that point). if a user clicks on a
notification, the callback is fired. if it is a url callback, it is
easy enough to handle, and if it is a desktop/local app callback, that
app is then passed back the appropriate message (this implementation
could be platform-specific, as long as the callback indicator was
generic). something in the callback indicator could identify which
window/app to send the notification to, so there is no need to keep
any queue of information. that also means that if the user doesnt
click on the notification, no more memory/processing is held/required.
>
> Good idea. Eating our own dog food, as the saying goes.
>
> I'm not sure that we'll do that, as I think it's more work for the
> framework—NSConnection is just *so easy*, and we already have it
> working. Nonetheless, using the new network protocol for both is a
> sound idea, and you may be able to convince me that we should do it.
available to us poor windows (or linux) folks. in order for an app to
have a common way to fire notifications across platforms would be
nice, and i definitely like the idea of 'eating our own dogfood'.
plus, it puts networked/internet apps on the same 'level' as desktop
apps (in other words, they arent limited by a different or more
restrictive protocol - they can do everything that a desktop app can
do).
anyway, that proposal was just a first draft and obvisouly needs
everyone elses input, but i really think that having a single data
format for all types of apps (local/network/internet) and not relying
on platform specific functions (NSConnection) will truly make this a
powerful cross-platform protocol.
Chris Forsythe
On Aug 4, 2008, at 10:16 PM, Peter Hosey wrote:
>
> On Aug 04, 2008, at 10:56:43, briandunnington wrote:
>> i would second the JSON approach - it is nice and easy, less verbose
>> than XML, and parsers should be available for every platform.
>
> I agree that they should be, but are they? :)
>
> - Is there a JSON parser for Windows?
> - Is there a JSON parser for Linux?
Is there JSON for AIR?
Chris
briandunnington
support. i also found this snippet on their site:
"A very limited form of eval() is also available after load, but
restricted to evaluating object literals and constants, which
basically means you can use it for strict JSON notation but not
generation of arbitrary code. "
Evan Schoenberg
corelib [1] provides JSON serialization as well as MD5/SHA1 hashing. :)
-Evan
Evan Schoenberg
On Aug 4, 8:16 pm, Peter Hosey <p...@growl.info> wrote:- Is there a JSON parser for Windows?- Is there a JSON parser for Linux?
.NET has JSON support built-in.
Good point. I'm really not sure how to handle callbacks.
i think having a robust callback mechanism that works for any type of
app (local or remote) should be a key goal, so i definitely think this
area should get some serious thought.
In the case of TCP, a two-way connection that lasts until thenotification fades out or is clicked could work. (On the other hand,if we use a single connection for *everything*, from registrationonward, click feedback could become more problematic. Am I wrong inthat?)I'm not sure what to do about UDP. Maintaining a queue ofnotifications waiting for click feedback is just asking for SYN-flood-type problems.
i was thinking it wouldnt be a queue-based system. instead, the UDP
notifcations are received and that is it (growl can forget about the
sending application at that point). if a user clicks on a
notification, the callback is fired. if it is a url callback, it is
easy enough to handle, and if it is a desktop/local app callback, that
app is then passed back the appropriate message (this implementation
could be platform-specific, as long as the callback indicator was
generic). something in the callback indicator could identify which
window/app to send the notification to, so there is no need to keep
any queue of information. that also means that if the user doesnt
click on the notification, no more memory/processing is held/required.
I'm not sure that we'll do that, as I think it's more work for theframework—NSConnection is just *so easy*, and we already have itworking. Nonetheless, using the new network protocol for both is asound idea, and you may be able to convince me that we should do it.
the idea here is that to be cross-platform, NSConnection might not be
available to us poor windows (or linux) folks. in order for an app to
have a common way to fire notifications across platforms would be
nice
Peter Hosey
> The more I think about it, the more I like JSON for this; it strikes
> a balance between writing our own markup language and the verbosity
> of XML.
The problem I have with JSON (and, to an extent, XML) is having to
receive and parse the entire notification before you can process it.
Consider the case of a notification server that can't handle images.
With the MIME-like protocol, it could receive a notification like this:
NOTIFY Download succeeded
Growl-Protocol-Version: 1.0
Application: GrowlSafari
Title: Download succeeded
Payloads: Description Image
Description-Length: 41
Image-Length: 1048576
File: Growl-1.2.dmg
Location: ~/Downloads
And it would stop—that is, close the connection and post the
notification—as soon as it's finished reading the description. After
all, it doesn't need the image if it's not going to display it. This
would save bandwidth on the receiving computer (think of places like
the UK, where internet service is often charged by the minute) as well
as the sending computer (may be a shared host with a bandwidth cap).
(For that matter, it could stop reading after “Application:” if it
finds that that notification for that application is turned off. Even
greater savings!)
JSON doesn't allow this. You have to either receive the image whether
you can use it or not, or have something like a URL that allows you to
request it separately. XML allows it, but you have to undergo the pain
of a stream-based parser, so it's an unpleasant choice.
And I'm not a fan of putting the image behind a URL. It adds
complexity to both sides: One side, the receiver, must either parse
the URL or have something that can handle the URL for it; the other
side, the notifier, must also run a server to serve the resource
(e.g., image) specified by the URL.
The only reason I can think of for using a URL is that it allows you
to not get the image if you don't need it—and the MIME-like protocol
allows that as well, while not requiring anything more than Growl
already does (no server on the notifying end and no URL-retrieval
machinery on the receiving end).
Evan Schoenberg
On Aug 5, 2008, at 6:09 PM, Peter Hosey wrote:
> The problem I have with JSON (and, to an extent, XML) is having to
> receive and parse the entire notification before you can process it.
> Consider the case of a notification server that can't handle images.
I don't know of any such notification server, nor do I envision
developing one. That doesn't at all invalidate the value of your
proposal, though - see below.
> With the MIME-like protocol, it could receive a notification like
> this:
>
> NOTIFY Download succeeded
> Growl-Protocol-Version: 1.0
> Application: GrowlSafari
> Title: Download succeeded
> Payloads: Description Image
> Description-Length: 41
> Image-Length: 1048576
>
> File: Growl-1.2.dmg
> Location: ~/Downloads
>
>
> And it would stop—that is, close the connection and post the
> notification—as soon as it's finished reading the description.
This would also be quite nice if the notification could include a
unique name or perhaps a hash for the image... and the receiving
server could choose to cache images and not re-download them.
> (For that matter, it could stop reading after “Application:” if it
> finds that that notification for that application is turned off. Even
> greater savings!)
>
> JSON doesn't allow this. You have to either receive the image whether
> you can use it or not, or have something like a URL that allows you to
> request it separately. XML allows it, but you have to undergo the pain
> of a stream-based parser, so it's an unpleasant choice.
A stream-based parser really isn't that big a deal.... and if you
don't want to take advantage of this possibility, you can always use a
more straightforward all-at-once parser.
Of course, for the stream-based parser for XML to be any savings in
terms of bandwidth, you'd need to specify that the image must be the
last thing to be sent, noting that anything sent after the image may,
at the server's option, be ignored. That's a pretty nasty violation
of how XML should work.
> And I'm not a fan of putting the image behind a URL. It adds
> complexity to both sides: One side, the receiver, must either parse
> the URL or have something that can handle the URL for it; the other
> side, the notifier, must also run a server to serve the resource
> (e.g., image) specified by the URL.
> The only reason I can think of for using a URL is that it allows you
> to not get the image if you don't need it—and the MIME-like protocol
> allows that as well, while not requiring anything more than Growl
> already does (no server on the notifying end and no URL-retrieval
> machinery on the receiving end).
I envisioned this more likely being used in a setting like the Gears
Notification API which is intended to notify from web to desktop... I
don't imagine it would be used preferentially over sending the image
directly by a desktop app, typically.
Cheers,
Evan
Peter Hosey
> I don't know of any such notification server, nor do I envision
> developing one.
Growl isn't currently one, but could be. It's possible that we could
make it ask displays whether they support images—SMS, Speech, and
current MailMe wouldn't.
> This would also be quite nice if the notification could include a
> unique name or perhaps a hash for the image... and the receiving
> server could choose to cache images and not re-download them.
Indeed.
> A stream-based parser really isn't that big a deal....
I have both used them and written one (LMX). In my experience, it was
pain. ☺
More importantly, while writing the code isn't *too* hard, I dread
ever having to read parser-delegate code again. It's spaghetti code,
necessarily.
> Of course, for the stream-based parser for XML to be any savings in
> terms of bandwidth, you'd need to specify that the image must be the
> last thing to be sent, noting that anything sent after the image
> may, at the server's option, be ignored. That's a pretty nasty
> violation of how XML should work.
Agreed.
>> And I'm not a fan of putting the image behind a URL. …
>
> I envisioned this more likely being used in a setting like the Gears
> Notification API which is intended to notify from web to desktop...
Hm, yeah. That's a hard problem.
What about AIR?
briandunnington
etc) was to allow the optimal transport type for each use case. a web
site sending notifications is already acting as a server and hosting/
serving images, so it is a no brainer to send just the url (instead of
sending the whole image down). displays that dont want the image can
ignore it with minimal overhead. displays that are webkit/browser-
based can use the url as-is and let the rendering engine fetch the
image for them - growl is off the hook completely.
for desktop apps that arent already serving up images or acting as a
server, using http:// probably wouldnt be the best fit. sending the
raw image data might be easier, and in that case, the network hit of
downloading the entire binary blob is negated by the fact that it isnt
going over the wire. or simply pointing to an image on the local file
system (file://) removes the performance hit of sending the binary
data altogether. the end result is flexibility for all different kinds
of notifiying apps.
the idea of some kind of hash to allow for caching is great. even if
it were not explicity passed, the growl receiver could generate the
hash based on the url/data and use that to identify duplicates. one
less thing for the notifying app to generate/track/etc.
i do like the idea of the MIME-type format being able to ignore pieces
it doesnt want to deal with. will it be limited though by future
additions to the format? for instance, what if notifications
eventually can send in a sound to play, or a video to play, or
multiple images - suddenly the source order is very important. what if
one notification type wants to display images but doesnt care about
sounds, but another (Speech?) wants to play the sound but not display
the image? which one do we optimize for? maybe that is over thinking
it and we just optimize for the most common case.
i think it will be important to decide early on if local applications
will or will not be using this format. depending on if they are/are
not, then some of the considerations might be removed or more
important to consider based on that criteria (ie: not acting as a
server, but 'bandwidth' is much cheaper; local image support vs. url
based, etc).
Evan Schoenberg
i think it will be important to decide early on if local applications
will or will not be using this format. depending on if they are/are
not, then some of the considerations might be removed or more
important to consider based on that criteria (ie: not acting as a
server, but 'bandwidth' is much cheaper; local image support vs. url
based, etc).
Ofri
header holds the offsets to the data section. Images, movies, whatever
will be located in the data section and organized by some logical
order allowing them to be skipped easily. I think the best way to do
this in practice will be to include hashes and if non of the hashes
are cached just download the entire data section.
Honestly, all of this discussion sounds like premature optimization.
Can anyone proof sending images with every message is too slow?
Ofri
Evan Schoenberg
On Aug 5, 2008, at 7:29 PM, Ofri wrote:
> Honestly, all of this discussion sounds like premature optimization.
> Can anyone proof sending images with every message is too slow?
Full-quality images can be pretty sizeable, especially relative to the
otherwise minimal size of the notification. It'd be nice to consider
in this protocol the possibility of notifications not being passed
solely around a local network. For example, once we're up and
running, I intend to set my home computer to forward notifications to
my laptop, which will have a dyndns-style address; I will then receive
notifications wherever I may be so long as I'm not stuck behind a NAT.
-Evan
briandunnington
>
> One important target platform on
> Mac OS X which will use this, for example, is Adobe AIR, which can't
> access Growl directly but can easily send 'network' notifications to
> the local instance.
>
>
> 1KDownload
understood. i think i was hoping that the concept of 'network'
notification vs. 'local' notification would go away (at least be very
blurred). in essence, there would just be a notification format, and
any app that wanted to send notifications would send that format,
regarless of where it lived. that forces the 'network' format to
always be as up-to-date and feature-rich as the 'local' format. (it
might be too easy to just pass a little extra information in the
NSConnection object that isnt part of the network protocol, etc -
similar to how the current network protocol doesnt support images or
callbacks). if there was only one format, it would always be in sync
for all types of apps.
i understand that local apps and network/air/internet apps are
different things, but the protocol could handle that. for instance,
when i was adding support for web-originated notifications in the
windows port of Growl, i made it so that web notifications were
disabled by default (unless overridden by the user) - that meant that
a web app couldnt just start spamming you with notifications until you
went in and turned it off - you had to explicitly take action first,
just like you would by installing an application on your machine.
same goes for security - requests that originate on the same machine
may have more relaxed requirements, but the protocol/format could be
the same.
maybe i am barking up the wrong tree with this idea though. i
definitely am on board for getting a unified format in place, even if
locally installed apps can use a different (perhaps better?) way of
communicating.
Peter Hosey
> sending the raw image data might be easier, and in that case, the
> network hit of downloading the entire binary blob is negated by the
> fact that it isnt going over the wire.
How isn't it?
> or simply pointing to an image on the local file system (file://)
> removes the performance hit of sending the binary data altogether.
But only works if the image already exists on the server—something I
doubt most people can rely on.
> will it be limited though by future additions to the format? for
> instance, what if notifications
> eventually can send in a sound to play, or a video to play, or
> multiple images - suddenly the source order is very important. what
> if one notification type wants to display images but doesnt care
> about sounds, but another (Speech?) wants to play the sound but not
> display the image? which one do we optimize for? maybe that is over
> thinking it and we just optimize for the most common case.
Surely the server can choose to the stop the connection at the correct
point—i.e., after it has everything it will use.
This is, however, an advantage of using URLs.
> i think it will be important to decide early on if local
> applications will or will not be using this format.
Not really. Even a local notification can still become a network
notification—that's forwarding.
Ofri
larger than the max notification size?
The only place i can think of when sending images will be a problem is
when sending many messages in a short period, in which case a hash +
internal cache will work, or the sender could somehow merge
notifications to a single POST command with only few image included.
Peter Hosey
> Can anyone proof sending images with every message is too slow?
Depends on the image, and on the connection. Satellite internet has
long latency, and dial-up (which some people still rely on, because
broadband still isn't available in some areas) has low bandwidth.
On a 53K dial-up connection (5 K/sec), a 100 K image (common in iTunes
libraries) transfers in at least 20 seconds. Per notification. Imagine
your GrowlTunes notification showing up 20 seconds after the track
starts.
When we implement the protocol, whenever the protocol is finished, we
should add a setting to the Growl prefpane to not forward images.
briandunnington
> On Aug 05, 2008, at 16:14:57, briandunnington wrote:
>
> > sending the raw image data might be easier, and in that case, the
> > network hit of downloading the entire binary blob is negated by the
> > fact that it isnt going over the wire.
>
> How isn't it?
>
> > or simply pointing to an image on the local file system (file://)
> > removes the performance hit of sending the binary data altogether.
>
> But only works if the image already exists on the server—something I
> doubt most people can rely on.
>
using http:// type urls made sense for web apps sending notifications
(since they already were hosting the images and acting as servers),
but for locally-installed applications, sending raw data or file://-type
urls might be better. the local sending application should know if its
image/icon is installed on the system and where - in which case it
doesnt need to 'send' anything but the filename. (and if it send the
entire binary data of the image, it isnt going 'over the wire' in the
sense that it doesnt go over any network - just passed between to
local applications on the same machine).
as for forwarding, again i was hoping that there would be no real
difference between normal notifications and forwarded notifications.
if a locally installed app sends a notification to growl using some
kind of platform-specific function (NSConnection or whatever), and the
user has set up growl to forward notifications, then Growl must
convert the notification to the new format and then send it on. if the
notification came from the original app already in the new format,
growl could simply forward the raw data as-is (and possibly not have
to download any external resources if it is not also displaying the
notifications locally). the growl instance at the end of the line
receives exactly what the original application sent, not a network-
ized version of it.
Peter Hosey
> if there was only one format, it would always be in sync for all
> types of apps.
Yup. I think that after we add support for this protocol in Growl,
future versions of the Growl framework should use it instead of
NSConnection.
Furthermore, once we release such a framework, we should deprecate the
NSConnection pathway, and all frameworks that use it.
Peter Hosey
> as for forwarding, again i was hoping that there would be no real
> difference between normal notifications and forwarded notifications.
That was my point. You can't really optimize for local notification
(e.g., with file://) and keep the ability of the server to just
forward the notification on to another server unmodified.
> if the notification came from the original app already in the new
> format, growl could simply forward the raw data as-is …. the growl
> instance at the end of the line receives exactly what the original
> application sent, not a network-ized version of it.
It couldn't, because the notification may contain references (e.g.,
file:// URLs) that any forwarded-to servers wouldn't be able to
dereference. The forwarding server needs to dereference them on the
recipients' behalf in such cases.
Christopher Forsythe
Chris
briandunnington
On Aug 5, 5:04 pm, Peter Hosey <p...@growl.info> wrote:
> On Aug 05, 2008, at 16:50:53, briandunnington wrote:
>
> > as for forwarding, again i was hoping that there would be no real
> > difference between normal notifications and forwarded notifications.
>
> That was my point. You can't really optimize for local notification
> (e.g., with file://) and keep the ability of the server to just
> forward the notification on to another server unmodified.
>
> file:// URLs) that any forwarded-to servers wouldn't be able to
> dereference. The forwarding server needs to dereference them on the
> recipients' behalf in such cases.
the window.
briandunnington
idea of being able to partially receive the notification if you are
not interested in some of the binary data, etc. i have a couple of
questions about how the MIME-style format handles things though. how
would the registration request deal with the variable length list of
notification types (send the same headers multiple times, assuming
some kind of order? that sounds error-prone). also, what if each
notification type also (optionally) specifies some image data?
Peter Hosey
> i have a couple of questions about how the MIME-style format handles
> things though. how would the registration request deal with the
I hadn't thought that far ahead yet. :)
Probably something like this:
REGISTER SurfWriter[1]
Growl-Protocol-Version: 1.0
Application-Bundle-Identifier: com.apple.surfwriter
Payloads: Notifications Application-Icon
Notifications-Count: 2
Application-Icon-Length: 1048576
Notification-Name: Document published
Notification-Enabled-By-Default: No
Notification-Description-Length: 64
You successfully published your document to another application.
Notification-Name: Document updated
Notification-Enabled-By-Default: No
Notification-Description-Length: 74
A document you've subscribed to has been updated from another
application.
<application icon bytes>
> also, what if each notification type also (optionally) specifies
> some image data?
I don't know what you mean. Image data is included at notification
time, not registration time.
The only image in a registration is the app icon.
[1]: SurfWriter was the name of the example app in the Inside
Macintosh books.
briandunnington
>
> I don't know what you mean. Image data is included at notification
> time, not registration time.
>
> The only image in a registration is the app icon.
>
> [1]: SurfWriter was the name of the example app in the Inside
> Macintosh books.
to specify the icon at registration time as well (it is optional). if
an app wants to display a different image for each notification type,
they could register it once and then not have to pass it with every
single notification (saving on bandwidth). sticking with the
SurfWriter example, perhaps the icon for publishing is some kind of
arrow or something, but the icon for subscriptions is more like the
RSS logo - they could register once and then every single notification
could re-use them.
for notifications that use a different image for every single
notification (itunes, etc), they dont have to register an image per
type if they dont want, but it is sort of like a default (with the
application icon being the absolute default).
just an idea since i envision a lot of applications *not* needing a
unique image for every single instance of a notification.
Peter Hosey
> sticking with the SurfWriter example, perhaps the icon for
> publishing is some kind of arrow or something, but the icon for
> subscriptions is more like the RSS logo - they could register once
> and then every single notification could re-use them.
Publish and Subscribe is actually a separate technology, described
best in one of the Inside Macintosh books:
http://developer.apple.com/documentation/mac/IAC/IAC-14.html
The modern equivalent (also on the Mac) is LinkBack:
http://linkback.nisus.com/about/
> for notifications that use a different image for every single
> notification (itunes, etc), they dont have to register an image per
> type if they dont want, but it is sort of like a default (with the
> application icon being the absolute default).
Makes sense.
Evan Schoenberg
On Aug 6, 2008, at 2:48 PM, Peter Hosey wrote:
> REGISTER SurfWriter[1]
> Growl-Protocol-Version: 1.0
> Application-Bundle-Identifier: com.apple.surfwriter
> Payloads: Notifications Application-Icon
> Notifications-Count: 2
> Application-Icon-Length: 1048576
>
> Notification-Name: Document published
> Notification-Enabled-By-Default: No
> Notification-Description-Length: 64
>
> You successfully published your document to another application.
>
> Notification-Name: Document updated
> Notification-Enabled-By-Default: No
> Notification-Description-Length: 74
>
> A document you've subscribed to has been updated from another
> application.
>
> <application icon bytes>
Is it a newline that separates 'application.' from the beginning of
the application's icon bytes? Or nothing at all, since we know the
expected length of the last description and of the icon?
To support per-notification icons, a Notification-Payload field could
be used:
----
Notification-Payload: Name Enabled-By-Default Description Icon
Notification-Name: Document published
Notification-Enabled-By-Default: No
Notification-Description-Length: 64
You successfully published your document to another application.
<notification icon bytes>
Notification-Payload: Name Enabled-By-Default Description Icon
Notification-Name: Document updated
Notification-Enabled-By-Default: No
Notification-Description-Length: 74
A document you've subscribed to has been updated from another
application.
<notification icon bytes>
<application icon bytes>
----
-Evan
Peter Hosey
> Is it a newline that separates 'application.' from the beginning of
> the application's icon bytes? Or nothing at all, since we know the
> expected length of the last description and of the icon?
There's a blank line between every two payloads for readability (not
just here, but also in the protocol), but the length determines where
the payload ends.
See also HTTP and MIME, which put a blank line between the headers and
the body.
> To support per-notification icons, a Notification-Payload field
> could be used:
No need for a separate header—any payload consisting of headers/
payloads (such as a notification description) could have its own
Payloads: header.
Also, if something is specified entirely on one line as the value of a
header, it's not a payload. The Payloads header means “look for an X-
Length header for each name X in the list of names, and read that many
bytes for each payload's value”.
Corrected example:
---
Payloads: Description Icon
Notification-Name: Document published
Notification-Enabled-By-Default: No
Notification-Description-Length: 64
You successfully published your document to another application.
<notification icon bytes>
Notification-Payload: Description Icon
Notification-Name: Document updated
Notification-Enabled-By-Default: No
Notification-Description-Length: 74
A document you've subscribed to has been updated from another
application.
<notification icon bytes>
---
briandunnington
MIME implementation, here is a new proposal (also uploaded to the
Files section). instead of defining payloads explicitly, any binary
data is specified the the *-Identifer: and Identifier: headers. the
format is Length;ID, where the ID corresponds to another section in
the data.
all non-binary data should be included first (parsers should assume
that the first instance of the Identifier: header is the end of the
headers/start of binary data). that way, if an app wants to ignore
binary data, it can stop reading at that point. this also allows
multiple binary chunks to be sent in the payload easily. the ID part
of the identifier could also be used as the key for caching data
(perhaps including the length as well just in case). this allows the
protocol to handle any future binary data that might be added in
addition (i added the example of a default sound for a notification as
a hypothetical case).
lastly, i move the application name/notification text out of the first
line of the protocol. that was simply to allow for unforseen future
changes where maybe the application name or notification name headers
needed to be specified differently.
REGISTER
Growl-Protocol-Version: 1.0
Application-Name: SurfWriter
Application-Bundle-Identifier: com.apple.surfwriter
Application-Icon-Identifier: 1028;6027F6C0-64AF-11DD-9779-EEDA55D89593
Notifications-Count: 2
Notification-Type: Document Published
Notification-Enabled-By-Default: No
C0DD55D89593
Notification-Sound-Identifier: 1048576;1201D5F0-64B0-11DD-8047-
A7E255D89593
Notification-Type: Download Complete
Notification-Name: Finished Downloading
Notification-Enabled-By-Default: Yes
Notification-Icon-Identifier: 1152;DCF70ABA-64AF-11DD-9CA9-
F1E055D89593
Identifier: 6027F6C0-64AF-11DD-9779-EEDA55D89593
<icon bytes>
Identifier: A01DA766-64AF-11DD-B5D5-C0DD55D89593
<icon bytes>
Identifier: DCF70ABA-64AF-11DD-9CA9-F1E055D89593
<icon bytes>
Identifier: 1201D5F0-64B0-11DD-8047-A7E255D89593
<sound bytes>
--------------------------------------------------------------
NOTIFY
Growl-Protocol-Version: 1.0
Notification-Type: Download Complete
Notification-Title: Your file has finished downloading.
Notification-Text: The file 'file.txt' was downloaded to 'c:\your files
\'.
Notification-Sticky: No
Notification-Icon-Identifier: 2689;7FCBD086-64B0-11DD-
B77C-3BE955D89593
X-Filename: file.txt
X-Timestamp: 8:57PM
Identifier: 7FCBD086-64B0-11DD-B77C-3BE955D89593
<icon bytes>
Peter Hosey
> instead of defining payloads explicitly, any binary data is
> specified the the *-Identifer: and Identifier: headers. the format
> is Length;ID, where the ID corresponds to another section in the data.
It seems like you should have the length after the Identifier: header
for the payload, not specified with every reference to the payload.
Partial payloads may make some (very little) sense for .icns files,
but for practically everything else, neither side should use anything
but the whole payload. We should just require that.
> all non-binary data should be included first (parsers should assume
> that the first instance of the Identifier: header is the end of the
> headers/start of binary data). that way, if an app wants to ignore
> binary data, it can stop reading at that point. this also allows
> multiple binary chunks to be sent in the payload easily. the ID part
> of the identifier could also be used as the key for caching data
> (perhaps including the length as well just in case). this allows the
> protocol to handle any future binary data that might be added in
> addition (i added the example of a default sound for a notification as
> a hypothetical case).
This is sounding more and more like actual MIME multi-part encoding.
Maybe we should just use that.
Downside: Harder to make a parser. (More details welcome. My knowledge
of RFC 2822 is vague.)
Upside: Could probably use an existing parser, like Python's rfc822
module.
Upside: Doesn't reinvent the wheel.
> lastly, i move the application name/notification text out of the
> first line of the protocol. that was simply to allow for unforseen
> future changes where maybe the application name or notification name
> headers needed to be specified differently.
Yeah, maybe we should.
> REGISTER
> ⋮
> Notification-Type: Document Published
> Notification-Name: Document published
?
> NOTIFY
> Notification-Type: Download Complete
The name Growl uses is “notification name”. “Type”, to me,
implies something more like subclasses of notification.
briandunnington
> On Aug 07, 2008, at 11:51:45, briandunnington wrote:
>
>
> It seems like you should have the length after the Identifier: header
> for the payload, not specified with every reference to the payload.
>
> This is sounding more and more like actual MIME multi-part encoding.
> Maybe we should just use that.
>
than we needed (since we will only be using a subset of it). however,
not re-inventing the wheel does have its appeal. we also need to make
sure the format is easy for clients to generate from a wide range of
programming languages (which it should be either way, since it is just
text/byte arrangement).
>
> Downside: Harder to make a parser. (More details welcome. My knowledge
> of RFC 2822 is vague.)
> Upside: Could probably use an existing parser, like Python's rfc822
> module.
> Upside: Doesn't reinvent the wheel.
>
> > ⋮
> > Notification-Type: Document Published
> > Notification-Name: Document published
>
> ?
>
> > NOTIFY
> > Notification-Type: Download Complete
>
> The name Growl uses is “notification name”. “Type”, to me,
> implies something more like subclasses of notification.
could use to define notifications, seperate from the name. for
instance, the Type might be 'Download Completed', but the name might
be 'When downloads complete' or 'transferencia directa completa' or
who knows what else. that way if an app wanted to update its
notification type icon or display name or whatever, there would be an
ID that doesnt change to link it to. (so instead of removing the
'Download Completed' notification and replacing it with 'Download
Finished' and thus losing the end users settings for Display, Sticky,
etc, they could update the Name of the 'Download Complete'
notification type. does that make any sense at all?
Peter Hosey
> yeah, i looked at [just using MIME] as well, but it seems more
> it).
Yeah, that's another downside.
>> The name Growl uses is “notification name”. “Type”, to me,
>> implies something more like subclasses of notification.
>
> the 'Type' was supposed to be like an identifier that the client app
> could use to define notifications, seperate from the name. for
> instance, the Type might be 'Download Completed', but the name might
> be 'When downloads complete' or 'transferencia directa completa' or
> who knows what else.
Growl already has something like this, and calls it the notification
display name. The idea, as I understand it (I didn't come up with it,
although I support it), is that an application could define the
notification name (non-display) in its .growlRegDict file, and then
store display names somewhere in each of its <language>.lproj folders:
en.lproj/GrowlRegDict.strings: "Download complete" = "Download
complete";
de.lproj/GrowlRegDict.strings: "Download complete" = "Download beendet";
es.lproj/GrowlRegDict.strings: "Download complete" = "Descarga
completa";
fr.lproj/GrowlRegDict.strings: "Download complete" = "Téléchargement
terminé";
(Spanish and French versions from Google Language Tools.)
I thought of including this in the protocol, but it doesn't really
make sense there, at least not in a single Notification-Display-Name
(or Notification-Name) header.
What we could do is have the sender send one header for every language:
Growl-Notification-Name: Download complete
Growl-Notification-Display-Name-en: Download complete
Growl-Notification-Display-Name-de: Download beendet
Growl-Notification-Display-Name-es: Descarga completa
Growl-Notification-Display-Name-fr: Téléchargement terminé
I have no idea where we would store this information upon receiving
it, though.
briandunnington
>
> Growl already has something like this, and calls it the notification
> display name. The idea, as I understand it (I didn't come up with it,
> although I support it), is that an application could define the
> notification name (non-display) in its .growlRegDict file, and then
> store display names somewhere in each of its <language>.lproj folders:
>
so i am glad it is already taken into account.
as for localizations (i know this is getting off topic a bit), perhaps
it is just up to the notifying app to use the correct localized
strings. so if the app registers (and sends) its notifications in
spanish, that is what you get. if you prefer french, you need to tell
the notifying app and hope that it has support for your request. that
way, Growl (and the protocol we are working on) only cares about the
Name and DisplayName, regardless of language.
in other words, leave localization of notifications up to the
notifying app and just let Growl display what it receives. (in theory,
you could have notifications from different apps in different
languages, depending on how you configure those apps).
but back to the protocol discussion - how do we get concensus on
something? i know there are still details to work out and more
discussion to be had, but how do we decide on (or rule out)
possibilities. you guys are the developers of growl and i am just some
guy with in interest in a cross-platform protocol, so it is obviously
your call. is JSON still a possibility? is the MIME-like scenario the
way to go? if we make some over-arching decisions, then we can all put
our efforts into working out the details.
also, i know this is premature, but what is the timeline for 1)
deciding on this new protocol, and 2) releasing a version of Growl
that supports it? i mean, is this something you want to get in place
for an upcoming release, or just something to eventually support?
Evan Schoenberg
On Aug 7, 2008, at 4:45 PM, briandunnington wrote:
> as for localizations (i know this is getting off topic a bit), perhaps
> it is just up to the notifying app to use the correct localized
> strings. so if the app registers (and sends) its notifications in
> spanish, that is what you get. if you prefer french, you need to tell
> the notifying app and hope that it has support for your request. that
> way, Growl (and the protocol we are working on) only cares about the
> Name and DisplayName, regardless of language.
>
> in other words, leave localization of notifications up to the
> notifying app and just let Growl display what it receives. (in theory,
> you could have notifications from different apps in different
> languages, depending on how you configure those apps).
This is precisely how Growl expects apps to handle it at present. The
notification name should be computer-readable, while the display name
should be localized to the user's locale.
> but back to the protocol discussion - how do we get concensus on
> something?
We figure out a protocol that everyone agrees rocks :)
> i know there are still details to work out and more
> discussion to be had, but how do we decide on (or rule out)
> possibilities. you guys are the developers of growl and i am just some
> guy with in interest in a cross-platform protocol, so it is obviously
> your call. is JSON still a possibility? is the MIME-like scenario the
> way to go? if we make some over-arching decisions, then we can all put
> our efforts into working out the details.
I think the MIME-like scenario is looking increasingly elegant.
> also, i know this is premature, but what is the timeline for 1)
> deciding on this new protocol, and 2) releasing a version of Growl
> that supports it? i mean, is this something you want to get in place
> for an upcoming release, or just something to eventually support?
I would like to have Growl at least receiving with the protocol we
work out together by the end of the month. A release would follow
after an appropriate period of testing.
-Evan
briandunnington
> On Aug 7, 2008, at 4:45 PM, briandunnington wrote:
>
>
> We figure out a protocol that everyone agrees rocks :)
>
comments? we started off with a few other ideas (xml, json, etc) and
just wanted to see what everyone else was thinking (and i apologize if
i am posting too much - i just have wanted to see this come to
fruition for a long time)
>
> I think the MIME-like scenario is looking increasingly elegant.
>
style format so far. you can go to the Files section to take a look
(it is called growl_v1.txt) - hopefully everyone else can use that as
a jumping off point to correct any mistakes, make additions/changes,
clarify the rules, etc.
>
> I would like to have Growl at least receiving with the protocol we
> work out together by the end of the month. A release would follow
> after an appropriate period of testing.
>
vision, so i am glad to hear it is something more immediate.
Peter Hosey
> On Aug 7, 1:02 pm, Peter Hosey <p...@growl.info> wrote:
>>
> as for localizations (i know this is getting off topic a bit),
> perhaps it is just up to the notifying app to use the correct
> localized strings. so if the app registers (and sends) its
> notifications in spanish, that is what you get.
I'm imagining that the notifying app includes all the available
display names in the registration. The listening app uses the correct
one according to whatever locale is selected on its machine.
So, for example, if you have the sending machine set to English and
the receiving machine set to Spanish, then the receiving machine will
list the notifications in Spanish, not English.
Also note that Growl only uses the display names in the list of
notifications in the Preference Pane—the application doesn't ever need
to send a notification. If it does, Growl uses the *title* in
displaying it, not the display name. The name (or display name) is
only for the list.
Another example: A notification may be named “Download succeeded”, and
may have a title such as “Download of "Growl-1.1.5.dmg" succeeded” and
description such as “4.4 MiB\n~/Downloads”.
> how do we get concensus on something? i know there are still details
> to work out and more discussion to be had, but how do we decide on
> (or rule out) possibilities. you guys are the developers of growl
> and i am just some guy with in interest in a cross-platform
> protocol, so it is obviously your call.
I'd rather not just issue an edict from “above”; I'd rather get
agreement from everybody.
> is JSON still a possibility?
Sure, if somebody can come up with a reason why it's better than the
MIME or MIME-like protocol.
So far, the only one I've seen is that there are more ready-made
parsers. I don't think that this outweighs its disadvantages (most of
which it shares with XML).
> is the MIME-like scenario the way to go?
I think so.
Peter Hosey
> I would like to have Growl at least receiving with the protocol we
> work out together by the end of the month. A release would follow
> after an appropriate period of testing.
I do think the earliest release that should have this protocol is 1.2.
I'd like to keep 1.1.5 fairly small—bug fixes only.
Peter Hosey
> i apologize if i am posting too much - i just have wanted to see
> this come to fruition for a long time
Nobody is posting too much. :)
> i took a stab at writing up a very rough draft describing the MIME-
> style format so far.
I suggest using CRLF instead of a single “newline character”. This is
what MIME and HTTP both use, and it's the traditional line-break
sequence in network protocols.
(Historical note: CR means “return the print head or typewriter
carriage to the start of the line”; LF means “advance the paper by one
line”. That's why the traditional down-one-line-then-move-to-line-
start sequence is both CR and LF.)
Regarding URLs for resources: Can all platforms easily grab the
contents of a URL? I know Cocoa can. By “easily”, I mean something
like this:
NSData *data = [NSData dataWithContentsOfURL:URL];
I still think length should be a separate header. Something like:
---
Identifier: 6027F6C0-64AF-11DD-9779-EEDA55D89593
Length: 1028
<icon bytes>
---
Rather than wrapping an identifier reference in {}, which is harder
for the C standard library to parse, I suggest using an URL instead.
Something like:
x-growl-resource://id/6027F6C0-64AF-11DD-9779-EEDA55D89593
briandunnington
> On Aug 07, 2008, at 15:08:03, briandunnington wrote:
>
> I suggest using CRLF instead of a single “newline character”. This is
> what MIME and HTTP both use, and it's the traditional line-break
> sequence in network protocols.
definitely want to keep consistent with other standards.
> Regarding URLs for resources: Can all platforms easily grab the
> contents of a URL? I know Cocoa can. By “easily”, I mean something
> like this:
>
> NSData *data = [NSData dataWithContentsOfURL:URL];
in windows (although, i dont think i can do it in one line as nicely
as your example).
> I still think length should be a separate header. Something like:
>
> ---
> Identifier: 6027F6C0-64AF-11DD-9779-EEDA55D89593
> Length: 1028
>
> <icon bytes>
> ---
> Rather than wrapping an identifier reference in {}, which is harder
> for the C standard library to parse, I suggest using an URL instead.
> Something like:
>
> x-growl-resource://id/6027F6C0-64AF-11DD-9779-EEDA55D89593
was easily distinguishable from a url that was pointing to a remote
resource vs. pointing to binary data included in the payload. since a
'Uniform Resource Locator' is, by definition, used to locate a
resource (regardless of where/how), i guess the proposed usage here is
ok. since it is a url though, someone could get confused and pass it
to something like your NSData method above and it wouldnt be able to
resolve obviously. since the data isnt accessible by other url-capable
means, i was wondering if we should create some other kind of pointer
syntax. but i wont lose any sleep over it either way.
Peter Hosey
> On Aug 7, 7:25 pm, Peter Hosey <p...@growl.info> wrote:
>>
>> x-growl-resource://id/6027F6C0-64AF-11DD-9779-EEDA55D89593
I would actually like to propose an amendment to my own suggestion:
x-growl-resource://Application_Name/Notification_Name/
6027F6C0-64AF-11DD-9779-EEDA55D89593
Notifying Growl-like apps should use the following algorithm on the
application and notification name:
1. Change space to '_'.
2. Delete anything that isn't in the set [-_A-Za-z0-9].
(Amendments welcome.)
Furthermore, we don't really even need to use UUIDs. Within one
connection/datagram, the sender could just use a counter. Could leave
it up to the sender (I would limit the set of acceptable identifier
characters to [-_A-Za-z0-9].[1]
> since it is a url though, someone could get confused and pass it to
> something like your NSData method above and it wouldnt be able to
> resolve obviously.
At least in Cocoa, it's possible to add new URL schemes, so it
wouldn't be impossible for Growl to add such a scheme and then handle
it through that mechanism.
> since the data isnt accessible by other url-capable means, i was
> wondering if we should create some other kind of pointer syntax.
I'm open to suggestions, as long as they're easy to parse and not
mistakable for an URL.
[1]: One amusing ramification of this limit/allowance is that a
notifier could use “_”, “__”, “___”, “____”, etc. as identifiers. :)
Evan Schoenberg
Then I think this should be the only new feature of 1.2, along with
general networking usability improvements. I really hate the mess we
get into regularly with Adium in which a double handful of new
features mean we can't release for months.
-Evan
Chris Forsythe
That's the general consensus for Growl, that updates should be small
and light from now on so we get stuff out to users faster. We need to
define a testing cycle, localization times, etc etc, but overall I can
see 1-2 features being enough to do a 1.x.0.
Mind starting a new thread to discuss this sort of thing?
Chris
briandunnington
thinking that it would be nice if websites could send notifications,
we have to consider the limitations they face. javascript cant do
Sockets, only HTTP requests, so javascript wont be able to send a
notification using the protocol we are migrating toward. (of course
there are alternatives, such as using a .swf file to do the
communication or whatever). websites wont be able to send the
notifications from the server-side since they 1) wont know where to
send them, 2) servers can initiate connections with clients, and 3)
even if the server could push notifications, many computers/firewalls
wouldnt allow incoming traffic, especially on non-standard ports.
going even further, browsers using AJAX are also limited by the cross-
domain security policy, so they cant post data to the localhost. if
you use an alternative approach to get around that limitation (hidden
iframe, dynamic script tag, etc), then you can only do standard GET
requests (no POST data, no custom headers, etc), which makes it very
hard to pass much data.
speaking of .swf/flash/air - although they can do Sockets, they are
limited by the cross-domain security policy. it wont help to have an
AIR app be able to send a notification back to its hosting domain -
the notification needs to get to the client's machine (localhost), not
the server. i know you can set up a crossdomain.xml policy file, but
then Growl has to act as a HTTP webserver also to serve up the policy
file. someone with more knowledge should chime in as to what ports
flash/air is allowed to connect to as well (if there are any further
restrictions).
i assume that part of creating this cross-platform format/protocol is
to allow additional apps like these (websites, AIR, etc) to send
notifications. if we get the format nailed down but clients arent
equipped to send the data, i am not sure that all parts of the goal
will have been met.
maybe i am not thinking this through clearly, but when i implemented
the ability for websites to send notifications in the windows version,
i went through all kinds of approaches before settling on one that
worked within the restrictions imposed.
briandunnington
> On Aug 07, 2008, at 23:00:56, briandunnington wrote:
>
> > On Aug 7, 7:25 pm, Peter Hosey <p...@growl.info> wrote:
> >> Something like:
>
> >> x-growl-resource://id/6027F6C0-64AF-11DD-9779-EEDA55D89593
>
> I would actually like to propose an amendment to my own suggestion:
>
> x-growl-resource://Application_Name/Notification_Name/
> 6027F6C0-64AF-11DD-9779-EEDA55D89593
>
> Notifying Growl-like apps should use the following algorithm on the
> application and notification name:
>
> 1. Change space to '_'.
> 2. Delete anything that isn't in the set [-_A-Za-z0-9].
>
> (Amendments welcome.)
>
> Furthermore, we don't really even need to use UUIDs. Within one
> connection/datagram, the sender could just use a counter. Could leave
> it up to the sender (I would limit the set of acceptable identifier
> characters to [-_A-Za-z0-9].[1]
>
> mistakable for an URL.
>
UUID, it was more akin to the MIME 'boundary' identifier. it was a way
to mark, within that message only, the various sections of data. since
the data is transient and only really exists in that form during the
lifetime of the message, it doesnt need to contain any other
identifying information (such as app name, etc). if the receiving
growl server wants to cache data per-application/notification, that
information is already available in the datagram and doesnt need to be
re-specified. along your statement of 'we don't really even need to
use UUIDs' - i agree completely. let the application set an arbitrary
string (with the limitations you proposed) and we just use that
(again, almost exactly like the MIME boundary identifier).
on an unrelated note - i also propose that we change the datagram
header from:
REGISTER
Growl-Protocol-Version: 1.0
to:
GNTP/1.0 REGISTER
(where GNTP (Growl Notificaiton Transport Protocol) can be changed if
you dont like it). this does two things:
1. identifies the datagram as a GNTP-type message (as opposed to HTTP
or FTP or POP or whatever else).
2. it puts the version number earlier in the datagram and in a
consistent place. this helps avoid the cases where we might have
changed from the original header of 'Growl-Protocol-Version' to
'Protocol-Version' or something. also, the version number is in the
same physical place (number of bytes offset from beginning) regardless
of the action type.
Peter Hosey
> … the ability for websites to send notifications in the windows
> version …
How does that work in your implementation?
briandunnington
requests on localhost:9888. i created a JSON string containing the
data (based on what is required/allowed in the current UDP network
protocol) and pass it via the querystring to http://localhost:9888/register?<json>.
to get around the cross-domain restrictions, the data is passed by
updating the url of a hidden iframe.
within Growl, i grab the querystring, parse the JSON and at that point
can treat the data like any other incoming notification. this approach
is not optimal for several reasons:
1) the size of the querystring is limited (for most browsers, it is
<2048 characters). the current UDP protocol doesnt support images or
other binary data, so for the most part, the notifications were small
enough to work.
2) Growl must listen on yet another port for yet another data format
and have yet another parser in place to process the request.
i tried several solutions before settling on this one, including:
- java applet
- flash or silverlight control
- custom protocol handler (growl://)
(if you are so inclined, you can read about the entire thought process
here:
http://ajaxian.com/archives/growls-for-windows-and-a-web-notification-api
)
between most options not supporting UDP and the cross-domain
restrictions i kept hitting, options were very limited. even if we got
a new format figured out, the size of including binary data would
render the querystring approach useless. i still dont have a good
solution for that issue, but that is why i brought it up =)
Evan Schoenberg
On Aug 8, 2008, at 3:26 AM, Peter Hosey wrote:
> Furthermore, we don't really even need to use UUIDs. Within one
> connection/datagram, the sender could just use a counter. Could leave
> it up to the sender (I would limit the set of acceptable identifier
> characters to [-_A-Za-z0-9].[1]
Using an identifier unique to the resource provides for the most
common reason that the server might tell the client not to transmit
any further data - all resources which are being offered in the
notification have been previously transmitted and cached locally.
-Evan
Peter Hosey
> Using an identifier unique to the resource provides for the most
> common reason that the server might tell the client not to transmit
> any further data - all resources which are being offered in the
> notification have been previously transmitted and cached locally.
That implies a hash as identifier. Perhaps MD5.
Evan Schoenberg
MD5 could be recommended, but it doesn't matter so long as it's
unique. A program might prefer to use a locally unique integer, a
representation of the pointer being sent, etc.
On the other hand, requiring MD5 would mean that Growl could cache
notifications coming in from multiple sources as a single file... the
odds of this being used ever, or being used with a frequency to
matter, are pretty slim, though.
-Evan
Peter Hosey
> On Aug 8, 2008, at 4:49 PM, Peter Hosey wrote:
>> On Aug 08, 2008, at 13:47:02, Evan Schoenberg wrote:
>>> Using an identifier unique to the resource provides for the most
>>> common reason that the server might tell the client not to
>>> transmit any further data - all resources which are being offered
>>> in the notification have been previously transmitted and cached
>>> locally.
>>
>> That implies a hash as identifier. Perhaps MD5.
>
> MD5 could be recommended, but it doesn't matter so long as it's
> unique.
Depends on whether the receiver generates its own hash, or just
associates it with whatever identifier the resource came in with.
briandunnington
comparison purposes. whatever the notifying application has used as an
identifier is what Growl can use as a cache key/identifier.
since many apps will probably be connecting to Growl via some kind of
provided framework, we could make md5 the standard, but i agree with
Evan that it really doesnt matter what the value is as long as it is
unique (to that application).
Peter Hosey
> since many apps will probably be connecting to Growl via some kind
Alternatively, we could use whichever hash we use for password hashing
(which would probably be SHA-1).
briandunnington
i think we should), i still think that the value should be enclosed in
some sort of brackets or required to always start with a special
delimiter (it need not be the curly brace, that was just an idea) or
somehow be indicative that it is a pointer to binary data rather than
a literal string.
the reason for this requirement is that if an application wants to
pass in a custom header using the X-* syntax, they should be allowed
to pass either a string value or a binary value just like the defined
headers. (this applies to defined headers as well, but at least with
those we sort of know what type of data we should be expecting). if we
dont enclose/delimit the special binary identifiers in some way, there
will be no way to distinguish between a literal string and a binary
identifier.
if an app passes:
X-Sound: dece8077759a2b36b96fbd1203a832ab
is that a pointer to some binary data later in the datagram, or is it
the literal string "dece8077759a2b36b96fbd1203a832ab"? and what if
they are using some other system of generating their IDs? of course,
even if we enclose/delimit the value, there is always the chance of
this happening, but it is much easier to recognize something if we
standardize on it, and if we say something like "you can pass anything
you want except strings that are enclosed by/start with x", then that
should allow for almost complete flexibility and our code will still
be able to intelligently handle the data by knowing if it should/
should not be expecting to find corresponding binary data.
briandunnington
Application-Name: SurfWriter
Application-Icon: http://www.site.org/image.jpg
Notifications-Count: 2
X-Creator: Apple Software
X-Application-ID: 08d6c05a21512a79a1dfeb9d2a8f262f
Notification-Name: Download Complete
Notification-Display-Name: Download completed
Notification-Enabled-By-Default: Yes
Notification-Icon: [08d6c05a21512a79a1dfeb9d2a8f262f]
Notification-Callback: fake callback
X-Language: English
X-Timezone: PST
Notification-Name: Document Published
Notification-Display-Name: Document successfully published
Notification-Enabled-By-Default: No
Notification-Icon: http://fake.net/image.png
Notification-Callback: fake callback
X-Sound: http://fake.net/sound.wav
X-Sound-Alt: [dece8077759a2b36b96fbd1203a832ab]
Identifier: 08d6c05a21512a79a1dfeb9d2a8f262f
Length: 4
<bytes>
Identifier: dece8077759a2b36b96fbd1203a832ab
Length: 4
<bytes>
Peter Hosey
> if we decide to use a hash/uuid/unique string as the identifier
Why is that? What's the advantage over data being inline?
> … i still think that the value should be enclosed in some sort of
> brackets or required to always start with a special delimiter (it
> need not be the curly brace, that was just an idea) or somehow be
> indicative that it is a pointer to binary data rather than a literal
> string.
The idea of my x-growl-resource: suggestion was that that would be the
special prefix. It has the same effect of starting it with '$$' or
something, while allowing systems that can (such as Growl) to handle
it through the URL system. The terminator would be the end of the line.
However, putting the identifier in-band (i.e., using the same header
either way) means that there's always some value that somebody may
want to use as the description for some reason, which the sender would
have to take care to escape in some way (by using an identifier to
specify it).
That means that we should do one of two things:
- Require Description to *always* be a payload, never a single-line
header. I think that this is overkill.
- Go back to allowing separate headers. In this case, Description is a
single-line header, and Description-Identifier refers to a payload.
(What happens if you provide both? Implementation-defined.)
What's the reason for identifiers in the first place? Why not just
have such payloads inline, referred to by order? (We can implement
Evan's caching idea by hashing it on the server side.)
(If we do go back to payloads by order, then mentally replace
“Description-Identifier” with “Description-Length” in the above.)
> the reason for this requirement is that if an application wants to
> pass in a custom header using the X-* syntax, they should be allowed
> to pass either a string value or a binary value just like the
> defined headers.
Or they could pass X-Foo-Identifier/X-Foo-Length instead of X-Foo. X-
Foo is always a single line, whereas X-Foo-(Identifier|Length) is a
payload.
Peter Hosey
> Notification-Callback: fake callback
We probably should start bandying about suggestions for implementing
click feedback.
For those developers of systems that don't (yet) have click feedback,
here's how it works, currently:
- The application specifies a “click context” object, which (on OS X)
is a property list object: a string, number, date, data object, array,
or dictionary. (You can't pass just arbitrary Cocoa objects here, even
now.)
- Growl keeps this object associated with the notification for its
lifetime.
- If the notification times out, Growl tells the application that the
notification timed out.
- If the user closes the notification window using its close box,
(Actually, Growl doesn't do anything here, yet, but a future version
will tell the application that the user closed the notification window.)
- If the user clicks on the notification window, Growl tells the
application window that the user clicked on the notification.
- No matter which message Growl sent back (timed out/[closed]/
clicked), Growl includes the click context with it.
- The application then reacts to the click context in whatever way it
sees fit. *Usually*, this is by revealing the subject of the
notification (e.g., chat window, Finder window for downloaded file,
Finder window for mounted volume), but we don't require this and
there's no way we could enforce it.
Here's my suggestion, still working from the MIME-like protocol (which
I think we've settled on, at this point):
In the sender's notification message:
Notification-Click-Context: TYPE <representation>
Or, to provide the click context as a payload:
Notification-Click-Context-Type: TYPE
Notification-Click-Context-(Identifier|Length): <identifier|length>
TYPE can be one of:
- STRING
- INTEGER
- REAL
- DATE
- DATA
- ARRAY
- DICTIONARY
Representation is:
- String: The string, with appropriate newline conversion already
performed (e.g., on a UNIX machine, replace \n with \r\n)
- Integer: Representation suitable for passing to strtoul(, , 0)
- Decimal (example: “32”
- Octal, prefixed with “0” (example: 040)
- Hexadecimal, prefixed with “0x” or “0X” (examples: “0x20”, “0X20”)
- Real: Decimal representation, preferably in scientific notification
- Example: 1.0
- Example: 1.000000e+00
- Date: ISO 8601 string
- Example: 2008-08-08T20:08:08-0800
- Data: The data
I'm not sure what we should do about arrays or dictionaries.
Maybe we should just use JSON for the value of the click context.
Unlike the whole protocol, I think it's a perfect fit here.
Another option would be to use the property-list formats, but I doubt
Windows has an easy way to handle property lists. (Maybe a library? I
don't know.)
briandunnington
> What's the reason for identifiers in the first place? Why not just
> have such payloads inline, referred to by order?
datagram so that the receiver could optimize its parsing routine. if
the parser got to the end of the regular (non-binary) headers and
found that any linked binary data was already cached, it could skip
reading the bulkier binary section. one of the first arguments against
an XML or JSON format was that the receiver would have to read the
entire datagram before it was able to parse anything. putting the data
inline removes that advantage. also, we become even more reliant on
the ordering of headers, which sort of goes against the MIME
inspiration (of course, there are still other things reliant on
ordering, but the regular header list should not be one of them in my
opinion).
> The idea of my x-growl-resource: suggestion was that that would be the
> special prefix. It has the same effect of starting it with '$$' or
> something, while allowing systems that can (such as Growl) to handle
> it through the URL system. The terminator would be the end of the line.
signal that it is a special data type. the only reason i was
suggesting brackets/braces/etc was that 1) it was slightly less
verbose, and 2) the difference between an ID and a url was more clear
at a glance (to me). if using the x-growl-resource identifier helps in
other ways (allows Growl to register that as a url handler, etc) then
that is an advantage.
briandunnington
>
> Here's my suggestion, still working from the MIME-like protocol (which
> I think we've settled on, at this point):
>
> Maybe we should just use JSON for the value of the click context.
> Unlike the whole protocol, I think it's a perfect fit here.
>
works - i dont even own a mac, so it was very helpful to understand.
i think your suggestion looks good, although i would go so far as to
say that we should let the app pass in any value they want for the
Type and Representation. an app might want to pass a primitive type,
or JSON (as you suggested), or maybe a small XML snippet, or maybe a
comma-seperated list of values, or maybe some other representation of
their data that only has meaning within their app. if they tell us
what Type they are passing and the value, we simply pass it back to
them and they know how to deal with it. since Growl doesnt need to use
or understand the actual data, i think that would give the app the
most flexibility.
Notification-Click-Context-Type: <string>
Notification-Click-Context: <string>
> Another option would be to use the property-list formats, but I doubt
> Windows has an easy way to handle property lists. (Maybe a library? I
> don't know.)
right? as such, it would be easy enough for Windows (or any other OS i
assume) to deal with the XML. however, i think it is unimportant if we
just let the app pass what it wants and dont try to understand the
meaning - simply pass it back when necessary.
that also brings up the question of 'how' the callbacks will be
returned. if the notifying app is a website, AIR app, or networked app
on another OS, we wont be able to use native methods of sending the
data back. one idea off the top of my head is to simply send the data
back using the same protocol we are defining (CALLBACK action instead
of REGISTER/NOTIFY), sending the same Notification-Click-Context-
(Type) headers that we received (and possibly also returning any
additional custom headers that were received, but that is not
important). it will require that the notifying app be able to act as a
simple server to receive the callback, but for a networked app, i am
not sure of any other way to send data back to them. if the app is
using a framework to send notifications, the ability to receive
callbacks would be baked right in (so the app developer doesnt have to
worry about writing the server logic).
briandunnington
send TCP packets from javascript. by using the Socket functionality
found in Flash, i was able to use a .swf file as a bridge from
javascript to send TCP packets. the .js wrapper exposes a standard
socket interface (connect, send, etc) and simply passes through the
calls into the .swf file, which can do raw TCP. in order for the .swf
file to talk to an app running on the local machine, Flash requires a
security policy file to be served up first. so the only additional
requirement on the Growl side of things to support this architecture
is to be able to respond to one additional type of request (in
addition to the formats we define for the actual notification
protocol). the policy file request is a simple '<policy-file-request/>
\0', and the response is an xml snippet that defines the hosts/ports
that area allowed to connect. i was able to make my simple server
return the contents of a policy file and it satisfied the Flash
security mechanism and i was able to send arbitrary TCP data after
that.
i think that adding the ability to respond to the policy-file-request
is pretty easy (since we are creating a TCP server anyway), so i think
we should try to include that in whatever Growl implementation gets
created. once that is in place, any app that can host a .swf file
(website, AIR, etc) would be able to send TCP notifications.
Peter Hosey
> i think your suggestion looks good, although i would go so far as to
> say that we should let the app pass in any value they want for the
> understand the actual data …
See below.
>> Another option would be to use the property-list formats, …
>
> i dont know much about plists, but i think they are just XML files,
> right?
Notice I said “formatS”. ;)
Here's a property-list representation of an array of two strings and a
number:
<array>
<string>This is a test</string>
<string>Growl</string>
<integer>42</integer>
</array>
Here's a property-list representation of an array of two strings and a
number:
(
"This is a test",
Growl,
42
)
Here's a (base64'd) property-list representation of an array of two
strings and a number:
YnBsaXN0MDCjAQIDXlRoaXMgaXMgYSB0ZXN0VUdyb3dsECoIDBshAAAAAAAAAQEA
AAAAAAAABAAAAAAAAAAAAAAAAAAAACM=
> however, i think it is unimportant if we just let the app pass what
> it wants and dont try to understand the meaning - simply pass it
> back when necessary.
Doesn't work so well on the client side when our Growl framework
passes the property-list object back to the application. ☺
A bit more edification: Each of the property-list types maps to one of
the Cocoa/CF classes:
Plist Cocoa CF
string NSString CFString
integer NSNumber CFNumber
real NSNumber CFNumber
date NSDate CFDate
data NSData CFData
array NSArray CFArray
dict NSDictionary CFDictionary
So, our framework needs to convert the plist data back into an
instance of one of those classes.
> one idea off the top of my head is to simply send the data back
> using the same protocol we are defining (CALLBACK action instead of
> REGISTER/NOTIFY), sending the same Notification-Click-Context-(Type)
> headers that we received (and possibly also returning any additional
> custom headers that were received, but that is not important).
Sounds reasonable.
How should we represent the difference between timed-out, closed, and
clicked? Use a header (User-Action: (Timed out|Closed|Clicked)), or
different actions (TIMEDOUT/CLOSED/CLICKED)?
I'm leaning toward the header, since it's one place to extend all the
different callbacks at once, if necessary.
> it will require that the notifying app be able to act as a simple
> server to receive the callback, but for a networked app, i am not
> sure of any other way to send data back to them. if the app is using
> a framework to send notifications, the ability to receive callbacks
> would be baked right in (so the app developer doesnt have to worry
> about writing the server logic).
An app that uses our framework doesn't receive responses directly from
the remote Growl anyway; the framework communicates with the local
Growl, which communicates with a remote Growl (forwarding). The remote
Growl then communicates back to the local Growl, which communicates
back to the app.
On the Mac, only growlnotify sends original notifications over the
network (upon explicit demand by the user), so only growlnotify would
need to have its own listener. All the other apps go through the local
Growl.
Perhaps we can factor out the emission/reception code so that
growlnotify can use the same code that the Growl pathway does.
Peter Hosey
> i think that adding the ability to respond to the policy-file-
> request is pretty easy (since we are creating a TCP server anyway),
> so i think we should try to include that in whatever Growl
> implementation gets created.
Sounds good. I'm not sure where we should document it, though—it's not
really part of the protocol; it's just something separate that all
Growl servers should implement.
Maybe an appendix.
briandunnington
> On Aug 09, 2008, at 09:23:39, briandunnington wrote:
>
> Notice I said “formatS”. ;)
shedding some light on it.
> Doesn't work so well on the client side when our Growl framework
> passes the property-list object back to the application. ☺
platform, then the data would have to be passed in an OS-agnostic way.
say a PHP script sends a notification over the network (not through a
local instance of Growl, but using the network protocol directly) and
a Windows system (running Growl) receives the notification. the script
could have passed a string, or JSON, or a property list or whatever it
wanted - the receiving Growl instance doesnt need to recognize/
understand the the data (and may not even be able to understand it),
but it can simply pass it back in the callback.
> On the Mac, only growlnotify sends original notifications over the
> network (upon explicit demand by the user), so only growlnotify would
> need to have its own listener. All the other apps go through the local
> Growl.
the difference between growlnotify and growl. in the version i hacked
together, there is just one 'thing' - the growl server (app bridge?).
it receives notifications from both local and remote apps, and also
handles forwarding notifications if so configured. if it supported
callbacks, they wouldnt be marshalled back through growl, but sent
directly to the original notifying application.
> I'm leaning toward the header, since it's one place to extend all the
> different callbacks at once, if necessary.
> An app that uses our framework doesn't receive responses directly from
> the remote Growl anyway; the framework communicates with the local
> Growl, which communicates with a remote Growl (forwarding). The remote
> Growl then communicates back to the local Growl, which communicates
> back to the app.
were forwarded multiple times, the callback would be directed to the
original application, not to the forwarding Growl instance. of course,
that might not be possible if the original app came from a machine
that is firewalled or otherwise not directly accessible. in the
current version on the Mac, can forwarded notifications even generate
a callback? the UDP network protocol doesnt have any provision for
passing callback context, so i assume it cant. perhaps that is how it
should be restricted: only the original receiving instance of Growl
allows for callbacks - if a notification is forwarded by Growl, the
downstream displays cant do callbacks. (maybe that is good anyway,
since what would happen if you received a notification on several
computers and all of them fired the callback action - it could cause
issues if the notifying app didnt properly account for multiple
callbacks from the same notification). it would be optimal if there
were no differences based on where the notification originated or
where it was eventually displayed, but that may just not be possible.
Peter Hosey
> say [an app] sends a notification over the network (not through a
> the script could have passed … whatever it wanted - the receiving
> Growl instance doesnt need to recognize/understand the the data (and
> back in the callback.
Yup.
BTW, Growl (i.e., GrowlHelperApp) is separate from the Growl
framework. Notifying apps use the Growl framework to contact
GrowlHelperApp. GrowlHelperApp is the background process that actually
shows the notifications.
(I'm not quite sure how the other notification systems work. Their
APIs are completely different.)
Currently, we keep the click context as a property list object
internally. It'd be somewhat difficult to change Growl to accept and
return any data without interpreting it as a plist (our pathway API is
currently built around the assumption that it's a plist), but I think
it's reasonable to expect that applications could come up with uses
for it.
> i didnt know about/understand the difference between growlnotify and
> growl.
growlnotify is just a notifying app that you use from the command-
line. It's one of the extras we include with Growl.
> in the version i hacked together, there is just one 'thing' - the
> growl server (app bridge?).
GrowlHelperApp. GrowlApplicationBridge is part of the framework, so it
runs in the app's process, not GrowlHelperApp's. It's the API that the
app uses to communicate with GHA.
> i was under that impression that, even if a notification were
> forwarded multiple times, the callback would be directed to the
> original application, not to the forwarding Growl instance.
That's a problem if any of the Growls along the line forward the
notification to multiple machines (which is an option in current Growl).
On the other hand, if the feedback backtracks along the same path of
Growls that the notification took, then it eventually goes through a
single Growl (the one that the application contacted locally). That
Growl (or every Growl on the network) can uniquify the feedback,
ignoring subsequent clicks, closes, or times-out on the same
notification.
> in the current version on the Mac, can forwarded notifications even
> generate a callback?
Seemingly not. It would be a nifty feature, though.
> the UDP network protocol doesnt have any provision for passing
> callback context, so i assume it cant.
The UDP protocol isn't used for forwarding.
Growl currently has two protocols:
- Cocoa Distributed Objects over TCP, used for forwarding
- Growl UDP protocol over (duh) UDP, used for original notifications
from growlnotify
The forwarding protocol is Cocoa-only, so you and all other users of
non-Mac systems can't use it. That sucks.
The UDP protocol doesn't support images or click feedback. That sucks.
So, you can see why we're behind a new protocol. :)
> perhaps that is how it should be restricted: only the original
> receiving instance of Growl allows for callbacks - if a notification
> is forwarded by Growl, the downstream displays cant do callbacks.
> (maybe that is good anyway, since what would happen if you received
> a notification on several computers and all of them fired the
> callback action - it could cause issues if the notifying app didnt
> properly account for multiple callbacks from the same notification).
See above.
Evan Schoenberg
On Aug 9, 2008, at 1:31 PM, briandunnington wrote:
>
> On Aug 9, 10:11 am, Peter Hosey <p...@growl.info> wrote:
>> Doesn't work so well on the client side when our Growl framework
>> passes the property-list object back to the application. ☺
>
> i guess i was thinking that if the callback mechanism was to be cross-
> platform, then the data would have to be passed in an OS-agnostic way.
> say a PHP script sends a notification over the network (not through a
> local instance of Growl, but using the network protocol directly) and
> a Windows system (running Growl) receives the notification. the script
> could have passed a string, or JSON, or a property list or whatever it
> wanted - the receiving Growl instance doesnt need to recognize/
> understand the the data (and may not even be able to understand it),
> but it can simply pass it back in the callback.
The Growl network server implemented in GrowlHelperApp can handle
parsing context information to/from a plist-representable datatype to
a compacted string representation (e.g. the base64 rep). This would
then still be OS-agnostic while Cocoa applications would transparently
send and receive NSDictionary/NSArray objects.
>> An app that uses our framework doesn't receive responses directly
>> from
>> the remote Growl anyway; the framework communicates with the local
>> Growl, which communicates with a remote Growl (forwarding). The
>> remote
>> Growl then communicates back to the local Growl, which communicates
>> back to the app.
>
> see above. i was under that impression that, even if a notification
> were forwarded multiple times, the callback would be directed to the
> original application, not to the forwarding Growl instance.
To avoid the sender needing to implement a separate listener and
needing to worry about NAT issues, I think we should do the following:
1. Client connects to server. This may be GrowlHelperApp on computer
foo connecting to GrowlHelperApp on computer bar for forwarding
purposes, or this may be Network-Aware Growl Sender connecting to a
GrowlHelperApp.
2. Connection is maintained so long as the notification is on-screen.
3. Callbacks are sent using the same connection, which is terminated
when the notification is dismissed or goes away.
> in the
> current version on the Mac, can forwarded notifications even generate
> a callback?
Nope.
> (maybe that is good anyway,
> since what would happen if you received a notification on several
> computers and all of them fired the callback action - it could cause
> issues if the notifying app didnt properly account for multiple
> callbacks from the same notification).
That's an interesting point. My first thought is that the connections
should be dropped once a click callback occurs, be it locally or from
any network source. This means the first to act happens while the
others are ignored.
-Evan
Ofri
user is away? The notification may stay on screen for hours.
Evan Schoenberg
On Aug 9, 2008, at 4:18 PM, Ofri wrote:
> What happens when the notification is set to stay on screen and the
> user is away? The notification may stay on screen for hours.
Is that really a problem? An open but idle socket is a pretty small
resource.
-Evan
briandunnington
> Currently, we keep the click context as a property list object
> internally. It'd be somewhat difficult to change Growl to accept and
> return any data without interpreting it as a plist (our pathway API is
> currently built around the assumption that it's a plist), but I think
> it's reasonable to expect that applications could come up with uses
> for it.
(not require it to be a plist data type)? i know it would be a pain to
change how the Mac version of Growl is already implemented, but i
think it is important to design the protocol around what is best for
all platforms and let the implementations follow and be wary of
designing the protocol around the implementation. since a theoretical
Growl app doesnt need to understand or convert the data in the click
context, it shouldnt have to fit a predefined implementation.
also, if the Mac version of Growl will support this new protocol and
it will eventually replace the current NSConnection architecture, then
there wont be a native object to send the plist data to anyway. if the
callback will be a MIME-like datagram, the click context data is going
to have to be 'serialized' (even if into raw bytes) anyway. i dont
think i am describing it well, but i dont think there is any need/
advantage to parse the click context data as a plist (other than the
current Mac implementation is already doing it).
> On the other hand, if the feedback backtracks along the same path of
> Growls that the notification took, then it eventually goes through a
> single Growl (the one that the application contacted locally). That
> Growl (or every Growl on the network) can uniquify the feedback,
> ignoring subsequent clicks, closes, or times-out on the same
> notification.
having the callback go back through the original growl instance allows
for much better control over duplicate callbacks, etc. plus, it allows
Evan's idea of keeping the TCP connection open to return callbacks and
should avoid any firewall issues. (if the end-of-the-line growl
instance tried to contact the original notifying app otherwise, it may
not even be able to).
i think that any forwarding growl instances might want to add some
kind of 'Forwarded' header to the datagram as well. (not exactly sure
of what the use would be, but i feel like it might be useful later
on).
> The Growl network server implemented in GrowlHelperApp can handle
> parsing context information to/from a plist-representable datatype to
> a compacted string representation (e.g. the base64 rep). This would
> then still be OS-agnostic while Cocoa applications would transparently
> send and receive NSDictionary/NSArray objects.
if you guys are suggesting the data be required to be in plist format
or not and what the advantage(s) are, and how this will work in the
future if native NSConnection transports are phased out.
> 3. Callbacks are sent using the same connection, which is terminated
> when the notification is dismissed or goes away.
connection open was not optimal, but the more i think about it, i
think it is the only way we are going to be able to do callbacks
properly when dealing with somenetworked/internet applications.
early in this discussion, i asked if this discussion was specifically
about a TCP protocol or just a datagram format and Peter agreed that
at this point, it was just a discussion on the format. i think that
using TCP opens up a lot of possibilities regarding callbacks and
allowing web/AIR apps to send notifications, but i think that Growl
should support this same data format over a UDP port. it wont require
any additional parsing work (same data format), but it allows several
additional features:
1. apps that dont care about callbacks can 'fire and forget' about the
notification and avoid the overhead of the TCP handshake/response or
handling connection errors if growl is not running, etc.
2. apps that want to broadcast notifications can do so using the
multicast features built into UDP
3. apps that want to do callbacks via url (instead of notification
response) can still 'fire and forget', close the connection, and
handle the callback if/when it comes via other means. (see below)
> That's an interesting point. My first thought is that the connections
> should be dropped once a click callback occurs, be it locally or from
> any network source. This means the first to act happens while the
> others are ignored.
the callbacks traverse back up the forwarding chain, as soon as an
instance fires the callback, the other connections can be dropped.
> Is that really a problem? An open but idle socket is a pretty small
> resource.
solutions as well. i think that we should allow alternative callback
mechanisms (url) for apps that want to avoid managing sockets, etc.
in Peter's original proposal, he suggested something like:
Notification-Callback-Context-Type: <value>
Notification-Callback-Context: <value>
i propose adding something similar to:
Notification-Callback-Target: <url>
this header is optional and if not specified, callbacks work like we
are talking about. however, if this header is specified, the TCP
connection can be closed and when the callback fires, the data will be
sent to the target url instead. this enables UDP connections to
utilize a version of callbacks, and allows web apps an easy way to
handle callbacks without managing TCP sockets. the data can either be
POSTed or whatever we decide (or perhaps it is yet another header that
can be set), but since it is optional, i dont think it hurts anything
and opens up some possibilities.
one last thing to add to my already-too-long post: should there be a
way to specify if the callback is 'silent' or not? by that i mean if a
user clicks a notification, the callback is sent to the notifying app,
but is it also brought to the forefront of the desktop automatically
(not sure how Growl currently works)? for some types of callback, that
is the desired behavior (click on a file, open Finder, etc). however,
for some notifications, the desired callback behavior is more of a
'record this action and move on'. imagine a calendar app that notifies
the user of an upcoming meeting. perhaps when clicked, the
notification is set to snooze the notification and have it pop up
again in a few minutes. in this case, it isnt useful to launch your
calendar app and interrupt your workflow. it would be nice to simply
notify the calendar app behind the scenes to snooze the appointment
and move on. thoughts?
Peter Hosey
> so are you saying it is or is not a good idea to allow arbitrary
> data (not require it to be a plist data type)?
I think that it is. A data plist object covers anything, but would
require other platforms to implement some sort of plist parsing. I
don't want to do that.
I think it'd be better if we allowed any data type, and Growl wrapped
that data in a plist object for internal use, if necessary.
> i think it is important to design the protocol around what is best
> for all platforms and let the implementations follow and be wary of
> designing the protocol around the implementation.
I agree. I'd still like to hear a reason why we should keep it plist-
only, since it would save us work, but I certainly can't think of
one. :)
> also, if the Mac version of Growl will support this new protocol and
> it will eventually replace the current NSConnection architecture,
> then there wont be a native object to send the plist data to anyway.
Yes there will. The object that sends plists to Growl and receives
plists from Growl is in the notifying app, not GrowlHelperApp or the
GrowlApplicationBridge.
We would convert this to a platform-independent representation, where
possible. We should also have a header for suggesting a particular
plist type:
Notification-Click-Context-Property-List-Type: (String|Integer|Real|
Data|Date|Array|Dictionary)
This will help keep things unambiguous—especially if you want to send,
say, a string containing “42” (as opposed to a number) as the click
context.
This is optional, of course. The Growl framework would use it (for API
compatibility), but other frameworks on other platforms would not need
to, although they could.
> i agree that backtracking along the forward chain is the way to go.
> … it allows Evan's idea of keeping the TCP connection open to return
> callbacks and should avoid any firewall issues. (if the end-of-the-
> line growl instance tried to contact the original notifying app
> otherwise, it may not even be able to).
Yup.
> i think that any forwarding growl instances might want to add some
> kind of 'Forwarded' header to the datagram as well. (not exactly
> sure of what the use would be, but i feel like it might be useful
> later on).
Maybe something like SMTP. Each MTA adds a Received: header to every
message it processes. For example:
Received: by 10.141.155.2 with SMTP id h2cs253307rvo; Sat, 9 Aug 2008
21:31:53
-0700 (PDT)
Received: by 10.150.148.1 with SMTP id v1mr9974738ybd.
67.1218342712688; Sat,
09 Aug 2008 21:31:52 -0700 (PDT)
Received: from yw-out-2526.google.com (yw-out-2526.google.com
[74.125.46.34])
by mx.google.com with ESMTP id 6si269949ywn.0.2008.08.09.21.31.51;
Sat, 09
Aug 2008 21:31:52 -0700 (PDT)
Received: by yw-out-2526.google.com with SMTP id 7so3048172ywf.42 for
<prh@growl…>; Sat, 09 Aug 2008 21:31:51 -0700 (PDT)
Received: by 10.100.96.9 with SMTP id t9mr285781anb.8.1218342711192;
Sat, 09
Aug 2008 21:31:51 -0700 (PDT)
Received: by 10.44.187.78 with SMTP id k78gr159hsf.0; Sat, 09 Aug 2008
21:31:51 -0700 (PDT)
We could borrow (and slightly improve) the same format, which is
specified on page 32 of RFC 821. Here's my suggestion:
Received: from <hostname> by <hostname> [with Growl] [id
<identifier>]; <ISO 8601 date>
ISO 8601 is the YYYY-MM-DD format:
2008-08-09T22:21:00-0800
> i initially agreed with Ofri in thinking that keeping the TCP
> connection open was not optimal, but the more i think about it, i
> think it is the only way we are going to be able to do callbacks
> properly when dealing with somenetworked/internet applications.
Use a single connection from registration onward? What's the advantage?
>> My first thought is that the connections should be dropped once a
>> click callback occurs, be it locally or from any network source.
>> This means the first to act happens while the others are ignored.
>
> i agree and i think this is also what Peter was describing above.
Yup.
> i think that we should allow alternative callback mechanisms (url)
> for apps that want to avoid managing sockets, etc.
It shouldn't be a terrible hassle for an app to manage sockets, since
they'd be local only (communicating with an instance of a
GrowlHelperApp-esque daemon* on the same box). Moreover, all apps
should be going through some sort of library (even if it's just drop-
in code) anyway.
Is there an environment in which any sort of socket communication is
untenable? You mentioned that you even got it to work with JavaScript
+ Flash.
> in Peter's original proposal, he suggested something like:
>
> Notification-Callback-Context-Type: <value>
> Notification-Callback-Context: <value>
>
> i propose adding something similar to:
>
> Notification-Callback-Target: <url>
>
> … if this header is specified, the TCP connection can be closed and
> when the callback fires, the data will be sent to the target url
> instead.
I'm not sure that this is necessary (see above).
> the data can either be POSTed or whatever we decide (or perhaps it
> is yet another header that can be set) …
We should include a header:
Notification-Callback-HTTP-Method: (GET|POST)
If POST, the notification system process (see footnote) sends the
click context data as the message body. If GET, then the app doesn't
get its click context back. (An app may want to just use different
URLs, instead of sharing a URL between multiple notifications and
switching based on the click context.)
It should be required for using Notification-Callback-Target, the same
way an HTML form element requires both a method and an action (target
URL).
> should there be a way to specify if the callback is 'silent' or not?
> by that i mean if a user clicks a notification, the callback is sent
> to the notifying app, but is it also brought to the forefront of the
> desktop automatically (not sure how Growl currently works)?
In Growl, it's up to the app to bring itself frontmost, if that's
appropriate. That's because, sometimes, it's not: One example being
opening a Finder window to reveal a file.
> imagine a calendar app that notifies the user of an upcoming
> meeting. perhaps when clicked, the notification is set to snooze the
> notification and have it pop up again in a few minutes.
I don't think any user would expect that. The calendar app should use
a dialog box (either a standard one, or, as iCal does, a custom one).
* Sentence structure is getting awkward, so I suggest some definitions:
- Notifying app: An application (or JavaScript script on a webpage, or
some other program) that the user is directly interacting with, and
that will post a notification.
- Notification framework: Library that a notifying app uses to post
notifications easily. (In Growl: Growl.framework.)
- Notification system process: Process that receives notifications
from a notifying app, then displays them, forwards them to another
notification system process on another machine, or both. (In Growl:
GrowlHelperApp.)
Anything else we should define?
Ofri
problems in them :)
What happens if you have 100 sticky notifications, the machine goes to
sleep, awakes an the user dismisses all notifications?
- Ofri
Evan Schoenberg
On Aug 10, 2008, at 5:56 AM, Ofri <ofri....@gmail.com> wrote:
>
> I honestly don't know.. I'm just running scenarios with potential
> problems in them :)
> What happens if you have 100 sticky notifications, the machine goes to
> sleep, awakes an the user dismisses all notifications?
Nothing, since the system going to sleep dropped the TCP sockets :)
-Evan
briandunnington
> Data|Date|Array|Dictionary)
> compatibility), but other frameworks on other platforms would not need
> to, although they could.
headers that are not custom (X-*), but are also kind of platform-
specific? kind of like CSS does with vender-specific types? (-moz-
scrollbars-horizontal, -webkit-border-bottom-radius, etc).
http://reference.sitepoint.com/css/vendorspecific
might be useful for things like the property list type and app bundle
identifier (and probably a lot of other things we havent dealt with
yet). or maybe it is fine to just define them as regular headers and
not worry about which platforms may or may not use them internally.
> We could borrow (and slightly improve) the same format, which is
> specified on page 32 of RFC 821. Here's my suggestion:
> Received: from <hostname> by <hostname> [with Growl] [id
<identifier>]; <ISO 8601 date>
> Use a single connection from registration onward? What's the advantage?
notification is received until it is closed/clicked/times out.
> In Growl, it's up to the app to bring itself frontmost, if that's
> appropriate. That's because, sometimes, it's not: One example being
> opening a Finder window to reveal a file.
it presents an issue. consider a website running in the browser - the
actual application that controls focus is the browser, but the
application that sent the notification is the website. if Growl tries
to make the http callback, it will have to decide to pass the url to
the browser (in which case the browser will mostly likely always come
to the front), or make the http request internally, which will never
allow the website to come to the front (since the browser will not be
involved). maybe that is just a limitation of url callbacks and if a
developer decides to use them, they have to understand that
limitation.
> * Sentence structure is getting awkward, so I suggest some definitions:
afraid i didnt fully understand the current Mac implementation and
didnt want to confuse the issue even more. by the way, when you guys
refer to 'Growl' in general, is that just referring to the collection
of pieces (GrowlApplicationBridge, GrowlHelperApp, etc) or does that
refer specifically to the pref-pane portion or? just curious.
the other day we had discussed possibly having the callbacks use a
format similar to the register/notify protocol with a message type of
CALLBACK and the appropriate headers. i was thinking about it and
wanted to throw another idea out for consideration: instead of using
the message type of CALLBACK, maybe we should consider all of the
various response types that we might need to return. for instance,
when an app sends a REGISTER message, the CALLBACK response does not
apply (i dont think). an if we add something like a STATUS command
where an app can query to see if growl is running and if so, if the
app is already registered, what notifications are enabled, etc, then a
different type of response would be appropriate. so maybe instead of a
message type in the response, we have -OK or -ERROR as the 'type' in
the response. then the rest of the message can be filled with the
appropriate headers for whatever the response type is. if it is an -
ERROR response, then there could be a header like 'Error-Description'
or whatever else was appropriate. that would 1) tell the app if their
request succeeded (or more importantly, failed), and 2) keep the
response datagrams easily identifiable from request datagrams. it
really is a small difference, but (to me) it is slightly more clear.
briandunnington
real jobs), but i added a bunch of information to the proposal
document in the Files section (the newest document is growl_v3.txt)
based on the discussions over the weekend. the document contains
sections for the following topics:
request format, including defined headers
response format, including defined headers
callback handling
forwarding
error conditions and responses
appendix dealing with Flash policy file
examples
of course, all of this is my interpretation of where the discussions
have headed, and none of it is set in stone, but i find it helpful to
have one place to review the whole working draft of the protocol and
not constantly have to review the entire discussion to see where
things stand.
take a look and feel free to make corrections, additions, deletions,
or bring up any issues you see. i think the decision to use the MIME
format is pretty much agreed upon, and most of the other items in the
document have had at least preliminary agreement. can we start to
solidify any other details at this point, or are there still some over-
arching concepts to discuss?
briandunnington
up alternatives, but should we have considered using XMPP/Jabber and
building off of that ( http://www.jabber.org/technology-detail )? i
know we ruled out XML, but i am sure lots of XMPP messages get passed
around every day and i havent heard anyone complaining of the
performance. as far as i can tell (i am not very well versed in the
details of the technology), it can be used as the basis for pretty
much any type of communication system. it sends data over TCP from the
client (notifying app) to the server (growl daemon) and back allows
two-way communication (for callbacks). since the data format is XML,
it is extensible. i am not sure of the details with regard to how
binary data is transferred, but i do know that XMPP clients are able
to send/receive files, so i assume there is handling for that built
in.
the advantages of course are that it is already built, there are
libraries for a variety of programming languages, and somebody else
already went through the trial-and-error process to get it polished.
downsides are that it is a generic solution (not fully optimized for
Growl) and has a connotation of being associated with instant
messaging (although it doesnt take too much of a stretch to consider
Growl as a specialized form of IM).
anyway, just thought i would throw it out there before we made any
final decisions to see if anybody felt like it was a better solution.
Chris Forsythe
simple as well.
Chris
Ofri
Evan Schoenberg
On Aug 12, 2008, at 2:19 AM, briandunnington
<briandu...@gmail.com> wrote:
The only area I still see largely lacking is security. Optimally,
notifications should be encrypted such that they can't be modified
when the server is password protected. I'm inclined towards one of the
password-based ciphers to avoid needing a complex back-and-forth of
public/private key exchange. Localhost-originating notifications need
not use the password, allowing any app to act like
GrowlApplicationBridge currently does.
-Evan
>
>
> >
briandunnington
>
> I'm inclined towards one of the password-based ciphers to avoid needing
> a complex back-and-forth of public/private key exchange.
>
use something similar to what the current UDP protocol uses (hash the
message contents using the password), we would need to keep in mind
the idea that the receiving growl daemon might not be reading the
entire datagram (ignoring cached binary data, etc). if the entire
message is not read, it cant be properly hashed*, and if the entire
message is required, it defeats the goal of not having to receiving
the entire message before parsing begins. if we only include certain
pieces in the hash, then the contents of the other pieces could get
modified in transit.
is the idea of 'security' in this context supposed to:
1. show that the notifying application is authorized to send
notifications?
2. ensure the contents of the message are not tampered with in
transit?
3. protect the contents of the message from being read while in
transit?
4. some combination of the above?
if it is only #1, then simply hashing the Application-Name header (or
something simple like that) would suffice. if it included #2 and/or
#3, we have the issue outlined above. we could come up with all kinds
of key-passing approaches, but like you said, it would be best to
avoid too much complexity. i guess we need to know the desired goal so
we can figure out the solution that provides that goal with the least
overhead.
* after thinking about it some more, i guess that the receiving daemon
_might_ be able to generate the hash by substituting its cached binary
data for the data not read from the stream. however, things like extra
blank lines or even untrimmed spaces in the headers could make this
approach unreliable.
Peter Hosey
> should we have considered using XMPP/Jabber and building off of that
> ( http://www.jabber.org/technology-detail )?
Before the other replies came in, I asked David Smith (one of Adium's
XMPP gurus) about it.
It's possible to do it over XMPP, but I agree with Evan: It's too
heavyweight.
In particular, consider the fact that every notification system would
need to include an XMPP server and client. And they'd need to either
write a small XMPP server themselves, or include an existing (large)
XMPP server.
I think we're better off sticking with the MIME-like protocol.
Evan Schoenberg
On Aug 14, 2008, at 2:53 PM, briandunnington wrote:
> is the idea of 'security' in this context supposed to:
> 1. show that the notifying application is authorized to send
> notifications?
> 2. ensure the contents of the message are not tampered with in
> transit?
> 3. protect the contents of the message from being read while in
> transit?
> 4. some combination of the above?
>
> if it is only #1, then simply hashing the Application-Name header (or
> something simple like that) would suffice. if it included #2 and/or
> #3, we have the issue outlined above. we could come up with all kinds
> of key-passing approaches, but like you said, it would be best to
> avoid too much complexity. i guess we need to know the desired goal so
> we can figure out the solution that provides that goal with the least
> overhead.
Some combination of the above, I think. Messages should only be
accepted from authorized parties and should be both protected from
tampering and secure from being read by recipients who don't have the
password.
What if rather than encrypting the entire message, we encrypted the
data parts but not the headers?
So:
Notification-Name: <encrypted string>
Notification-Title: <encrypted string>
and so on.
If the server has a password, it can decrypt each string. It can do
so on a line-by-line basis and can therefore stop accepting more
information when it wants to.
---
I just realized a possible problem. How can the server say "stop
sending me this notification data" other than dropping the
connectIon? If that's the only way, it'll break keeping the socket
open for a callback.
-Evan
Brian Dunnington
I just realized a possible problem. How can the server say "stop sending me this notification data" other than dropping the connectIon? If that's the only way, it'll break keeping the socket open for a callback.
Peter Hosey
> i was actually implementing a tcp server and once i read the
> information i needed, i tried to write my response back to the
> client. however, the remaining incoming data was still in the buffer/
> stream, so subsequent reads still fetched that data.
That just means that the incoming data was still queued, not that you
weren't able to write. (I think it depends on network hardware. I
doubt that people still make NICs that can't do full-duplex, though.)
The most reliable way is to use select() (or something similar) to
multiplex: write when you can write, and read when you can read. The
select() call will tell you which you can do.
Peter Hosey
> accept the entire datagram and still keep the connection open and
> write a response back.
Forgot to respond to this part.
Yes, there is. The ideal way is to use select(), and write when you
can write and read when you can read, at least until you have written
everything (and error out if you have read everything, as indicated by
count and length headers, but still haven't been able to write the
rest of your response for N seconds).
There's also blocking I/O, but if both sides write or both sides read,
they'll deadlock. select() is much more reliable. (And in Cocoa, we
don't even need to use select—we can just use the Cocoa classes, and
let them and the run loop handle it for us.)
Evan Schoenberg
based on AsyncSocket does appear to handle this correctly.
-Evan
>
>
>
> >
briandunnington
> On Aug 14, 2008, at 22:26:56, Brian Dunnington wrote:
>
> > i was actually implementing a tcp server and once i read the
> > information i needed, i tried to write my response back to the
> > client. however, the remaining incoming data was still in the buffer/
> > stream, so subsequent reads still fetched that data.
>
> That just means that the incoming data was still queued, not that you
> weren't able to write.
request. that wasnt the problem really. the problem was that after i
wrote my response, i check to see if the socket is readable/data is
available and it is - because the previous request data is still in
the buffer.
i am sure this is totally off-topic and there are better places for me
to bone up on my socket programming skills, so i wont clutter up this
thread. at the moment i am mostly concerned with making sure a Flash
socket implementation is workable, since that is the only way (that i
know of) for websites to do TCP across domains. if that doesnt work,
then websites wont be able to send notifications using this protocol,
so i just wanted to have a simple proof of concept (i have a working
version, was running into issues when dealing with servers that wrote
their response early). i will get it sorted out though - thanks for
the clarifications.