General web hook discovery and registration

[Jeff Lindsay]

I'm starting to think about a standard way for web hook event discovery and callback registration. I've kind of known GetPingd does pretty much that, but since I haven't actively been thinking about this, I haven't put them together. I'd be interested in working with you guys to think about this in the context of web hooks in general. The main thing (and maybe why I didn't think of it sooner) is that web hooks are a more ... general case. Web hook callbacks can be used for update pings OR triggering code with a payload, which goes into a completely different domain than content updates --- yet the mechanics of this interface are pretty much the same.

What do you all think about this?

--
Jeff Lindsay
http://webhooks.org -- HTTP callbacks are the future
http://shdh.org -- A party for hackers and thinkers
http://tigdb.com -- Discover indie games

[Paul Jones]

Hi Jeff,

I'm certainly interested in discussing the topic more - things have kind of stalled because we were in part waiting to see what the outcomes of the various discovery projects (OAuth/XRDS for instance) were before proceeding too much further. However, I do think we could kick the discussion back off again - the "format" of the discovery mechanism isn't the only thing to consider!

I'm curious what you mean when you talk about triggering code with a payload? Are you perhaps referring to event delivery in general? My vision for GetPingd is that we can provide a discovery mechanism that allows a myriad of different eventing mechanism to be discovered - web hooks, XMPP or perhaps even more standard message queueing infrastructure (AMQP for instance). I'd be very interested to hear if you've come up with an even more general case that hadn't yet been considered!

Cheers,
Paul.

[Jeff Lindsay]

The killer feature of web hooks to me (defined as user-defined HTTP callbacks) is that by requesting a web URL, you have a mechanism to run code because of the ubiquity of CGI. XMPP, email, SMS, Twitter... none of these do that, however, from code you can push into those channels very easily. That's why I focus on HTTP for web hooks, otherwise I would have called them web notifications or some other generic message-passy name. The key is programmability.

But I'm glad somebody else is starting to write a spec'ish document for web hooks. However, I'd personally like to see a more decoupled approach in standardization. The discovery and registration standard can be decoupled from delivery standards. I would *really* like to hold off on trying to standardize delivery, Daniel. Nonetheless, really nice work so far with shario... where are you located?

Paul, like I said above, that's what I meant about triggering code with a payload. If you look at Daniel's rough web hooks spec, you can see he has an option for just "notify" vs "full payload." I'm not entirely sold on why that needs to be an option, however. Anyway... some things I'd like to see with GetPingd would be less pubsub framing (updates, notifications, etc) and more general eventing language ("events"). I'd also like to see it simpler. Some nuances are specific to content updates, like requesting a diff mode. It would be nice if this was a very simple but extendable standard. The Base case would be the bare minimum.

That said, I have some interesting preferences that sort of stray away from the approach of GetPingd so far, or at least it seems that way. I would prefer a more RESTful approach, with the option of XML or JSON, or even just HTTP request parameters. For example, most people implementing web hooks now have a web UI for registering the callback URL. It would be nice if they could reuse the endpoint for this programmatic discovery with their web UI form. Yeah?

-jeff

[Daniel Parker]

Paul,

I had replied under-the-radar, if you hadn't picked that up. (I don't want to publicize shario yet, as it's really not yet operational.)

The standards document Jeff is referring to is at http://tinyurl.com/webhooks. I wrote it up last month to begin organizing and communicating my thoughts on the whole idea of event-pushing.

Jeff, I understand what you're saying about the "programmability" of http callbacks. I also understand that it is the lightest of the push technologies when it comes to network footprint and ease of implementation. However, I do believe some of these others (xmpp, sms, email, twitter) have their merit, and I believe the "programmability" of them is growing slowly but consistently. I just don't want to limit the overall idea. The cool thing about xmpp in the mix is that desktop applications can then be included in the whole subscribing-receiving end, and then there is no boundary to web hooks.

Therefore I may be stubborn, but I will insist on not binding the term "web hooks" to simply http.

When it comes to decoupling the standards for Discovery / Subscription / Delivery, I agree -- I just haven't yet understood quite how that should be done. And as I've moved onto the agile crowd a while back, I just have to get something moving, even if it's not quite right. I do like having them described right next to each other so that it's easy to see the connections and how they play together. I'd love to get some more feedback though on what you mean by "decoupling the standards" and perhaps how we could do that.

I'm not sure how you're thinking of registration, Jeff, but it might be slightly different than I am: I'm thinking of a completely automated system here -- a person could tell their new rss reader to subscribe to a feed, and it will automatically discover the subscribing endpoint for that specific feed, and automatically post a subscription request to that feed. Perhaps since it's a desktop application, the rss client subscribes to xmpp, full-payload. Then also from the rss client the user can unsubscribe at any time -- the unsubscribe link is somehow included in a standard way in every pushed event. Subscribe your facebook photos to your flickr account and your myspace photos to your facebook photos, subscribe your desktop background to your friend's photoblog, subscribe your cell phone (sms) to your house's security system, subscribe to your pool's chemical levels by email, *dreamy thoughts continue...* - These are all automated systems, and I wouldn't want a web browser involved. Just to clarify that the reason we need a standard for subscription is because we want it automatable. But you probably already knew that. :)

Lastly, What do you think of the minimal set of standards I have written so far? Every part of it is up for discussion, since I know that it's often hard for one person to be right first try on these kinds of things. ;) A few parts that are shaky for me are:
    • Discovery: HTTP header or meta tag? If we want discovery on an xml document, we can't insert a meta tag, so maybe we need it to be an http header. Then again, perhaps not all web publishers can insert http headers in their published articles. Should both methods be encouraged?
    • Subscription: What standard options should exist? From getpingd and other plain webhook implementations (github, for example) I've gleaned that people often want at least "notification" and "full" mode. Should a "duration" option be available (subscribe for x number of updates, for a time period, or forever until unsubscribed)?
    • Delivery: How much standardization should go into this? It seems that the more standardization, the less flexible, but the more automatable. Is it possible to get around that trade-off?

- daniel parker -

"You have granted me life and steadfast love, and your care has preserved my spirit." Job 10:12
"The LORD is my chosen portion and my cup . . . indeed, I have a beautiful inheritance." Psalm 16:5-6
"Give what you can ... take nothing back!"

[Joe Cascio, Jr.]

Have you already looked into the rssCloud spec, which is part of RSS 2.0? http://cyber.law.harvard.edu/rss/soapMeetsRss.html
It allows a subscriber to provide a callback URL which is to be called when an update is made to the RSS stream. Yeah, there's SOAPiness about it, but might provide some guidance.

Apparently some people did an implementation and found it did not scale up. That may have had to do with the fact that the sites being called were individual users' feedreaders. In that case, of course, you get into all kinds of proxy and firewall problems.

But scale-up would need to be considered. If a feed has thousands of subscribers and is updated frequently (like a Twitter stream).

Not sure what type of applications this is being targeted at, but sounds like blog publishing.

I prototyped an alternative messaging system a while back that used UDP as the notification mechanism, and HTML for everything else. But really anything that proposes a "push" type of protocol has to deal with the proxy and firewall problems.

Great convo!

Joe Cascio

On Fri, Feb 27, 2009 at 6:34 AM, Jeff Lindsay <prog...@gmail.com> wrote:

[Jeff Lindsay]

True. I've talked to Flickr, Google and Twitter and they are all very convincing that popular, one to many feeds are impractical right now for web hooks. There are still *lots* of use-cases outside of this realm and I think we can figure out the one to many issue eventually.

Understand that I'm not going to push (ahem) push for server to client yet. I honestly don't find it worth the work or even that interesting YET. So I'm operating on the assumption of server-to-server and I ignore complaints about firewalls and all this because I can. That said, obviously 80 can usually get through firewalls so I'm not necessarily taking us to a dead end for server to client.

About XMPP being associated with web hooks. I've thought about this before, however I didn't think about email, Twitter, SMS or whatever else. I think have a standard discovery and subscription protocol is a Good Thing to allow whatever transport. However, this is why I think this standard should be called something other than Web Hooks. Perhaps WebSub. Slightly different angle than pubsub, more lightweight... and just an interface. So thinking ahead, what I associate with web hooks should use this standard that works with any transport/delivery mechanism... but I'm still going to focus on HTTP callbacks with regards to what "web hooks" are. For now.

But I LOVE your dreamy thoughts, and I do expect this to all be entirely automatic eventually. But it will come slowly. I don't think we should try to standardize things that don't already have some proven implementation. Like OAuth was built on existing good ideas. Microformats bootstrapped off existing formats (vcard, etc) and then adopted things from the greater POSH world as they become popular. Same as the way HTML adopts new tags from popular CSS class names. We should AVOID trying to standardize things that aren't proven by the masses, or else we risk poor adoption and more.

Anyway, *fully* automatic in the greater sense would probably require standardizing the delivery. But like I said, I'd like to put that off. For now, I'd like to try and put something together for discovery and registration alone that's simple enough to get the job done, and general enough to be used for web hook events or pubsub. I mentioned some of my ideal requirements already... but Daniel, I've always thought of the same thing for discovery: meta or link tags on pages, perhaps Headers. In fact, very similar to XRDS and what GetPingd is doing now. I'm just wondering if XRDS can be avoided because it's not necessarily important. In fact, it makes up the bulk of the spec when you look at GetPingd and it's not even necessary! Frankly, I like RSS discovery: just a link tag, no description junk.

-jeff 

[Joe Cascio, Jr.]

Totally agree that the discovery/subscription interface should not be wedded to any particular delivery (fulfillment?) protocol. A publishing node may be able to offer several methods of notification. The subscriber should be able to choose.

I too tend to think of this as a server-to-server mechanism where there could be many different implementations of server-to-end-user, including one in which the server and UI are integrated in to a single-user program that is installed on a machine on which the user has total control of the router, IP addresses, firewall, etc.

Are notification mechanism (vs. discovery/subscription) discussions out of scope for this group? If so, anyone know a group where they are the main topic

Thanks,
JoeC

[Jeff Lindsay]

Well, there's always the Web Hooks group: http://groups.google.com/group/webhooks

It's definitely more of the context I'm biased towards. But the goals seemed pretty aligned with GetPingd... only we may come up with something totally different?? So many means to an end...

Tangent: My process for solving problems has always been more than just solving. Identify the problem (so you're not just fixing a symptom), design the *ideal* solution without constraints of existing solutions or any kind of implementation, then find iterative solutions/implementations towards the ideal. See Ackoff on Idealized Design. Starting with requirements you do want is more effective than starting with what you have or don't want.

So, some requirements we might have are:

- Machine friendly for automatic discovery and registration
- Human friendly for debugging and ease of understanding
- Easy to learn, easy to adopt
- Perhaps a way to see current callback (one scenario I want requires this)
- As close to the way people are already doing it as possible
- No more complex than needed to get the job done
- Ideally aligned with RESTful design principles (this is starting to get too caught up in implementation)

I'm sure there's more. Your thoughts?

-jeff

[Daniel Parker]

Jeff, I've been away from this project/discussion for quite a while now, but I'm coming back today with a new angle of attack: What is the problem with a simple standardized HTTP POST request that subscribes to a live Atom push to a url (webhook)? Answer: Authentication. How do you verify that the post-back url belongs to the same user/server/process as the subscription requester? I think the typical solution to that problem is to do just that - actively verify that other end with a secret token and require the user/server/process to complete the circle. (Like verifying that two doors enter the same house, without going into the house: give something to the guy inside and have him give it back to you at the other door.) So for an HTTP example: 1) A <link> tag is included on an HTML page or in an RSS/Atom feed. 2) A Subscriber sends a subscription request, which includes a) post-back url, b) secret string, c) other subscription options 3) The Publisher immediately sends the secret string to a POST request to the post-back url. The post-back receiver responds to accept or decline the content updates from this Publisher. 4) If the post-back receiver responded favorably, the Subscription is active and the Publisher begins sending updates. In this process, the Subscriber can either come up with its own secret string, in which case it must either communicate this secret with the post-back Receiver or trust that the post-back Receiver will accept the subscription; or the Subscriber can request a secret string to be shared with it by the post-back Receiver prior to subscribing. The important points are that a) the push Receiver has a chance to approve or deny future content delivery, and b) these verification requests can be linked to subscription requests, if needed. I think a similar authorization/verification mechanism could be worked out for any transport. We do this with email already every time we sign up for an online service. We do similar things with SMS, and it is also very easy to do with XMPP. - daniel parker -

"You have granted me life and steadfast love, and your care has preserved my spirit." Job 10:12
"The LORD is my chosen portion and my cup . . . indeed, I have a beautiful inheritance." Psalm 16:5-6
"Give what you can ... take nothing back!"

[Jeff Lindsay]

Daniel,

Do you think this can be simplified if the Publisher provides a randomly generated key on subscription? Although, that would require changes to the hook script on subscribe. Also, this focuses on the initial subscription. I'm trying to lead into how this is similar/different to the HMAC authentication approach Google uses for their post-commit webhook. They provide a key, and sign all messages with an hmac signature based on that key. That seems to achieve the same Subscriber-Receiver verification and allows for ongoing authentication/spoof prevention (as in, it ensures to the receiver it's a message from the Publisher). When combined with a nonce it can prevent replay attacks (if concerned). It's a much simpler approach than the PayPal method of verification where you have to post the whole request BACK to PayPal to get assurance it came from them.

I imagine a process like much like yours although with a couple changes considering the hmac signature approach.

Step 1, the link tag also includes authentication protocol and if there is one necessary. Some people don't care!
Step 3 becomes: send an initial message with hmac signature, and if returns 200, all is good. Perhaps it makes sense to return 400 if the message and signature don't match, or if the receiver doesn't want to subscribe (in all likelihood, it's not that it doesn't "want" to, it wasn't "made" to).

Interestingly enough, we're discussing on the webhooks list myqron.com and pubsubhubbub ... and the issue of authentication and discovery came up (just several hours ago: http://groups.google.com/group/webhooks/browse_thread/thread/206f1eec1301b1e7/47267b1f3f9ed0ed#47267b1f3f9ed0ed). Apparently the hubbub guys are now coming back to that issue. We should all be talking about it together methinks.

-jeff

http://webhooks.org -- Make the web more programmable

http://shdh.org -- A party for hackers and thinkers
http://tigdb.com -- Discover indie games

http://progrium.com -- More interesting things

[Daniel Parker]

I'll reply to points in your message here for completeness' sake...

For reference: I see at http://pubsubhubbub.appspot.com/, they have a wonderful 10-slide presentation on the skeleton of the protocol.

Do you think this can be simplified if the Publisher provides a randomly generated key on subscription?

I originally mapped it out that way, but it seems to me that having the subscriber propose a key opens up more possibilities:

    • Note, if the publisher provides a key, the subscriber must communicate that key to the Hook Receiver (if they're different) AFTER subscribing.

    • If the subscriber generates the key, you have several options for authentication:

        1) A promiscuous Hook Receiver may not require a key at all, in which case the subscriber need not generate a key.

        2) The Hook Receiver may accept keys based on a pre-determined rule, and therefore the subscriber need not communicate the key, because it could generate a valid key on its own.

        3) If the publisher provides the key, the Hook Receiver must have a way to receive the key from the subscriber, and store that key until the publisher asks for it. Also, this ends up being a race condition.

    • Last positive: If the publisher can verify the Hook Receiver immediately, it can respond success/failure to the initial subscription request itself:

Subscriber --------------------> Publisher

           HookUrl <------------- Publisher

           HookUrl -------------> Publisher

Subscriber <-------------------- Publisher

if returns 200, all is good. Perhaps it makes sense to return 400 if the message and signature don't match, or if the receiver doesn't want to subscribe (in all likelihood, it's not that it doesn't "want" to, it wasn't "made" to).

I suggest using a 2xx response other than 200, to be further sure that the HookUrl is indeed an intelligent Hook Receiver. "202 Accepted" might be reasonable. This is just for an "HTTP Subscription Authorization Request".

I would like to make an HMAC signature for the payloads optional, as well as SSL. I imagine the first "subscription authorization request" would operate identical to the payload requests, but be just an empty payload, with the addition of the subscriber-generated key.

This method, I believe, leaves the door open for any range of authorization you want to accomplish. The Hook Receiver would usually require a key but wouldn't be required to; it could also ask for an hmac signature or ssl on that initial Subscription Authorization response.

- daniel parker -

"You have granted me life and steadfast love, and your care has preserved my spirit." Job 10:12
"The LORD is my chosen portion and my cup . . . indeed, I have a beautiful inheritance." Psalm 16:5-6
"Give what you can ... take nothing back!"

[Jeff Lindsay]

Ok, I like this. Optional key supplied by subscriber. And optional HMAC. And I'm a little hesitant about 202, but I guess as long as we formalize this protocol and people explicitly adopt it, it's not a big deal at all. I was just concerned about existing implementations that are looking for 200. However, thinking about the intended use of 202, I'm hesitant again because it's usually for processes that are not completed by the end of the response (which actually makes it a good return code for successful Hookah posts ... sorry, thinking out loud). Maybe 201 Created, as in subscription created?

[Daniel Parker]

I'm not settled on 202 either. My thought was to get away from 200 because some poorly-written end-points might accept the request and respond with 200 even if they weren't written for webhooks; but perhaps that could be a desired result in some cases. People will probably know what they're doing after all. Might as well allow a simple 200 response.

By the way, I'm working on making a service like pubsubhubbub and myqron, but based on this discussion. I started building it at least a year ago, but hadn't really figured out the subscription mechanism yet. So at the first iteration of this site, visitors will be able to enter two pieces of information: "URL" and "Deliver to". (The site will poll the URL and deliver to the endpoint the user specifies.) "Deliver to" has three options: WebHook, Email, and Jabber. After submitting the form, another input is provided, "Authorization message". For Email and Jabber, a user-created message will work great, since they'll be "inviting" themself, or a friend. For WebHook, the instruction will be "If the WebHook you entered requires that you have permission to use it, then get a webhook permission key from that website; otherwise leave this blank." On submitting the form, the "hook" end-point is sent a message -- email includes the message and a link to activate the subscription; jabber includes the message and a link OR instruction to reply with a short given code to activate; http just includes the message, and a 200 response will activate the subscription.

That will accomplish all these transports using essentially the same authorization mechanism. It wouldn't be too hard to add SMS: as long as messages are small.

- daniel parker -

"You have granted me life and steadfast love, and your care has preserved my spirit." Job 10:12
"The LORD is my chosen portion and my cup . . . indeed, I have a beautiful inheritance." Psalm 16:5-6
"Give what you can ... take nothing back!"

[Jeff Lindsay]

This is a big BTW, but the majority of webhook cases *right now* the subscriber owns/wrote the hook script. Hence the "people will probably know what they're doing" ... as well as letting people get "creative" with hook endpoints suggests 200 sounds good to start with. To be honest, in the current webhooks ecosystem, authorizing a hook script isn't necessary because it's a) not public and secure by obscurity, b) owned by the subscriber. Obviously, we do want to consider cases in the future when that's not the case. But I just wanted to point that out.

Also, a couple of people have been talking to me about capability based auth ... since you can wrap auth info into a URL and that becomes a key with certain privileges attached to it.

Anyway, I look forward to playing with your authorization prototype.

-jeff

[Pedro Melo]

On Apr 29, 2009, at 5:52 AM, Daniel Parker wrote:

> So for an HTTP example:
> 1) A <link> tag is included on an HTML page or in an RSS/Atom feed.
> 2) A Subscriber sends a subscription request, which includes a) post-
> back url, b) secret string, c) other subscription options
> 3) The Publisher immediately sends the secret string to a POST
> request to the post-back url. The post-back receiver responds to
> accept or decline the content updates from this Publisher.
> 4) If the post-back receiver responded favorably, the Subscription
> is active and the Publisher begins sending updates.

> We do this with email already every time we sign up for an online
> service. We do similar things with SMS, and it is also very easy to
> do with XMPP.

FYI, the protocol that you suggest has the same logic as dialback
authentication in XMPP federation links.

See http://xmpp.org/rfcs/rfc3920.html#dialback for a complete
specification.

Best regards,