Formal spec

[Chris Obdam]

Hi all,

Is there a formal spec for webhooks somewhere available?

Cheers,

Chris Obdam
Stichting OpenID NL (Dutch OpenID foundation)

[Jeff Lindsay]

Not yet. A lot of work is being done with PubSubHubbub though that is
relevant. What exactly are you looking for from a spec?

[Chris Obdam]

I was thinking about using Webhooks in de OpenID Attribute Exchange,
for the update URL. Which is, in a way, a web hook. A call back URL.

But I want to reffer to a spec from the AX spec.

[Jeff Lindsay]

Well, I'm not sure what you can refer to, especially if you aren't using Hubbub (which doesn't seem like its necessary). I'm trying to carve out some time to make the wiki a better resource for existing conventions that might someday become a standard.

Until then, just roll your own interface and just include it in the AX spec? Maybe you'll come up with something useful for the rest of us.

-jeff

--
Jeff Lindsay
http://webhooks.org -- Make the web more programmable
http://shdh.org -- A party for hackers and thinkers
http://tigdb.com -- Discover indie games
http://progrium.com -- More interesting things

[Nathan]

A wiki would be great, it would be nice to know what others have
attempted and what works well etc.

> Jeff Lindsayhttp://webhooks.org-- Make the web more programmablehttp://shdh.org-- A party for hackers and thinkershttp://tigdb.com-- Discover indie gameshttp://progrium.com-- More interesting things

[Chris Obdam]

A official spec is not needed?

[Troy Davis]

On Fri, Oct 2, 2009 at 9:14 AM, Chris Obdam <chris...@holder.nl> wrote:
> A official spec is not needed?

That's one of the strongest attractions to webhooks, IMO. An official
spec doesn't exist and better, doesn't need to exist.

Troy

[Rick Olson]

On Fri, Oct 2, 2009 at 9:14 AM, Chris Obdam <chris...@holder.nl> wrote:

A official spec is not needed?

Webhooks are a generic term for a variety of techniques.  The only rule is, "send some data to an endpoint that the customer defines."  POSTing values over HTTP is the most common method, but I see no reason why you couldn't choose something else, like Jabber.  

Each service definitely needs some good specs for their webhook implementation, however.  If you expect customers to do amazing things with your service, they'll want something stable and well defined.  Two examples:

Paypal has a very comprehensive spec for Instant Payment Notifications: https://www.paypal.com/ipn

Github describes their Post Receive hooks in a much more straight forward manner: http://github.com/guides/post-receive-hooks

 

--
Rick Olson
http://entp.com
http://lighthouseapp.com
http://tenderapp.com

[Daniel Parker]

I agree that WebHooks is particularly attractive because you can do it however you want, and I agree that there shouldn't be a standard that locks down "how you should use webhooks."

But I also know for a fact that you can accomplish so much more when everyone is working on the same page. For example, the ability to discover webhooks and have your site work with them automatically without customizing how you interact with it for every site you want to work with -- that would be wonderful. A spec is absolutely necessary for that. (So is domain-specific format standards, but that's the next problem and will be taken care of when we get there.) In other words, if we all want to play our own game, it may be fun and we may accomplish a lot, but if we all want to play the same game, we all have to follow the same rules. And that's really all it takes: if we're following the same rules, then we are by definition here playing the same game. The rules are the standards, and if created right, they help us accomplish a specific goal together, that we couldn't accomplish separately.

Without standards we are like the age where everybody does their own thing, and to create a nice piece of equipment you would have to go to 10 different blacksmiths, silversmiths, glass-blowers, etc; and then probably tweak their product a little between steps. With standards, we have a factory with an assembly line. We really aren't efficient until we get to that point. Of course we don't want to force all blacksmiths to work in the factory, but in order to make certain products efficiently, we need a factory. In other words, a standard for each particular end-product.

I have a Google Document where I've placed most of my thoughts of what could/should go into a standard for webhook-based data event subscription (which is what most webhooks are). It's by no means ready to be followed in detail, although you could implement some of it if you wanted to. Here's the link: http://tinyurl.com/webhooks.

(I've also copied that document into a wave at Google Wave as well. If you have an account, add my gmail address as a contact and ask me to share it with you -- that way you can edit too, and I can see your edits and comments and work out the details together.)

- daniel parker -

"You have granted me life and steadfast love, and your care has preserved my spirit." Job 10:12
"The LORD is my chosen portion and my cup . . . indeed, I have a beautiful inheritance." Psalm 16:5-6
"Give what you can ... take nothing back!"

[Nathan]

I agree Daniel,
If you're Sales Force or Google you can implement WebHooks in anyway
you see fit, but for the rest of us a standard, or at least strong
guidelines, go a long way. I like the idea of a standard message
structure, but not necessarily format. I can represent an xml doc with
json so why not RESTfuly offer both formats. What about subscription
forms as XHTML; part of the appeal of WebHooks is in allowing regular
web users to link their web apps. I see value in subscription
resources served up as json/xml for machines or xhtml for humans. I
think a description language like WADL is desirable, I am a fan of
code-gen. I would like to join you on Google wave, I feel like a
schmuck for asking... if you have a spare invitation I would gladly
accept :) (no pressure, I completely understand if you say no).

On Oct 2, 12:05 pm, Daniel Parker <dcpar...@gmail.com> wrote:
> I agree that WebHooks is particularly attractive because you can do it
> however you want, and I agree that there shouldn't be a standard that locks
> down "how you should use webhooks."
> But I also know for a fact that you can accomplish so much more when
> everyone is working on the same page. For example, the ability to discover
> webhooks and have your site work with them automatically without customizing
> how you interact with it for every site you want to work with -- that would
> be wonderful. A spec is absolutely necessary for that. (So is
> domain-specific format standards, but that's the next problem and will be
> taken care of when we get there.) In other words, if we all want to play our
> own game, it may be fun and we may accomplish a lot, but if we all want to
> play the same game, we all have to follow the same rules. And that's really
> all it takes: if we're following the same rules, then we are by definition
> here playing the same game. The rules are the standards, and if created

> right, they *help* us accomplish *a specific goal* together, that we

> couldn't accomplish separately.
> Without standards we are like the age where everybody does their own thing,
> and to create a nice piece of equipment you would have to go to 10 different
> blacksmiths, silversmiths, glass-blowers, etc; and then probably tweak their
> product a little between steps. With standards, we have a factory with an
> assembly line. We really aren't efficient until we get to that point. Of
> course we don't want to force all blacksmiths to work in the factory, but in
> order to make certain products efficiently, we need a factory. In other
> words, a standard for each particular end-product.
>
> I have a Google Document where I've placed most of my thoughts of what

> could/should go into a *standard for webhook-based data event
> subscription* (which

> is what most webhooks are). It's by no means ready to be followed in detail,
> although you could implement some of it if you wanted to. Here's the link:http://tinyurl.com/webhooks.
>
> (I've also copied that document into a wave at Google Wave as well. If you
> have an account, add my gmail address as a contact and ask me to share it
> with you -- that way you can edit too, and I can see your edits and comments
> and work out the details together.)
>
> - daniel parker -
>
> "You have granted me life and steadfast love, and your care has preserved my
> spirit." Job 10:12
> "The LORD is my chosen portion and my cup . . . indeed, I have a beautiful
> inheritance." Psalm 16:5-6
> "Give what you can ... take nothing back!"
>
>
>

> On Fri, Oct 2, 2009 at 12:16 PM, Troy Davis <t...@yort.com> wrote:

[Nathan]

Daniel,
Regarding section 1.4.4 Delivery; why not just put the metadata in the
header and leave the body for "real" payload (image, document, json).
We already have MIME types why repeat the type in the xml container?
Essentially your creating SOAP light. Why not use built in HTTP
headers. Eg.

XML: These should be name/value pairs inside of the xml element
'payload' -- for example, <payload source-url="http://flickr.com/user/
photo/456" cancel-url="http://flickr.com/subscriptions?
id=883421&action=cancel" content-type="image/jpeg">[JPEG-IMAGE-IN-
BASE64-ENCODING]</payload>

Could be:

POST http://www.url.com/
MIME-Version: 1.0
Content-Type: image/jpeg
Content-Location : http://flickr.com/user/photo/456
WebHooks-Cancel-Url : http://flickr.com/subscriptions?id=883421&action=cancel

[JPEG-IMAGE-IN-BASE64-ENCODING]

Or better yet RESTify it:

WebHooks-Url : http://flickr.com/webhooks/883421

unsubscribe:

DELETE http://flickr.com/webhooks/883421

[Jeff Lindsay]

I, too, prefer that data in the headers. However, I know a number of people that prefer the light envelope model. :\

[Chris Obdam]

Great start! Is it possible to put it on the wiki?

That's maybe more accesible?

Cheers,

Chris Obdam

Stichting OpenID NL (Dutch OpenID foundation)

[Chris Obdam]

About the "1.4.1 Discovery" Part:

A HTML tag is suggested, maybe we can use XRDS as discovery mechanism?
Like OpenID: http://xrds-simple.net/

"XRDS-Simple is a protocol for service discovery. The level of
discovery is kept intentionally simple to accommodate existing use
cases and apply established patterns. Service discovery at this level
is technically trivial. After all, how hard can it be to list a bunch
of URIs and add some descriptions next to each one? The only issue is
agreeing on some format that everyone can understand and implement. In
developing a discovery protocol, we are trying to answer two questions:
• How to find information about a resource through its URI?
• What format is this information provided in?"

Op 2 okt 2009, om 16:10 heeft Nathan het volgende geschreven:

>

[Nathan]

XRDS vs WADL. I did not read the full spec just glanced at the example
implementation so I apologize if this is a stupid questions but what
does XRDS have over WADL? Any reason to use XRDS over WADL or are they
considered roughly comparable.?

> >> Jeff Lindsayhttp://webhooks.org--Make the web more programmablehttp://shdh.org

[Daniel Parker]

First of all, I guess I'm glad of the interest. I mean, if several of you are interested in talking about standardizing *something*, then that's great. We do have to be focused on what that very specific *something* is, and make sure that the specific focus is communicated loud and clear.

I've conceded to Jeff's advice in the past to move slowly in the area of standards, giving things time to sort-of grow on their own, time to be tried and tested, new people to find and comment on things as they grow, etc.

In any case, here are my replies to a couple things that were brought up here:

HTTP Headers vs. "light envelope" model.

    I personally like HTTP Headers better --- for HTTP. When I wrote up that spec (it was a while ago) I had settled on the light envelope model. I won't claim that my conclusions are the "right" way, but ... if I remember correctly, this was my thought process: The standard I was working on writing was actually all about the big idea of push subscription as a whole, and that encompasses several different medium: HTTP, XMPP, SMTP, SMS and more. This in itself necessitates that if a single spec can span the various mediums, the payload itself must contain the meta-data related to the subscription content delivered, while the regular medium-specific meta-data should only carry data regarding the transmission of that content. This also makes it easy and simple to relay subscription content from one medium to another without the need to understand and translate headers or other meta-data.

Illustration: Using HTTP Headers in this case might be like putting the newspaper's date on the bag it came in instead of on the actual paper. This makes sense for most types of delivery, but not for newspapers. The content makes the difference.

The same reasons hold for base64-encoding it. Encoding it may easily look silly and extraneous even to me, but it solves the problem of XMPP clients that could be confused by badly-formed XML that could be in the payload body.

- - - -

After this email was sitting in my drafts box for a day waiting for me to finish it, I guess I'll send as-is, see what thoughts you guys have in response.

- daniel parker -

"You have granted me life and steadfast love, and your care has preserved my spirit." Job 10:12
"The LORD is my chosen portion and my cup . . . indeed, I have a beautiful inheritance." Psalm 16:5-6
"Give what you can ... take nothing back!"

[Nathan]

Hey Daniel,
You have some good points. I also think moving slowly and not forcing
standards is a good idea *but* I think it is important to get groups
of people rallying behind different sets of standards. Some of us
working towards a REST standard and perhaps others towards an envelope
standard etc. There are a lot of developers out there that don't want
to re-invent the wheel, they have enough on their plates without
having to create a custom WebHooks implementation. They would rather
see some guideliness and then say "well, that doesn't quite work so
I'll tweak this or that." I think in the end we get more adoption and
better insight because the "best practices" are out in the open for
people to see, criticize and rework. If we have this "open"
hodgepodge, there will be 20 organizations making all the same
mistakes instead of learning from each other and advancing.

Regarding your envelope comments, I think a universal enveloped
payload is still unnecessary even in your example. The reality is that
if I am going to send a text message, IM, etc. I have to change
networks/protocols anyway, I must have some sort of "adapter" to send
a message from HTTP to SMS. Furthermore, if I send my message in a
envelope the target network either has to understand my native
envelope or transform it at which point the envelope is moot. I would
rather see a REST approach where we have adapters that convert a
RESTful HTTP message to an envelope'less SMS message at the same time
moving it onto the target network. Why wrap stuff up in envelopes? Are
there a bunch of different envelopes for SMS (I hope not otherwise
that hurts my point)? If there are not several SMS envelopes then why
wrap-up HTTP in envelopes? To give you an example I am currently
prepping for a project that requires adapters to Quickbooks and other
supply chain tools. I plan to expose RESTful (true REST not REST-RPC
hybrid) services and WebHooks from the core web app. I then plan to
write the adapters as RESTful+WebHooked services. When a sales order
is added to my system the web based Quickbooks adapter is notified
through its WebHooks subscription. This notification gets added to a
queue of notifications that Quickbooks (a desktop adapter) will poll
for updates. Any time Quickbooks changes, the notification will be
posted to my Quickbooks adapter which will publish a WebHooks event
that my core system has subscribed to (this has an added benefit in
that I could allow others to subscribe directly to my web based QB
adapter thereby expanding my sales universe). I can envision the rest
of the web working in this way. I publish a WebHooks event that an SMS
adapter has subscribed to, it sends the text message using native
protocols (no envelopes); my target texts me back, the SMS adapter
publishes a WebHooks message that I subscribe to. Lets be honest about
what our protocol is/supports and use adapters to move those messages
to other protocols instead of creating yet another xml envelope
standard.

My ten cents and then some :)
Nathan

On Oct 4, 8:58 pm, Daniel Parker <dcpar...@gmail.com> wrote:
> First of all, I guess I'm glad of the interest. I mean, if several of you
> are interested in talking about standardizing *something*, then that's
> great. We do have to be focused on what that very specific *something* is,
> and make sure that the specific focus is communicated loud and clear.
> I've conceded to Jeff's advice in the past to move slowly in the area of
> standards, giving things time to sort-of grow on their own, time to be tried
> and tested, new people to find and comment on things as they grow, etc.
>
> In any case, here are my replies to a couple things that were brought up
> here:
>
> HTTP Headers vs. "light envelope" model.
>     I personally like HTTP Headers better --- for HTTP. When I wrote up that
> spec (it was a while ago) I had settled on the light envelope model. I won't
> claim that my conclusions are the "right" way, but ... if I remember
> correctly, this was my thought process: The standard I was working on
> writing was actually all about the big idea of push subscription as a whole,
> and that encompasses several different medium: HTTP, XMPP, SMTP, SMS and
> more. This in itself necessitates that if a single spec can span the various
> mediums, the payload itself must contain the meta-data related to the

> *subscription
> content delivered*, while the regular medium-specific meta-data should only
> carry data regarding the *transmission* of that content. This also makes it

> easy and simple to relay subscription content from one medium to another
> without the need to understand and translate headers or other meta-data.
>
> Illustration: Using HTTP Headers in this case might be like putting the
> newspaper's date on the bag it came in instead of on the actual paper. This
> makes sense for most types of delivery, but not for newspapers. The content
> makes the difference.
>
> The same reasons hold for base64-encoding it. Encoding it may easily look
> silly and extraneous even to me, but it solves the problem of XMPP clients

> that could be confused by badly-formed XML that *could* be in the payload

> > > >> Jeff Lindsayhttp://webhooks.org--Makethe web more programmablehttp://
> > shdh.org

[Jeff Lindsay]

I want to start revamping the almost entirely untouched webhooks wiki (wiki.webhooks.org) ... I'd love it if you guys could use that (just make new pages) for talking about these patterns and conventions with pros/cons etc. Eventually I'll start populating it with tons of sites, example implementations, tools, and all that great stuff that SHOULD be on a wiki. It would help if you guys helped get it started. :)

[Nathan]

To get involved in defining RESTful WebHooks visit:
http://webhooks.pbworks.com/RESTful-WebHooks

> > > > > >> Jeff Lindsayhttp://webhooks.org--Maketheweb more

> > programmablehttp://
> > > > shdh.org
> > > > > >> -- A party for hackers and thinkershttp://tigdb.com--Discover
> > > > > >> indie gameshttp://progrium.com--Moreinterestingthings
>

> --
> Jeff Lindsayhttp://webhooks.org-- Make the web more programmablehttp://shdh.org-- A party for hackers and thinkershttp://tigdb.com-- Discover indie gameshttp://progrium.com-- More ...
>
> read more »

[Daniel Parker]

Yeah, I got the other email. Try not to post twice.
- daniel parker -

[Chris Obdam]

Nathan,

I'm not familiar with WADL. The reason to use XRDS is that it's simple
and easy to use and it's already used in OpenID.

As far as i've read, WADL is more in line with WSDL, while XRDS is
more on the server side.

Example of a file: http://en.wikipedia.org/wiki/XRDS#Example_XRDS_Document

Cheers,

Chris

[Chris Obdam]

Hi all,

> You have some good points. I also think moving slowly and not forcing
> standards is a good idea *but* I think it is important to get groups
> of people rallying behind different sets of standards. Some of us
> working towards a REST standard and perhaps others towards an envelope
> standard etc. There are a lot of developers out there that don't want
> to re-invent the wheel, they have enough on their plates without
> having to create a custom WebHooks implementation. They would rather
> see some guideliness and then say "well, that doesn't quite work so
> I'll tweak this or that." I think in the end we get more adoption and
> better insight because the "best practices" are out in the open for
> people to see, criticize and rework. If we have this "open"
> hodgepodge, there will be 20 organizations making all the same
> mistakes instead of learning from each other and advancing.

Totally agree. I was fairly new to webhooks, knew the abstracts but
didn't knew how and where to start. The current WIKI is a great start!
If we can try to work towards a 'official' spec that would clear
things out for a lot of people.

And.... We can incorporate in into other standards? I guess that
that's a valid argument too?

I don't quite understand how the envelope should work, but as I have
seen the HTTP header I think that's an easy and clear approach.
Don't know what's against it, but in case you wanted to know: I like
the HTTP headers.. ;-)

Cheers,

Chris

[Daniel Parker]

I hadn't replied on this yet -- Your point on the envelopes is right on: We don't need another envelope standard, even if it is very light. Now, someone *could* have a use case where envelope works best, but for the most part we shouldn't need it. Regular HTTP headers (used REST-fully) works fine in most cases.

- daniel parker -

"You have granted me life and steadfast love, and your care has preserved my spirit." Job 10:12
"The LORD is my chosen portion and my cup . . . indeed, I have a beautiful inheritance." Psalm 16:5-6
"Give what you can ... take nothing back!"

[Daniel Parker]

The current WIKI is a great start! If we can try to work towards a 'official' spec that would clear things out for a lot of people.

One thing I want to clarify for everyone: In making a spec, there are going to be a lot of differing opinions, because many people will have different values, goals, purposes and use cases in mind. According to Roy Fielding, the originator of REST, to define a new application protocol (WAKA presentation, slide #2), you must "Document desired architectural properties" -- meaning, what we're going for, the purpose and intent of what the protocol should achieve.

There will ideally be several specs / standards that come out of our discussions. I am purposing to work on one of them, one whose purpose is to integrate with the existing fabric of HTTP and REST, while being stupid-simple to understand and implement on most sensible platforms.

- - - - - - -

That said, I'd like to ask if anyone knows of a better resource that can be used for a "collaborative wiki".

• I think Google Wave might be the best solution, but not everyone has an account. (However, I do think we get invitations to share after you've been on a few days, so perhaps we could each agree to invite 1 person.)
• Github might also be an option, as we can fork each others' pages and edit them, but it is a little limiting to those who know how to operate it, and git would be a little clunky for viewing differences, etc.

Any better ideas? A web app I don't know about? Or should we go with Google Wave for great dynamic collaboration and document-creation all in one? I think we can embed the wave on a page, maybe even the wiki page, for live publication.

[Nathan]

Wave would be AWSOME and I could promise to reserve at least one
invitation for WebHook participants.

On Oct 7, 10:50 am, Daniel Parker <dcpar...@gmail.com> wrote:
> > The current WIKI is a great start! If we can try to work towards a
> > 'official' spec that would clear things out for a lot of people.
>
> One thing I want to clarify for everyone: In making a spec, there are going
> to be a lot of differing opinions, because many people will have different
> values, goals, purposes and use cases in mind. According to Roy Fielding,
> the originator of REST, to define a new application protocol (WAKA

> presentation<http://www.slideshare.net/nilezparker/200211-fielding-apachecon>,

> slide #2), you must "Document desired architectural properties" -- meaning,
> what we're going for, the purpose and intent of what the protocol should
> achieve.
>
> There will ideally be several specs / standards that come out of our
> discussions. I am purposing to work on one of them, one whose purpose is to
> integrate with the existing fabric of HTTP and REST, while being
> stupid-simple to understand and implement on most sensible platforms.
>
> - - - - - - -
>
> That said, I'd like to ask if anyone knows of a better resource that can be
> used for a "collaborative wiki".
>

>    - I think Google Wave might be the best solution, but not everyone has an

>    account. (However, I do think we get invitations to share after you've been
>    on a few days, so perhaps we could each agree to invite 1 person.)

>    - Github might also be an option, as we can fork each others' pages and

[Chris Obdam]

All more the reason to participate! :-)

[Brad Jones]

I suggest the latest MindTouch core – 9.08:  http://www.mindtouch.com/ . It’s open source, strong community, but you will need to find a place to host it. – Brad Jones

[Jeff Lindsay]

I would prefer it if we just used the webhooks wiki: http://wiki.webhooks.org

[Jeff Lindsay]

WADL is not very popular. I went to the wikipedia and clicked links to official sites and they were all down. The only one that worked was this post by Joe Gregario saying it shouldn't be used: http://bitworking.org/news/193/Do-we-need-WADL

Personally, I don't think we *need* the complexity XRDS brings. If all we're trying to solve is discovery, I think we can create a simple, conventions oriented way that's specific to our needs. PubSubHubbub does this quite well. Their discovery mechanism? A link tag in feeds that use a URL to the Hub endpoint with rel "hub". That's all they need. And that works quite well. I'd start there.

XRDS ... again, I've seen specs that are 1/10 the complexity of XRDS that never get picked up because they had to first introduce XRDS as the discovery mechanism. I think XRDS doesn't get crap from OAuth because OAuth is actually more complicated. (Ugh)

[Daniel Parker]

Just to keep others in the loop, we're looking at this idea for discovery and subscription:

https://gist.github.com/780ccb1fa8943700515a

No complicated XRDS necessary. Just simple HTTP headers. Only a few pieces of information, and it seems to us to make sense with the usual HTTP + REST semantics.

- daniel parker -

"You have granted me life and steadfast love, and your care has preserved my spirit." Job 10:12
"The LORD is my chosen portion and my cup . . . indeed, I have a beautiful inheritance." Psalm 16:5-6
"Give what you can ... take nothing back!"