Cases for Web Hooks vs XMPP?
115 views
Skip to first unread message
Leah Culver
Nov 11, 2008, 1:33:16 PM11/11/08
to Web Hooks
Hi all,
Mike (other Pownce dev) and I had a good chat about web hooks vs.
XMPP. Most web services with public APIs (Flickr, Pownce, Twitter
etc...) will probably lean a bit towards XMPP due to the high volume
of updates.
However, something to note is that web hooks are significantly easier
for the client if they are already running a webserver. PayPal is a
great example because they can assume the client is an online store
(shopping!). It's significanly easier to use a web hook than expect
the client to run an XMPP server.
Another example we thought of was blogs and feed readers. A blog can
assume a feed reader is running a webserver and registering a callback
for a feedreader makes some sense (the push to RSS's pull). Mike
points to http://en.wikipedia.org/wiki/Ping_(blogging)
So, let's not say this is good solution for Twitter talking with
Pownce, Jaiku, or FriendFeed because we're likely to get spanked.
Instead, Mike and I see web hooks as useful with the following
criteria:
- can reasonably assume the client is running a web server
- pings are public (or obscured by a messy URL, as pointed out by Cal)
- low volume, not critical, or easy to re-try
Thoughts?
Leah
P.S.
However, Twitter is already behaving a bit like a client since they
publish a REST endpoint for accepting pings. FriendFeed will ping this
endpoint for individual users based on user pref. So if Twitter had
registered their endpoint with FriendFeed (as oppose to FriendFeed
special casing this), you could probably call it a web hook.
Mike (other Pownce dev) and I had a good chat about web hooks vs.
XMPP. Most web services with public APIs (Flickr, Pownce, Twitter
etc...) will probably lean a bit towards XMPP due to the high volume
of updates.
However, something to note is that web hooks are significantly easier
for the client if they are already running a webserver. PayPal is a
great example because they can assume the client is an online store
(shopping!). It's significanly easier to use a web hook than expect
the client to run an XMPP server.
Another example we thought of was blogs and feed readers. A blog can
assume a feed reader is running a webserver and registering a callback
for a feedreader makes some sense (the push to RSS's pull). Mike
points to http://en.wikipedia.org/wiki/Ping_(blogging)
So, let's not say this is good solution for Twitter talking with
Pownce, Jaiku, or FriendFeed because we're likely to get spanked.
Instead, Mike and I see web hooks as useful with the following
criteria:
- can reasonably assume the client is running a web server
- pings are public (or obscured by a messy URL, as pointed out by Cal)
- low volume, not critical, or easy to re-try
Thoughts?
Leah
P.S.
However, Twitter is already behaving a bit like a client since they
publish a REST endpoint for accepting pings. FriendFeed will ping this
endpoint for individual users based on user pref. So if Twitter had
registered their endpoint with FriendFeed (as oppose to FriendFeed
special casing this), you could probably call it a web hook.
Mike Malone
Nov 11, 2008, 2:15:38 PM11/11/08
to Web Hooks
Hey, this is Mike (other Pownce dev),
I didn't see the Hackers talk, so some of this may have been discussed
already, but here are my thoughts...
It seems like there are only a limited number of cases where a "web
hook" type callback would be more appropriate than a persistent
connection like XMPP. The only real advantage I see is simplicity. If
you're already running a web server, handling an HTTP callback is
straightforward, while setting up an XMPP server (or even running a
client) is significantly more complicated. Aside from that, XMPP (or
something similar) seems more reliable, easier to scale, and useful to
a wider variety of clients (trying to run a web server to receive
callbacks for a desktop or mobile app seems silly to me).
So I guess my point is (and I'm playing a bit of devil's advocate
here): how would you sell me on this vs. XMPP? If, hypothetically, I
was a web developer at Pownce and was trying to build a better event
notification mechanism for the Pownce API, why would I use "web hooks"
instead of XMPP? Or would I use both?
Mike
P.S. All that said, I think PayPal is a great example of the type of
situation where this pattern works. The "clients" tend to already be
running web servers, there aren't a lot of them, and event volume
isn't terribly high. Surely there are similar situations where web
hooks could be used, but aren't, so I think generating some discussion
around the topic is definitely worthwhile.
On Nov 11, 10:33 am, Leah Culver <l...@pownce.com> wrote:
> Hi all,
>
> Mike (other Pownce dev) and I had a good chat about web hooks vs.
> XMPP. Most web services with public APIs (Flickr, Pownce, Twitter
> etc...) will probably lean a bit towards XMPP due to the high volume
> of updates.
>
> However, something to note is that web hooks are significantly easier
> for the client if they are already running a webserver. PayPal is a
> great example because they can assume the client is an online store
> (shopping!). It's significanly easier to use a web hook than expect
> the client to run an XMPP server.
>
> Another example we thought of was blogs and feed readers. A blog can
> assume a feed reader is running a webserver and registering a callback
> for a feedreader makes some sense (the push to RSS's pull). Mike
> points tohttp://en.wikipedia.org/wiki/Ping_(blogging)
I didn't see the Hackers talk, so some of this may have been discussed
already, but here are my thoughts...
It seems like there are only a limited number of cases where a "web
hook" type callback would be more appropriate than a persistent
connection like XMPP. The only real advantage I see is simplicity. If
you're already running a web server, handling an HTTP callback is
straightforward, while setting up an XMPP server (or even running a
client) is significantly more complicated. Aside from that, XMPP (or
something similar) seems more reliable, easier to scale, and useful to
a wider variety of clients (trying to run a web server to receive
callbacks for a desktop or mobile app seems silly to me).
So I guess my point is (and I'm playing a bit of devil's advocate
here): how would you sell me on this vs. XMPP? If, hypothetically, I
was a web developer at Pownce and was trying to build a better event
notification mechanism for the Pownce API, why would I use "web hooks"
instead of XMPP? Or would I use both?
Mike
P.S. All that said, I think PayPal is a great example of the type of
situation where this pattern works. The "clients" tend to already be
running web servers, there aren't a lot of them, and event volume
isn't terribly high. Surely there are similar situations where web
hooks could be used, but aren't, so I think generating some discussion
around the topic is definitely worthwhile.
On Nov 11, 10:33 am, Leah Culver <l...@pownce.com> wrote:
> Hi all,
>
> Mike (other Pownce dev) and I had a good chat about web hooks vs.
> XMPP. Most web services with public APIs (Flickr, Pownce, Twitter
> etc...) will probably lean a bit towards XMPP due to the high volume
> of updates.
>
> However, something to note is that web hooks are significantly easier
> for the client if they are already running a webserver. PayPal is a
> great example because they can assume the client is an online store
> (shopping!). It's significanly easier to use a web hook than expect
> the client to run an XMPP server.
>
> Another example we thought of was blogs and feed readers. A blog can
> assume a feed reader is running a webserver and registering a callback
> for a feedreader makes some sense (the push to RSS's pull). Mike
Andy Kent
Nov 11, 2008, 2:43:33 PM11/11/08
to Web Hooks
I think both web hooks and XMPP have a place in a push ecosystem.
The key to web hooks lie in their simplicity, it's such a low barrier
of entry to consume services but obviously that can raise scalability
issues. XMPP gives you a lot more but this comes at a cost of
significantly raising the barrier of entry for third parties.
I like to think of web hooks vs XMPP as a similar question to
microformats vs RDF. XMPP is the 'proper' way but web hooks is the
simple solution for end users that can help solve problems in the real
world and get people using push messaging right now. Unfortunately
it's not quite that easy when it comes to scaling web hooks on the
distribution end, having said that several high volume sites such as
PayPal, Shopify and of course Google have managed to make this work
quite successfully.
Another way of thinking about web hooks is to consider them as a more
machine friendly alternative to email, emails to users, hooks to
machines. The hook provides additional flexibility to consume and
redirect messages in real time to whatever outputs are required.
XMPP is a great protocol but it's not without it's issues either, the
gnip guys troubles spring to mind http://blog.gnipcentral.com/2008/11/03/winding-down-xmpp-for-now/
on the other hand web hooks work over plain http which is one of the
most understood protocols in existence.
As you both mentioned, not every API makes a suitable candidate for
web hooks. I'm not trying to sell web hooks to you but I do personally
believe they have a big part to play in the future of push API's.
Andy
The key to web hooks lie in their simplicity, it's such a low barrier
of entry to consume services but obviously that can raise scalability
issues. XMPP gives you a lot more but this comes at a cost of
significantly raising the barrier of entry for third parties.
I like to think of web hooks vs XMPP as a similar question to
microformats vs RDF. XMPP is the 'proper' way but web hooks is the
simple solution for end users that can help solve problems in the real
world and get people using push messaging right now. Unfortunately
it's not quite that easy when it comes to scaling web hooks on the
distribution end, having said that several high volume sites such as
PayPal, Shopify and of course Google have managed to make this work
quite successfully.
Another way of thinking about web hooks is to consider them as a more
machine friendly alternative to email, emails to users, hooks to
machines. The hook provides additional flexibility to consume and
redirect messages in real time to whatever outputs are required.
XMPP is a great protocol but it's not without it's issues either, the
gnip guys troubles spring to mind http://blog.gnipcentral.com/2008/11/03/winding-down-xmpp-for-now/
on the other hand web hooks work over plain http which is one of the
most understood protocols in existence.
As you both mentioned, not every API makes a suitable candidate for
web hooks. I'm not trying to sell web hooks to you but I do personally
believe they have a big part to play in the future of push API's.
Andy
Timothy Fitz
Nov 11, 2008, 4:20:36 PM11/11/08
to webh...@googlegroups.com
On Tue, Nov 11, 2008 at 11:15 AM, Mike Malone <mjma...@gmail.com> wrote:
> So I guess my point is (and I'm playing a bit of devil's advocate
> here): how would you sell me on this vs. XMPP? If, hypothetically, I
> was a web developer at Pownce and was trying to build a better event
> notification mechanism for the Pownce API, why would I use "web hooks"
> instead of XMPP? Or would I use both?
> So I guess my point is (and I'm playing a bit of devil's advocate
> here): how would you sell me on this vs. XMPP? If, hypothetically, I
> was a web developer at Pownce and was trying to build a better event
> notification mechanism for the Pownce API, why would I use "web hooks"
> instead of XMPP? Or would I use both?
I think there are two arguments.
One is simplicity. (Andy's response is good, here's my 2c) Not for
simplicities sake, but because simplicity means dramatically larger
amounts of users who will benefit from the feature. The easiest
example of this is an SMS notification service. I register there, and
plug their URL into your service. Done.
The second is infrastructure. XMPP is an unknown beast. The biggest
player (Google) uses a proprietary implementation. The rest of the
ecosystem is underdeveloped. Where's my rock solid XMPP hardware load
balancer? Where are my debugging tools? Where are my simple servers?
My documentation?
I'm all for using new protocols when warranted, but I just don't see
why XMPP is better than HTTP. Exactly what features of XMPP made you
choose it over HTTP?
We did the math in-house (at IMVU) where we do chat / realtime
messaging over HTTP already, and the only reason to switch to XMPP was
that ejabberd solves the scalability problems behind presence
information (who's online status), a factor that doesn't affect your
decision here.
Jeff Lindsay
Nov 11, 2008, 5:46:13 PM11/11/08
to Web Hooks
Leah and everybody,
I'm not entirely convinced web hooks aren't a good solution for
Twitter talking to Pownce, Jaiku, etc, but you're probably right.
However, I wouldn't assume the use cases for web hooks are limited to
where users have a web server.
The great thing about the simplicity of web hooks is that you *can*
register your own handler that you wrote and put on your own web
server (or on App Engine, or AppJet, or random cheap PHP hosting), but
that's obviously for the advanced users that can write simple handler
scripts. However, it very likely that there would be third parties out
there that would provide you with a handler endpoint that gives you
some great functionality without you needing a web server.
Tim's example is a web hook to SMS provider/adapter. They give you a
custom endpoint (post-receive URL as some call them) that you can drop
into whatever application, and then it magically takes the payload and
gives you some kind of SMS message to your phone. (In fact, since SMS
can be sent by email, you Switchub guys could probably put together an
SMS output handler pretty easily (just email to their number at all
the major networks or let them select their network in
configuration)).
But another great example of an actual service that gives value to end
users that don't have a web server is Scrobbld (http://
www.scrobbld.com), which integrates with PayPal without the need of
any username or password (simpler integration/authentication model)
and gives you all this extra value thanks to IPN (as a web hook).
One thing to keep in mind about comparing most of these web hook use
cases to trackback pings (and I'm working on a blog post about this),
is that the security model is inverted. With Trackbacks, a blog will
advertise (or will be known by convention) the URL to post to in a
public way. This is why there is a problem with spam. But with most
web hook use cases, the URL is not conventional or publicly
advertised. Unless you can crawl to it, the only people that know it
are the owner and the service you give it to, which we assume is not
going to spam you, but instead post useful and expected data (like
PayPal IPN--although you can argue whether it's all expected, but
that's PayPal's fault for lack of clear documentation).
-jeff
I'm not entirely convinced web hooks aren't a good solution for
Twitter talking to Pownce, Jaiku, etc, but you're probably right.
However, I wouldn't assume the use cases for web hooks are limited to
where users have a web server.
The great thing about the simplicity of web hooks is that you *can*
register your own handler that you wrote and put on your own web
server (or on App Engine, or AppJet, or random cheap PHP hosting), but
that's obviously for the advanced users that can write simple handler
scripts. However, it very likely that there would be third parties out
there that would provide you with a handler endpoint that gives you
some great functionality without you needing a web server.
Tim's example is a web hook to SMS provider/adapter. They give you a
custom endpoint (post-receive URL as some call them) that you can drop
into whatever application, and then it magically takes the payload and
gives you some kind of SMS message to your phone. (In fact, since SMS
can be sent by email, you Switchub guys could probably put together an
SMS output handler pretty easily (just email to their number at all
the major networks or let them select their network in
configuration)).
But another great example of an actual service that gives value to end
users that don't have a web server is Scrobbld (http://
www.scrobbld.com), which integrates with PayPal without the need of
any username or password (simpler integration/authentication model)
and gives you all this extra value thanks to IPN (as a web hook).
One thing to keep in mind about comparing most of these web hook use
cases to trackback pings (and I'm working on a blog post about this),
is that the security model is inverted. With Trackbacks, a blog will
advertise (or will be known by convention) the URL to post to in a
public way. This is why there is a problem with spam. But with most
web hook use cases, the URL is not conventional or publicly
advertised. Unless you can crawl to it, the only people that know it
are the owner and the service you give it to, which we assume is not
going to spam you, but instead post useful and expected data (like
PayPal IPN--although you can argue whether it's all expected, but
that's PayPal's fault for lack of clear documentation).
-jeff
> Hi all,
>
> Mike (other Pownce dev) and I had a good chat about web hooks vs.
> XMPP. Most web services with public APIs (Flickr, Pownce, Twitter
> etc...) will probably lean a bit towards XMPP due to the high volume
> of updates.
>
> However, something to note is that web hooks are significantly easier
> for the client if they are already running a webserver. PayPal is a
> great example because they can assume the client is an online store
> (shopping!). It's significanly easier to use a web hook than expect
> the client to run an XMPP server.
>
> Another example we thought of was blogs and feed readers. A blog can
> assume a feed reader is running a webserver and registering a callback
> for a feedreader makes some sense (the push to RSS's pull). Mike
> points tohttp://en.wikipedia.org/wiki/Ping_(blogging)
>
> Mike (other Pownce dev) and I had a good chat about web hooks vs.
> XMPP. Most web services with public APIs (Flickr, Pownce, Twitter
> etc...) will probably lean a bit towards XMPP due to the high volume
> of updates.
>
> However, something to note is that web hooks are significantly easier
> for the client if they are already running a webserver. PayPal is a
> great example because they can assume the client is an online store
> (shopping!). It's significanly easier to use a web hook than expect
> the client to run an XMPP server.
>
> Another example we thought of was blogs and feed readers. A blog can
> assume a feed reader is running a webserver and registering a callback
> for a feedreader makes some sense (the push to RSS's pull). Mike
Mike Malone
Nov 11, 2008, 9:32:13 PM11/11/08
to Web Hooks
Hey Tim,
At the risk of making this thread become about XMPP I'll respond to a
couple of your points (I can't help myself :).
> The second is infrastructure. XMPP is an unknown beast. The biggest
> player (Google) uses a proprietary implementation. The rest of the
> ecosystem is underdeveloped. Where's my rock solid XMPP hardware load
> balancer? Where are my debugging tools? Where are my simple servers?
> My documentation?
I agree that much of the XMPP infrastructure is underdeveloped, but
there are many promising projects in the space. Despite its obtuse
documentation and stupidly complicated configuration ejabberd is a
robust server that is more or less battle hardened (jabber.org runs
it). Other projects exist to develop various tools and utilities
(https://launchpad.net/poetry comes to mind). There probably won't be
any truly "simple" servers (like thttpd simple) since the protocol is
significantly more complicated than HTTP, but there's no shortage of
tools for working in XMPP in your language of choice (sleekxmpp is
great in Python, and I've heard good things about XMPP4R for Ruby).
> I'm all for using new protocols when warranted, but I just don't see
> why XMPP is better than HTTP. Exactly what features of XMPP made you
> choose it over HTTP?
The simple answer is that HTTP was designed as a stateless protocol
where clients make requests to servers and servers respond. XMPP, on
the other hand, was designed for two-way, persistent, stateful, real-
time communication. There's generally lower overhead with XMPP once
the connection is established, it should be easier to scale (once you
get it set up), and a lot of thought has been put into designing the
protocol and the various extensions (e.g., xep-0060 pubsub, xep-0206
bosh, xep-0114 components). The result is a robust mechanism for event
notifications with a bunch of well thought out extras like logging and
presence.
Mike
At the risk of making this thread become about XMPP I'll respond to a
couple of your points (I can't help myself :).
> The second is infrastructure. XMPP is an unknown beast. The biggest
> player (Google) uses a proprietary implementation. The rest of the
> ecosystem is underdeveloped. Where's my rock solid XMPP hardware load
> balancer? Where are my debugging tools? Where are my simple servers?
> My documentation?
there are many promising projects in the space. Despite its obtuse
documentation and stupidly complicated configuration ejabberd is a
robust server that is more or less battle hardened (jabber.org runs
it). Other projects exist to develop various tools and utilities
(https://launchpad.net/poetry comes to mind). There probably won't be
any truly "simple" servers (like thttpd simple) since the protocol is
significantly more complicated than HTTP, but there's no shortage of
tools for working in XMPP in your language of choice (sleekxmpp is
great in Python, and I've heard good things about XMPP4R for Ruby).
> I'm all for using new protocols when warranted, but I just don't see
> why XMPP is better than HTTP. Exactly what features of XMPP made you
> choose it over HTTP?
where clients make requests to servers and servers respond. XMPP, on
the other hand, was designed for two-way, persistent, stateful, real-
time communication. There's generally lower overhead with XMPP once
the connection is established, it should be easier to scale (once you
get it set up), and a lot of thought has been put into designing the
protocol and the various extensions (e.g., xep-0060 pubsub, xep-0206
bosh, xep-0114 components). The result is a robust mechanism for event
notifications with a bunch of well thought out extras like logging and
presence.
Mike
Mike Malone
Nov 11, 2008, 9:35:33 PM11/11/08
to Web Hooks
> Tim's example is a web hook to SMS provider/adapter. They give you a
> custom endpoint (post-receive URL as some call them) that you can drop
> into whatever application, and then it magically takes the payload and
> gives you some kind of SMS message to your phone.
Interesting... so knowledge of the URL would be the authentication?> custom endpoint (post-receive URL as some call them) that you can drop
> into whatever application, and then it magically takes the payload and
> gives you some kind of SMS message to your phone.
Andy Kent
Nov 14, 2008, 7:21:08 AM11/14/08
to Web Hooks
> Interesting... so knowledge of the URL would be the authentication?
Yep, this is how we make switchub.com work as well.(you can use the
code HEAD08 if you want to have a play)
As this thread seems to be bringing up some interesting points around
XMPP and scaling which is a combined area I will admit to not having a
vast amount of experience, I sought some advice from a friend who
does. It seems that XMPP is not without it's scaling headaches either
and in some situations web hooks may actually scale better.
From what he was saying it seems XMPP works very well in situations
where you have a steady number of clients and a regular flow of
messages but it struggles with unknown quantities and spikes. eJabberd
has a limit of around 3000 connections per machine which sets a
ceiling based on the number of machines you have available, you need
to ensure you have enough available connections to allow all clients
to connect and once connected a client is tethered to a physical
machine making it hard to dynamically change the cluster size. Compare
this to web hooks where pings can be queued and the queue can be
distributed across a variable size cluster of machines, whilst each
individual delivery would likely be slower the load can be more
intelligently distributed over both machines and time.
This debate has really caught my interest so I'd love to hear more
opinions especially if anyone has some solid numbers around all of
this speculation.
Andy.
Reply all
Reply to author
Forward
0 new messages