Deferred HTTP, or HTTP Futures

[Jeff Lindsay]

I've been coming across a special case of webhooks for an asynchronous version of a standard request/response cycle. In particular, my use case is using RPC (JSON-RPC *or* XML-RPC, hence this shouldn't be a specific extension to either) that may take days or weeks to respond. A normal long-lived HTTP connection will not do the trick. Webhooks are ideal here, obviously.

The SIMPLEST implementation of this concept looks like this:

--> Request (GET, POST, PUT, or DELETE)

[standard headers]

Link: <callback-url>; rel="callback"

[arbitrary body]

<-- 202 Accepted Response

[ignore body]

Then later, when the response is available, the responding server above makes a request (webhook):

--> POST Request

[standard headers]

X-Status: 200

[response body]

<-- 200 OK (else perhaps fall-off retry)

And that's it. So in a nutshell, it makes HTTP async over long periods of time. This basically gives us the semantics familiar in evented programming (Twisted, EventMachine, nodejs, etc). 

Optionally, perhaps borrowing the Status-URI header, we could give a URI that represents the deferred response / HTTP future (as well as help tie the callback request to the initial request). That allows you to query the status of the deferred/future, but also possible exposes semantics to cancel it by calling DELETE on it. 

Thoughts? I don't want it to be complicated...

--
Jeff Lindsay
http://progrium.com

[Martyn Loughran]

Jeff,

Just saw this message, not been following the list...

I really like this idea!

In many applications webhooks are an all or nothing thing. I typically
configure a callback url for a given event in the system which I'm
interested in and it will be delivered to me forever more. This allows
a much more selective approach.

A variation would be an event emitter style abstraction - something
like X-Event: your-status rather than X-Status: 200. The use of http
status codes you suggest is neat and elegant, but I'm not convinced
that it's actually that useful! In general I find event emitters much
more flexible.

I wouldn't recommend adding the DELETE functionality on the callback
url in most cases - it would make the implementation for the callback
provider much more complex, and I can't see it being useful very
often. But it's a nice idea!

All the best,

Martyn

> --
> You received this message because you are subscribed to the Google Groups
> "WebHooks" group.
> To post to this group, send email to webh...@googlegroups.com.
> To unsubscribe from this group, send email to
> webhooks+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/webhooks?hl=en.
>

[Jeff Lindsay]

The reason it was using HTTP status codes was because this is literally for async HTTP. It's not necessarily a general event system pattern, even though it could probably be used that way. But I actually want to get status 200 or whatever in the callback because you need that if you're going to be using this for async HTTP request/response.

[eli]

Is this something you could implement using the TaskQueue api on
appengine? (I'm trying to figure out if I understand the specific use
case you are describing)

Mainly..

1. Send your RPC to a task handler on Appengine that includes where
to send the completed work.
2. Task handler takes request, builds queue item for future work,
adds it to task queue, and returns the name of the queue item to you.
3. At future date, you receive your response.

Thus.. if you wanted the status of your work item (or wanted to delete
it).. you could just query the handler and give it the name of your
task queue item etc.

Though, are you suggesting maybe some formalized definition of this
process? (or is it something different than just queued work to be
processed in the future?)

> > > webhooks+u...@googlegroups.com<webhooks%2Bunsubscribe@googlegroups.c om>
> > .

> > > For more options, visit this group at
> > >http://groups.google.com/group/webhooks?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "WebHooks" group.
> > To post to this group, send email to webh...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > webhooks+u...@googlegroups.com<webhooks%2Bunsubscribe@googlegroups.c om>
> > .

[Jeff Lindsay]

It's not really about explicitly queuing work. There could be a queue behind it. I imagine a gateway from HTTP to Deferred HTTP might do that, but it would have to hold the original connection open (which app engine isn't suitable for). Mongrel2 is probably more likely to be useful for implementing this. But pretty much any evented server could do it pretty easily. I'm just specing out what could be implemented to serve this purpose (going from sync req/resp to async req/resp).

To unsubscribe from this group, send email to webhooks+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/webhooks?hl=en.