Using ServiceStack + Redis for Message Queue, Translation, and WebHook callbacks?

[Joshua Gough]

I've been chatting on jabbr about this with Mythz.

I'm trying to figure out how I can approach this with SS and Redis in mind.

I want to have a web service that:

1) Listens for POST bodies at myserver/CommitService that will be coming from various cloud-hosted source control systems, like GitHub and BitBucket, after repo commits. Example POST body: https://help.github.com/articles/post-receive-hooks

1.1) In customized cases, these POST bodies may also include a ReplyTo URL

2) Queues up the raw POST into a "To Be Translated and Processed" queue -- and sends an immediate response with a status URL for clients to GET for later status info.

3) The queue processor does something like the "Message Translator" EIP pattern to convert from whatever format came in to a common "Commit" DTO.

4) Passes the Commit dto to a service for internal processing.

5) After this processing is done, our system will:

a) If ReplyTo was specified, send a POST to it.

b) If the user has registered ReplyTo urls (basically WebHooks, http://en.wikipedia.org/wiki/Webhook) ahead of time, we'll POST to them instead.

Those servers are then free to do whatever they please in response to knowing that we have successfully processed the Commit. Typically, they would examine some data and turn around and make a REST API call back into our systems to modify something.

I also want steps a) and b) to done with Retries, because I don't want outbound callbacks to get dumped if HTTP errors or network failures happen. And, any outbound callbacks that ultimately fail, I want to be in the DLQ for debugging and follow up.

I want step 3 to leave the messages or move them into an "unknown" queue or something, such that I can examine it for failed conversions, but also so I can tell customers to point their WebHooks at us temporarily so we can capture their formats, and then implement our plugin translator for it.

After talking with Mythz, it sounds like the capabilities here: https://github.com/ServiceStack/ServiceStack/wiki/Messaging-and-redis, are a good place to start, but I am not entirely sure what the nuts and bolts will look like.

I thought about using RequestBinders.Add(...) to do the initial translate, and load translator plugins via MEF, but I want to get that incoming POST into a queue as soon as possible and not lose it, so not sure that RequestBinders is the right thing to start with.

I will make it post a message into the ServiceStack chatroom and say "ServiceStack wuz here" though :-P

Thanks!

Josh