subscriber programming style
Joe Williams
to json and sends it as a message to rabbitmq. Then I have subscriber
processes pop messages from the queue and parse the json and do
something with it. My interest is on the subscriber side of things,
currently the json is parsed and I have a case statement that determines
what happens next based on a field in the json. It seems hacky to have a
case with more than a couple clauses but I am unsure how to make it
cleaner. How are most folks doing this sort of thing? Any suggestions?
Thanks for the advice.
-Joe
--
Name: Joseph A. Williams
Email: j...@joetify.com
Blog: http://www.joeandmotorboat.com/
Chuck Remes
On May 5, 2009, at 4:25 PM, Joe Williams wrote:
>
> Currently I have a prototype that takes input from a webapp, encodes
> it
> to json and sends it as a message to rabbitmq. Then I have subscriber
> processes pop messages from the queue and parse the json and do
> something with it. My interest is on the subscriber side of things,
> currently the json is parsed and I have a case statement that
> determines
> what happens next based on a field in the json. It seems hacky to
> have a
> case with more than a couple clauses but I am unsure how to make it
> cleaner. How are most folks doing this sort of thing? Any suggestions?
Joe,
there are two ways that I have tried to solve this.
1. Use a "case" statement like you described.
2. Subdivide all of the possible events more intelligently by using a
routing key and have subscribers subscribe to specific events by
matching against that routing key.
Let's say you generate a JSON structure that has a "type" field. That
field can take on 3 values: foo, bar, baz.
Under #1, you would do something like:
case json_object.type do
when "foo"
# do foo processing
when "bar"
# do bar...
when "baz"
# do baz...
end
Under #2, the JSON object's type field would be moved into the AMQP
realm. It would be part of the routing key in a fanout exchange, for
example. Your subscribers would then subscribe doing a match against
that routing key.
This essentially pushes the "case" statement into the AMQP
conversation and let's the library handle the matching. I don't think
there is any way to escape this kind of matching behavior whether it
is done by AMQP or explicitly in your subscriber code.
Make sense?
cr
Joe Williams
not far off.
Regarding the exchange and fanout, although probably off topic for this
list, does this mean that there is a possibility of a message getting
processed more than once since it gets published to all queues in the
exchange? Or is that once it popped in one queue its removed from all?
In my use case I need my messages to get processed once and only once.
-Joe
Chuck Remes
On May 5, 2009, at 5:12 PM, Joe Williams wrote:
>
> Chuck, thanks for the explanation, makes sense to me. Glad to know I
> am
> not far off.
>
> Regarding the exchange and fanout, although probably off topic for
> this
> list, does this mean that there is a possibility of a message getting
> processed more than once since it gets published to all queues in the
> exchange? Or is that once it popped in one queue its removed from all?
> In my use case I need my messages to get processed once and only once.
>
When you publish to a fanout exchange, the message is delivered to
each queue. It doesn't support a first-come-first-served constraint
since that is an application-level requirement.
Using a direct exchange (the other obvious choice) doesn't necessarily
solve this dilemma either. Even in a direct exchange a copy of the
message would be delivered to every queue bound to that exchange. You
need to enforce this constraint inside your application(s).
My other thought would be to use a "shared queue" setup (see section
2.1.5.1 in the AMQP 0-9-1 protocol specification doc available from
amqp.org which is what rabbitmq targets). You would have multiple
subscribers all "popping" from the same queue. Whoever got to each
message first would be the sole owner of that message. Note that this
likely puts you back to using a (potentially) large "case" statement
to choose the correct processing behavior based on the message's
contents.
You might want to subscribe to the rabbitmq list and ask there (or the
list of your favorite amqp broker). They may have some ideas since my
understanding is likely not as sophisticated as theirs.
cr
Joe Williams
throw them an email.
-Joe
Joe Williams
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-May/003921.html
> For your use case, the following setup should work:
> - one direct exchange
> - one queue for each of the case branches
> - one binding per queue to the direct exchange, with the binding key set
> to the case branch discriminator
> - one consumer per queue, handling the messages for one particular
> branch discriminator
>
-Joe