Google Groups

Re: Messaging queue


Josiah Carlson Dec 16, 2011 12:37 PM
Posted in group: NoSQL Databases
On Fri, Dec 16, 2011 at 11:14 AM, alexis <alexis.r...@gmail.com> wrote:
> Josiah,
>
> On Dec 15, 11:54 pm, Josiah Carlson <josiah.carl...@gmail.com> wrote:
>>
>> I would heartily discourage RabbitMQ. Reddit had problems with it,
>
> To the best of my knowledge Reddit are still using RabbitMQ, and
> happily.

The last thing I heard about it was:
http://blog.reddit.com/2010/05/reddits-may-2010-state-of-servers.html
, which we read just before our problem. The lesson we took from it,
when we had our problems, was: "Things have improved thus far, but
replacing rabbitmq is at the top end of our extremely long list of
things to do."

We had an engineer that had had zero problems with ActiveMQ at his
previous employer, where it worked well for them for over a year. He
spent a couple hours updating our bindings, and we were good to go.

>> I
>> have had problems with it,
>
> I'm sorry you had problems with it.
>
> Did you report them to the RabbitMQ team?  Typically this makes
> problems go away fast.  We help people who ask.

We read documentation, searched the internet for our exact problems,
and read your responses to the Reddit folks. At the time, the RabbitMQ
response was (paraphrased): we will have a new persistence engine in
the next version which will solve the problem, we don't know when that
version will be released. We needed a solution that day, so we
executed a change in backend.

> Almost all problems are caused by using one of the hundreds of clients
> on the web.  Not all of these are implemented right.  I believe that
> the Redis team get as frustrated as we do, when we get blamed for
> someone's work that we have no knowledge of ...

Indeed.

> There is a subset of problems that are solved by upgrading RabbitMQ.
> We are now on 2.7.0 with 2.7.1 due soon.  The move to 2.0 was very
> important since it introduced disk paging, allowing a disk-bounded
> persistent address space instead of RAM-bounded.
>
> See here: http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/
>
>
>> and I have friends who have problems with
>> it (RabbitMQ has died on all of us when sending messages to it "too
>> fast").
>
> A messaging system should support multiple fast producers and slow
> consumers at scale.  In practice this means:
>
> - disk paging (see above) to flow data to disk when consumers
> disappear
> - producer back pressure and other 'self protection' modes
>
> Also useful for this may be:
>
> - a management capability
> - a way to use multiple machines to scale with different topologies
> - various ack & nack modes
>
> Rabbit provides all these but people often do not use them.  This can
> lead to problems.
>
> I love Redis and would not want to discourage its use.  But as you
> scale, you may find you need the above things.  We make a big effort
> to support them well.  As with all this, please make sure you
> understand your use case, test software for yourself, and make sure
> you ask for help when you want it.

It's great that it has all of those options now. But, having been bit
in the ass in the past and lost data, I'm not one to go back to that
same abusive mistress, even if "she is better now, no more hitting
you!".

Also, those benchmarks are in bulk streaming mode (rather than 1 at a
time) and are not applicable to what I would consider to be the vast
majority of message queue applications. In that mode, everything looks
great. In particular, you may as well use something like ZeroMQ and
move millions of messages/second.

>> It's possible that the Erlang release in the last few days
>> fixed the problems, but I wouldn't rely on it in any sort of
>> production scenario, and definitely not with a fresh Erlang release.
>
> Very many folks are delighted with RabbitMQ in production.
>
> Here are two examples, one talking about availability, one about
> throughput (ingress).
>
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-July/013981.html
> http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-April/012321.html
>
>
>> If a non-Redis message queue is necessary, I would recommend ActiveMQ.
>
> http://news.ycombinator.com/item?id=2588072

Note the quote from the article: "well, ActiveMQ is used by lots of
very happy users, so you must be doing something wrong."

Note your quote from above: "Very many folks are delighted with
RabbitMQ in production." and "Rabbit provides all these but people
often do not use them.  This can lead to problems."

I'm seeing no significant difference in your words vs. the words from
the ActiveMQ folks, and can only go by my experience.

My experience differs from others. It's like going to a restaurant,
maybe the waitress had a bad day, and that's why my strawberry
milkshake comes to me as strawberry milk. I'm not going to go back.
But you go in on a different day, get the best apple pie you've ever
eaten, and tell everyone about it. We will have to agree to disagree.

Regards,
 - Josiah

>> Regards,
>>  - Josiah
>
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.