Running RabbitMQ inside container - cgroup (docker) memory limit

[mthenw]

Hi,

I'm running RabbitMQ cluster on multiple container across few EC2 instances. I use mirrored persistent queue. Under high load I've noticed that one of slave is crushing. It's caused by Docker's OOM killer.

I guess that this is caused because RabbitMQ is not aware of cgroup memory limits and uses host memory info. Is there a way to set memory limit through RabbitMQ configuration? Or are there any plans for supporting containers as runtime environment? I'm aware of vm_memory_high_watermark but but "Note that this does not prevent the RabbitMQ server from using more than 40%, it is merely the point at which publishers are throttled."

[Michael Klishin]

VM memory watermark is the only mechanism. I'm not sure there is such thing as "Docker OOM killer" — pretty it is the OS that does it.

Unless there is a way for the OS to notify process about approaching a memory limit (à la iOS), we can't do much.

Not sure what isn't clear about publisher throttling: RabbitMQ won't read from the socket which is the most effective way we know to stop ingesting data.

We are working on some things that will eliminate fixed RAM cost per message and RAM cache use in userspace for 3.6.0.

MK

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mthenw]

On Tuesday, June 30, 2015 at 10:02:59 PM UTC+2, Michael Klishin wrote:

VM memory watermark is the only mechanism. I'm not sure there is such thing as "Docker OOM killer" — pretty it is the OS that does it.

Yes of course, this is what I meant.

 

Unless there is a way for the OS to notify process about approaching a memory limit (à la iOS), we can't do much.

Not sure what isn't clear about publisher throttling: RabbitMQ won't read from the socket which is the most effective way we know to stop ingesting data.

 

Generally, my point is, if currently there is no way to get cgroup memory limit, maybe there should be a way to set available memory manually? The main advantage of containers it that they can be ran on different hosts without thinking about that layer. I'm not able to predict how much memory will have host running docker while I'm writing Dockerfile with RabbitMQ config so vm_memory_high_watermark is useless. 

[Michael Klishin]

How are you going to set the limit manually then? Setting a value manually means you know the range of possible values ahead of time.

MK

[mthenw]

I can assume that container will always run with some memory limit (`docker run` with -m option). Same value I would put inside rabbitmq.config. That way I don't care about underlying OS, I just specify that my container has some memory requirements. 

[Michael Klishin]

On 1 July 2015 at 11:35:43, mthenw (maciej....@gmail.com) wrote:
> I can assume that container will always run with some memory
> limit (`docker run` with -m option). Same value I would put inside
> rabbitmq.config. That way I don't care about underlying OS,
> I just specify that my container has some memory requirements.

It is not currently possible to set RAM limit as an absolute value, although
we've had a bug filed for that for a while. Disk limit can be configured like
that, so it's a reasonable request.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[mthenw]

Great! Is there a way to track this issue? Jira or something?

[Michael Klishin]

It was in our closed bugzilla. We use github for everything these days.

MK

On 1/7/2015, at 12:11, mthenw <maciej....@gmail.com> wrote:

Great! Is there a way to track this issue? Jira or something?

--

[Michael Klishin]

Heads-up: ability to set VM memory watermark will be in RabbitMQ 3.5.7:

https://github.com/rabbitmq/rabbitmq-server/issues/207

On Wednesday, 1 July 2015 12:14:51 UTC+3, Michael Klishin wrote:

It was in our closed bugzilla. We use github for everything these days.

MK