Forcing a server to block connections for testing purposes

[Jason]

Is there a way to force a RabbitMQ server into the state where all client connections go into the blocking state (e.g., as if the broker was running low on resources, like memory or disk space)? I want to test code in my RabbitMQ client instances that handle this condition, but obviously need to be able to put the server INTO the condition in order to run those tests.

Basically, last night, we had a RabbitMQ server hiccup (v3.5.6, Erlang R16B03) where all incoming connections went into the blocking state; it was due to a log file that started growing exponentially on the server and ended up exhausting all the available disk space. (Sigh.) All of our apps that publish to the server use a common library (which we wrote, using Lyra as the client), and every one of them puked horribly and died. So obviously I'd like to figure out what we need to do to at least prevent all the end-user-facing apps from suffering an ugly death when our server has an issue...

Thanks in advance for any advice you might have!

Jason

[Michael Klishin]

On 30 November 2015 at 17:36:09, Jason (ja...@queso.com) wrote:
> Is there a way to force a RabbitMQ server into the state where
> all client connections go into the blocking state (e.g., as if
> the broker was running low on resources, like memory or disk space)?
> I want to test code in my RabbitMQ client instances that handle
> this condition, but obviously need to be able to put the server
> INTO the condition in order to run those tests.
>
> Basically, last night, we had a RabbitMQ server hiccup (v3.5.6,
> Erlang R16B03) where all incoming connections went into the
> blocking state; it was due to a log file that started growing exponentially
> on the server and ended up exhausting all the available disk space.
> (Sigh.) All of our apps that publish to the server use a common

> library (which we wrote, using Lyra(https://github.com/jhalterman/lyra)

> as the client), and every one of them puked horribly and died.
> So obviously I'd like to figure out what we need to do to at least
> prevent all the end-user-facing apps from suffering an ugly
> death when our server has an issue...

We typically use rabbitmqctl to set RAM watermark to 0.000000001 or so. It will effectively
block every connection that sends a basic.publish. Blocking individual connections
is much more involved but the above should be sufficient for client testing. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Jason]

Awesome — I presume you're talking about the set_vm_memory_high_watermark parameter?

[Michael Klishin]

On 30 November 2015 at 17:53:03, Jason (ja...@queso.com) wrote:
> I presume you're talking about the set_vm_memory_high_watermark
> parameter?

Correct. Forgot to mention: our test suites use it, too.