How to simulate multiple concurrent producers?

[Philipp Schumacher]

Hi,

so far i did a few basic steps with the python pika client and sending messages in a loop with 1 open connection, sending thoudands of messages, like:

channel = connection.channel()

channel.exchange_declare()

for x in range(0, 500000):

  channel.basic_publish():

connection.close()

In the next step, i would like to simulate a scenario with thousand of concurrent producers instead of sending thousands of messages with one producer. So each producer should open it's own connection and send only a single message, in parallel.

Can i do this with the pika client (or at least any other client)? Could you give me a hint on how to achieve this.

Thanks!

[Michael Klishin]

On 26 January 2016 at 11:23:48, Philipp Schumacher (de.schumac...@gmail.com) wrote:
> In the next step, i would like to simulate a scenario with thousand
> of concurrent producers instead of sending thousands of messages
> with one producer. So each producer should open it's own connection
> and send only a single message, in parallel.
>
> Can i do this with the pika client (or at least any other client)?
> Could you give me a hint on how to achieve this.

You can do it with most clients but after a certain point, you will begin hitting
(OS imposed) limits such the number of threads ("ulimit -u") a single OS process can use.

And then there are things that need to be tuned to sustain a high number of connections:
http://rabbitmq.com/networking.html

Other than that, starting N threads (or similar) with its own connection each sounds straightforward
enough in any language because they won't share anything beyond server connection information
(immutable) and maybe exchange name (also immutable). 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Philipp Schumacher]

Thanks for the quick response and the tuning tips :-)

A moment a go i saw that there is a JMeter Plugin for AMQP. I will try to do some load testing with this.

[vitaly numenta]

JMeter is probably what you need. If you still need to implement massively concurrent tests outside of JMeter, you can avoid the thread limits/issues by using pika's asynchronous adapters, such as SelectConnection or TornadoConnection. https://github.com/agoragames/haigha with Gevent "green threads" is another way to use 1000's of concurrent AMQP connections without Posix threads.