rabbitmq streams - number of confirmations for published messages

[Artur Wroblewski]

Hello,

One of my applications sends very small messages - 7 or 14 bytes.

Therefore, when testing performance of my library (RbFly) and RabbitMQ
Streams, I am using messages of 10 bytes.

When I send 100 messages in a batch, then RabbitMQ Streams broker sends
back confirmation in two phases

- 64 message published ids
- then 36 message published ids

Is there a configuration option I could use to change the minimal number of
message published ids sent by the broker? I hope I could increase the
throughput of my application by setting it to 100 (or 128).

I skimmed the documentation, and I cannot find anything.

Best regards,

Artur

--
https://mortgage.diy-labs.eu/

[Arnaud Cogoluègnes]

There's no such mechanism. Confirmation depends on several parameters (ingress, replication, persistence), so it would not be trivial to implement.

What is the throughput you get and what is the target throughput?

[kjnilsson]

For the use case where you already have a delineated batch using sub batches makes a lot of sense. Then the batch will always travel as a single "entry".

[Artur Wroblewski]

> When I send 100 messages in a batch, then RabbitMQ Streams broker sends
> back confirmation in two phases
>
> - 64 message published ids
> - then 36 message published ids

At the moment, when sending 100 messages in a batch to RabbitMQ Streams
broker, I have throughput of 230,000 msg/sec.

I have modified my library to accept first tranche of message publishing
ids confirmations as whole batch confirmation (this is, I receive
confirmation for 64 messages, and mark batch of 100 messages as done).

Thanks to above the throughput increased to about 450,000 msg/sec. Not sure
if I made some mistake here (not enough sleep!).

Is there a way to make a simple code change in RabbitMQ to perform a better
test?

[Arnaud Cogoluègnes]

With small messages (~10 bytes) and batching at 100, we usually get around 1 M messages / second. Can you experiment with stream-perf-test [1]?

Batching publish confirms more aggressively means delaying them, meaning adding latency. This could also slow down publishers that have an outstanding confirms limit (most of the client libraries we maintain do). Moreover it could be awkward and complex to implement on the broker side.

We could change the publish confirm frame (bump a version more precisely) and implement publish confirm ranges to optimize network traffic. This is not a trivial change either and we don't really know the benefits beforehand.

We consider streams to be quite fast already. I'm not saying we are not open to implementing quick-win optimizations and fixing critical bottlenecks, but we are trying to focus our efforts on robustness and reliability these days. We'll review contributions and PRs of course.

[1] https://rabbitmq.github.io/rabbitmq-stream-java-client/snapshot/htmlsingle/#the-performance-tool

[Artur Wroblewski]

On Wed, Jun 14, 2023 at 12:45:38AM -0700, Arnaud Cogoluègnes wrote:
> With small messages (~10 bytes) and batching at 100, we usually get around
> 1 M messages / second. Can you experiment with stream-perf-test [1]?

I am aware of these numbers, but let's avoid comparing different testing
conditions (Python vs Java, and I suspect: single threaded vs
multithreaded, waiting for confirmations vs background thread to receive
those, no cpu turbo vs cpu turbo enabled, laptop vs server).

> Batching publish confirms more aggressively means delaying them, meaning
> adding latency. This could also slow down publishers that have an
> outstanding confirms limit (most of the client libraries we maintain do).
> Moreover it could be awkward and complex to implement on the broker side.

I do understand, but this is what I observe.

For reference, if I send a batch of

- 100 messages, the confirmations come back in two tranches (64 and 36 ids)
- 64 messages, the confirmations come back in two tranches (32 and 32 ids)
- 32 messages, the confirmations come back in a single tranche of 32 ids

On the protocol level, when sending batch of 100 small messages, RabbitMQ
Streams receives a single "Publish" protocol frame. That single frame says
what number of published messages is. There is nothing to guess, and there
is nothing to wait for here.

Also, 128 message publishing ids (as the broker responds in power of 2)
encoded within "PublishConfirm" protocol frame is 1029 bytes, if I am not
mistaken. Well below a typical ethernet MTU, for example?

[...]

> We consider streams to be quite fast already.

Indeed, there is nothing to worry about here (I hope to publish some
performance tests comparing to aiokafka this month). Just trying to see if
I can optimize for my use case.

[...]