RabbitMQ Streams server offset-store

[Francesco Pessina]

Hello everyone.

I'm doing some experiments in order to decide the new "streams" feature of rabbitmq can be useful for my use case.

I'm experiencing some weird behaviour I wish you could explain me.

I wrote a dummy producer which sends a fixed number of messages (i.e.: 1000) to a stream, with a dummy consumer which consumes from it.

I don't want to lose any message and I want to avoid to consume duplicates, so I set, producer side, "batchSize" to 1 and "maxUnconfirmedMessages" to 1.

On consumer side, instead, I set to commit the offset to the server every message.

The weird behaviour is that, with this configuration, I get on the stream more or less 1900 messages (for 1000 messages I sent); this overhead, I think is due to the offset commit, isn't it?

But if I set, producer side, "maxUnconfirmedMessages" to the default (10000), the final number of messages on the streams are more or less 1100 (a lot less!).

Why do I get this behaviour?

I'm using the Java client, last version.

Thanks

[Michal Kuratczyk]

Yes, offset tracking information is stored in the stream as well. This metric inaccuracy is documented:

https://www.rabbitmq.com/streams.html#limitations-ui-metrics

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/42ca2537-4250-4da7-9bc1-cf42318caff2n%40googlegroups.com.

--

Michał

RabbitMQ team

[kjnilsson]

The built-in offset tracking is probably not recommended for your use case (assuming your test matches your actual use case). If an offset storage request is received in the same internal batch as some message write requests the offset tracking info will be appended to the chunk containing messages, however, if (as in your test) there are no writes at the same time most offset storage requests will be written as a new chunk. 

For your use case it may be better to use some kind of mutable database to update the offset frequently.

Cheers

Karl