rabbitmq stream: start consume - offset vs timestamp
Дмитрий Николаевич
Does `basic_consume` delay differ using timestamp as offset comparing to offset value ?
pvladMQ
Hi Dmitry,
Assuming you mean RabbitMQ streams consumed via AMQP 0.9.1 basic.consume with x-stream-offset: after the consumer is attached, there should be no meaningful consumption speed difference between using a numeric offset and using a timestamp.
The difference is mainly at subscription/startup time:
x-stream-offset = 5000 means: attach at this exact stream offset, or clamp to start/end if that offset is no longer available.
x-stream-offset = <timestamp> means: RabbitMQ first resolves that timestamp to a real stream offset, then starts consuming from there. The RabbitMQ stream protocol describes this as resolving first, last, next, offset, or timestamp to a concrete offset.
Important detail: timestamp offset is not based on your application message timestamp. RabbitMQ uses the message arrival time in the stream. With AMQP 0.9.1 the timestamp is POSIX time with one-second accuracy. Also, streams are stored in chunks, so when consuming by timestamp you can receive messages published slightly before the timestamp. For production restart/resume logic, I’d prefer storing the numeric offset, because it is exact and deterministic. RabbitMQ’s own stream offset tracking docs describe offsets as the consumer’s position in the stream and recommend storing them periodically, not necessarily every message. Use timestamp offsets when you want a replay like “start from messages around 10:00 CET” or “replay the last hour”. Use numeric offsets when you want “continue exactly where this consumer stopped”.
Also remember that for AMQP 0.9.1 stream consumption, RabbitMQ requires basic.qos / prefetch and acknowledgements; acks act as credit to advance consumption.
Docs:
RabbitMQ Streams consuming / x-stream-offset: https://www.rabbitmq.com/docs/streams
RabbitMQ Stream protocol offset resolution: https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_stream/docs/PROTOCOL.adoc
RabbitMQ Streams offset tracking: https://www.rabbitmq.com/blog/2021/09/13/rabbitmq-streams-offset-tracking
I hope this helps!
Vlad