RabbitMQ Streams manually set Timestamp in V3.12.8

[Nadav Peled]

Hi,

Trying to make use of RabbitMQ Streams (using amqp-node) and load a very large Log of events stored in a an archive database.

My goal is to play them one by one starting from time X, allowing jumping forward and backward using the timestamp offset feature.

Each event has a timestamp (time it occured).

When loading them to the stream, they are automatically assigned with a timestamp consisting the time they arrived to the queue.

I was able to set the message timestamp property and a timestamp_in_ms header but could not get the timestamp offset to work according to the event's original time.

I have tried providing the following arguments according to amqp-node readme : 

{ "x-stream-offset": { "!": "timestamp", value: (some number) }}

I have also tried (long shot..) playing with rabbitmq.conf and set to true/false

message_interceptors.incoming.set_header_timestamp.overwrite = true

Is it possible to override the message timestamp with a manually set value ? 

Best regards

[Arnaud Cogoluègnes]

The timestamp you set on subscription does not match anything in messages themselves. RabbitMQ uses the timestamp of a block of messages (a chunk, the unit of storage in streams). See the documentation on the timestamp offset specification [1].

You should try to set a timestamp a bit backward from what you want (to make sure to have all the messages you expect) and filter out messages on the client side. The filtering would apply only at the beginning of the subscription and should not have much impact on performance.

[1] https://www.rabbitmq.com/docs/streams#consuming

[Nadav Peled]

Thanks Arnaud,

I went through the documentation but could not set the timestamp of the published message to a time in the past.

My log records has a timestamp somewhere in the past (lets say, last 30 days).

Seems like all messages published to the stream receive a timestamp from rabbitmq-server code-behind and ignore the timestamp I attach.

I was able to set the timestamp field in the message property, but couldn't read the stream according to this timestamp.

I was able to set the timestamp_in_ms header as well, but again, couldn't read the stream according to this timestamp.

I chose the streams following your talk in youtube because the time-travel use case seemed relevant to my case -

I have a very large data lake with plenty of logs, I want to fill the stream with log records from the DB and replay them.

So although all messages were published to the stream now, their 'logical' timestamp might be some days in the past... as for now, they all receive "new" timestamp which I cannot override...

Thanks again