x-max-age not working

[Myke Brenan]

I seem to have a problem with the stream queue and x-max-age.  Ive tried setting this as both a queue argument as as a policy and still am not seeing the stream messages shrink.  

Is there any logging or anything additional I could provide to be more helpful. 

[Michal Kuratczyk]

Hi,

Are you publishing to the stream? See the last paragraph of https://www.rabbitmq.com/streams.html#retention

Retention works on a per-segment-file basis. Messages are never removed from a file - either the whole file remains (if it contains any messages younger than max-age or if it is the only segment file of the stream) or the whole file is deleted.

Best,

On Thu, Mar 24, 2022 at 6:09 PM Myke Brenan <myke....@gmail.com> wrote:

I seem to have a problem with the stream queue and x-max-age.  Ive tried setting this as both a queue argument as as a policy and still am not seeing the stream messages shrink.  

Is there any logging or anything additional I could provide to be more helpful. 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/8a0069e8-a737-46ff-ae2f-92365b430a15n%40googlegroups.com.

--

Michał

RabbitMQ team

[Myke Brenan]

SO I tried turning the segment size pretty far down.  Im not publishing directly to the stream. The stream is bound to an exchange.  

Myke 

[Wes Peng]

May I ask if rabbitmq will provide a SQL like engine for its streaming
query in future?

Thanks

[Michal Kuratczyk]

Hi,

Never say never, but this is not something we are working on or even planning at this point. Some form of filtered consumption is quite likely (publish with different keys to a single stream but consume only messages with a given key from that stream) but this is obviously nowhere near a SQL-like query engine.

If you have any specific use cases in mind - please let us know.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7e8a3c00-c590-ff2f-ee06-e1a3fe066b8a%40stackdev.eu.

--

Michał

RabbitMQ team

[Myke Brenan]

Using this value 

stream-max-segment-size-bytes:100

Im still never seeing items leave the stream. 

Myke

[Michal Kuratczyk]

1. It doesn't matter how small you set it, if you don't publish. As I mentioned in the beginning - at least one file is always there for a stream. You need to publish enough for a new segment file to be created and only when this happens, RabbitMQ will check if the old segment file contains only expired messages and if yes - it will immediately delete that old segment.

2. This is not a reasonable size for a stream segment. You basically can't fit even a single message in there (apart from the message payload, there is additional metadata in the segment files). You have to accept that messages don't just disappear from the stream - it's an immutable data structure and messages won't be removed from a segment file - only the whole file can be deleted.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/835d1c3f-5750-4288-81b5-db6a19beea79n%40googlegroups.com.

--

Michał

RabbitMQ team

[Myke Brenan]

1. It doesn't matter how small you set it, if you don't publish. As I mentioned in the beginning - at least one file is always there for a stream. You need to publish enough for a new segment file to be created and only when this happens, RabbitMQ will check if the old segment file contains only expired messages and if yes - it will immediately delete that old segment.

In the case of the stream being bound to an exchange doesn't that count as publishing?  I am publishing to the exchange and I do see messages added. 

Basically I would like to publish to an exchange and be able to have in the stream the ability to go back 1 day multiple times. 

[Michal Kuratczyk]

Sorry, I must have misunderstood then. Please check the segment files in your streams folder - are there any files older than your retention period?

If you still believe this can be a bug, would you be able to create an executable test case (for example using rabbitmq-perf-test to declare and publish)?

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/18dc3aac-1ec7-413f-a5d3-f37fc8492f3en%40googlegroups.com.

--

Michał

[kjnilsson]

I replied to a similar question here: https://github.com/rabbitmq/rabbitmq-server/discussions/4384#discussioncomment-2475952

Basically to go back 1d you just use a time based consumer specification rather than reading the whole stream and relying on retention to remove older entries.

https://www.rabbitmq.com/streams.html#consuming

Cheers

Karl

[Myke Brenan]

There seems to be a single .segment file that is constantly growing in size but that's all I see.  Let me try to get a test case together over docker. 

[Myke Brenan]

actually no longer seeing any issue with RabbitMQ 3.9.14.  I was seeing it in 3.9.13 but appears fixed.