Trimming only processed messages from streams

[Chris Hanks]

Hi everyone, I'm evaluating Redis streams for a project, and I'm trying to figure out the most efficient way to make sure that old messages are deleted from streams once they've been processed by all consumer groups. It seems like the generally recommended way to trim streams is using XTRIM, but it's important for this use case that I not delete any messages that haven't yet been processed, which XTRIM doesn't guarantee.

I've been considering having a process that inspects the state of the stream and uses XDEL on individual messages once it knows that all existing groups have seen them. It seems like I'd need to use XPENDING for each consumer group to find the oldest pending ids, then XRANGE to fetch all messages below the single oldest pending id, then XDEL all of their ids? And if XPENDING is empty, then the whole stream has been processed, and I should just be able to XTRIM the whole stream to zero. I'm assuming I'll do all this in a Lua script, so that there's no chance of any messages being added between the XPENDING and the XTRIM.

This all seems pretty inefficient, though. Has anyone solved this problem in a better/cleaner way? Also, does anyone see any flaws in this plan?

Thanks!

Chris

[Chris Hanks]

I was feeling industrious so I coded this up as a Lua script, if anyone needs it: https://gist.github.com/chanks/c2e7e0efbd3d038775208047abb68524

Still open to better suggestions! Or, ideally, an option to XTRIM that does all this more efficiently.

Chris

[Itamar Haber]

Hello Chris,

The challenge is real indeed and I hope to see Redis supporting this type of "protection" natively. BTW, the same challenge exists with using `XADD ... MAXLEN` and perhaps with `XDEL` as well. It would make sense, in my opinion, to document this feature request in the repository.

The Lua script is damn nice (haven't scrutinized but looks good :)) - thanks for sharing it. One could also port that script to a Redis Module for extra speed, although the current API does not provide low-level access to Streams (yet).

Cheers,

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

--

Itamar Haber Technicalist Evangely Phone: +972.54.567.9692

[Chris Hanks]

Thanks Itamar! I went ahead and opened a feature request here: https://github.com/antirez/redis/issues/5774

Chris