Redis Streams grab Nth element or Nth offset?

[Vincent Marquez]

Hello.  I'm developing a streaming application with Redis Streams and it occured to me that it would be more efficient if there was a way to retrieve the Nth offset, starting from a given offset. 

So if I start with offset ABCD-0, and there are 5,000 elements in a stream, and I'd like to 'jump ahead' by 1000, I'd need to know the offset of the 1000th element.  Currently the only way I know how to do this is to iterate through with a count of 1000, but that is inefficient.  

Since the first part of the offset is a timestamp, I have no way of knowing how many elements were added at what time, so I can't just 'bump' the offset.  

Presumably there's an internal way for Redis to know the offset of the 1000th element, no?  

Any ideas?  Thank you in advance!

[Benjamin Sergeant]

I can't see any commands that would provide what you need. The closest would be xinfo.

https://redis.io/commands/xinfo

> XINFO STREAM mystream
 1) length
 2) (integer) 2
 3) radix-tree-keys
 4) (integer) 1
 5) radix-tree-nodes
 6) (integer) 2
 7) groups
 8) (integer) 2
 9) last-generated-id
10) 1538385846314-0
11) first-entry
12) 1) 1538385820729-0
    2) 1) "foo"
       2) "bar"
13) last-entry
14) 1) 1538385846314-0
    2) 1) "field"
       2) "value"

Maybe xinfo could take an extra parameter to fetch all the stream ids (and not the values I assume) / it already fetch the first entry and the last entry.

I noticed that redis 6.2 has a lot of stream commands fixes planned.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/067c910d-efb3-4341-a72e-91b6541f7d2en%40googlegroups.com.

[Itamar Haber]

Hola Vincent,

I'm wondering about the use case.

Cheers,

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/44420ECE-19F4-4D61-A304-AD5E9EA45338%40gmail.com.

--

Itamar Haber Technicalist Evangely Phone: +972.54.567.9692

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

[Vincent Marquez]

On Thu, Sep 10, 2020 at 1:41 PM Itamar Haber <ita...@redislabs.com> wrote:

Hola Vincent,

I'm wondering about the use case.

Cheers,

I've re-implemented the spark-redis connector to respect trigger limits, so I can now consume from a stream and ONLY read N elements per trigger.  To do this though, I need to see the next available offset to read UP to, which means bumping the offset from the last-read.  So i'm currently doing this the inefficient way, but this would be much better if there was a redis command for this. 

~Vincent

 

You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/bT00XGMq4DM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/CAFWLiegLYGqmfPWtsbabdS7SjsgfoYA17tyrT%2BrynV-x7frq8g%40mail.gmail.com.

[Itamar Haber]

Hrm. Have you considered using `XREADGROUP ... COUNT 1000`? I may be missing the point here, but if you need to distribute 1K messages to each of trigger's instances that's what consumer groups are for.

> Presumably there's an internal way for Redis to know the offset of the 1000th element, no?

AFAIK no. Streams are managed as radix trees of the ids. These are very good for searching by id but don't offer an efficient way to find ids by an offset, hence the lack of such functionality. To facilitate an efficient means of "ranking" one would have to use another data structure. This can be implemented in userspace with a Sorted Set that is populated with the message IDs and then `ZRANK`ed,  for example.

Cheers,

Itamar

[James Harbal]

Only way would to use a secondary sorted set and maybe use a lua script to get the stream elements via the zrangebyscore

Sent from Yahoo Mail on Android

--