Stream XADD less than top id

[Alejandro Wainzinger]

I have a set of time series data that fits well with streams XRANGE form of access, but I need to sometimes XADD items into the past. At the moment that seems not to be possible, and I understand that it's because the main use case for streams is for streaming data where we always append to the end, but is there some way I can get the advantage of streams nice API for timeseries range queries but insert into the past?

The only way I can think to do that right now is to either:

A. Recreate the stream, but inserting the one item in the right place, which is very expensive.

B. Linearly XDEL back to the point of insertion, append, then reinsert the rest.

Both of these interfere heavily with using the data structure while doing this potentially expensive operation, and aren't particularly elegant.

Should I use other data types to model time series data that can be inserted to in the middle?

Thanks!

[Richard Pragnell]

Hello,

I also have the same requirement of XADD items before an existing ID. I did some quick modifications to the source code in order to accept this behaviour but broke streams in the process...

I understand this could alternatively be implemented using sets, but missing out on the really useful tools that streams provide.

Any thoughts?

Thank you

[David Maier]

Hello Alejandro,

could you give us a bit more background regarding the concrete use case? Time series use cases are often append-only (as events are usually occuring in a kind of timely order).

I could imagine that you might have multiple sources and you want to update the stream from these sources in batches. Maybe one batch has an earlier timestamp in it than one which you did already add to the stream. If this is the case then one solution could be to:

C. Use one stream per source (e.g, sensor) and then combine the results by using XREAD

Here an example:

127.0.0.1:6379> FLUSHDB
OK
127.0.0.1:6379> XADD my:sensor:1 * "id" "1"
"1564558318886-0"
127.0.0.1:6379> XADD my:sensor:1 * "id" "2"
"1564558330094-0"
127.0.0.1:6379> XADD my:sensor:2 "1564558318886-1" "id" "3"
"1564558318886-1"
127.0.0.1:6379> XREAD STREAMS my:sensor:1 my:sensor:2 0-0 0-0
1) 1) "my:sensor:1"
   2) 1) 1) "1564558318886-0"
         2) 1) "id"
            2) "1"
      2) 1) "1564558330094-0"
         2) 1) "id"
            2) "2"
2) 1) "my:sensor:2"
   2) 1) 1) "1564558318886-1"
         2) 1) "id"
            2) "3"

BTW: The result is grouped by stream. So you might want to flatten it down into one result stream again.

You might also want to take a look at RedisTimeSeries. RedisTimeSeries is also 'append-only'. However, it comes with a TS.MRANGE command which is allowing to do a range query across multiple time-series. It also allows you to filter by specific criteria (e.g. all time-series of a specific group of sensors). Further details can be found here: https://github.com/RedisTimeSeries/RedisTimeSeries/blob/master/docs/commands.md#tsmrange

Hope this helps.

Regards,

David

[Richard Pragnell]

Hello David,

Thanks a lot for the ideas and example!

The batching idea is indeed interesting but doesn't fit my use case, that is: produce events with timestamps (IDs) in the future which are added to the stream, consumers obtain a range of events from the stream (current time - some delta) and process the events in order.

The concept would be like a scheduler, example:

Add an event to be processed in two days time -> XADD my:events:1 "currentTimeStamp + 2 days" "id" "1"

Add an event to be processed in 30 minutes time -> XADD my:events:1 "currentTimeStamp + 30 min" "id" "2" ->  Error! ID is smaller than last ID in the stream.

For now I'll use sorted sets as an alternative https://redislabs.com/redis-best-practices/time-series/

Thanks again!

Regards,

Richard

PS: Sorry for highjacking Alejandro's original message. I hope this discussion will still be helpful for other readers :)