streams questions

[Ulrich Schreiner]

hi,

i have two questions about redis streams. 

1. ordering of ID

we receive location messages from different devices all over the world. many of then send the data with a data connection (for example iridium) and as the connection is not always active (depends on the weather, or your exact location, etc.) they can be delayed, sometimes we have delays up to more hours (seldom, but happens). i thought about putting all incoming messages in a stream and use the timestamp of the event as a key. but the timestamp of the event is not the timestamp when our system received the message. and so i cannot insert a delayed message in a stream if i already received a newer one. is there any chance to achive this? if i use the receive timestamp then the stream-ordering would be ok, but now i cannot detect the position of a device with one lookup in the stream. and i also cannot use the stream as a sort of "playback medium" because the events are not ordered with the time they were genereted but with the time they where recevied. 

the only idea i have is to create a different stream: one for the receiving timestamp and another one with the event-timestamp which is recreated as a whole when we receive a delayed message? is there any other way? or am i missing something?

2. consumer name in consumer group

i like the concept of a consumer group. but i have a little problem with the "consumer-name". as my consumers will run in a kubernetes cluster, they do not have a well-known name (they are stateless; i could make them a stateful set, but i dislike this). in our current solution i use nsq channels, which i think are very compareable. but instead of consumer groups a client connects to a channel and only has to know the topic/channel name. the client does not need a identifying "consumer name". so my question is: is it a problem, if a client uses "xreadgroup" with a fix "consumer name" that will change, when the client restarts? in this case my clients will generate a dynamic consumer name which is created at startup. will this approach be ok? or will redis complain later that there are "missing consumers" in a group?

thx

Ulrich Schreiner

[Nick Farrell]

1: the streams data type is not intended to be used as a static object to query the current state. You should use other data types for this. It is intended to allow downstream systems to both subscribe for real-time pushes of new messages while also being able to "catch up" following a disconnect, with no risk of data loss.

If your incoming data is out of order, the obvious solution to me is to use a few lines of Lua to compare the incoming timestamp to the current timestamp, and to discard the incoming message if it is older than what is already there. So maybe one hash of weather values, one hash of latest timestamps, in each case using the weather station as the key.

2: I imagine there is no problem with new containers having a random ID, as long as you are sure it has not been used before. You could even use redis to log your consumer name, ensuring subsequent clients to not pick it (perhaps throw in an expiry time and other tickle your key every so often, or terminate the instance before the expiry time would be reached (e.g. 1 month)).

If a consumer isn't seen by redis for a while it should not be a problem. Other consumers will pretty soon process its unacked messages and there will be negligible overhead to its being there.