Scaling strategies for Redis Streams

[Swapnonil Mukherjee]

Hi,

I am a new user to Redis. I have built an application using Redis Streams, Spring Data Redis and Spring Web Flux to be able to stream events to any browser supporting HTML 5 Server Side Events. And had a lot of fun building it. Diagram attached for reference.

3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BfontFamily%3DHelvetica%3BstrokeColor%3D%23999999%3BstrokeWidth%3D2%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%221210%22%20y%3D%22268%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%221260%22%20y%3D%22268%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%221230%22%20y%3D%22430%22%2F%3E%3CmxPoint%20x%3D%221240%22%20y%3D%22268%22%2F%3E%3CmxPoint%20x%3D%221260%22%20y%3D%22268%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E

%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BfontFamily%3DHelvetica%3BstrokeColor%3D%23999999%3BstrokeWidth%3D2%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%221210%22%20y%3D%22268%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%221260%22%20y%3D%22268%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%221230%22%20y%3D%22430%22%2F%3E%3CmxPoint%20x%3D%221240%22%20y%3D%22268%22%2F%3E%3CmxPoint%20x%3D%221260%22%20y%3D%22268%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3EMy question is with regards to scaling and somewhat with regards to persistence.

My requirements are modest. I will have roughly up to 200 msg/sec and about 2500 plus HTML 5 Clients connected to receive updates from the Spring Web FLux App.

Given that I will have 2500+ consumers,

• Can I scale out the reads with Redis Sentinel? Or does streams require any special scaling strategy?
• How do I make sure streams are replicated to slaves as quickly as possible? Should I use RDB or AOF?

Advance apologies, if I am asking the wrong question.

[Jonathan Mendoza]

Depends on if you are using consumer groups or not.  Consumer groups are stateful, so reading from a "replica" in a consumer group will not affect the redis master's state.  

Are you planning on using consumer groups?  Or just XREAD?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/245dda72-9750-43ac-84c5-4784702ac02f%40googlegroups.com.

[Jonathan Mendoza]

From Antirez (https://gist.github.com/antirez/68e67f3251d10f026861be2d0fe0d2f4)

Client group and read-only slaves

While XREAD and XRANGE are read only commands that can be scaled with slaves, XREAD-GROUP alters the state of the stream, because groups are part of the stream data structure itself, being persisted and replicated.

For this reason XREAD-GROUP can only be used in master instances.

[Swapnonil Mukherjee]

Jonathan,

No, I am not planning to use consumer groups. In my use case, all 2500 HTML 5 clients need to receive all events since the time they launch the browser. 

To unsubscribe from this group and stop receiving emails from it, send an email to redi...@googlegroups.com.