How does rebalancing work in storm-kafka?

[Varun Vijayaraghavan]

I am checking out at the storm-kafka spout in the storm-contrib repo, to use as a replacement for our current python kafka spout. Specifically, we want to build a kafka-spout system where the brokers, partitions and consumers are dynamic.

I was just going through the code, and it looks like this can handle the case when partitions and brokers change. 

However, I could not find anything obvious that handles the case where a consumer dies, or a new consumer is added. Can someone explain how partitions get redistributed amongst each spout thread in this case?

Thanks.

[Jason Jackson]

The # of kafka consumers can't change in a running topology because # of tasks are fixed in a topology. Also if a task dies it restarts, and continues where it left off because KafkaSpout stores last read offset in Zookeeper. 

The partitions are mapped to spout tasks by some hashing algorithm. If you redeploy a topology with more spout tasks, then a new mapping will be done. Each task will continue reading where the last mapped task last read. 

Does that answer your Q? 

[vinh]

If a topology is redeployed, but with less tasks than earlier, then I assume some partitions may no longer be read?

[Jason Jackson]

One spout task can read multiple partitions. The mapping is recalculated with each deploy. There's no state surrounding the mapping, just the last read offset. 

[Jason Jackson]

(state that persists between redeploys) 

[Varun Vijayaraghavan]

Thanks, that makes sense.

--
- varun :)