Curator recipe to fairly distribute data streams

[John Weatherford]

Greetings,

I am new to curator, and reading through the recipes I think there might be some way to solve my problem, so I figured I would ask the experts. Here is my situation.

I have several (let's say 10) data streams that are being continually filled with information. I would like to be able to start up in a distributed fashion a number of clients to read from these 10 streams in a fair way. So, if one client is started up, it will see that it is the sole client, and consume from all 10 streams. Then, if another 4 clients are started, they will all reset their connections to accept 2 data streams. 

Any ideas would be appreciated. If curator has nothing for this, would it be a useful recipe to contribute? 

  -John

[Jordan Zimmerman]

You could use PathChildrenCache. You'd get notifications as nodes come and go and each worker could adjust itself based on that. 

-JZ

--
You received this message because you are subscribed to the Google Groups "curator-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to curator-user...@googlegroups.com.
To post to this group, send email to curato...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/curator-users/-/sMJqyc5EGtMJ.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[John Weatherford]

Thanks Jordan! I will look into this!

 -jW

[Eric Pederson]

Hi Jordan - 

Speaking of PathChildrenCache - in the JavaDocs it says:

IMPORTANT - it's not possible to stay transactionally in sync. Users of this class must be prepared for false-positives and false-negatives. 

Can you elaborate on what this means in practice?

Thanks,

[Jordan Zimmerman]

It's just summarizing that ZooKeeper is an eventually consistent system. The PathChildrenCache will not be exactly accurate at any given point in time. Eventually, however, it will see all changes.

-JZ

To view this discussion on the web visit https://groups.google.com/d/msg/curator-users/-/XZyosUqwlaEJ.

[Eric Pederson]

Got it - thanks!

-- Eric