Handling a dynamic set of zookeeper hosts

[Mark Gius]

I'm using zookeeper in an environment where the number and addresses of zookeeper hosts don't remain static over the runtime of the application and it is possible that at the time a client is created the list of hosts will change to include none of the original hosts.  It's the same cluster, but the cluster hosts may have been moved.  

In Kazoo 1.0 I took advantage of the implementation detail of the ConnectionHandler and wrote a little session event handler that triggered a replacement of KazooClient.hosts whenever we ended up in a disconnected state.  I'm working on upgrading to Kazoo 1.3 but the implementation of ConnectionHandler now uses itertools.cycle, which prevents me from sliding my new host list in sideways.  

I'm looking for a supported (or new backdoor) way to get a new list of hosts into an existing kazoo client without having to stop() and then start() it.  It doesn't look like there is a public API on the client or connectionmanager to allow this now.  Would the developers be opposed to me adding one in?  

Alternative suggestions for how to work around this are also welcome, but I think being able to change the list of host endpoints would be beneficial to more than just myself.  FWIW, Curator (kind-of) supports this via Exhibitor, although my reading suggests that only new clients get the updated list so perhaps my use case is more unique.

Mark

[Hanno Schlichting]

Hi.

 

On Fri, Sep 27, 2013, at 0:20, Mark Gius wrote:

I'm looking for a supported (or new backdoor) way to get a new list of hosts into an existing kazoo client without having to stop() and then start() it.  It doesn't look like there is a public API on the client or connectionmanager to allow this now.  Would the developers be opposed to me adding one in?  

 

Adding a new API sound good to me. We could turn client.hosts into a writable property or add a new client.set_hosts method.

 

With Zookeeper 3.5 and dynamic reconfiguration support, this is likely doing to be more common (https://issues.apache.org/jira/browse/ZOOKEEPER-1660). Another use-case might be reloading kazoo configuration data from some ini file, triggered by sending a signal to the process.

 

Alternative suggestions for how to work around this are also welcome, but I think being able to change the list of host endpoints would be beneficial to more than just myself.  FWIW, Curator (kind-of) supports this via Exhibitor, although my reading suggests that only new clients get the updated list so perhaps my use case is more unique.

 

Your use-case does sound more unique. I'd expect it to be rather uncommon to add new ZK servers to a cluster, especially since the ZK server itself made it kinda hard until the upcoming 3.5.

 

Hanno