Change Apache Kafka properties dynamically without restarting servers using Archaius

[vijj...@gmail.com]

Hello,

I have some questions related to properties of Apache Kafka. We have a four node Kafka Cluster which is on Amazon EMR. Currently, in order to change any properties we do restart the servers to pick the new changes made. For the following:

1. Can we change the topic name that a producer is writing to
2. Can we change the bootstrap server list
3. Can we change the consumer group name
4. Can we change the number of consumers etc. and be able to verify and measure that the change is picked up and did not result in lost or duplicate message nor introduce any consumer lag

Is it possible to change these properties without restarting the servers ? I mean can we change these dynamically using Archaius (https://github.com/Netflix/archaius/wiki)?

Reply me at your earliest convenience.

Thanks,

Vijju

[Ewen Cheslack-Postava]

Vijju,

I'm not really familiar with archaius, but broker settings are configured via a properties file and are currently fixed . However, there are some settings (topic level configs) that are stored in ZK and can be changed dynamically. Finally, some of the items you mentioned in your list aren't on the broker at all -- for example, the topic name a producer is writing to isn't even fixed for the entire broker, you can set it on a per-message basis. Generally, for each of the items you've listed, if it is specified in a config file or is specified for an entire client instance at configure()-time, then it can't be changed without restarting/reinstantiating; if it can be set via a tool or API during execution (topic-level configs, client parameters set in Record classes), then they can be changed whenever you want.

-Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6e54fbc2-f1a5-46c6-8bd7-936bbe4c5db8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen