Programmatically set replication factor per topic

650 views
Skip to first unread message

Scott Cranford

unread,
Apr 5, 2017, 7:18:58 PM4/5/17
to Confluent Platform
Hi,

I am adding a new use case to our Kafka 0.10 cluster that needs different replication and retention values. Is there a way to programmatically set these properties with the Java producer on a per topic basis?

Those two particular properties don't appear to be available: https://kafka.apache.org/documentation/#producerconfigs

Wonder how others have set these properties for dynamically created topics.

Thanks!

Michael Noll

unread,
Apr 6, 2017, 3:19:10 AM4/6/17
to confluent...@googlegroups.com
Scott,

first, setting the replication and retention values is a broker-side operation.  It's not something "the producer does".

Kafka KIP-4 [1] is an ongoing work -- where part of the functionality is already available today -- to have an official "admin API" for creating topics, setting replication factor, etc.

In the meantime, an "unofficial" option you could use is described in my StackOverflow answer at [2].  If you need an up-to-date code example, take a look at [3], which is for Confluent 3.2 and Apache Kafka 0.10.2.0.

Hope this helps!
Michael








--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/c942fa46-dc90-451c-b455-bb2236245f7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Michael G. Noll
Product Manager | Confluent
Follow us: Twitter | Blog

Michael Noll

unread,
Apr 6, 2017, 3:20:18 AM4/6/17
to confluent...@googlegroups.com
PS: Since you were looking for settings like topic retention:  They are listed under the Broker Configuration section https://kafka.apache.org/documentation/#configuration (not the Producer Configuration).

On Thu, Apr 6, 2017 at 9:19 AM, Michael Noll <mic...@confluent.io> wrote:
Scott,

first, setting the replication and retention values is a broker-side operation.  It's not something "the producer does".

Kafka KIP-4 [1] is an ongoing work -- where part of the functionality is already available today -- to have an official "admin API" for creating topics, setting replication factor, etc.

In the meantime, an "unofficial" option you could use is described in my StackOverflow answer at [2].  If you need an up-to-date code example, take a look at [3], which is for Confluent 3.2 and Apache Kafka 0.10.2.0.

Hope this helps!
Michael







On Thu, Apr 6, 2017 at 1:18 AM, Scott Cranford <scottc...@gmail.com> wrote:
Hi,

I am adding a new use case to our Kafka 0.10 cluster that needs different replication and retention values. Is there a way to programmatically set these properties with the Java producer on a per topic basis?

Those two particular properties don't appear to be available: https://kafka.apache.org/documentation/#producerconfigs

Wonder how others have set these properties for dynamically created topics.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
--
Michael G. Noll
Product Manager | Confluent
Follow us: Twitter | Blog

Michael Noll

unread,
Apr 6, 2017, 3:21:37 AM4/6/17
to confluent...@googlegroups.com
PPS: And these settings can be specified on the broker-level or with per-topic overrides:

Scott Cranford

unread,
Apr 6, 2017, 2:40:58 PM4/6/17
to Confluent Platform
Thanks for the info Michael. It seems KAFKA-3265 would add similar functionality and has a fix version of 0.11. 


On Thursday, April 6, 2017 at 12:19:10 AM UTC-7, Michael Noll wrote:
Scott,

first, setting the replication and retention values is a broker-side operation.  It's not something "the producer does".

Kafka KIP-4 [1] is an ongoing work -- where part of the functionality is already available today -- to have an official "admin API" for creating topics, setting replication factor, etc.

In the meantime, an "unofficial" option you could use is described in my StackOverflow answer at [2].  If you need an up-to-date code example, take a look at [3], which is for Confluent 3.2 and Apache Kafka 0.10.2.0.

Hope this helps!
Michael







On Thu, Apr 6, 2017 at 1:18 AM, Scott Cranford <scottc...@gmail.com> wrote:
Hi,

I am adding a new use case to our Kafka 0.10 cluster that needs different replication and retention values. Is there a way to programmatically set these properties with the Java producer on a per topic basis?

Those two particular properties don't appear to be available: https://kafka.apache.org/documentation/#producerconfigs

Wonder how others have set these properties for dynamically created topics.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Michael Noll

unread,
Apr 6, 2017, 4:22:26 PM4/6/17
to confluent...@googlegroups.com
Scott,

yes, https://issues.apache.org/jira/browse/KAFKA-3265 is part of our KIP-4 work (see the breadcrumb links above the ticket title "Create Java Admin Client").

-Michael


On Thu, Apr 6, 2017 at 8:40 PM, Scott Cranford <scottc...@gmail.com> wrote:
Thanks for the info Michael. It seems KAFKA-3265 would add similar functionality and has a fix version of 0.11. 

On Thursday, April 6, 2017 at 12:19:10 AM UTC-7, Michael Noll wrote:
Scott,

first, setting the replication and retention values is a broker-side operation.  It's not something "the producer does".

Kafka KIP-4 [1] is an ongoing work -- where part of the functionality is already available today -- to have an official "admin API" for creating topics, setting replication factor, etc.

In the meantime, an "unofficial" option you could use is described in my StackOverflow answer at [2].  If you need an up-to-date code example, take a look at [3], which is for Confluent 3.2 and Apache Kafka 0.10.2.0.

Hope this helps!
Michael







On Thu, Apr 6, 2017 at 1:18 AM, Scott Cranford <scottc...@gmail.com> wrote:
Hi,

I am adding a new use case to our Kafka 0.10 cluster that needs different replication and retention values. Is there a way to programmatically set these properties with the Java producer on a per topic basis?

Those two particular properties don't appear to be available: https://kafka.apache.org/documentation/#producerconfigs

Wonder how others have set these properties for dynamically created topics.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.



--
Michael G. Noll
Product Manager | Confluent
Follow us: Twitter | Blog

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a930aa1f-4d29-4132-9db0-7a486ed48e2d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages