Partition Key in Kafka Output

[David Kleiner]

Hello,

I'm trying to assign a partition key in kafka output but can't find the right spell.  

Fluentd has default_partition_key, I can't find a setting in librdkafka.

Librdkafka has paritioner strategy which defaults to consistent_random on the key but I can't find a way to specify that key of if it's possible - is it referring to the whole payload as the key? 

The reasoning: getting  time-ordered search results from a specific partition.

Thanks,

David

[Eduardo Silva]

Hi, 

Since Fluent Bit Kafka connector depends on librdkafka, I "think" (not sure) you can tweak the option "partitioner":

https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

Any configuration option that you see in the file above can be set in Fluent Configuration with the prefix rdkafka., e.g:

to set "partitioner" you use "rdkafka.partitioner  SOMEVALUE". 

regards, 

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/f82899f3-a3cb-4e3b-85d5-90f21f15be95%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva
Principal Engineer  | Arm
. . . . . . . . . . . . . . . . . . . . . . . . . . . 
m. +506 70138007
Arm.com
Treasuredata.com

   

[David Kleiner]

Thank you for the pointer, Eduardo.  Which key is being used for partitioning?  I can't find its definition in the librdkafka config  Is it set on the kafka side? 

[Eduardo Silva]

I am not a Kafka user/expert, but looking at the config file I see the relevant keys:

- partition.assignment.strategy: range,roundrobin
- partitioner: random, consistent, consistent_random, murmur2 and murmur2_random

- partitioner_cb -> looks like a callback that can be used to set a partitioner when a producing to RD_KAFKA_PARTITION_UA, but that's not our case

you have to use some of the partitioner's available

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/c9473dc5-2cf1-46dc-893b-7e63a4f060f0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[David Kleiner]

Thanks again, I'll keep looking!