how to set kafka consumer groupid using storm-kafka.

[mykidong]

Hi,

I am using storm-kafka.

From a topic, I want to consume same messages from two different consumer groups, namely 'groupA', 'groupB':

                   msg1, msg2, ...
<a topic> ------------------------->  groupA (consumerA1, consumerA2, ...)
      |
      |
      |           msg1, msg2, ...
     +----------------------------------> groupB(consumerB1, consumerB2, ...)

That is, I want to have two kafka spouts with different groupid for a specific topic.

It seems that there is Kafka Consumer Property, groupid.
 
SpoutConfig() constructor has id argument, is this kafka consumer groupid? 
Or how to set groupid in SpoutConfig for my two different consumer groups?



- Kidong.

[Homer Strong]

No, it's not exactly the same as the consumer groupid, although it is similar. The kafka spout uses the low-level kafka client, whereas the consumer groupid is only a configuration for the high-level/ZK kafka client.

I think you could use SpoutConfig's id as the groupid and get the behavior you want. If you wanted to be super careful you could also store the data under a different ZK root.

[Kidong Lee]

Thank you for your advice, Homer.

I have another question.

In trident topology, TransactionalTridentKafkaSpout can be used.

TransactionalTridentKafkaSpout has no configuration for id or zkRoot using TridentKafkaConfig.

If I want to use it, how to set groupid? or what to do for my case?

- Kidong.

2013/6/15 Homer Strong <homer....@gmail.com>

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Homer Strong]

Trident will use the configured transactional zookeeper root: https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml#L47

Note that trident spout IDs need to be globally unique.

--
You received this message because you are subscribed to a topic in the Google Groups "storm-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/storm-user/zxQunjvjkRI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to storm-user+...@googlegroups.com.