Error while Get Data from Kafka (Soruce)

[Yogesh Sharma]

Hi,

We are getting below error when we try to get the data from kafka topic

2015-09-27 04:49:05 +0530 [error]: Poseidon::Errors::UnknownTopicOrPartition

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/poseidon-0.0.5/lib/poseidon/partition_consumer.rb:193:in `resolve_offset_if_necessary'

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/poseidon-0.0.5/lib/poseidon/partition_consumer.rb:136:in `next_offset'

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/poseidon-0.0.5/lib/poseidon/partition_consumer.rb:218:in `build_topic_fetch_request'

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/poseidon-0.0.5/lib/poseidon/partition_consumer.rb:107:in `fetch'

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-kafka-0.0.17/lib/fluent/plugin/in_kafka.rb:137:in `consume'

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-kafka-0.0.17/lib/fluent/plugin/in_kafka.rb:125:in `call'

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-kafka-0.0.17/lib/fluent/plugin/in_kafka.rb:125:in `on_timer'

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'

  2015-09-27 04:49:05 +0530 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-kafka-0.0.17/lib/fluent/plugin/in_kafka.rb:96:in `run'

2015-09-27 04:49:06 +0530 [error]: Poseidon::Errors::UnknownTopicOrPartition

Below is source:

  <source>

    type kafka

    host 10.2.18.1

    port 9092

    topics mobapi

    format text

  </source>

All connectivity and Topic and kafka is good.

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic mobapi

Topic:mobapi PartitionCount:4 ReplicationFactor:2 Configs:

Topic: mobapi Partition: 0 Leader: 30022973 Replicas: 30022973,300224209 Isr: 30022973,300224209

Topic: mobapi Partition: 1 Leader: 300224209 Replicas: 300224209,300231128 Isr: 300224209,300231128

Topic: mobapi Partition: 2 Leader: 300231128 Replicas: 300231128,30022973 Isr: 300231128,30022973

Topic: mobapi Partition: 3 Leader: 30022973 Replicas: 30022973,300231128 Isr: 30022973,300231128

Please suggest.

Best Regards,

__________________________________________

Yogesh Sharma

Email: yks...@gmail.com | Web: www.initd.in 

RHCE, VCE-CIA, RACKSPACE CLOUD U Certified

      

[Kazuki Ohta]

Yogesh,

We're working on having Kafka -> Fluentd connector. It's still in alpha phase, but it's written in Java. So will be more robust and capable compared to Ruby's implementation. Maybe you want to give it a try.

> https://github.com/treasure-data/kafka-fluentd-consumer

Thanks -K

==

Kazuki Ohta | Founder and CTO, Treasure Data

Cell: +1-650-223-5679

k...@treasure-data.com | www.treasuredata.com

Follow us:    Check the Service Intro Slide!

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.