Can I use Kafka Connect to pull the data from Event Hubs to Kafka Topics?

[vivek sharma]

Hi Mates - Can I use Kafka Connect to pull the data from Event Hubs to Kafka Topics? If not is there any other API I need to use? Any help will be very much appreciated !!!!

[ha...@confluent.io]

Most likely yes. Which specific Event Hubs did you have mind?

-hans 

[vivek sharma]

trying to pull from Microsoft Azure Event Hubs. I know it is very easy with Spark Streaming but trying to figure out how to do without micro-batch with spark but with Kafka Sreams

[ha...@confluent.io]

I haven't integrated with Azure Event hub before but thought these connectors might be helpful

https://github.com/Azure/toketi-kafka-connect-iothub

https://github.com/ppatierno/kafka-connect-amqp

-hans

Sent from my iPhone

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/EAh51yESyts/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a715dd24-6993-4c24-b556-a8d712f21be6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vivek sharma]

Thanks Hans to share these links, all this information was pretty useful.. however I am still not able to use Kafka Connect, it says it started but may be I don't know how to use it to consume the messages from event hubs, there are couple questions if you or someone from your team could help, it will be great, I am using the same command for distributed in the same way as stand alone, not sure what I am missing:

1. Is it mandatory to use Rest API for Kafka Connect in distributed mode? For Post method I am getting "Method Not Allowed" error.

2. Do I need to ignore the below error I am seeing in my Kafka Manager logs?

[warn] k.m.a.c.KafkaManagedOffsetCache - Failed to process a message from offset topic on cluster lashmdevkafkav2-ssh.azurehdinsight.net!
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topics': Error reading array of size 1468532, only 28 bytes available
        at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na]
        at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeSubscription(ConsumerProtocol.java:97) ~[org.apache.kafka.kafka-clients-0.
        at kafka.manager.utils.zero90.GroupMetadataManager$$anonfun$readGroupMessageValue$1.apply(GroupMetadataManager.scala:311) ~[kafka-manager.kafka-manag
        at kafka.manager.utils.zero90.GroupMetadataManager$$anonfun$readGroupMessageValue$1.apply(GroupMetadataManager.scala:304) ~[kafka-manager.kafka-manag

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ~[org.scala-lang.scala-library-2.11.8.jar:na]

3. Is there any other way than Control Center to see how many records Kafka Connect ingesting?

Once I am able to successfully use Kafka Connect, I will have couple more questions, Sorry about that :)

On Sunday, February 19, 2017 at 1:17:40 PM UTC-8, ha...@confluent.io wrote:

I haven't integrated with Azure Event hub before but thought these connectors might be helpful

https://github.com/Azure/toketi-kafka-connect-iothub

https://github.com/ppatierno/kafka-connect-amqp

-hans

Sent from my iPhone

On Feb 19, 2017, at 12:30 PM, vivek sharma <vivek.gu...@gmail.com> wrote:

trying to pull from Microsoft Azure Event Hubs. I know it is very easy with Spark Streaming but trying to figure out how to do without micro-batch with spark but with Kafka Sreams

On Sunday, February 19, 2017 at 12:03:59 PM UTC-8, ha...@confluent.io wrote:

Most likely yes. Which specific Event Hubs did you have mind?

-hans 

On Thursday, February 16, 2017 at 3:36:51 PM UTC-8, vivek sharma wrote:

Hi Mates - Can I use Kafka Connect to pull the data from Event Hubs to Kafka Topics? If not is there any other API I need to use? Any help will be very much appreciated !!!!

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/EAh51yESyts/unsubscribe.

To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.