Kafka Producer (MirrorMaker) interaction with ZooKeeper

[Stuart Wong]

Hi all,

In a Dockerized setup, I've got a local ZooKeeper and Kafka cluster and a remote ZooKeeper and Kafka cluster in AWS. Example config below:

Local

ZK: 172.17.8.101:2181,172.17.8.102:2181,172.17.8.103:2181

Kafka: 172.17.10.101:2181,172.17.10.102:2181,172.17.10.103:2181

SSH tunnel host: 172.17.10.101

AWS

ZK: 10.183.0.220:2181,10.183.0.219:2181,10.183.0.218:2181

Kafka: 10.183.2.220:9092,10.183.2.219:9092,10.183.2.218:9092

Bastion: 52.50.185.90

I'm trying to setup MirrorMaker using an SSH tunnel to gain access from the local to the remote (in AWS) via a bastion host and am not having much success. The SSH tunnel runs inside of the MirrorMaker container and I've used some `--add-host` trickery to update the MirrorMaker Docker container host for mapping the AWS DNS and IPs to localhost and am able to list and create topics successfully so I know the tunnel works. However, I am unable to produce messages from local to remote via the SSH tunnel as well as using MirrorMaker (expected given the former).

Does anyone have any insights into what I'm missing or how the producer/MirrorMaker is connecting to the remote Brokers and uses ZooKeeper? Any help would be greatly appreciated. I get errors as given below:

[2015-09-08 17:16:20,096] WARN Failed to send producer request with correlation id 2 to broker 1 with data for partitions [cg-mirror,0] (kafka.producer.async.DefaultEventHandler)

java.nio.channels.ClosedChannelException

at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)

at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)

at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)

at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:103)

at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)

at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)

at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)

at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:102)

at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:102)

at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:102)

at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)

at kafka.producer.SyncProducer.send(SyncProducer.scala:101)

at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:255)

at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:106)

at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:100)

at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:778)

at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)

at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)

at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)

at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)

at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)

at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:777)

at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)

at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)

at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)

at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:94)

at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)

-----------------------

Thanks.

[Stuart Wong]

I resolved this by using hostnames for 'advertised.host.name' instead of IPs. That allowed the trickery via `add-host` in the Docker run command to "resolve" to localhost and the SSH tunnel to work.

Thanks for all who viewed and tried their hand to resolve this issue.

[Michael Noll]

Happy to hear you could solve the issue, Stuart -- and thanks for reporting back how you actually fixed it!

--Michael