java.lang.OutOfMemoryError: Java heap space - With mirror maker tool

[Varun Saini]

Hi,

I am trying mirror-maker to mirror some topics. But when I run it I get OutOfMemoryError exception. 

[2016-09-14 21:41:14,270] ERROR Uncaught exception in kafka-producer-network-thread | producer-1:  (org.apache.kafka.common.utils.KafkaThread)

java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
        at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
        at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
        at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:323)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:229)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
        at java.lang.Thread.run(Thread.java:745)

I tried to add heap memory by doing 

java -Xmx1G ./kafka-mirror-maker --consumer.config ../etc/kafka/mirror-consumer.properties --producer.config ../etc/kafka/mirror-producer.properties --whitelist snapshot-users-1 num.producers 2 --num.streams 4

but then it is not able to find or load the class. Please help.

Thanks,

Varun

[ha...@confluent.io]

--num.producers 2

You are missing the "--"

-hans

[Varun Saini]

Well, I tried that and it error out - 

./kafka-mirror-maker --consumer.config ../etc/kafka/mirror-consumer.properties --producer.config ../etc/kafka/mirror-producer.properties --num.producers 2 --num.streams 4 --whitelist snapshot-users-1
[2016-09-15 16:26:35,146] ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$)
joptsimple.UnrecognizedOptionException: num.producers is not a recognized option
        at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
        at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:449)
        at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
        at joptsimple.OptionParser.parse(OptionParser.java:381)
        at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:167)
        at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
Exception in thread "main" java.lang.NullPointerException
        at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)

        at kafka.tools.MirrorMaker.main(MirrorMaker.scala)

[Varun Saini]

It is working now. Thanks.

[fumin wang]

Hi Varun, How did you resolve the issue? I am seeing the same problem..

在 2016年9月15日星期四 UTC-4下午6:05:24，Varun Saini写道：