Why simple kafka create topic does not work on CDAP virtual machine

[manje...@gmail.com]

Objective to run kafka-cdap-sample  as per this page http://docs.cask.co/cdap/3.5.0/en/examples-manual/how-to-guides/cdap-kafka-ingest-guide.html#build-and-run-application

1. Downloaded CDAP 3.5.1

2. netstat -tl ==> Port 2181, and 9092 are listening which means Zookeeper and Kafka are already started

3. Just want to create a topic for testing, so followed instruction, as per section "Publish Messages to a Kakfa topic"

downloaded kafka, https://archive.apache.org/dist/kafka/0.8.0/kafka_2.8.0-0.8.0.tar.gz

you cannot start new kafka-instance as port 9092 is taken, so I went ahead and created a topic 

$ bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 1 --partition 1 --topic MyTopic

reation failed because of org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids

org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids

what needs to be done to create a topic ??

Thanks

[Terence Yim]

Hi,

How do you start the zookeeper and kafka? Are you using the one that comes with CDAP SDK? If that’s the case, you need to pass in the zookeeper root namespace to the kafka-create-topic command as well since the kafka that started by CDAP SDK doesn’t use “/“ in zookeeper.

kaka-create-topic.sh --zookeeper localhost:2181/kafka ….

Terence

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/57f24723-a343-4e0d-b261-81b900698968%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[manje...@gmail.com]

Hi Terence, Thanks for your response.

1 . I gave up on using CDAP kafka to create topic, so instead what I did is started an new zookeeper and kafka at 2182 and 9093. I believe that should be fine.

2.  I download the sample from  https://github.com/cdap-guides/cdap-kafka-ingest-guide

3.  add a missing log4j dependecy and build that package as is , but for version CDAp 3.5.1

4. When I start the flow "start flow KafkaIngestionApp.KafkaIngestionFlow", I get a null pointer

I just want to build and run a simple kafka-application on CDAP 3.5.1 ... thanks for you help

2016-09-24 13:50:40,030 - INFO  [FlowletRuntimeService STARTING:c.c.c.i.a.r.f.FlowletRuntimeService@119] - Flowlet initialized: flowlet=kafkaConsumer, instance=0, groupsize=1, namespaceId=default, applicationId=KafkaIngestionApp, program=KafkaIngestionFlow, runid=dfdf94c7-825d-11e6-9c86-080027e799c9
2016-09-24 13:50:40,660 - WARN  [FlowletProcessDriver-kafkaConsumer-0-executor:c.c.c.i.a.r.f.FlowletProcessDriver@364] - Process failure: flowlet=kafkaConsumer, instance=0, groupsize=1, namespaceId=default, applicationId=KafkaIngestionApp, program=KafkaIngestionFlow, runid=dfdf94c7-825d-11e6-9c86-080027e799c9, null, input: NullInputDatum{retries=0}
java.lang.NullPointerException: null
	at co.cask.cdap.kafka.flow.KafkaConsumerFlowlet$4.apply(KafkaConsumerFlowlet.java:390) ~[cdap-kafka-flow-core-0.9.0.jar:na]
	at co.cask.cdap.kafka.flow.KafkaConsumerFlowlet$4.apply(KafkaConsumerFlowlet.java:387) ~[cdap-kafka-flow-core-0.9.0.jar:na]
	at co.cask.cdap.kafka.flow.KafkaConsumerFlowlet.decodeKey(KafkaConsumerFlowlet.java:275) ~[cdap-kafka-flow-core-0.9.0.jar:na]
	at co.cask.cdap.kafka.flow.KafkaConsumerFlowlet.processMessage(KafkaConsumerFlowlet.java:246) ~[cdap-kafka-flow-core-0.9.0.jar:na]
	at co.cask.cdap.kafka.flow.KafkaConsumerFlowlet.pollMessages(KafkaConsumerFlowlet.java:144) ~[cdap-kafka-flow-core-0.9.0.jar:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_75]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_75]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_75]
	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_75]
	at co.cask.cdap.internal.app.runtime.flow.ReflectionProcessMethod.invoke(ReflectionProcessMethod.java:95) ~[co.cask.cdap.cdap-app-fabric-3.5.1.jar:na]
	at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.handleProcessEntry(FlowletProcessDriver.java:246) [co.cask.cdap.cdap-app-fabric-3.5.1.jar:na]
	at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver.access$000(FlowletProcessDriver.java:61) [co.cask.cdap.cdap-app-fabric-3.5.1.jar:na]
	at co.cask.cdap.internal.app.runtime.flow.FlowletProcessDriver$1.run(FlowletProcessDriver.java:194) [co.cask.cdap.cdap-app-fabric-3.5.1.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_75]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

[Terence Yim]

Hi,

It is caused by a Kafka passing in a null ByteBuffer instead of an empty one to denote a message without key. This behavior changed between Kafka 0.8.1 and 0.8.2.

For the meantime, I’ve put in a fix in the guide to avoid the null point exception to happen. Please pull the latest from github and try it again.

Thanks,

Terence

-- 
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/80690c34-1b4f-4c4d-a4a6-5481ad98d05b%40googlegroups.com.

[manje...@gmail.com]

It worked like a charm ... thanks Terence for your help .. 

We have started using CDAP in our organization, for example if I want to learn a framework and understand the details I'll read a book, for CDAP is there any document that you would like to refer ... I have trying the sample and learning it .. if I want to formally learn the framework .. what is he best way ? and if you could provide any helpful links it would be great   .. thanks again ...

 

On Saturday, September 24, 2016 at 3:57:01 AM UTC, manje...@gmail.com wrote:

[Terence Yim]

Hi,

Learning from examples and guides are definitely good place to start. You can also find extensive documentations about CDAP in http://docs.cask.co/cdap/3.5.1/en/index.html

Terence

Sent from my iPhone

--

You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/1cde1ed9-72f9-425f-93ab-c952188de2ae%40googlegroups.com.