Guys - any advice?HDFS connectivity should be straight forward, not sure what I am missing?Also, where do we include kerberos keytab file if my authentication with hadoop hdfs is via kerberos ?Any configuration parameter where I can specify this ?I am in the process of evaluating Druid for a very large data set.Thanks!
On Sunday, April 17, 2016 at 9:47:46 PM UTC-4, JagadeeshM wrote:Need help integrating hadoop hdfs as a deep storage.[FYI - I modified all server names to some random text]To begin, I have few questions -1 - with version 0.9.0, what needs to be included in the classpath for hadoop while starting the real-time node?2 - If I did a local mount of hdfs directory in my host server, what sort of performance issues may occur?3 - Does the host needs hadoop installation?[host is where I am running real-time node] Realtime node logs are attached.Here is how I am starting my realtime node -
#!/bin/bash
export HADOOP_HOME=/apps/home/eng/eagle-eyes/druid-0.9.0/hadoop-dependencies/hadoop-client/2.3.0
export PATH=$PATH:$HADOOP_HOME:$HADOOP_HOME/bin:
java `cat conf/druid/realtime/jvm.config | xargs` -cp conf/druid/_common:conf/druid/realtime:lib/* io.druid.cli.Main server realtime &
jvm.config
-server
-Xms2g
-Xmx2g
-Dlogfilename=realtime
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Ddruid.realtime.specFile=examples/cache.spec
-Dlogfilename=realtime
-Djava.io.tmpdir=var/tmpI see the following errors / warning in the log file after pre-aggregation process starts2016-04-18T01:16:03,809 DEBUG o.a.h.i.Client [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] Connecting to production-hadoop/10.32.82.1:500702016-04-18T01:16:03,810 DEBUG o.a.h.s.UserGroupInformation [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] PrivilegedAction as:eng (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:708)2016-04-18T01:16:03,810 DEBUG o.a.h.s.SaslRpcClient [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] Sending sasl message state: NEGOTIATE2016-04-18T01:16:03,810 WARN o.a.h.s.UserGroupInformation [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] PriviledgedActionException as:eng (auth:KERBEROS) cause:com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.2016-04-18T01:16:03,811 DEBUG o.a.h.s.UserGroupInformation [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] PrivilegedAction as:eng (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:637)2016-04-18T01:16:03,811 WARN o.a.h.i.Client [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] Exception encountered while connecting to the server : com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.2016-04-18T01:16:03,811 WARN o.a.h.s.UserGroupInformation [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] PriviledgedActionException as:eng (auth:KERBEROS) cause:java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.2016-04-18T01:16:03,811 DEBUG o.a.h.i.Client [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] closing ipc connection to production-hadoop/10.32.82.1:50070: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:674) ~[hadoop-common-2.3.0.jar:?]at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_79]at javax.security.auth.Subject.doAs(Subject.java:415) ~[?:1.7.0_79]at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) ~[hadoop-common-2.3.0.jar:?]at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:637) [hadoop-common-2.3.0.jar:?]at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:721) [hadoop-common-2.3.0.jar:?]at org.apache.hadoop.ipc.Client$Connection.access$2700(Client.java:367) [hadoop-common-2.3.0.jar:?]at org.apache.hadoop.ipc.Client.getConnection(Client.java:1458) [hadoop-common-2.3.0.jar:?]at org.apache.hadoop.ipc.Client.call(Client.java:1377) [hadoop-common-2.3.0.jar:?]at org.apache.hadoop.ipc.Client.call(Client.java:1359) [hadoop-common-2.3.0.jar:?]at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) [hadoop-common-2.3.0.jar:?]at com.sun.proxy.$Proxy56.mkdirs(Unknown Source) [?:?]at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) ~[?:?]at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) [hadoop-common-2.3.0.jar:?]at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) [hadoop-common-2.3.0.jar:?]at com.sun.proxy.$Proxy56.mkdirs(Unknown Source) [?:?]at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:487) [hadoop-hdfs-2.3.0.jar:?]at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2532) [hadoop-hdfs-2.3.0.jar:?]at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2503) [hadoop-hdfs-2.3.0.jar:?]at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:823) [hadoop-hdfs-2.3.0.jar:?]at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:819) [hadoop-hdfs-2.3.0.jar:?]at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) [hadoop-common-2.3.0.jar:?]at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:819) [hadoop-hdfs-2.3.0.jar:?]at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:812) [hadoop-hdfs-2.3.0.jar:?]at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1816) [hadoop-common-2.3.0.jar:?]at io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:87) [druid-hdfs-storage-0.9.0.jar:0.9.0]at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:550) [druid-server-0.9.0.jar:0.9.0]at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:42) [druid-common-0.9.0.jar:0.9.0]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) ~[protobuf-java-2.5.0.jar:?]at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) ~[protobuf-java-2.5.0.jar:?]at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:143) ~[protobuf-java-2.5.0.jar:?]at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176) ~[protobuf-java-2.5.0.jar:?]at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188) ~[protobuf-java-2.5.0.jar:?]at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193) ~[protobuf-java-2.5.0.jar:?]at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) ~[protobuf-java-2.5.0.jar:?]at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseFrom(RpcHeaderProtos.java:2344) ~[?:?]at org.apache.hadoop.ipc.ProtobufRpcEngine$RpcResponseMessageWrapper.parseHeaderFrom(ProtobufRpcEngine.java:412) ~[?:?]at org.apache.hadoop.ipc.ProtobufRpcEngine$RpcResponseMessageWrapper.parseHeaderFrom(ProtobufRpcEngine.java:388) ~[?:?]at org.apache.hadoop.ipc.ProtobufRpcEngine$RpcMessageWithHeader.readFields(ProtobufRpcEngine.java:308) ~[?:?]at org.apache.hadoop.ipc.ProtobufRpcEngine$RpcResponseMessageWrapper.readFields(ProtobufRpcEngine.java:388) ~[?:?]at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:361) ~[?:?]at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:547) ~[?:?]at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367) ~[?:?]at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713) ~[?:?]at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:709) ~[?:?]at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_79]at javax.security.auth.Subject.doAs(Subject.java:415) ~[?:1.7.0_79]at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) ~[?:?]at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:708) ~[?:?]... 27 moreTo view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/80d9c091-beb1-4fe5-9408-e7fa05dddad6%40googlegroups.com.--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.