Re: [druid-dev] Re: Druid - Hadoop as Deep Storage - InvalidProtocolBufferException

[Fangjin]

Jagadeesh, if you need dedicated assistance for a POC, please try http://imply.io/. We try to offer community assistance as much as we are able but there will be some delay before you can get help

For Hadoop batch ingestion, please try creating a directory with your hadoop *.xml configuration files and putting that directory in the cp of all your Druid processes.

Read more here: http://imply.io/docs/latest/ingestion-batch

On Mon, Apr 18, 2016 at 9:56 AM, JagadeeshM <jagadeesh...@gmail.com> wrote:

Guys - any advice?

HDFS connectivity should be straight forward, not sure what I am missing?

Also, where do we include kerberos keytab file if my authentication with hadoop hdfs is via kerberos ?

Any configuration parameter where  I can specify this ?

I am in the process of evaluating Druid for a very large data set.

Thanks!

On Sunday, April 17, 2016 at 9:47:46 PM UTC-4, JagadeeshM wrote:

Need help integrating hadoop hdfs as a deep storage.

[FYI - I modified all server names to some random text]

To begin, I have few questions - 

1 - with version 0.9.0, what needs to be included in the classpath for hadoop while starting the real-time node?

2 - If I did a local mount of hdfs directory in my host server, what sort of performance issues may occur?

3 - Does the host needs hadoop installation?

[host is where I am running real-time node] Realtime node logs are attached.

Here is how I am starting my realtime node - 

#!/bin/bash


export HADOOP_HOME=/apps/home/eng/eagle-eyes/druid-0.9.0/hadoop-dependencies/hadoop-client/2.3.0


export PATH=$PATH:$HADOOP_HOME:$HADOOP_HOME/bin:


java `cat conf/druid/realtime/jvm.config | xargs` -cp conf/druid/_common:conf/druid/realtime:lib/* io.druid.cli.Main server realtime &

jvm.config

-server
-Xms2g
-Xmx2g
-Dlogfilename=realtime
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Ddruid.realtime.specFile=examples/cache.spec
-Dlogfilename=realtime
-Djava.io.tmpdir=var/tmp

I see the following errors / warning in the log file after pre-aggregation process starts

2016-04-18T01:16:03,809 DEBUG o.a.h.i.Client [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] Connecting to production-hadoop/10.32.82.1:50070

2016-04-18T01:16:03,810 DEBUG o.a.h.s.UserGroupInformation [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] PrivilegedAction as:eng (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:708)

2016-04-18T01:16:03,810 DEBUG o.a.h.s.SaslRpcClient [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] Sending sasl message state: NEGOTIATE

2016-04-18T01:16:03,810 WARN o.a.h.s.UserGroupInformation [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] PriviledgedActionException as:eng (auth:KERBEROS) cause:com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.

2016-04-18T01:16:03,811 DEBUG o.a.h.s.UserGroupInformation [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] PrivilegedAction as:eng (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:637)

2016-04-18T01:16:03,811 WARN o.a.h.i.Client [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] Exception encountered while connecting to the server : com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.

2016-04-18T01:16:03,811 WARN o.a.h.s.UserGroupInformation [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] PriviledgedActionException as:eng (auth:KERBEROS) cause:java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.

2016-04-18T01:16:03,811 DEBUG o.a.h.i.Client [ActivityStats-2016-04-17T15:04:00.000Z-persist-n-merge] closing ipc connection to production-hadoop/10.32.82.1:50070: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.

java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.

at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:674) ~[hadoop-common-2.3.0.jar:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_79]

at javax.security.auth.Subject.doAs(Subject.java:415) ~[?:1.7.0_79]

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) ~[hadoop-common-2.3.0.jar:?]

at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:637) [hadoop-common-2.3.0.jar:?]

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:721) [hadoop-common-2.3.0.jar:?]

at org.apache.hadoop.ipc.Client$Connection.access$2700(Client.java:367) [hadoop-common-2.3.0.jar:?]

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1458) [hadoop-common-2.3.0.jar:?]

at org.apache.hadoop.ipc.Client.call(Client.java:1377) [hadoop-common-2.3.0.jar:?]

at org.apache.hadoop.ipc.Client.call(Client.java:1359) [hadoop-common-2.3.0.jar:?]

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) [hadoop-common-2.3.0.jar:?]

at com.sun.proxy.$Proxy56.mkdirs(Unknown Source) [?:?]

at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) ~[?:?]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]

at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) [hadoop-common-2.3.0.jar:?]

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) [hadoop-common-2.3.0.jar:?]

at com.sun.proxy.$Proxy56.mkdirs(Unknown Source) [?:?]

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:487) [hadoop-hdfs-2.3.0.jar:?]

at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2532) [hadoop-hdfs-2.3.0.jar:?]

at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2503) [hadoop-hdfs-2.3.0.jar:?]

at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:823) [hadoop-hdfs-2.3.0.jar:?]

at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:819) [hadoop-hdfs-2.3.0.jar:?]

at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) [hadoop-common-2.3.0.jar:?]

at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:819) [hadoop-hdfs-2.3.0.jar:?]

at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:812) [hadoop-hdfs-2.3.0.jar:?]

at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1816) [hadoop-common-2.3.0.jar:?]

at io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:87) [druid-hdfs-storage-0.9.0.jar:0.9.0]

at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:550) [druid-server-0.9.0.jar:0.9.0]

at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:42) [druid-common-0.9.0.jar:0.9.0]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]

Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.

at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) ~[protobuf-java-2.5.0.jar:?]

at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) ~[protobuf-java-2.5.0.jar:?]

at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:143) ~[protobuf-java-2.5.0.jar:?]

at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176) ~[protobuf-java-2.5.0.jar:?]

at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188) ~[protobuf-java-2.5.0.jar:?]

at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193) ~[protobuf-java-2.5.0.jar:?]

at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) ~[protobuf-java-2.5.0.jar:?]

at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseFrom(RpcHeaderProtos.java:2344) ~[?:?]

at org.apache.hadoop.ipc.ProtobufRpcEngine$RpcResponseMessageWrapper.parseHeaderFrom(ProtobufRpcEngine.java:412) ~[?:?]

at org.apache.hadoop.ipc.ProtobufRpcEngine$RpcResponseMessageWrapper.parseHeaderFrom(ProtobufRpcEngine.java:388) ~[?:?]

at org.apache.hadoop.ipc.ProtobufRpcEngine$RpcMessageWithHeader.readFields(ProtobufRpcEngine.java:308) ~[?:?]

at org.apache.hadoop.ipc.ProtobufRpcEngine$RpcResponseMessageWrapper.readFields(ProtobufRpcEngine.java:388) ~[?:?]

at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:361) ~[?:?]

at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:547) ~[?:?]

at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367) ~[?:?]

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713) ~[?:?]

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:709) ~[?:?]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_79]

at javax.security.auth.Subject.doAs(Subject.java:415) ~[?:1.7.0_79]

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) ~[?:?]

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:708) ~[?:?]

... 27 more

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/80d9c091-beb1-4fe5-9408-e7fa05dddad6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Charles Allen]

A few things to note (hopefully useful to the general community):

• To include hadoop in the classpath of a task that is launched by the overlord (like the hadoop indexing task) we use http://druid.io/docs/0.9.0/ingestion/batch-ingestion.html hadoopDependencyCoordinates
• There is a way to set default dependency coordinates http://druid.io/docs/0.9.0/operations/other-hadoop.html druid.indexer.task.defaultHadoopCoordinates
• For the HDFS deep storage *extension*, the hadoop jars are packaged in the hdfs extension directory. You can either take those or replace them with ones from your distribution. If you replace them, I *HIGHLY* suggest re-packaging druid with the correct hadoop version, and letting druid figure out the dependency jars.
• There is an issue https://github.com/druid-io/druid/issues/2476 where the hdfs extension pulls in the hadoop jars and those are distributed as part of a hadoop (or spark) task. There isn't a good workaround for this yet, but should only impact folks who may use different versions of hadoop in their cluster.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.