realnode deepstore promble

[北斗七]

hi：

when  i use realnoe by set deepstore to HDFS, an error happend, is there someting wrong ?thanks

2015-03-04 07:50:47,710 ERROR [druidtestwww12-2015-03-04T04:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[druidtestwww12]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.io.FileNotFoundException, exceptionMessage=hdfs:/mycluster:8020/user/test/druidtest/druidtestwww12/2015-03-04T04:00:00.000Z_2015-03-04T05:00:00.000Z/2015-03-04T04:00:00.000Z/0/index.zip (No such file or directory), interval=2015-03-04T04:00:00.000Z/2015-03-04T05:00:00.000Z}

java.io.FileNotFoundException: hdfs:/mycluster:8020/user/test/druidtest/druidtestwww12/2015-03-04T04:00:00.000Z_2015-03-04T05:00:00.000Z/2015-03-04T04:00:00.000Z/0/index.zip (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at io.druid.utils.CompressionUtils.zip(CompressionUtils.java:35)
at io.druid.segment.loading.LocalDataSegmentPusher.push(LocalDataSegmentPusher.java:82)
at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:362)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

[Fangjin Yang]

Hi, the error is stating that you tried to write a merged index file to hdfs:/mycluster:8020, but the operation appears to have failed. Are you sure you have a valid URI configured?

[北斗七]

i use cdh4.2 version, druid-0.6.171

here is the realtime config:

druid.storage.type=Hdfs

druid.storage.storageDirectory=hdfs://mycluster\:8020/user/qixing/druidtest

i don't know why  // to  /

and start it by  java -server -Xmx1512m -XX:+UseG1GC -XX:MaxGCPauseMillis=100   -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=realtime_www2kafka.spec -classpath lib/*:config/realtime:`hadoop classpath` io.druid.cli.Main server realtime

so  search , the https://github.com/druid-io/druid/pull/1022

but not ok

then i found  some 

a) druid.storage.type=Hdfs ,druid.storage.type=hdfs is not different 

b)7 ERROR [druidtestwww16-2015-03-05T09:39:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[druidtestwww16]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.lang.UnsupportedOperationException, exceptionMessage=This is supposed to be overridden by subclasses., interval=2015-03-05T09:39:00.000Z/2015-03-05T09:40:00.000Z}

java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

        at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)

        at org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407)

        at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749)

        at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:530)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$MkdirsRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:16969)

        at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193)

        at com.sun.proxy.$Proxy108.mkdirs(Unknown Source)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:425)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:483)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)

        at com.sun.proxy.$Proxy109.mkdirs(Unknown Source)

        at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2108)

        at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2079)

        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:543)

        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1867)

        at io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:77)

        at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:362)

        at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:42)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

c) mkdri hdfs:  dir  on local hdfs 

sorry for my poor language,thanks very much for help. 

[Fangjin Yang]

Hi, if you haven't had a chance to read this yet, please do so:

http://druid.io/docs/0.7.0/Other-Hadoop.html (the version is 0.7.0 but the information is still relevant for your druid version)

What is the version of Hadoop that is included with CDH 4.2?

[北斗七]

is CDH4.2 

--
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/c2dvJOi7f30/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/98f0f882-10aa-4b2f-a05d-f2c1a76414ed%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fangjin Yang]

Hi,

Getting different versions of Hadoop to work with Druid is always interesting. Looking online, it looks like CDH4.2 bundles Hadopo 2.0.0 with it (although I don't know if CDH's versioning scheme matches the community version at all). Please see our notes on getting 2.x versions of Hadoop to work with Druid.

http://druid.io/docs/0.7.0/Other-Hadoop.html

is CDH4.2 

To unsubscribe from this group and all its topics, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.

[北斗七]

hello.i use the version druid0.7,and recompile with  cdh5.

but when i start realtime node with deepstore type ad hdfs

is not work well. merge data to local,like hdfs:

info:

merged] to [hdfs:/mycluster:8020 

is parse  hdfs:// to hdfs:/.so merge to local filesystem

is there something wrong to me .

here is the config:

:

druid.extensions.coordinates=["io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage"]

hadoopDependencyCoordinates=["org.apache.hadoop:hadoop-client:2.3.0-cdh5.0.0"]

druid.storage.type=hdfs                                                                                                             

druid.storage.storageDirectory=hdfs\://mycluster\:8020/user/****/druidtest7

thanks.

To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/65c94169-54f2-4c82-97ec-610740b556ff%40googlegroups.com.

[Fangjin Yang]

Hi, do you mind showing me the command line you are using to start up the node that runs your hadoop job?

[北斗七]

i recompile druid0.7 with CDH5.3,  code patch  https://github.com/druid-io/druid/pull/1022 from this ，then maybe works. Thanks. 

To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/c39f2043-2a2c-4121-b42d-ba8a05b3bc66%40googlegroups.com.

[Fangjin Yang]

Hi,

You may also want to check out this recent thread about Druid and CDH:

https://groups.google.com/forum/#!topic/druid-user/bDLJ3iY8RRQ