2017-01-03T09:36:05,219 INFO [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] com.metamx.common.CompressionUtils - Adding file[/data/druid/var/druid/task/index_realtime_user_feedback_hour_08081200_2017-01-03T00:00:00.000Z_0_0/work/persist/user_feedback_hour_08081200/2017-01-03T08:00:00.000+08:00_2017-01-03T09:00:00.000+08:00/merged/00000.smoosh] with size[115,722,350]. Total size so far[620]
2017-01-03T09:36:20,876 WARN [ResponseProcessor for block BP-273804461-192.168.1.17-1454415673434:blk_1151925682_95871733] org.apache.hadoop.hdfs.DFSClient - DFSOutputStream ResponseProcessor exception for block BP-273804461-192.168.1.17-1454415673434:blk_1151925682_95871733
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1883) ~[hadoop-hdfs-2.3.0.jar:?]
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:116) ~[hadoop-hdfs-2.3.0.jar:?]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:726) [hadoop-hdfs-2.3.0.jar:?]
2017-01-03T09:36:20,882 WARN [DataStreamer for file /druid-new/storage/user_feedback_hour_08081200/20170103T080000.000+0800_20170103T090000.000+0800/2017-01-03T08_06_00.337+08_00/0/index.zip block BP-273804461-192.168.1.17-1454415673434:blk_1151925682_95871733] org.apache.hadoop.hdfs.DFSClient - Error Recovery for block BP-273804461-192.168.1.17-1454415673434:blk_1151925682_95871733 in pipeline
192.168.1.44:50010,
192.168.1.16:50010: bad datanode
192.168.1.44:500102017-01-03T09:36:20,888 WARN [DataStreamer for file /druid-new/storage/user_feedback_hour_08081200/20170103T080000.000+0800_20170103T090000.000+0800/2017-01-03T08_06_00.337+08_00/0/index.zip block BP-273804461-192.168.1.17-1454415673434:blk_1151925682_95871733] org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception
org.apache.hadoop.ipc.RemoteException: BP-273804461-192.168.1.17-1454415673434:blk_1151925682_95871733 does not exist or is not under Constructionnull
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:7066)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:7133)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:757)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.updateBlockForPipeline(AuthorizationProviderProxyClientProtocol.java:637)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:932)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
at org.apache.hadoop.ipc.Client.call(Client.java:1406) ~[hadoop-common-2.3.0.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1359) ~[hadoop-common-2.3.0.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) ~[hadoop-common-2.3.0.jar:?]
at com.sun.proxy.$Proxy63.updateBlockForPipeline(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:794) ~[hadoop-hdfs-2.3.0.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_45]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) ~[hadoop-common-2.3.0.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[hadoop-common-2.3.0.jar:?]
at com.sun.proxy.$Proxy64.updateBlockForPipeline(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1057) ~[hadoop-hdfs-2.3.0.jar:?]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:838) ~[hadoop-hdfs-2.3.0.jar:?]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:483) [hadoop-hdfs-2.3.0.jar:?]
2017-01-03T09:36:20,895 ERROR [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[user_feedback_hour_08081200]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.lang.IllegalArgumentException, exceptionMessage=Self-suppression not permitted, interval=2017-01-03T08:00:00.000+08:00/2017-01-03T09:00:00.000+08:00}
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[?:1.7.0_45]
at io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:93) ~[?:?]
at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:550) [druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:42) [druid-common-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [?:1.7.0_45]
Caused by: org.apache.hadoop.ipc.RemoteException: BP-273804461-192.168.1.17-1454415673434:blk_1151925682_95871733 does not exist or is not under Constructionnull
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:7066)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:7133)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:757)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.updateBlockForPipeline(AuthorizationProviderProxyClientProtocol.java:637)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:932)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
at org.apache.hadoop.ipc.Client.call(Client.java:1406) ~[?:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1359) ~[?:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) ~[?:?]
at com.sun.proxy.$Proxy63.updateBlockForPipeline(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:794) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_45]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) ~[?:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[?:?]
at com.sun.proxy.$Proxy64.updateBlockForPipeline(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1057) ~[?:?]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:838) ~[?:?]
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:483) ~[?:?]
2017-01-03T09:36:20,910 INFO [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2017-01-03T09:36:20.906+08:00","service":"druid-new/middleManager","host":"nj-p8-192-168-1-135:8105","severity":"component-failure","description":"Failed to persist merged index[user_feedback_hour_08081200]","data":{"class":"io.druid.segment.realtime.plumber.RealtimePlumber","exceptionType":"java.lang.IllegalArgumentException","exceptionMessage":"Self-suppression not permitted","exceptionStackTrace":"java.lang.IllegalArgumentException: Self-suppression not permitted\n\tat java.lang.Throwable.addSuppressed(Throwable.java:1043)\n\tat io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:93)\n\tat io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:550)\n\tat io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:42)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\nCaused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-273804461-192.168.1.17-1454415673434:blk_1151925682_95871733 does not exist or is not under Constructionnull\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:7066)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:7133)\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:757)\n\tat org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.updateBlockForPipeline(AuthorizationProviderProxyClientProtocol.java:637)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:932)\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)\n\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1406)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1359)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)\n\tat com.sun.proxy.$Proxy63.updateBlockForPipeline(Unknown Source)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:794)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:606)\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)\n\tat com.sun.proxy.$Proxy64.updateBlockForPipeline(Unknown Source)\n\tat org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1057)\n\tat org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:838)\n\tat org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:483)\n","interval":"2017-01-03T08:00:00.000+08:00/2017-01-03T09:00:00.000+08:00"}}]
2017-01-03T09:36:20,912 INFO [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[user_feedback_hour_08081200_2017-01-03T08:00:00.000+08:00_2017-01-03T09:00:00.000+08:00_2017-01-03T08:06:00.337+08:00] at path[/druid-new/druid/segments/nj-p8-192-168-1-135:8105/nj-p8-192-168-1-135:8105_realtime__default_tier_2017-01-03T08:06:00.960+08:00_03f38d1781b548e4b8f95b89ed23843b0]
2017-01-03T09:36:20,912 INFO [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] io.druid.curator.announcement.Announcer - unannouncing [/druid-new/druid/segments/nj-p8-192-168-1-135:8105/nj-p8-192-168-1-135:8105_realtime__default_tier_2017-01-03T08:06:00.960+08:00_03f38d1781b548e4b8f95b89ed23843b0]
2017-01-03T09:36:20,926 INFO [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_realtime_user_feedback_hour_08081200_2017-01-03T00:00:00.000Z_0_0]: LockReleaseAction{interval=2017-01-03T08:00:00.000+08:00/2017-01-03T09:00:00.000+08:00}
2017-01-03T09:36:20,928 INFO [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_realtime_user_feedback_hour_08081200_2017-01-03T00:00:00.000Z_0_0] to overlord[
http://druid001:8090/druid/indexer/v1/action]: LockReleaseAction{interval=2017-01-03T08:00:00.000+08:00/2017-01-03T09:00:00.000+08:00}
2017-01-03T09:36:20,928 INFO [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] com.metamx.http.client.pool.ChannelResourceFactory - Generating:
http://druid001:80902017-01-03T09:36:20,936 INFO [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Deleting Index File[/data/druid/var/druid/task/index_realtime_user_feedback_hour_08081200_2017-01-03T00:00:00.000Z_0_0/work/persist/user_feedback_hour_08081200/2017-01-03T08:00:00.000+08:00_2017-01-03T09:00:00.000+08:00]
2017-01-03T09:36:20,937 INFO [user_feedback_hour_08081200-2017-01-03T08:00:00.000+08:00-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Removing sinkKey 1483401600000 for segment user_feedback_hour_08081200_2017-01-03T08:00:00.000+08:00_2017-01-03T09:00:00.000+08:00_2017-01-03T08:06:00.337+08:00
2017-01-03T09:36:20,940 ERROR [task-runner-0-priority-0] io.druid.indexing.common.task.RealtimeIndexTask - Failed to finish realtime task: {class=io.druid.indexing.common.task.RealtimeIndexTask, exceptionType=class com.metamx.common.ISE, exceptionMessage=Exception occurred during persist and merge.}
com.metamx.common.ISE: Exception occurred during persist and merge.
at io.druid.segment.realtime.plumber.RealtimePlumber.finishJob(RealtimePlumber.java:644) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at io.druid.indexing.common.task.RealtimeIndexTask.run(RealtimeIndexTask.java:405) [druid-indexing-service-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:338) [druid-indexing-service-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:318) [druid-indexing-service-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [?:1.7.0_45]
2017-01-03T09:36:20,940 INFO [task-runner-0-priority-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2017-01-03T09:36:20.940+08:00","service":"druid-new/middleManager","host":"nj-p8-192-168-1-135:8105","severity":"component-failure","description":"Failed to finish realtime task","data":{"class":"io.druid.indexing.common.task.RealtimeIndexTask","exceptionType":"com.metamx.common.ISE","exceptionMessage":"Exception occurred during persist and merge.","exceptionStackTrace":"com.metamx.common.ISE: Exception occurred during persist and merge.\n\tat io.druid.segment.realtime.plumber.RealtimePlumber.finishJob(RealtimePlumber.java:644)\n\tat io.druid.indexing.common.task.RealtimeIndexTask.run(RealtimeIndexTask.java:405)\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:338)\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:318)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n"}}]
2017-01-03T09:36:20,942 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[RealtimeIndexTask{id=index_realtime_user_feedback_hour_08081200_2017-01-03T00:00:00.000Z_0_0, type=index_realtime, dataSource=user_feedback_hour_08081200}]
com.metamx.common.ISE: Exception occurred during persist and merge.
at io.druid.segment.realtime.plumber.RealtimePlumber.finishJob(RealtimePlumber.java:644) ~[druid-server-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at io.druid.indexing.common.task.RealtimeIndexTask.run(RealtimeIndexTask.java:405) ~[druid-indexing-service-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:338) [druid-indexing-service-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:318) [druid-indexing-service-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [?:1.7.0_45]
2017-01-03T09:36:20,948 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
"id" : "index_realtime_user_feedback_hour_08081200_2017-01-03T00:00:00.000Z_0_0",
"status" : "FAILED",
"duration" : 5424521
}
2017-01-03T09:36:20,957 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.coordination.AbstractDataSegmentAnnouncer.stop()] on object[io.druid.server.coordination.BatchDataSegmentAnnouncer@56db21c3].
2017-01-03T09:36:20,957 INFO [main] io.druid.server.coordination.AbstractDataSegmentAnnouncer - Stopping class io.druid.server.coordination.BatchDataSegmentAnnouncer with config[io.druid.server.initialization.ZkPathsConfig@775fe927]
2017-01-03T09:36:20,957 INFO [main] io.druid.curator.announcement.Announcer - unannouncing [/druid-new/druid/announcements/nj-p8-192-168-1-135:8105]
2017-01-03T09:36:20,961 INFO [main] org.eclipse.jetty.server.ServerConnector - Stopped ServerConnector@1ec1c237{HTTP/1.1}{
0.0.0.0:8105}
2017-01-03T09:36:20,963 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@26341503{/,null,UNAVAILABLE}
there is always one task failed and other task success.did any one known how to solve it??