Historical node: io.druid.segment.loading.SegmentLoadingException for some segments

190 views

Skip to first unread message

Alexander Makarenko

unread,

Mar 23, 2015, 8:28:30 PM3/23/15

to druid...@googlegroups.com

Hello, guys!

I'm using druid 0.7.0. For local FS for deep storage: /druid_san points to SAN storage, filesystem is NFS.

I've got following error message in logs of Historical nodes:

2015-03-23T21:22:43,794 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[d018_2015-03-20T05:00:00.000Z_2015-03-20T06:00:00.000Z_2015-03-20T05:00:00.000Z], segment=DataSegment{size=721834, shardSpec=NoneShardSpec, metrics=[m1, m2], dimensions=[d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15, d16, d17, d18], version='2015-03-20T05:00:00.000Z', loadSpec={type=local, path=/druid_san/deep/d018/2015-03-20T05:00:00.000Z_2015-03-20T06:00:00.000Z/2015-03-20T05:00:00.000Z/0/index.zip}, interval=2015-03-20T05:00:00.000Z/2015-03-20T06:00:00.000Z, dataSource='d018', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[d018_2015-03-20T05:00:00.000Z_2015-03-20T06:00:00.000Z_2015-03-20T05:00:00.000Z]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.0.jar:0.7.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: MALFORMED
at java.util.zip.ZipCoder.toString(ZipCoder.java:58) ~[?:1.7.0_71]
at java.util.zip.ZipCoder.toStringUTF8(ZipCoder.java:117) ~[?:1.7.0_71]
at java.util.zip.ZipInputStream.readLOC(ZipInputStream.java:297) ~[?:1.7.0_71]
at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:121) ~[?:1.7.0_71]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:93) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:81) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.segment.loading.LocalDataSegmentPuller.getSegmentFiles(LocalDataSegmentPuller.java:66) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.0.jar:0.7.0]
... 18 more

2015-03-23T21:22:43,803 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[d018_2015-03-19T20:00:00.000Z_2015-03-19T21:00:00.000Z_2015-03-19T20:00:00.000Z], segment=DataSegment{size=1244861, shardSpec=NoneShardSpec, metrics=[m1, m2], dimensions=[d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15, d16, d17, d18], version='2015-03-19T20:00:00.000Z', loadSpec={type=local, path=/druid_san/deep/d018/2015-03-19T20:00:00.000Z_2015-03-19T21:00:00.000Z/2015-03-19T20:00:00.000Z/0/index.zip}, interval=2015-03-19T20:00:00.000Z/2015-03-19T21:00:00.000Z, dataSource='d018', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[d018_2015-03-19T20:00:00.000Z_2015-03-19T21:00:00.000Z_2015-03-19T20:00:00.000Z]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.0.jar:0.7.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: io.druid.segment.loading.SegmentLoadingException: Unable to unzip file[/druid_san/deep/d018/2015-03-19T20:00:00.000Z_2015-03-19T21:00:00.000Z/2015-03-19T20:00:00.000Z/0/index.zip]
at io.druid.segment.loading.LocalDataSegmentPuller.getSegmentFiles(LocalDataSegmentPuller.java:69) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.0.jar:0.7.0]
... 18 more
Caused by: java.util.zip.ZipException: invalid code lengths set
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164) ~[?:1.7.0_71]
at java.util.zip.ZipInputStream.read(ZipInputStream.java:193) ~[?:1.7.0_71]
at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.7.0_71]
at com.google.common.io.ByteStreams.copy(ByteStreams.java:175) ~[guava-16.0.1.jar:?]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:95) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:81) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.segment.loading.LocalDataSegmentPuller.getSegmentFiles(LocalDataSegmentPuller.java:66) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.0.jar:0.7.0]
... 18 more

2015-03-23T21:23:43,731 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[d018_2015-03-22T22:00:00.000Z_2015-03-22T23:00:00.000Z_2015-03-22T22:00:00.000Z], segment=DataSegment{size=1015195, shardSpec=NoneShardSpec, metrics=[m1, m2], dimensions=[d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15, d16, d17, d18], version='2015-03-22T22:00:00.000Z', loadSpec={type=local, path=/druid_san/deep/d018/2015-03-22T22:00:00.000Z_2015-03-22T23:00:00.000Z/2015-03-22T22:00:00.000Z/0/index.zip}, interval=2015-03-22T22:00:00.000Z/2015-03-22T23:00:00.000Z, dataSource='d018', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[d018_2015-03-22T22:00:00.000Z_2015-03-22T23:00:00.000Z_2015-03-22T22:00:00.000Z]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.0.jar:0.7.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: io.druid.segment.loading.SegmentLoadingException: Unable to unzip file[/druid_san/deep/d018/2015-03-22T22:00:00.000Z_2015-03-22T23:00:00.000Z/2015-03-22T22:00:00.000Z/0/index.zip]
at io.druid.segment.loading.LocalDataSegmentPuller.getSegmentFiles(LocalDataSegmentPuller.java:69) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.0.jar:0.7.0]
... 18 more
Caused by: java.util.zip.ZipException: invalid entry size (expected 1342177814 but got 534 bytes)
at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:403) ~[?:1.7.0_71]
at java.util.zip.ZipInputStream.read(ZipInputStream.java:195) ~[?:1.7.0_71]
at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.7.0_71]
at com.google.common.io.ByteStreams.copy(ByteStreams.java:175) ~[guava-16.0.1.jar:?]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:95) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:81) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.segment.loading.LocalDataSegmentPuller.getSegmentFiles(LocalDataSegmentPuller.java:66) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.0.jar:0.7.0]
... 18 more

2015-03-23T22:14:45,587 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[d018_2015-03-22T04:00:00.000Z_2015-03-22T05:00:00.000Z_2015-03-22T04:00:00.000Z], segment=DataSegment{size=763627, shardSpec=NoneShardSpec, metrics=[m1, m2], dimensions=[d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15, d16, d17, d18], version='2015-03-22T04:00:00.000Z', loadSpec={type=local, path=/druid_san/deep/d018/2015-03-22T04:00:00.000Z_2015-03-22T05:00:00.000Z/2015-03-22T04:00:00.000Z/0/index.zip}, interval=2015-03-22T04:00:00.000Z/2015-03-22T05:00:00.000Z, dataSource='d018', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[d018_2015-03-22T04:00:00.000Z_2015-03-22T05:00:00.000Z_2015-03-22T04:00:00.000Z]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.0.jar:0.7.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: io.druid.segment.loading.SegmentLoadingException: Unable to unzip file[/druid_san/deep/d018/2015-03-22T04:00:00.000Z_2015-03-22T05:00:00.000Z/2015-03-22T04:00:00.000Z/0/index.zip]
at io.druid.segment.loading.LocalDataSegmentPuller.getSegmentFiles(LocalDataSegmentPuller.java:69) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.0.jar:0.7.0]
... 18 more
Caused by: java.util.zip.ZipException: invalid stored block lengths
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164) ~[?:1.7.0_71]
at java.util.zip.ZipInputStream.read(ZipInputStream.java:193) ~[?:1.7.0_71]
at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.7.0_71]
at com.google.common.io.ByteStreams.copy(ByteStreams.java:175) ~[guava-16.0.1.jar:?]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:95) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:81) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.segment.loading.LocalDataSegmentPuller.getSegmentFiles(LocalDataSegmentPuller.java:66) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.0.jar:0.7.0]
... 18 more

2015-03-23T22:40:46,490 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[d018_2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z_2015-03-23T06:00:00.000Z], segment=DataSegment{size=736842, shardSpec=NoneShardSpec, metrics=[m1, m2], dimensions=[d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15, d16, d17, d18], version='2015-03-23T06:00:00.000Z', loadSpec={type=local, path=/druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/index.zip}, interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, dataSource='d018', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[d018_2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z_2015-03-23T06:00:00.000Z]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.0.jar:0.7.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: io.druid.segment.loading.SegmentLoadingException: Unable to unzip file[/druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/index.zip]
at io.druid.segment.loading.LocalDataSegmentPuller.getSegmentFiles(LocalDataSegmentPuller.java:69) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.0.jar:0.7.0]
... 18 more
Caused by: java.util.zip.ZipException: invalid code lengths set
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164) ~[?:1.7.0_71]
at java.util.zip.ZipInputStream.read(ZipInputStream.java:193) ~[?:1.7.0_71]
at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.7.0_71]
at com.google.common.io.ByteStreams.copy(ByteStreams.java:175) ~[guava-16.0.1.jar:?]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:95) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.utils.CompressionUtils.unzip(CompressionUtils.java:81) ~[druid-api-0.3.4.jar:0.3.4]
at io.druid.segment.loading.LocalDataSegmentPuller.getSegmentFiles(LocalDataSegmentPuller.java:66) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.0.jar:0.7.0]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.0.jar:0.7.0]
... 18 more

When I check these index.zip files they are corrupted. Each one (as you may see from ZIP errors) is corrupted in it's own way. For example /druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/index.zip:

[druid@drd5 ~]# zip -T /druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/index.zip
error: invalid compressed data to inflate 00000.smoosh
test of /druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/index.zip FAILED
zip error: Zip file invalid, could not spawn unzip, or wrong unzip (original files unmodified)

It's very strange because Realtime Nodes (the only kind of ingestion we use now) show clean persist-n-merge operations. Below I put logs showing persisting of 2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z segment. We have 2 Realtime Nodes (running on separate machines). One has 2 plumbers:

2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@1981010a, count=0}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@646b449a, count=1}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@3e650df7, count=2}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@5c77f321, count=3}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@46fb2af4, count=4}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@13c11d94, count=5}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@6b7ce9ad, count=6}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@55d193fc, count=7}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@324ab168, count=8}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@45c53ca3, count=9}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@1810151e, count=10}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@47da70e9, count=11}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@562af5d9, count=12}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@1e72a5c1, count=13}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@78e03184, count=14}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@46d30f1b, count=15}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4a84d9f0, count=16}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@580041c8, count=17}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@78dede9a, count=18}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@6fe88e74, count=19}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@ce09b12, count=20}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@6de4c158, count=21}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4551b0c5, count=22}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@762c7910, count=23}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@61c0460c, count=24}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@1636eff7, count=25}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@70c44c85, count=26}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@677be999, count=27}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@782570e2, count=28}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@6216430d, count=29}]
2015-03-23T07:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@238c3cca, count=30}]
2015-03-23T07:15:00,021 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed index.drd in 1 millis.
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@2a459385, count=0}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@3dd81671, count=1}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@35a9746f, count=2}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@81b3eef, count=3}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@34c0c111, count=4}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@5ac35e25, count=5}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@5059fdb0, count=6}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@441e5706, count=7}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@23e36dda, count=8}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@2841159e, count=9}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@48f24cb7, count=10}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@1b8aa7cb, count=11}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@2c0e0e8d, count=12}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4edd6193, count=13}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@28312258, count=14}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@31c5d405, count=15}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@32e2adca, count=16}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@3fffc2df, count=17}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@3052df4c, count=18}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@10c20f5d, count=19}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@347eab43, count=20}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@681dd6a1, count=21}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@7ef6196d, count=22}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4eff8439, count=23}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@6941e2b3, count=24}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@648a8bc1, count=25}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@43dd5f78, count=26}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@7aa6e216, count=27}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@62455142, count=28}]
2015-03-23T07:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@34d9f675, count=29}]
2015-03-23T07:15:00,032 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed index.drd in 1 millis.
2015-03-23T07:15:00,302 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed dim conversions in 281 millis.
2015-03-23T07:15:00,308 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed dim conversions in 276 millis.
2015-03-23T07:15:00,657 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed walk through of 4,923 rows in 355 millis.
2015-03-23T07:15:00,657 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d1] with cardinality[865]
2015-03-23T07:15:00,665 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed walk through of 4,925 rows in 357 millis.
2015-03-23T07:15:00,665 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d1] with cardinality[862]
2015-03-23T07:15:00,810 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d1] in 153 millis.
2015-03-23T07:15:00,810 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d2] with cardinality[6]
2015-03-23T07:15:00,811 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d1] in 146 millis.
2015-03-23T07:15:00,811 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d2] with cardinality[7]
2015-03-23T07:15:00,824 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d2] in 14 millis.
2015-03-23T07:15:00,824 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d3] with cardinality[778]
2015-03-23T07:15:00,826 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d2] in 15 millis.
2015-03-23T07:15:00,827 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d3] with cardinality[780]
2015-03-23T07:15:00,965 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d3] in 139 millis.
2015-03-23T07:15:00,966 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d4] with cardinality[12]
2015-03-23T07:15:00,967 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d3] in 143 millis.
2015-03-23T07:15:00,967 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d4] with cardinality[12]
2015-03-23T07:15:00,985 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d4] in 18 millis.
2015-03-23T07:15:00,985 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d5] with cardinality[99]
2015-03-23T07:15:00,986 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d4] in 21 millis.
2015-03-23T07:15:00,986 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d5] with cardinality[94]
2015-03-23T07:15:01,025 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d5] in 39 millis.
2015-03-23T07:15:01,026 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d6] with cardinality[790]
2015-03-23T07:15:01,026 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d5] in 41 millis.
2015-03-23T07:15:01,026 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d6] with cardinality[814]
2015-03-23T07:15:01,168 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d6] in 142 millis.
2015-03-23T07:15:01,168 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d7] with cardinality[1,000]
2015-03-23T07:15:01,174 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d6] in 148 millis.
2015-03-23T07:15:01,174 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d7] with cardinality[1,014]
2015-03-23T07:15:01,372 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d7] in 204 millis.
2015-03-23T07:15:01,373 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d8] with cardinality[872]
2015-03-23T07:15:01,392 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d7] in 218 millis.
2015-03-23T07:15:01,392 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d8] with cardinality[883]
2015-03-23T07:15:01,513 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d8] in 141 millis.
2015-03-23T07:15:01,513 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d9] with cardinality[25]
2015-03-23T07:15:01,530 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d9] in 17 millis.
2015-03-23T07:15:01,530 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d10] with cardinality[516]
2015-03-23T07:15:01,536 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d8] in 144 millis.
2015-03-23T07:15:01,537 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d9] with cardinality[25]
2015-03-23T07:15:01,557 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d9] in 21 millis.
2015-03-23T07:15:01,559 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d10] with cardinality[516]
2015-03-23T07:15:01,627 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d10] in 97 millis.
2015-03-23T07:15:01,630 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d11] with cardinality[519]
2015-03-23T07:15:01,650 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d10] in 93 millis.
2015-03-23T07:15:01,653 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d11] with cardinality[520]
2015-03-23T07:15:01,725 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d11] in 98 millis.
2015-03-23T07:15:01,728 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d12] with cardinality[301]
2015-03-23T07:15:01,762 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d11] in 111 millis.
2015-03-23T07:15:01,764 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d12] with cardinality[297]
2015-03-23T07:15:01,790 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d12] in 65 millis.
2015-03-23T07:15:01,791 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d13] with cardinality[3]
2015-03-23T07:15:01,800 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d13] in 10 millis.
2015-03-23T07:15:01,800 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d14] with cardinality[6]
2015-03-23T07:15:01,809 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d14] in 9 millis.
2015-03-23T07:15:01,809 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d15] with cardinality[177]
2015-03-23T07:15:01,828 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d12] in 66 millis.
2015-03-23T07:15:01,829 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d13] with cardinality[1]
2015-03-23T07:15:01,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d13] in 8 millis.
2015-03-23T07:15:01,838 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d14] with cardinality[2]
2015-03-23T07:15:01,846 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d14] in 9 millis.
2015-03-23T07:15:01,847 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d15] with cardinality[181]
2015-03-23T07:15:01,851 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d15] in 42 millis.
2015-03-23T07:15:01,852 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d16] with cardinality[6]
2015-03-23T07:15:01,864 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d16] in 13 millis.
2015-03-23T07:15:01,865 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d17] with cardinality[4,909]
2015-03-23T07:15:01,903 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d15] in 57 millis.
2015-03-23T07:15:01,904 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d16] with cardinality[6]
2015-03-23T07:15:01,915 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d16] in 12 millis.
2015-03-23T07:15:01,916 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d17] with cardinality[4,910]
2015-03-23T07:15:02,770 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d17] in 906 millis.
2015-03-23T07:15:02,771 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d18] with cardinality[4]
2015-03-23T07:15:02,771 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d17] in 856 millis.
2015-03-23T07:15:02,771 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d18] with cardinality[4]
2015-03-23T07:15:02,783 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d18] in 13 millis.
2015-03-23T07:15:02,783 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed inverted.drd in 2,118 millis.
2015-03-23T07:15:02,785 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d18] in 14 millis.
2015-03-23T07:15:02,785 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed inverted.drd in 2,128 millis.
2015-03-23T07:15:02,822 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Converting v8[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] to v9[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged]
2015-03-23T07:15:02,823 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d1.drd]
2015-03-23T07:15:02,823 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Converting v8[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] to v9[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged]
2015-03-23T07:15:02,823 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d1] is single value, converting...
2015-03-23T07:15:02,824 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d1.drd]
2015-03-23T07:15:02,824 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d1] is single value, converting...
2015-03-23T07:15:02,824 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d2.drd]
2015-03-23T07:15:02,825 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d2] is single value, converting...
2015-03-23T07:15:02,825 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d2.drd]
2015-03-23T07:15:02,825 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d2] is single value, converting...
2015-03-23T07:15:02,825 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d3.drd]
2015-03-23T07:15:02,826 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d3] is single value, converting...
2015-03-23T07:15:02,826 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d3.drd]
2015-03-23T07:15:02,826 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d3] is single value, converting...
2015-03-23T07:15:02,826 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d4.drd]
2015-03-23T07:15:02,827 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d4] is single value, converting...
2015-03-23T07:15:02,827 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d4.drd]
2015-03-23T07:15:02,827 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d4] is single value, converting...
2015-03-23T07:15:02,827 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d5.drd]
2015-03-23T07:15:02,827 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d5.drd]
2015-03-23T07:15:02,828 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d5] is single value, converting...
2015-03-23T07:15:02,828 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d5] is single value, converting...
2015-03-23T07:15:02,828 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d6.drd]
2015-03-23T07:15:02,828 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d6] is single value, converting...
2015-03-23T07:15:02,828 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d6.drd]
2015-03-23T07:15:02,829 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d6] is single value, converting...
2015-03-23T07:15:02,829 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d7.drd]
2015-03-23T07:15:02,829 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d7] is single value, converting...
2015-03-23T07:15:02,829 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d7.drd]
2015-03-23T07:15:02,830 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d8.drd]
2015-03-23T07:15:02,830 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d7] is single value, converting...
2015-03-23T07:15:02,830 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d8] is single value, converting...
2015-03-23T07:15:02,830 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d9.drd]
2015-03-23T07:15:02,830 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d9] is single value, converting...
2015-03-23T07:15:02,830 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d8.drd]
2015-03-23T07:15:02,831 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d8] is single value, converting...
2015-03-23T07:15:02,831 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d10.drd]
2015-03-23T07:15:02,831 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d10] is single value, converting...
2015-03-23T07:15:02,831 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d11.drd]
2015-03-23T07:15:02,831 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d9.drd]
2015-03-23T07:15:02,832 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d11] is single value, converting...
2015-03-23T07:15:02,832 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d9] is single value, converting...
2015-03-23T07:15:02,832 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d12.drd]
2015-03-23T07:15:02,832 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d12] is single value, converting...
2015-03-23T07:15:02,832 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d10.drd]
2015-03-23T07:15:02,833 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d10] is single value, converting...
2015-03-23T07:15:02,833 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d13.drd]
2015-03-23T07:15:02,833 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d13] is single value, converting...
2015-03-23T07:15:02,833 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d14.drd]
2015-03-23T07:15:02,833 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d11.drd]
2015-03-23T07:15:02,833 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d14] is single value, converting...
2015-03-23T07:15:02,833 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d14] has null rows.
2015-03-23T07:15:02,834 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d11] is single value, converting...
2015-03-23T07:15:02,834 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d15.drd]
2015-03-23T07:15:02,834 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d12.drd]
2015-03-23T07:15:02,834 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d15] is single value, converting...
2015-03-23T07:15:02,835 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d12] is single value, converting...
2015-03-23T07:15:02,835 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d16.drd]
2015-03-23T07:15:02,835 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d16] is single value, converting...
2015-03-23T07:15:02,835 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d13.drd]
2015-03-23T07:15:02,835 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d17.drd]
2015-03-23T07:15:02,836 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d17] is single value, converting...
2015-03-23T07:15:02,836 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d13] is single value, converting...
2015-03-23T07:15:02,836 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d14.drd]
2015-03-23T07:15:02,836 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d18.drd]
2015-03-23T07:15:02,836 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d14] is single value, converting...
2015-03-23T07:15:02,836 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d14] has null rows.
2015-03-23T07:15:02,836 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d18] is single value, converting...
2015-03-23T07:15:02,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[index.drd]
2015-03-23T07:15:02,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[inverted.drd]
2015-03-23T07:15:02,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_m1_LITTLE_ENDIAN.drd]
2015-03-23T07:15:02,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_m2_LITTLE_ENDIAN.drd]
2015-03-23T07:15:02,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[spatial.drd]
2015-03-23T07:15:02,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[time_LITTLE_ENDIAN.drd]
2015-03-23T07:15:02,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d15.drd]
2015-03-23T07:15:02,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Skipped files[[index.drd, inverted.drd, spatial.drd]]
2015-03-23T07:15:02,837 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d15] is single value, converting...
2015-03-23T07:15:02,838 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d16.drd]
2015-03-23T07:15:02,838 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d16] is single value, converting...
2015-03-23T07:15:02,839 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d17.drd]
2015-03-23T07:15:02,839 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d17] is single value, converting...
2015-03-23T07:15:02,840 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.loading.LocalDataSegmentPusher - Compressing files from[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged] to [/druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/index.zip]
2015-03-23T07:15:02,840 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d18.drd]
2015-03-23T07:15:02,840 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d18] is single value, converting...
2015-03-23T07:15:02,841 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[index.drd]
2015-03-23T07:15:02,841 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[inverted.drd]
2015-03-23T07:15:02,841 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_m1_LITTLE_ENDIAN.drd]
2015-03-23T07:15:02,841 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_m2_LITTLE_ENDIAN.drd]
2015-03-23T07:15:02,841 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.utils.CompressionUtils - Adding file[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/meta.smoosh] with size[523]. Total size so far[0]
2015-03-23T07:15:02,841 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[spatial.drd]
2015-03-23T07:15:02,841 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[time_LITTLE_ENDIAN.drd]
2015-03-23T07:15:02,841 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.utils.CompressionUtils - Adding file[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/00000.smoosh] with size[731,701]. Total size so far[523]
2015-03-23T07:15:02,841 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Skipped files[[index.drd, inverted.drd, spatial.drd]]
2015-03-23T07:15:02,842 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.loading.LocalDataSegmentPusher - Compressing files from[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged] to [/druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/index.zip]
2015-03-23T07:15:02,843 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.utils.CompressionUtils - Adding file[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/meta.smoosh] with size[523]. Total size so far[0]
2015-03-23T07:15:02,843 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.utils.CompressionUtils - Adding file[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/00000.smoosh] with size[733,284]. Total size so far[523]
2015-03-23T07:15:02,929 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.utils.CompressionUtils - Adding file[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/version.bin] with size[4]. Total size so far[732,224]
2015-03-23T07:15:02,933 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.loading.LocalDataSegmentPusher - Creating descriptor file at[/druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/descriptor.json]
2015-03-23T07:15:02,935 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.metadata.SQLMetadataSegmentPublisher - Found [d018_2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z_2015-03-23T06:00:00.000Z] in DB, not updating DB
2015-03-23T07:15:02,948 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.utils.CompressionUtils - Adding file[/tmp/realtime/basePersist/2/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/version.bin] with size[4]. Total size so far[733,807]
2015-03-23T07:15:02,950 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.loading.LocalDataSegmentPusher - Creating descriptor file at[/druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/descriptor.json]
2015-03-23T07:15:02,953 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.metadata.SQLMetadataSegmentPublisher - Found [d018_2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z_2015-03-23T06:00:00.000Z] in DB, not updating DB
2015-03-23T08:15:00,019 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43800f19}]
2015-03-23T08:15:00,022 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@47b2a552}]
2015-03-23T09:15:00,019 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43800f19}]
2015-03-23T09:15:00,029 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@47b2a552}]
2015-03-23T10:15:00,018 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43800f19}]
2015-03-23T10:15:18,520 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@47b2a552}]
2015-03-23T11:15:03,486 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43800f19}]
2015-03-23T11:15:03,589 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@47b2a552}]
2015-03-23T12:15:03,710 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@47b2a552}]
2015-03-23T12:15:03,789 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43800f19}]
2015-03-23T12:48:53,802 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@1f30323e}]
2015-03-23T12:48:53,804 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4001f49d}]
2015-03-23T13:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@1f30323e}]
2015-03-23T13:15:00,006 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4001f49d}]
2015-03-23T14:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@1f30323e}]
2015-03-23T14:15:00,005 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4001f49d}]
2015-03-23T15:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@1f30323e}]
2015-03-23T15:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4001f49d}]
2015-03-23T15:56:10,222 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@79610672}]
2015-03-23T15:56:10,246 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@5d3b2f61}]
2015-03-23T16:01:02,919 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4e4f880c}]
2015-03-23T16:01:03,074 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@39af2335}]
2015-03-23T16:41:11,533 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43201cd6}]
2015-03-23T16:41:11,549 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@8f2d5a8}]
2015-03-23T17:15:00,003 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@8f2d5a8}]
2015-03-23T17:15:00,005 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43201cd6}]
2015-03-23T18:15:00,003 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@8f2d5a8}]
2015-03-23T18:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43201cd6}]
2015-03-23T19:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@8f2d5a8}]
2015-03-23T19:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43201cd6}]
2015-03-23T20:15:00,003 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@8f2d5a8}]
2015-03-23T20:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43201cd6}]
2015-03-23T21:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@8f2d5a8}]
2015-03-23T21:15:00,006 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43201cd6}]
2015-03-23T22:15:00,004 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@8f2d5a8}]
2015-03-23T22:15:00,005 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43201cd6}]
2015-03-23T23:15:00,003 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@8f2d5a8}]
2015-03-23T23:15:00,005 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@43201cd6}]

Another one has 1 plumber:

2015-03-23T07:15:00,050 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@48dae47c, count=0}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@1aceae42, count=1}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@77127ccd, count=2}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@5ebfb3cd, count=3}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@7bb58853, count=4}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4b094098, count=5}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4c6fe1a4, count=6}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4546fd30, count=7}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@37987dd8, count=8}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@7c6e0c6b, count=9}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@1cf1849f, count=10}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@3439f191, count=11}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@48228160, count=12}]
2015-03-23T07:15:00,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@515bea9f, count=13}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@69889673, count=14}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@64d8e26, count=15}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4daf6405, count=16}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@35cfac1b, count=17}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@552c3c36, count=18}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4b2528e9, count=19}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@74a0ff89, count=20}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@75e8b730, count=21}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@152ecca, count=22}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@6b37c273, count=23}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@19bf4303, count=24}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@5bc88729, count=25}]
2015-03-23T07:15:00,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4a21c3d2, count=26}]
2015-03-23T07:15:00,053 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@6ebf3603, count=27}]
2015-03-23T07:15:00,053 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@4b7337c2, count=28}]
2015-03-23T07:15:00,053 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@7951c840, count=29}]
2015-03-23T07:15:00,053 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Adding hydrant[FireHydrant{index=null, queryable=io.druid.segment.ReferenceCountingSegment@68321ff9, count=30}]
2015-03-23T07:15:00,059 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed index.drd in 1 millis.
2015-03-23T07:15:00,315 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed dim conversions in 255 millis.
2015-03-23T07:15:00,552 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed walk through of 4,925 rows in 237 millis.
2015-03-23T07:15:00,552 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d1] with cardinality[889]
2015-03-23T07:15:00,779 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d1] in 227 millis.
2015-03-23T07:15:00,779 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d2] with cardinality[6]
2015-03-23T07:15:00,790 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d2] in 11 millis.
2015-03-23T07:15:00,790 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d3] with cardinality[801]
2015-03-23T07:15:00,959 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d3] in 169 millis.
2015-03-23T07:15:00,959 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d4] with cardinality[12]
2015-03-23T07:15:00,974 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d4] in 15 millis.
2015-03-23T07:15:00,974 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d5] with cardinality[95]
2015-03-23T07:15:01,005 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d5] in 31 millis.
2015-03-23T07:15:01,006 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d6] with cardinality[814]
2015-03-23T07:15:01,183 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d6] in 178 millis.
2015-03-23T07:15:01,184 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d7] with cardinality[1,028]
2015-03-23T07:15:01,391 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d7] in 208 millis.
2015-03-23T07:15:01,391 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d8] with cardinality[894]
2015-03-23T07:15:01,571 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d8] in 180 millis.
2015-03-23T07:15:01,572 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d9] with cardinality[25]
2015-03-23T07:15:01,589 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d9] in 17 millis.
2015-03-23T07:15:01,590 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d10] with cardinality[511]
2015-03-23T07:15:01,715 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d10] in 126 millis.
2015-03-23T07:15:01,716 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d11] with cardinality[513]
2015-03-23T07:15:01,834 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d11] in 119 millis.
2015-03-23T07:15:01,834 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d12] with cardinality[286]
2015-03-23T07:15:01,909 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d12] in 75 millis.
2015-03-23T07:15:01,909 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d13] with cardinality[2]
2015-03-23T07:15:01,920 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d13] in 11 millis.
2015-03-23T07:15:01,920 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d14] with cardinality[8]
2015-03-23T07:15:01,931 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d14] in 11 millis.
2015-03-23T07:15:01,931 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d15] with cardinality[185]
2015-03-23T07:15:01,981 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d15] in 50 millis.
2015-03-23T07:15:01,981 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d16] with cardinality[6]
2015-03-23T07:15:01,992 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d16] in 11 millis.
2015-03-23T07:15:01,993 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d17] with cardinality[4,916]
2015-03-23T07:15:03,028 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d17] in 1,034 millis.
2015-03-23T07:15:03,028 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Starting dimension[d18] with cardinality[4]
2015-03-23T07:15:03,039 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - Completed dimension[d18] in 11 millis.
2015-03-23T07:15:03,039 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] completed inverted.drd in 2,487 millis.
2015-03-23T07:15:03,047 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Converting v8[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/v8-tmp] to v9[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged]
2015-03-23T07:15:03,047 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d1.drd]
2015-03-23T07:15:03,048 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d1] is single value, converting...
2015-03-23T07:15:03,049 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d2.drd]
2015-03-23T07:15:03,049 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d2] is single value, converting...
2015-03-23T07:15:03,050 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d3.drd]
2015-03-23T07:15:03,050 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d3] is single value, converting...
2015-03-23T07:15:03,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d4.drd]
2015-03-23T07:15:03,051 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d4] is single value, converting...
2015-03-23T07:15:03,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d5.drd]
2015-03-23T07:15:03,052 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d5] is single value, converting...
2015-03-23T07:15:03,053 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d6.drd]
2015-03-23T07:15:03,053 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d6] is single value, converting...
2015-03-23T07:15:03,054 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d7.drd]
2015-03-23T07:15:03,055 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d7] is single value, converting...
2015-03-23T07:15:03,055 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d8.drd]
2015-03-23T07:15:03,056 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d8] is single value, converting...
2015-03-23T07:15:03,057 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d9.drd]
2015-03-23T07:15:03,057 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d9] is single value, converting...
2015-03-23T07:15:03,057 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d10.drd]
2015-03-23T07:15:03,058 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d10] is single value, converting...
2015-03-23T07:15:03,058 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d11.drd]
2015-03-23T07:15:03,058 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d11] is single value, converting...
2015-03-23T07:15:03,059 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d12.drd]
2015-03-23T07:15:03,059 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d12] is single value, converting...
2015-03-23T07:15:03,060 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d13.drd]
2015-03-23T07:15:03,060 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d13] is single value, converting...
2015-03-23T07:15:03,061 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d14.drd]
2015-03-23T07:15:03,061 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d14] is single value, converting...
2015-03-23T07:15:03,061 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d14] has null rows.
2015-03-23T07:15:03,062 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d15.drd]
2015-03-23T07:15:03,062 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d15] is single value, converting...
2015-03-23T07:15:03,063 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d16.drd]
2015-03-23T07:15:03,063 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d16] is single value, converting...
2015-03-23T07:15:03,064 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d17.drd]
2015-03-23T07:15:03,064 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d17] is single value, converting...
2015-03-23T07:15:03,065 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_d18.drd]
2015-03-23T07:15:03,065 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[d18] is single value, converting...
2015-03-23T07:15:03,066 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[index.drd]
2015-03-23T07:15:03,066 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[inverted.drd]
2015-03-23T07:15:03,066 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_m1_LITTLE_ENDIAN.drd]
2015-03-23T07:15:03,066 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_m2_LITTLE_ENDIAN.drd]
2015-03-23T07:15:03,066 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[spatial.drd]
2015-03-23T07:15:03,066 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[time_LITTLE_ENDIAN.drd]
2015-03-23T07:15:03,066 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.IndexIO$DefaultIndexIOHandler - Skipped files[[index.drd, inverted.drd, spatial.drd]]
2015-03-23T07:15:03,071 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.loading.LocalDataSegmentPusher - Compressing files from[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged] to [/druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/index.zip]
2015-03-23T07:15:03,072 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.utils.CompressionUtils - Adding file[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/meta.smoosh] with size[523]. Total size so far[0]
2015-03-23T07:15:03,073 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.utils.CompressionUtils - Adding file[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/version.bin] with size[4]. Total size so far[523]
2015-03-23T07:15:03,073 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.utils.CompressionUtils - Adding file[/tmp/realtime/basePersist/1/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/merged/00000.smoosh] with size[736,315]. Total size so far[527]
2015-03-23T07:15:03,167 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.loading.LocalDataSegmentPusher - Creating descriptor file at[/druid_san/deep/d018/2015-03-23T06:00:00.000Z_2015-03-23T07:00:00.000Z/2015-03-23T06:00:00.000Z/0/descriptor.json]
2015-03-23T08:15:00,047 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@6c1340f}]
2015-03-23T09:15:00,046 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@6c1340f}]
2015-03-23T10:15:00,047 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@6c1340f}]
2015-03-23T11:15:03,957 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@6c1340f}]
2015-03-23T12:15:04,198 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@6c1340f}]
2015-03-23T12:59:36,036 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4e18e0a6}]
2015-03-23T13:15:00,007 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4e18e0a6}]
2015-03-23T14:15:00,007 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4e18e0a6}]
2015-03-23T15:15:00,006 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4e18e0a6}]
2015-03-23T16:15:00,007 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4e18e0a6}]
2015-03-23T17:15:05,979 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@4e18e0a6}]
2015-03-23T17:27:59,435 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@425c4719}]
2015-03-23T18:15:00,006 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@425c4719}]
2015-03-23T19:15:00,006 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@425c4719}]
2015-03-23T20:15:00,006 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@425c4719}]
2015-03-23T21:15:05,015 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@425c4719}]
2015-03-23T22:15:00,006 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@425c4719}]
2015-03-23T23:15:00,007 INFO [d018-2015-03-23T06:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Already pushed sink[Sink{interval=2015-03-23T06:00:00.000Z/2015-03-23T07:00:00.000Z, schema=io.druid.segment.indexing.DataSchema@425c4719}]

We're checking bad-disk-blocks case but it's very unlikely thing.

So... What could cause such errors? Or what data should I also provide so you can know for sure?

Fangjin Yang

unread,

Mar 24, 2015, 7:37:23 PM3/24/15

to druid...@googlegroups.com

Hi Alexander, are you sure you are correctly writing segments to your storage?

java.util.zip.ZipException: invalid entry size (expected 1342177814 but got 534 bytes)

It seems like the problem is with your deep storage and not with Druid.

...

Alexander Makarenko

unread,

Apr 20, 2015, 5:28:56 PM4/20/15

to druid...@googlegroups.com

Hi, Fangjin. The error is still present, I suppose it may be caused by bad configuration of Realtime Nodes. Problem appeared only after I changed configuration of those nodes:

Previously 1st realtime node had 2 ingestion configurations (each had it's own basePersistDirectory) and 2nd realtime node on different machine had 1 ingestion config.

I shutdown 2nd node and moved it's ingestion configuration to 1st realtime node. So now there are three realtime streams each with different basePersistDirectory path.

May it be a problem ?

...

Alexander Makarenko

unread,

Apr 20, 2015, 5:57:44 PM4/20/15

to druid...@googlegroups.com

Btw, maybe this will also help somehow. Just tried to execute segmentMetadata query on Realtime Node mentioned in previous post (1st):

{
"queryType":"segmentMetadata",
"dataSource":"d018",
"intervals":["2015-04-20T18:00:00Z/2015-04-20T19:00:00Z"]
}

And got following error:

com.metamx.common.ISE: Merging when a merge isn't supposed to happen[SegmentAnalysis{id='d0180_2015-04-20T18:00:00.000Z_2015-04-20T19:00:00.000Z_2015-04-20T18:00:00.000Z', interval=[2015-04-20T18:00:00.000Z/2015-04-20T18:16:00.000Z], size=606329}], [SegmentAnalysis{id='d018_2015-04-20T18:00:00.000Z_2015-04-20T19:00:00.000Z_2015-04-20T18:00:00.000Z', interval=[2015-04-20T18:00:00.000Z/2015-04-20T19:00:00.000Z], size=129871}] (Exception handling request: {class=io.druid.server.QueryResource, exceptionType=class com.metamx.common.ISE, exceptionMessage=Merging when a merge isn't supposed to happen[SegmentAnalysis{id='d018_2015-04-20T18:00:00.000Z_2015-04-20T19:00:00.000Z_2015-04-20T18:00:00.000Z', interval=[2015-04-20T18:00:00.000Z/2015-04-20T18:16:00.000Z], size=606329}], [SegmentAnalysis{id='d018_2015-04-20T18:00:00.000Z_2015-04-20T19:00:00.000Z_2015-04-20T18:00:00.000Z', interval=[2015-04-20T18:00:00.000Z/2015-04-20T19:00:00.000Z], size=129871}], exception=com.metamx.common.ISE: Merging when a merge isn't supposed to happen[SegmentAnalysis{id='d018_2015-04-20T18:00:00.000Z_2015-04-20T19:00:00.000Z_2015-04-20T18:00:00.000Z', interval=[2015-04-20T18:00:00.000Z/2015-04-20T18:16:00.000Z], size=606329}], [SegmentAnalysis{id='d018_2015-04-20T18:00:00.000Z_2015-04-20T19:00:00.000Z_2015-04-20T18:00:00.000Z', interval=[2015-04-20T18:00:00.000Z/2015-04-20T19:00:00.000Z], size=129871}], query=io.druid.query.metadata.metadata.SegmentMetadataQuery@28b6a973, peer=10.18.3.152})
at io.druid.query.metadata.SegmentMetadataQueryQueryToolChest$2$2.apply(SegmentMetadataQueryQueryToolChest.java:103)
at io.druid.query.metadata.SegmentMetadataQueryQueryToolChest$2$2.apply(SegmentMetadataQueryQueryToolChest.java:88)
at io.druid.common.guava.CombiningSequence$CombiningAccumulator.accumulate(CombiningSequence.java:255)
at com.metamx.common.guava.YieldingAccumulators$1.accumulate(YieldingAccumulators.java:32)
at com.metamx.common.guava.BaseSequence.makeYielder(BaseSequence.java:104)
at com.metamx.common.guava.BaseSequence.toYielder(BaseSequence.java:81)
at com.metamx.common.guava.BaseSequence.accumulate(BaseSequence.java:67)
at io.druid.common.guava.CombiningSequence.accumulate(CombiningSequence.java:64)
at com.metamx.common.guava.MappedSequence.accumulate(MappedSequence.java:40)
at com.metamx.common.guava.Sequences.toList(Sequences.java:113)
at io.druid.query.ChainedExecutionQueryRunner$1$1$1.call(ChainedExecutionQueryRunner.java:130)
at io.druid.query.ChainedExecutionQueryRunner$1$1$1.call(ChainedExecutionQueryRunner.java:120)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
at io.druid.query.ChainedExecutionQueryRunner$1$1.apply(ChainedExecutionQueryRunner.java:118)
at io.druid.query.ChainedExecutionQueryRunner$1$1.apply(ChainedExecutionQueryRunner.java:110)
at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at com.google.common.collect.Iterators.addAll(Iterators.java:357)
at com.google.common.collect.Lists.newArrayList(Lists.java:147)
at com.google.common.collect.Lists.newArrayList(Lists.java:129)
at io.druid.query.ChainedExecutionQueryRunner$1.make(ChainedExecutionQueryRunner.java:106)
at com.metamx.common.guava.BaseSequence.toYielder(BaseSequence.java:78)
at io.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:76)
at io.druid.server.QueryResource.doPost(QueryResource.java:161)
at sun.reflect.GeneratedMethodAccessor84.invoke
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)
at java.lang.Thread.run(Thread.java:745)

On Wednesday, March 25, 2015 at 2:37:23 AM UTC+3, Fangjin Yang wrote:

...

Alexander Makarenko

unread,

Apr 22, 2015, 8:26:14 PM4/22/15

to druid...@googlegroups.com

Hi, Fangjin!

It was me who was really-really stupid. I treated shardSpec wrong thinking that partitionNum is somehow related to Kafka partition. So I had 3 ingestion configurations for same data source, with same basePersistDirectory with only difference in shardSpec.partitionNum which resulted in all the errors we previously saw. Now everything is good.

On Wednesday, March 25, 2015 at 2:37:23 AM UTC+3, Fangjin Yang wrote:

...

Eric Tschetter

unread,

Apr 22, 2015, 9:16:47 PM4/22/15

to Alexander Makarenko, druid...@googlegroups.com

Alexander,

Is there something we could have added in the docs that would've
helped clarify this for you more? If so, can you do a PR against the
docs with a suggestion on what you think might have helped you?

--Eric

> --
> You received this message because you are subscribed to the Google Groups
> "Druid User" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to druid-user+...@googlegroups.com.
> To post to this group, send email to druid...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/druid-user/e239e9ab-9c0b-4b00-ad2c-60224bad2264%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

Alexander Makarenko

unread,

Apr 22, 2015, 9:41:16 PM4/22/15

to druid...@googlegroups.com, estlib...@gmail.com

Hi, Eric! Well, as I see it, simple note that partitionNum is not related to Kafka topic partition would be very helpful. So people with good imagination and bad attention would not have a chance. :-) Will make a PR tmrw.

Reply all

Reply to author

Forward

0 new messages