java.util.zip.ZipException while loading data from S3 bucket to historical segment_cache

[anindita dey]

Hi, 

I am using druid 0.9.1 and facing the below error while trying to load from S3 bucket ( Deep Storage) to historical segment_cache. 

It will be great, if you could help.

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[customer-scoring_2016-11-28T15:00:00.000Z_2016-11-28T16:00:00.000Z_2017-01-16T22:16:12.103Z]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:309) ~[druid-server-0.9.1.1.jar:0.9.1.1]

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:350) [druid-server-0.9.1.1.jar:0.9.1.1]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:44) [druid-server-0.9.1.1.jar:0.9.1.1]

at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:152) [druid-server-0.9.1.1.jar:0.9.1.1]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522) [curator-recipes-2.10.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.10.0.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-2.10.0.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-2.10.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:514) [curator-recipes-2.10.0.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.10.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:772) [curator-recipes-2.10.0.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_121]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_121]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_121]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_121]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_121]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_121]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_121]

Caused by: io.druid.segment.loading.SegmentLoadingException: error in opening zip file

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:238) ~[?:?]

at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:62) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:143) ~[druid-server-0.9.1.1.jar:0.9.1.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:95) ~[druid-server-0.9.1.1.jar:0.9.1.1]

at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:152) ~[druid-server-0.9.1.1.jar:0.9.1.1]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:305) ~[druid-server-0.9.1.1.jar:0.9.1.1]

... 18 more

Caused by: java.util.zip.ZipException: error in opening zip file

at java.util.zip.ZipFile.open(Native Method) ~[?:1.7.0_121]

at java.util.zip.ZipFile.<init>(ZipFile.java:215) ~[?:1.7.0_121]

at java.util.zip.ZipFile.<init>(ZipFile.java:145) ~[?:1.7.0_121]

at java.util.zip.ZipFile.<init>(ZipFile.java:159) ~[?:1.7.0_121]

at com.metamx.common.CompressionUtils.unzip(CompressionUtils.java:205) ~[java-util-0.27.9.jar:?]

at com.metamx.common.CompressionUtils.unzip(CompressionUtils.java:157) ~[java-util-0.27.9.jar:?]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:207) ~[?:?]

at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:62) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:143) ~[druid-server-0.9.1.1.jar:0.9.1.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:95) ~[druid-server-0.9.1.1.jar:0.9.1.1]

at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:152) ~[druid-server-0.9.1.1.jar:0.9.1.1]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:305) ~[druid-server-0.9.1.1.jar:0.9.1.1]

... 18 more

[anindita dey]

Hi

Just to add to the existing question. I am encrypting the index.zip and descriptor.json files in my deep storage (S3). 

I have all the required *.xml files inside /conf/druid/_common directory.

I am not sure, if the error is due to encrypted files. 

Thanks

ani

[Jay R]

@deya

Currently we are facing the same issue.

By any chance are you able get root cause and solution for this issue.

Thanks

Jay R