historical node fail - Caused by: io.druid.segment.loading.SegmentLoadingException: S3 fail!

[David Montgomery]

Hi,

What is wrong my my historical server given the below config using 0.7.3?  I am write to s3 using the realtime node and query with the broker and the realtime code.  My historical server fails however.  The access and secret key are the same for the realtime and hsotorical node.  I am seeing data in s3.

So in s3 I see this file

https://console.aws.amazon.com/s3/home?region=eu-central-1#&bucket=druid-development&prefix=development-realtime/v1/datasource-topic-test/2015-07-17T03%3A16%3A00.000Z_2015-07-17T03%3A17%3A00.000Z/2015-07-17T03%3A16%3A00.000Z/0/

descriptor.json
index.zip

yet in the logs I see this:
 Caused by: io.druid.segment.loading.SegmentLoadingException: S3 fail!

druid.host=111.111.111.111:8082
druid.service=historical
druid.port=8082
druid.zk.service.host=222.222.222.222
druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions:0.7.3","io.druid.extensions:mysql-metadata-storage:0.7.3"]
druid.server.maxSize=10000000000
druid.processing.buffer.sizeBytes=100000000
druid.processing.numThreads=1
druid.segmentCache.locations=[{"path": "/tmp/druid/indexCache", "maxSize"\: 10000000000}]
druid.storage.type=s3
druid.s3.secretKey=xxxxxx
druid.s3.accessKey=xxxxxx
druid.storage.bucket=druid-development
druid.storage.baseKey=development-realtime/v1
druid.storage.disableAcl=true



015-07-17T03:33:48,728 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T13:23:00.000Z_2015-07-13T13:24:00.000Z_2015-07-13T13:23:00.000Z]
2015-07-17T03:33:48,736 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,736 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T13:22:00.000Z_2015-07-13T13:23:00.000Z_2015-07-13T13:22:00.000Z]
2015-07-17T03:33:48,750 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,750 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T13:17:00.000Z_2015-07-13T13:18:00.000Z_2015-07-13T13:17:00.000Z]
2015-07-17T03:33:48,757 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,757 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T13:15:00.000Z_2015-07-13T13:16:00.000Z_2015-07-13T13:15:00.000Z]
2015-07-17T03:33:48,771 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,771 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T12:59:00.000Z_2015-07-13T13:00:00.000Z_2015-07-13T12:59:00.000Z]
2015-07-17T03:33:48,777 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,777 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-12T06:12:00.000Z_2015-07-12T06:13:00.000Z_2015-07-12T06:12:00.000Z]

03:17:00.000Z_2015-07-17T03:18:00.000Z_2015-07-17T03:17:00.000Z]
2015-07-17T03:34:13,807 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[datasource-topic-test_2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z_2015-07-17T03:17:00.000Z], segment=DataSegment{size=94448, shardSpec=NoneShardSpec, metrics=[test], dimensions=[browser_name, browser_version, campaign_frequency_cap, campaign_id, carrier, connection_type, country, cpc, device_brand, device_name, device_type, fallback_redirect, hour, id, ip_address, is_robot, lead, offer_conversion_tracking, offer_frequency_cap, offer_id, operating_system, payout, pointing_method, referer, referer_host, traffic_source_id, url, window_top], version='2015-07-17T03:17:00.000Z', loadSpec={type=s3_zip, bucket=druid-development, key=development-realtime/v1/datasource-topic-test/2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z/2015-07-17T03:17:00.000Z/0/index.zip}, interval=2015-07-17T03:17:00.000Z/2015-07-17T03:18:00.000Z, dataSource='datasource-topic-test', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[datasource-topic-test_2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z_2015-07-17T03:17:00.000Z]
    at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.3.jar:0.7.3]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]
    at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
    at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]
    at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_79]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_79]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]
    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
Caused by: io.druid.segment.loading.SegmentLoadingException: S3 fail! Key[s3://druid-development/development-realtime/v1/datasource-topic-test/2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z/2015-07-17T03:17:00.000Z/0/index.zip]
    at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:325) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
    at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
    ... 18 more
Caused by: org.jets3t.service.S3ServiceException: Request Error.
    at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1775) ~[jets3t-0.9.3.jar:0.9.3]
    at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:93) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:319) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:315) ~[?:?]
    at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.27.0.jar:?]
    at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:86) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:313) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
    at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
    ... 18 more
Caused by: org.jets3t.service.impl.rest.HttpException
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:423) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:277) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1038) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2250) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2179) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:2574) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1773) ~[jets3t-0.9.3.jar:0.9.3]
    at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:93) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:319) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:315) ~[?:?]
    at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.27.0.jar:?]
    at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:86) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:313) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
    at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
    ... 18 more
2015-07-17T03:34:13,827 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - zNode[/druid/loadQueue/172.31.4.173:8082/datasource-topic-test_2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z_2015-07-17T03:17:00.000Z] was removed
2015-07-17T03:34:13,829 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - New request[LOAD: datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z] with zNode[/druid/loadQueue/172.31.4.173:8082/datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z].
2015-07-17T03:34:13,829 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Loading segment datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z
2015-07-17T03:34:13,829 INFO [ZkCoordinator-0] io.druid.storage.s3.S3DataSegmentPuller - Pulling index at path[s3://druid-development/development-realtime/v1/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0/index.zip] to outDir[/tmp/druid/indexCache/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0]
2015-07-17T03:34:13,902 INFO [ZkCoordinator-0] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/tmp/druid/indexCache/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0]
2015-07-17T03:34:13,902 INFO [ZkCoordinator-0] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/tmp/druid/indexCache/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z]
2015-07-17T03:34:13,902 INFO [ZkCoordinator-0] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/tmp/druid/indexCache/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z]
2015-07-17T03:34:13,903 WARN [ZkCoordinator-0] io.druid.server.coordination.BatchDataSegmentAnnouncer - No path to unannounce segment[datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z]
2015-07-17T03:34:13,903 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Completely removing [datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z] in [30,000] millis
2015-07-17T03:34:13,906 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Completed request [LOAD: datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z]
2015-07-17T03:34:13,906 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z], segment=DataSegment{size=92850, shardSpec=NoneShardSpec, metrics=[test], dimensions=[browser_name, browser_version, campaign_frequency_cap, campaign_id, carrier, connection_type, country, cpc, device_brand, device_name, device_type, fallback_redirect, hour, id, ip_address, is_robot, lead, offer_conversion_tracking, offer_frequency_cap, offer_id, operating_system, payout, pointing_method, referer, referer_host, traffic_source_id, url, window_top], version='2015-07-17T03:16:00.000Z', loadSpec={type=s3_zip, bucket=druid-development, key=development-realtime/v1/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0/index.zip}, interval=2015-07-17T03:16:00.000Z/2015-07-17T03:17:00.000Z, dataSource='datasource-topic-test', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z]
    at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.3.jar:0.7.3]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]
    at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
    at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]
    at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_79]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_79]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]
    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
Caused by: io.druid.segment.loading.SegmentLoadingException: S3 fail! Key[s3://druid-development/development-realtime/v1/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0/index.zip]
    at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:325) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
    at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
    ... 18 more
Caused by: org.jets3t.service.S3ServiceException: Request Error.
    at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1775) ~[jets3t-0.9.3.jar:0.9.3]
    at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:93) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:319) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:315) ~[?:?]
    at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.27.0.jar:?]
    at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:86) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:313) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
    at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
    ... 18 more
Caused by: org.jets3t.service.impl.rest.HttpException
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:423) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:277) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1038) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2250) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2179) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:2574) ~[jets3t-0.9.3.jar:0.9.3]
    at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1773) ~[jets3t-0.9.3.jar:0.9.3]
    at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:93) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:319) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:315) ~[?:?]
    at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.27.0.jar:?]
    at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:86) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:313) ~[?:?]
    at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
    at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
    at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
    ... 18 more
2015-07-17T03:34:13,916 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - zNode[/druid/loadQueue/172.31.4.173:8082/datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z] was removed
2015-07-17T03:34:13,925 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - New request[LOAD: datasource-topic-test_2015-07-17T03:15:00.000Z_2015-07-17T03:16:00.000Z_2015-07-17T03:15:00.000Z] with zNode[/druid/loadQueue/172.31.4.173:8082/datasource-topic-test_2015-07-17T03:15:00.000Z_2015-07-17T03:16:00.000Z_2015-07-17T03:15:00.000Z].

[David Montgomery]

PS I can download the index.zip from the s3 console.

[David Montgomery]

PPS

This is how I start my hasotorical server

/usr/bin/java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath
/var/druid-0.7.3/lib/*:/var/druid-0.7.3/config/historical io.druid.cli.Main server historical

On Friday, July 17, 2015 at 10:52:35 AM UTC+7, David Montgomery wrote:

[David Montgomery]

PPS...


The s3 bucket is in Frankfurt.  Will that have any issues?  i tried to use boto to test but there were some errors with that region due to some of the features of s3 not fully supported.

Thanks

On Friday, July 17, 2015 at 10:52:35 AM UTC+7, David Montgomery wrote:

[Gian Merlino]

Jets3t didn't work with eu-central-1 until 0.9.3, but we are using 0.9.3 so in theory it should work. But, that region is a bit special. Maybe there's some edge case that still isn't working in jets3t.

Can you turn on debug logging for org.jets3t and try again? The response and its headers should get printed in the logs when you do that.

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/a83cf3ba-76a1-41e1-9288-6bebeafa89ed%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Shuai Chang]

Gian, we are using Druid 0.8.3, where middle managers can upload to S3 fine in eu-central-1 but cannot download index. The version of Druid is using jets3t 0.9.4, any idea how we can make it work in such AWS regions?

On Friday, July 17, 2015 at 10:52:08 AM UTC-7, Gian Merlino wrote:

Jets3t didn't work with eu-central-1 until 0.8.3, but we are using 0.9.3 so in theory it should work. But, that region is a bit special. Maybe there's some edge case that still isn't working in jets3t.

[Shuai Chang]

Gian, I was able to investigate the issue more after turning on the debug log, the resolution is in https://groups.google.com/forum/#!topic/druid-user/i3qK0u5BDGM. it seems something is missing from Druid side rather than jets3t since the segment upload from MM was OK but not good from historical

On Friday, July 17, 2015 at 10:52:08 AM UTC-7, Gian Merlino wrote:

[Fangjin Yang]

V4 auth problems:

The underlying problems are https://issues.apache.org/jira/browse/HADOOP-9248 and https://issues.apache.org/jira/browse/HADOOP-13325

The confirmed workaround is:

1) Clone Druid master, add `case "s3a":` at line 404 of JobHelper.java, change aws-java-sdk to version 1.7.4 in pom.xml, rebuild

2) In common.runtime.properties, configure S3 deep storage as normal.

3) Save a file in conf/druid/_common/jets3t.properties with the contents:

s3service.s3-endpoint = s3.ap-northeast-2.amazonaws.com

storage-service.request-signature-version=AWS4-HMAC-SHA256

4) Run: java   -cp "dist/druid/lib/*"   -Ddruid.extensions.directory="dist/druid/extensions"   -Ddruid.extensions.hadoopDependenciesDir="dist/druid/hadoop-dependencies"   io.druid.cli.Main tools pull-deps   --no-default-hadoop   -h "org.apache.hadoop:hadoop-client:2.7.2" -h "org.apache.hadoop:hadoop-aws:2.7.2"

5) In druid.indexer.runner.javaOpts on middleManager, add -Dcom.amazonaws.services.s3.enableV4

6) In job json, "hadoopDependencyCoordinates" : ["org.apache.hadoop:hadoop-client:2.7.2", "org.apache.hadoop:hadoop-aws:2.7.2"]

7) In job json, "jobProperties" : {

     "fs.s3.impl" : "org.apache.hadoop.fs.s3a.S3AFileSystem",

     "fs.s3n.impl" : "org.apache.hadoop.fs.s3a.S3AFileSystem",

     "fs.s3a.endpoint" : "s3.ap-northeast-2.amazonaws.com",

     "fs.s3a.access.key" : "XXX",

     "fs.s3a.secret.key" : "YYY"

[anshul....@c1exchange.com]

Hi,

I'm facing the same issue. Any update to these instructions?

I have done everything except 1) 

I believe the code change is unnecessary now, due to https://github.com/druid-io/druid/pull/4116 and the aws-java-sdk version is also changed on druid master

I'm going to try with master but is there a workaround for older stabler versions? 

[Lawrence Huang]

See my comment here for using s3a deep storage:

https://groups.google.com/d/msg/druid-user/i3qK0u5BDGM/iyjShu8EAQAJ