Hi,
What is wrong my my historical server given the below config using 0.7.3? I am write to s3 using the realtime node and query with the broker and the realtime code. My historical server fails however. The access and secret key are the same for the realtime and hsotorical node. I am seeing data in s3.
So in s3 I see this file
https://console.aws.amazon.com/s3/home?region=eu-central-1#&bucket=druid-development&prefix=development-realtime/v1/datasource-topic-test/2015-07-17T03%3A16%3A00.000Z_2015-07-17T03%3A17%3A00.000Z/2015-07-17T03%3A16%3A00.000Z/0/descriptor.json
index.zip
yet in the logs I see this:
Caused by: io.druid.segment.loading.SegmentLoadingException: S3 fail!
druid.host=
111.111.111.111:8082druid.service=historical
druid.port=8082
druid.zk.service.host=222.222.222.222
druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions:0.7.3","io.druid.extensions:mysql-metadata-storage:0.7.3"]
druid.server.maxSize=10000000000
druid.processing.buffer.sizeBytes=100000000
druid.processing.numThreads=1
druid.segmentCache.locations=[{"path": "/tmp/druid/indexCache", "maxSize"\: 10000000000}]
druid.storage.type=s3
druid.s3.secretKey=xxxxxx
druid.s3.accessKey=xxxxxx
druid.storage.bucket=druid-development
druid.storage.baseKey=development-realtime/v1
druid.storage.disableAcl=true
015-07-17T03:33:48,728 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T13:23:00.000Z_2015-07-13T13:24:00.000Z_2015-07-13T13:23:00.000Z]
2015-07-17T03:33:48,736 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,736 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T13:22:00.000Z_2015-07-13T13:23:00.000Z_2015-07-13T13:22:00.000Z]
2015-07-17T03:33:48,750 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,750 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T13:17:00.000Z_2015-07-13T13:18:00.000Z_2015-07-13T13:17:00.000Z]
2015-07-17T03:33:48,757 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,757 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T13:15:00.000Z_2015-07-13T13:16:00.000Z_2015-07-13T13:15:00.000Z]
2015-07-17T03:33:48,771 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,771 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-13T12:59:00.000Z_2015-07-13T13:00:00.000Z_2015-07-13T12:59:00.000Z]
2015-07-17T03:33:48,777 INFO [ZkCoordinator-Exec--0] io.druid.server.coordination.ServerManager - Told to delete a queryable for a dataSource[datasource-topic-test] that doesn't exist.
2015-07-17T03:33:48,777 WARN [ZkCoordinator-Exec--0] io.druid.server.coordination.ZkCoordinator - Unable to delete segmentInfoCacheFile[/tmp/druid/indexCache/info_dir/datasource-topic-test_2015-07-12T06:12:00.000Z_2015-07-12T06:13:00.000Z_2015-07-12T06:12:00.000Z]
03:17:00.000Z_2015-07-17T03:18:00.000Z_2015-07-17T03:17:00.000Z]
2015-07-17T03:34:13,807 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[datasource-topic-test_2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z_2015-07-17T03:17:00.000Z], segment=DataSegment{size=94448, shardSpec=NoneShardSpec, metrics=[test], dimensions=[browser_name, browser_version, campaign_frequency_cap, campaign_id, carrier, connection_type, country, cpc, device_brand, device_name, device_type, fallback_redirect, hour, id, ip_address, is_robot, lead, offer_conversion_tracking, offer_frequency_cap, offer_id, operating_system, payout, pointing_method, referer, referer_host, traffic_source_id, url, window_top], version='2015-07-17T03:17:00.000Z', loadSpec={type=s3_zip, bucket=druid-development, key=development-realtime/v1/datasource-topic-test/2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z/2015-07-17T03:17:00.000Z/0/index.zip}, interval=2015-07-17T03:17:00.000Z/2015-07-17T03:18:00.000Z, dataSource='datasource-topic-test', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[datasource-topic-test_2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z_2015-07-17T03:17:00.000Z]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.3.jar:0.7.3]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
Caused by: io.druid.segment.loading.SegmentLoadingException: S3 fail! Key[s3://druid-development/development-realtime/v1/datasource-topic-test/2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z/2015-07-17T03:17:00.000Z/0/index.zip]
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:325) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
... 18 more
Caused by: org.jets3t.service.S3ServiceException: Request Error.
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1775) ~[jets3t-0.9.3.jar:0.9.3]
at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:93) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:319) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:315) ~[?:?]
at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.27.0.jar:?]
at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:86) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:313) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
... 18 more
Caused by: org.jets3t.service.impl.rest.HttpException
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:423) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:277) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1038) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2250) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2179) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:2574) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1773) ~[jets3t-0.9.3.jar:0.9.3]
at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:93) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:319) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:315) ~[?:?]
at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.27.0.jar:?]
at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:86) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:313) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
... 18 more
2015-07-17T03:34:13,827 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - zNode[/druid/loadQueue/
172.31.4.173:8082/datasource-topic-test_2015-07-17T03:17:00.000Z_2015-07-17T03:18:00.000Z_2015-07-17T03:17:00.000Z] was removed
2015-07-17T03:34:13,829 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - New request[LOAD: datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z] with zNode[/druid/loadQueue/
172.31.4.173:8082/datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z].
2015-07-17T03:34:13,829 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Loading segment datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z
2015-07-17T03:34:13,829 INFO [ZkCoordinator-0] io.druid.storage.s3.S3DataSegmentPuller - Pulling index at path[s3://druid-development/development-realtime/v1/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0/index.zip] to outDir[/tmp/druid/indexCache/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0]
2015-07-17T03:34:13,902 INFO [ZkCoordinator-0] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/tmp/druid/indexCache/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0]
2015-07-17T03:34:13,902 INFO [ZkCoordinator-0] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/tmp/druid/indexCache/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z]
2015-07-17T03:34:13,902 INFO [ZkCoordinator-0] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/tmp/druid/indexCache/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z]
2015-07-17T03:34:13,903 WARN [ZkCoordinator-0] io.druid.server.coordination.BatchDataSegmentAnnouncer - No path to unannounce segment[datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z]
2015-07-17T03:34:13,903 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Completely removing [datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z] in [30,000] millis
2015-07-17T03:34:13,906 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Completed request [LOAD: datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z]
2015-07-17T03:34:13,906 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z], segment=DataSegment{size=92850, shardSpec=NoneShardSpec, metrics=[test], dimensions=[browser_name, browser_version, campaign_frequency_cap, campaign_id, carrier, connection_type, country, cpc, device_brand, device_name, device_type, fallback_redirect, hour, id, ip_address, is_robot, lead, offer_conversion_tracking, offer_frequency_cap, offer_id, operating_system, payout, pointing_method, referer, referer_host, traffic_source_id, url, window_top], version='2015-07-17T03:16:00.000Z', loadSpec={type=s3_zip, bucket=druid-development, key=development-realtime/v1/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0/index.zip}, interval=2015-07-17T03:16:00.000Z/2015-07-17T03:17:00.000Z, dataSource='datasource-topic-test', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.3.jar:0.7.3]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_79]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
Caused by: io.druid.segment.loading.SegmentLoadingException: S3 fail! Key[s3://druid-development/development-realtime/v1/datasource-topic-test/2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z/2015-07-17T03:16:00.000Z/0/index.zip]
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:325) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
... 18 more
Caused by: org.jets3t.service.S3ServiceException: Request Error.
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1775) ~[jets3t-0.9.3.jar:0.9.3]
at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:93) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:319) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:315) ~[?:?]
at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.27.0.jar:?]
at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:86) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:313) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
... 18 more
Caused by: org.jets3t.service.impl.rest.HttpException
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:423) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:277) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1038) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2250) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2179) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:2574) ~[jets3t-0.9.3.jar:0.9.3]
at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1773) ~[jets3t-0.9.3.jar:0.9.3]
at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:93) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:319) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:315) ~[?:?]
at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.27.0.jar:?]
at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:86) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:313) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:162) ~[?:?]
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:88) ~[?:?]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:141) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:93) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.3.jar:0.7.3]
at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.3.jar:0.7.3]
... 18 more
2015-07-17T03:34:13,916 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - zNode[/druid/loadQueue/
172.31.4.173:8082/datasource-topic-test_2015-07-17T03:16:00.000Z_2015-07-17T03:17:00.000Z_2015-07-17T03:16:00.000Z] was removed
2015-07-17T03:34:13,925 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - New request[LOAD: datasource-topic-test_2015-07-17T03:15:00.000Z_2015-07-17T03:16:00.000Z_2015-07-17T03:15:00.000Z] with zNode[/druid/loadQueue/
172.31.4.173:8082/datasource-topic-test_2015-07-17T03:15:00.000Z_2015-07-17T03:16:00.000Z_2015-07-17T03:15:00.000Z].