Cannot connect to S3 Service with a null path

39 views
Skip to first unread message

Fenlon Xiong

unread,
Nov 15, 2018, 2:25:40 AM11/15/18
to Druid User

anybody has the problem?

deepStorageType=S3, and segments put to S3 success , but historical load failed with the exception...


exception info:

2018-11-15T06:58:19,265 INFO [ZkCoordinator] io.druid.server.coordination.ZkCoordinator - New request[LOAD: ibu.hotel.order_2018-11-14T17:00:00.000Z_2018-11-14T18:00:00.000Z_2018-11-14T17:08:04.769Z] with zNode[/druid/loadQueue/10.5.111.107:8083/ibu.hotel.order_2018-11-14T17:00:00.000Z_2018-11-14T18:00:00.000Z_2018-11-14T17:08:04.769Z].

2018-11-15T06:58:19,265 INFO [ZkCoordinator] io.druid.server.coordination.SegmentLoadDropHandler - Loading segment ibu.hotel.order_2018-11-14T17:00:00.000Z_2018-11-14T18:00:00.000Z_2018-11-14T17:08:04.769Z

2018-11-15T06:58:19,265 INFO [ZkCoordinator] io.druid.storage.s3.S3DataSegmentPuller - Pulling index at path[s3://ibu_plt_druid_data/druid/segments/ibu.hotel.order/2018-11-14T17:00:00.000Z_2018-11-14T18:00:00.000Z/2018-11-14T17:08:04.769Z/0/index.zip] to outDir[var/druid/segment-cache/ibu.hotel.order/2018-11-14T17:00:00.000Z_2018-11-14T18:00:00.000Z/2018-11-14T17:08:04.769Z/0]

2018-11-15T06:58:19,337 ERROR [ZkCoordinator] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Failed to load segment in current location /home/druid/imply/data/var/druid/segment-cache, try next location if any: {class=io.druid.segment.loading.SegmentLoaderLocalCacheManager, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=org.jets3t.service.S3ServiceException: Cannot connect to S3 Service with a null path, location=/home/druid/imply/data/var/druid/segment-cache}

io.druid.segment.loading.SegmentLoadingException: org.jets3t.service.S3ServiceException: Cannot connect to S3 Service with a null path

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:234) ~[?:?]

at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) [druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) [druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) [druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) [druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) [druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) [druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-server-0.12.3-iap9.jar:0.12.3-iap9]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]

at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [curator-client-4.0.0.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [curator-recipes-4.0.0.jar:4.0.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

Caused by: java.lang.RuntimeException: org.jets3t.service.S3ServiceException: Cannot connect to S3 Service with a null path

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.storage.s3.S3DataSegmentPuller$2.openStream(S3DataSegmentPuller.java:198) ~[?:?]

at io.druid.java.util.common.CompressionUtils$1.call(CompressionUtils.java:170) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.java.util.common.CompressionUtils$1.call(CompressionUtils.java:166) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.java.util.common.CompressionUtils.unzip(CompressionUtils.java:164) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:203) ~[?:?]

... 27 more

Caused by: org.jets3t.service.S3ServiceException: Cannot connect to S3 Service with a null path

at org.jets3t.service.impl.rest.httpclient.RestS3Service.setupConnection(RestS3Service.java:278) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1043) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[jets3t-0.9.4.jar:0.9.4]

at io.druid.storage.s3.S3DataSegmentPuller.buildFileObject(S3DataSegmentPuller.java:67) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$2.openStream(S3DataSegmentPuller.java:190) ~[?:?]

at io.druid.java.util.common.CompressionUtils$1.call(CompressionUtils.java:170) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.java.util.common.CompressionUtils$1.call(CompressionUtils.java:166) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.java.util.common.CompressionUtils.unzip(CompressionUtils.java:164) ~[java-util-0.12.3-iap9.jar:0.12.3-iap9]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:203) ~[?:?]

... 27 more

2018-11-15T06:58:19,343 INFO [ZkCoordinator] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[var/druid/segment-cache/ibu.hotel.order/2018-11-14T17:00:00.000Z_2018-11-14T18:00:00.000Z/2018-11-14T17:08:04.769Z/0]

2018-11-15T06:58:19,343 INFO [ZkCoordinator] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[var/druid/segment-cache/ibu.hotel.order/2018-11-14T17:00:00.000Z_2018-11-14T18:00:00.000Z/2018-11-14T17:08:04.769Z]

2018-11-15T06:58:19,343 INFO [ZkCoordinator] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[var/druid/segment-cache/ibu.hotel.order/2018-11-14T17:00:00.000Z_2018-11-14T18:00:00.000Z]

2018-11-15T06:58:19,344 INFO [ZkCoordinator] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[var/druid/segment-cache/ibu.hotel.order]


My Config Info:

1:  common.runtime.properties

# Extensions

#

druid.extensions.directory=dist/druid/extensions

druid.extensions.hadoopDependenciesDir=dist/druid/hadoop-dependencies

druid.extensions.loadList=["mysql-metadata-storage","druid-parser-route","druid-histogram","druid-s3-extensions"]

#

# Logging

#

# Log all runtime properties on startup. Disable to avoid logging properties on startup:

druid.startup.logging.logProperties=true

#

#

druid.zk.service.host=z1,z2,z3

druid.zk.paths.base=/druid

#

#

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI=jdbc:mysql://xxx/ibu_druid_meta_db

druid.metadata.storage.connector.user=xxx

druid.metadata.storage.connector.password=xxx

#

# Deep storage

#

druid.storage.type=s3

druid.storage.bucket=xxx

druid.storage.baseKey=druid/segments

druid.s3.accessKey=xxx

druid.s3.secretKey=xxxx

#

druid.indexer.logs.type=s3

druid.indexer.logs.s3Bucket=xxxx

druid.indexer.logs.s3Prefix=druid/indexing-logs

# Service discovery

#

druid.selectors.indexing.serviceName=druid/overlord

druid.selectors.coordinator.serviceName=druid/coordinator

#

# Monitoring

#

druid.monitoring.monitors=["io.druid.java.util.metrics.JvmMonitor"]

druid.emitter=logging

druid.emitter.logging.logLevel=debug

#

#SQL

#

druid.sql.enable=true

druid.javascript.enabled=true


2: historical runtime.properties

druid.host=ip

druid.service=druid/historical

druid.port=8083

druid.server.http.numThreads=40

druid.processing.buffer.sizeBytes=134217728

druid.processing.numMergeBuffers=2

druid.processing.numThreads=13

druid.processing.tmpDir=var/druid/processing

druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize"\:130000000000}]

druid.server.maxSize=130000000000

druid.historical.cache.useCache=true

druid.historical.cache.populateCache=true

druid.cache.type=caffeine

druid.cache.sizeInBytes=2000000000


3: jets3t.properties

s3service.s3-endpoint=host

s3service.s3-endpoint-http-port=80

s3service.disable-dns-buckets=false

s3service.https-only=false


Fenlon Xiong

unread,
Nov 16, 2018, 5:13:54 AM11/16/18
to Druid User

Problem Solved !

because of my s3 bucket name contains underscores('_') 

in java.net.URI hostname can't contain underscores, if contains , uri.getHost will return null ,so exceptionMessage "Cannot connect to S3 Service with a null path"

in druid when download segments from s3,the uri=s3://bucketName/key.....(which use http rest api)

so when get bucketName(uri.getHost) ,return null

best wishes 






在 2018年11月15日星期四 UTC+8下午3:25:40，Fenlon Xiong写道：
Reply all
Reply to author
Forward
0 new messages