S3 UnderFS - eu-central-1 AWS Signature Version 4 problem
閲覧: 165 回
最初の未読メッセージにスキップ
Sander van Dijk
2015/07/06 12:01:172015/07/06
To: tachyo...@googlegroups.com
Hey all,
I am trying to set up Tachyon with S3 as under-FS, using a bucket located in the eu-central-1 (Frankfurt) zone. This zone uses AWS Signature version 4 exclusively, and I'm having issues to get Tachyon to work with this.
I have tried both with 0.6.4 and current master. I have followed https://github.com/amplab/tachyon/blob/master/docs/Setup-UFS.md#amazon-s3 to pull in jets3t 0.9.3 as a dependency, which has v4 signature support. I have set the AWS credentials. Formatting with:
./bin/tachyon format
works (data_$folder$ and workers_$folder$ show up in the directory in the bucket), so the basic setup seems to work. However, when I run `./bin/tachyon runTest Basic CACHE_THROUGH` next, I get (some things censored out):
$ ./bin/tachyon runTest Basic CACHE_THROUGH
/default_tests_files/BasicFile_CACHE_THROUGH has been removed
2015-07-06 15:37:13,731 INFO (MasterClient.java:connect) - Tachyon client (version 0.7.0-SNAPSHOT) is trying to connect with master @ localhost/127.0.0.1:19998
2015-07-06 15:37:13,759 INFO (MasterClient.java:connect) - User registered with the master @ localhost/127.0.0.1:19998; got UserId 3
2015-07-06 15:37:13,782 INFO (CommonUtils.java:printTimeTakenMs) - createFile with fileId 3 took 54 ms.
2015-07-06 15:37:13,806 INFO (WorkerClient.java:connect) - Trying to get local worker host : tachyonhost
2015-07-06 15:37:13,818 INFO (WorkerClient.java:connect) - Connecting local worker @ tachyonhost/10.0.2.15:29998
2015-07-06 15:37:14,147 WARN NativeCodeLoader (NativeCodeLoader.java:<clinit>) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-07-06 15:37:15,475 WARN RestStorageService (RestStorageService.java:performRequest) - Retrying request with "AWS4-HMAC-SHA256" signing mechanism: GET https://<bucket>.s3.amazonaws.com:443/?delimiter=%2F&max-keys=1&prefix=tachyon%2Fworkers%2F1436197000001%2F3%2F HTTP/1.1
2015-07-06 15:37:15,476 WARN RestStorageService (RestStorageService.java:performRequest) - Retrying request following error response: GET '/?delimiter=/&max-keys=1&prefix=tachyon/workers/1436197000001/3/' -- ResponseCode: 400, ResponseStatus: Bad Request, Request Headers: [Date: Mon, 06 Jul 2015 15:37:15 GMT, Authorization: AWS <accesskeyid>:<foo>], Response Headers: [x-amz-request-id: 087B449D237FA925, x-amz-id-2: WdIYxuVrJj/3b/T18FEC98fLJbyl8j6CZra88IoFATqrcssSrC7s81CTSvdNgqRCgqz1v86n70E=, x-amz-region: eu-central-1, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Mon, 06 Jul 2015 15:37:12 GMT, Connection: close, Server: AmazonS3]
2015-07-06 15:37:15,668 WARN RestStorageService (RestStorageService.java:performRequest) - Retrying request after automatic adjustment of Host endpoint from "<bucket>.s3.amazonaws.com" to "<bucket>.s3-eu-central-1.amazonaws.com" following request signing error using AWS request signing version 4: GET https://<bucket>.s3-eu-central-1.amazonaws.com:443/?delimiter=/&max-keys=1&prefix=tachyon/workers/1436197000001/3/ HTTP/1.1
2015-07-06 15:37:15,668 WARN RestStorageService (RestStorageService.java:performRequest) - Retrying request following error response: GET '/?delimiter=/&max-keys=1&prefix=tachyon/workers/1436197000001/3/' -- ResponseCode: 400, ResponseStatus: Bad Request, Request Headers: [Date: Mon, 06 Jul 2015 15:37:15 GMT, x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, Host: <bucket>.s3.amazonaws.com, x-amz-date: 20150706T153715Z, Authorization: AWS4-HMAC-SHA256 Credential=<accesskeyid>/20150706/us-east-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=<sig>], Response Headers: [x-amz-request-id: 30CAA3FF4E35DA4D, x-amz-id-2: 7+NAhQeUsuyHWQ+1mEdv1VtxfwfJwO6yfAOVSDT7Rj/aVJAoCm4Rmf70tMNQhb9wxmh1IKYYLMc=, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Mon, 06 Jul 2015 15:37:13 GMT, Connection: close, Server: AmazonS3]
2015-07-06 15:37:17,503 INFO NativeS3FileSystem (NativeS3FileSystem.java:<init>) - OutputStream for key 'tachyon/workers/1436197000001/3/3' writing to tempfile '/tmp/hadoop-vagrant/s3/output-7934511847480226942.tmp'
2015-07-06 15:37:17,534 INFO (TachyonFS.java:getLocalBlockTemporaryPath) - Folder /mnt/ramdisk/3 was created!
2015-07-06 15:37:17,540 INFO (BlockOutStream.java:<init>) - /mnt/ramdisk/3/3221225472 was created!
2015-07-06 15:37:17,542 INFO NativeS3FileSystem (NativeS3FileSystem.java:close) - OutputStream for key 'tachyon/workers/1436197000001/3/3' closed. Now beginning upload
2015-07-06 15:37:57,555 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@362de08a
org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error: Failed to automatically set required header "x-amz-content-sha256" for request with entity org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity@28ee5194
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleS3ServiceException(Jets3tNativeFileSystemStore.java:405)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
at org.apache.hadoop.fs.s3native.$Proxy4.storeFile(Unknown Source)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:221)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
at tachyon.client.FileOutStream.close(FileOutStream.java:105)
at tachyon.examples.BasicOperations.writeFile(BasicOperations.java:81)
at tachyon.examples.BasicOperations.call(BasicOperations.java:55)
at tachyon.examples.BasicOperations.call(BasicOperations.java:37)
at tachyon.examples.Utils.runExample(Utils.java:102)
at tachyon.examples.BasicOperations.main(BasicOperations.java:115)
Caused by: org.jets3t.service.S3ServiceException: Request Error: Failed to automatically set required header "x-amz-content-sha256" for request with entity org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity@28ee5194
at org.jets3t.service.S3Service.putObject(S3Service.java:2358)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:113)
... 16 more
Caused by: java.lang.RuntimeException: Failed to automatically set required header "x-amz-content-sha256" for request with entity org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity@28ee5194
at org.jets3t.service.utils.SignatureUtils.awsV4GetOrCalculatePayloadHash(SignatureUtils.java:238)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.authorizeHttpRequest(RestStorageService.java:762)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:324)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:277)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1143)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:1954)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:1875)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectImpl(RestStorageService.java:1867)
at org.jets3t.service.StorageService.putObject(StorageService.java:840)
at org.jets3t.service.S3Service.putObject(S3Service.java:2212)
at org.jets3t.service.S3Service.putObject(S3Service.java:2356)
... 17 more
Caused by: java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
at java.io.BufferedInputStream.reset(BufferedInputStream.java:435)
at org.jets3t.service.utils.SignatureUtils.awsV4GetOrCalculatePayloadHash(SignatureUtils.java:236)
... 27 more
Failed the test!
After this, the path `tachyon/workers/1436197000001/3` exists in my bucket, with no object `3` in there. I have also tried the new native S3 client, by adding ` -Dtachyon.underfs.hadoop.prefixes=` to TACHYON_JAVA_OPTS, which fails with similar output, except for the stack trace at the end:
2015-07-06 15:53:52,974 INFO (BlockOutStream.java:<init>) - /mnt/ramdisk/3/4294967296 was created!
2015-07-06 15:53:54,020 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@57a3de41
java.io.IOException: FailedToCheckpointException(message:Failed to rename s3n://<bucket>/tachyon/workers/1436198000001/3/4 to s3n://<bucket>/tachyon/data/4)
at tachyon.worker.WorkerClient.addCheckpoint(WorkerClient.java:124)
at tachyon.client.TachyonFS.addCheckpoint(TachyonFS.java:228)
at tachyon.client.FileOutStream.close(FileOutStream.java:106)
at tachyon.examples.BasicOperations.writeFile(BasicOperations.java:81)
at tachyon.examples.BasicOperations.call(BasicOperations.java:55)
at tachyon.examples.BasicOperations.call(BasicOperations.java:37)
at tachyon.examples.Utils.runExample(Utils.java:102)
at tachyon.examples.BasicOperations.main(BasicOperations.java:115)
Caused by: FailedToCheckpointException(message:Failed to rename s3n://<bucket>/tachyon/workers/1436198000001/3/4 to s3n://<bucket>/tachyon/data/4)
at tachyon.thrift.WorkerService$addCheckpoint_result$addCheckpoint_resultStandardScheme.read(WorkerService.java:3509)
at tachyon.thrift.WorkerService$addCheckpoint_result$addCheckpoint_resultStandardScheme.read(WorkerService.java:3477)
at tachyon.thrift.WorkerService$addCheckpoint_result.read(WorkerService.java:3403)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at tachyon.thrift.WorkerService$Client.recv_addCheckpoint(WorkerService.java:221)
at tachyon.thrift.WorkerService$Client.addCheckpoint(WorkerService.java:207)
at tachyon.worker.WorkerClient.addCheckpoint(WorkerClient.java:118)
... 7 more
Failed the test!
Is there anything I can do to make it work, or is it something that needs to be addressed in Tachyon? I found the following jets3t issue with a resolution, though not sure it's completely related:
https://bitbucket.org/jmurty/jets3t/issue/203/synchronize-with-aws4-failing-ioexception
Many thanks for any help!
Sander
I am trying to set up Tachyon with S3 as under-FS, using a bucket located in the eu-central-1 (Frankfurt) zone. This zone uses AWS Signature version 4 exclusively, and I'm having issues to get Tachyon to work with this.
I have tried both with 0.6.4 and current master. I have followed https://github.com/amplab/tachyon/blob/master/docs/Setup-UFS.md#amazon-s3 to pull in jets3t 0.9.3 as a dependency, which has v4 signature support. I have set the AWS credentials. Formatting with:
./bin/tachyon format
works (data_$folder$ and workers_$folder$ show up in the directory in the bucket), so the basic setup seems to work. However, when I run `./bin/tachyon runTest Basic CACHE_THROUGH` next, I get (some things censored out):
$ ./bin/tachyon runTest Basic CACHE_THROUGH
/default_tests_files/BasicFile_CACHE_THROUGH has been removed
2015-07-06 15:37:13,731 INFO (MasterClient.java:connect) - Tachyon client (version 0.7.0-SNAPSHOT) is trying to connect with master @ localhost/127.0.0.1:19998
2015-07-06 15:37:13,759 INFO (MasterClient.java:connect) - User registered with the master @ localhost/127.0.0.1:19998; got UserId 3
2015-07-06 15:37:13,782 INFO (CommonUtils.java:printTimeTakenMs) - createFile with fileId 3 took 54 ms.
2015-07-06 15:37:13,806 INFO (WorkerClient.java:connect) - Trying to get local worker host : tachyonhost
2015-07-06 15:37:13,818 INFO (WorkerClient.java:connect) - Connecting local worker @ tachyonhost/10.0.2.15:29998
2015-07-06 15:37:14,147 WARN NativeCodeLoader (NativeCodeLoader.java:<clinit>) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-07-06 15:37:15,475 WARN RestStorageService (RestStorageService.java:performRequest) - Retrying request with "AWS4-HMAC-SHA256" signing mechanism: GET https://<bucket>.s3.amazonaws.com:443/?delimiter=%2F&max-keys=1&prefix=tachyon%2Fworkers%2F1436197000001%2F3%2F HTTP/1.1
2015-07-06 15:37:15,476 WARN RestStorageService (RestStorageService.java:performRequest) - Retrying request following error response: GET '/?delimiter=/&max-keys=1&prefix=tachyon/workers/1436197000001/3/' -- ResponseCode: 400, ResponseStatus: Bad Request, Request Headers: [Date: Mon, 06 Jul 2015 15:37:15 GMT, Authorization: AWS <accesskeyid>:<foo>], Response Headers: [x-amz-request-id: 087B449D237FA925, x-amz-id-2: WdIYxuVrJj/3b/T18FEC98fLJbyl8j6CZra88IoFATqrcssSrC7s81CTSvdNgqRCgqz1v86n70E=, x-amz-region: eu-central-1, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Mon, 06 Jul 2015 15:37:12 GMT, Connection: close, Server: AmazonS3]
2015-07-06 15:37:15,668 WARN RestStorageService (RestStorageService.java:performRequest) - Retrying request after automatic adjustment of Host endpoint from "<bucket>.s3.amazonaws.com" to "<bucket>.s3-eu-central-1.amazonaws.com" following request signing error using AWS request signing version 4: GET https://<bucket>.s3-eu-central-1.amazonaws.com:443/?delimiter=/&max-keys=1&prefix=tachyon/workers/1436197000001/3/ HTTP/1.1
2015-07-06 15:37:15,668 WARN RestStorageService (RestStorageService.java:performRequest) - Retrying request following error response: GET '/?delimiter=/&max-keys=1&prefix=tachyon/workers/1436197000001/3/' -- ResponseCode: 400, ResponseStatus: Bad Request, Request Headers: [Date: Mon, 06 Jul 2015 15:37:15 GMT, x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, Host: <bucket>.s3.amazonaws.com, x-amz-date: 20150706T153715Z, Authorization: AWS4-HMAC-SHA256 Credential=<accesskeyid>/20150706/us-east-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=<sig>], Response Headers: [x-amz-request-id: 30CAA3FF4E35DA4D, x-amz-id-2: 7+NAhQeUsuyHWQ+1mEdv1VtxfwfJwO6yfAOVSDT7Rj/aVJAoCm4Rmf70tMNQhb9wxmh1IKYYLMc=, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Mon, 06 Jul 2015 15:37:13 GMT, Connection: close, Server: AmazonS3]
2015-07-06 15:37:17,503 INFO NativeS3FileSystem (NativeS3FileSystem.java:<init>) - OutputStream for key 'tachyon/workers/1436197000001/3/3' writing to tempfile '/tmp/hadoop-vagrant/s3/output-7934511847480226942.tmp'
2015-07-06 15:37:17,534 INFO (TachyonFS.java:getLocalBlockTemporaryPath) - Folder /mnt/ramdisk/3 was created!
2015-07-06 15:37:17,540 INFO (BlockOutStream.java:<init>) - /mnt/ramdisk/3/3221225472 was created!
2015-07-06 15:37:17,542 INFO NativeS3FileSystem (NativeS3FileSystem.java:close) - OutputStream for key 'tachyon/workers/1436197000001/3/3' closed. Now beginning upload
2015-07-06 15:37:57,555 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@362de08a
org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error: Failed to automatically set required header "x-amz-content-sha256" for request with entity org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity@28ee5194
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleS3ServiceException(Jets3tNativeFileSystemStore.java:405)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
at org.apache.hadoop.fs.s3native.$Proxy4.storeFile(Unknown Source)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:221)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
at tachyon.client.FileOutStream.close(FileOutStream.java:105)
at tachyon.examples.BasicOperations.writeFile(BasicOperations.java:81)
at tachyon.examples.BasicOperations.call(BasicOperations.java:55)
at tachyon.examples.BasicOperations.call(BasicOperations.java:37)
at tachyon.examples.Utils.runExample(Utils.java:102)
at tachyon.examples.BasicOperations.main(BasicOperations.java:115)
Caused by: org.jets3t.service.S3ServiceException: Request Error: Failed to automatically set required header "x-amz-content-sha256" for request with entity org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity@28ee5194
at org.jets3t.service.S3Service.putObject(S3Service.java:2358)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:113)
... 16 more
Caused by: java.lang.RuntimeException: Failed to automatically set required header "x-amz-content-sha256" for request with entity org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity@28ee5194
at org.jets3t.service.utils.SignatureUtils.awsV4GetOrCalculatePayloadHash(SignatureUtils.java:238)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.authorizeHttpRequest(RestStorageService.java:762)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:324)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:277)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1143)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:1954)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:1875)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectImpl(RestStorageService.java:1867)
at org.jets3t.service.StorageService.putObject(StorageService.java:840)
at org.jets3t.service.S3Service.putObject(S3Service.java:2212)
at org.jets3t.service.S3Service.putObject(S3Service.java:2356)
... 17 more
Caused by: java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
at java.io.BufferedInputStream.reset(BufferedInputStream.java:435)
at org.jets3t.service.utils.SignatureUtils.awsV4GetOrCalculatePayloadHash(SignatureUtils.java:236)
... 27 more
Failed the test!
After this, the path `tachyon/workers/1436197000001/3` exists in my bucket, with no object `3` in there. I have also tried the new native S3 client, by adding ` -Dtachyon.underfs.hadoop.prefixes=` to TACHYON_JAVA_OPTS, which fails with similar output, except for the stack trace at the end:
2015-07-06 15:53:52,974 INFO (BlockOutStream.java:<init>) - /mnt/ramdisk/3/4294967296 was created!
2015-07-06 15:53:54,020 ERROR (Utils.java:runExample) - Exception running test: tachyon.examples.BasicOperations@57a3de41
java.io.IOException: FailedToCheckpointException(message:Failed to rename s3n://<bucket>/tachyon/workers/1436198000001/3/4 to s3n://<bucket>/tachyon/data/4)
at tachyon.worker.WorkerClient.addCheckpoint(WorkerClient.java:124)
at tachyon.client.TachyonFS.addCheckpoint(TachyonFS.java:228)
at tachyon.client.FileOutStream.close(FileOutStream.java:106)
at tachyon.examples.BasicOperations.writeFile(BasicOperations.java:81)
at tachyon.examples.BasicOperations.call(BasicOperations.java:55)
at tachyon.examples.BasicOperations.call(BasicOperations.java:37)
at tachyon.examples.Utils.runExample(Utils.java:102)
at tachyon.examples.BasicOperations.main(BasicOperations.java:115)
Caused by: FailedToCheckpointException(message:Failed to rename s3n://<bucket>/tachyon/workers/1436198000001/3/4 to s3n://<bucket>/tachyon/data/4)
at tachyon.thrift.WorkerService$addCheckpoint_result$addCheckpoint_resultStandardScheme.read(WorkerService.java:3509)
at tachyon.thrift.WorkerService$addCheckpoint_result$addCheckpoint_resultStandardScheme.read(WorkerService.java:3477)
at tachyon.thrift.WorkerService$addCheckpoint_result.read(WorkerService.java:3403)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at tachyon.thrift.WorkerService$Client.recv_addCheckpoint(WorkerService.java:221)
at tachyon.thrift.WorkerService$Client.addCheckpoint(WorkerService.java:207)
at tachyon.worker.WorkerClient.addCheckpoint(WorkerClient.java:118)
... 7 more
Failed the test!
Is there anything I can do to make it work, or is it something that needs to be addressed in Tachyon? I found the following jets3t issue with a resolution, though not sure it's completely related:
https://bitbucket.org/jmurty/jets3t/issue/203/synchronize-with-aws4-failing-ioexception
Many thanks for any help!
Sander
Calvin Jia
2015/07/13 0:51:242015/07/13
To: tachyo...@googlegroups.com、sgva...@gmail.com
Hi,
Currently the native client uses jets3t 0.8.1. You can recompile tachyon and modify the s3 native client dependency on jets3t to 0.9.3. However, the performance will not be as good in certain cases (small reads).
If you are planning to use the hdfs implementation, be sure to use the correct hadoop version which uses jets3t 0.9.x (I think 2.4+ should be fine).
Hope this helps,
Calvin
全員に返信
投稿者に返信
転送
新着メール 0 件