Is it able to connect S3-Compatible storage with Alluxio?

[zeng zhen]

Dear team

I am trying to connect S3-Compatible storage with Alluxio, but failed.

conf/alluxio-site.properties

----

alluxio.underfs.address=s3a://sso-test/lineitem_1g_orc

alluxio.underfs.s3.endpoint=kkk.storage.internal-domain.jp

alluxio.underfs.s3.endpoint.http.port=80

alluxio.underfs.s3.proxy.https.only=false

----

Endpoint server is S3-Compatible storage inside our company.

and set access keys in conf/alluxio-env.sh

```

ALLUXIO_MASTER_HOSTNAME=${ALLUXIO_MASTER_HOSTNAME:-"localhost"}

ALLUXIO_WORKER_MEMORY_SIZE=${ALLUXIO_WORKER_MEMORY_SIZE:-"85774MB"}

ALLUXIO_RAM_FOLDER=${ALLUXIO_RAM_FOLDER:-"/mnt/ramdisk"}

AWS_ACCESS_KEY_ID=aaaaaa

AWS_SECRET_ACCESS_KEY=*********

```

Then following the guide ( http://www.alluxio.org/docs/1.4/en/Configuring-Alluxio-with-S3.html ),

I tried test command:

$ ./bin/alluxio runTests

at last got error message in :

```

Exception in thread "main" java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3a://sso-test/lineitem_1g_orc
com.amazonaws.services.s3.model.AmazonS3Exception: ' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 57428d0e-787c-4424-48fc-d319984b1435), S3 Extended Request ID: api1114
        at alluxio.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:130)
        at alluxio.underfs.UnderFileSystem$Factory$Cache.get(UnderFileSystem.java:77)
        at alluxio.underfs.UnderFileSystem$Factory.get(UnderFileSystem.java:160)
        at alluxio.underfs.UnderFileSystem$Factory.get(UnderFileSystem.java:147)
        at alluxio.master.file.FileSystemMaster$Metrics.registerGauges(FileSystemMaster.java:3174)
        at alluxio.master.file.FileSystemMaster$Metrics.access$000(FileSystemMaster.java:3116)
        at alluxio.master.file.FileSystemMaster.<init>(FileSystemMaster.java:285)
        at alluxio.master.file.FileSystemMaster.<init>(FileSystemMaster.java:257)
        at alluxio.master.DefaultAlluxioMaster.createMasters(DefaultAlluxioMaster.java:165)
        at alluxio.master.DefaultAlluxioMaster.<init>(DefaultAlluxioMaster.java:140)
        at alluxio.master.AlluxioMasterService$Factory.create(AlluxioMasterService.java:42)
        at alluxio.master.AlluxioMaster.main(AlluxioMaster.java:42)

```

My Question:

1) Does Alluxio use AWS S3 Signature V4 to connect?

2) Is it able to connect S3-Compatible storage with Alluxio?

[Calvin Jia]

Hi,

As of 1.4.0 you should be able to use Alluxio to connect to S3-compatible storage. See this PR as an example. One additional configuration change you may need to do is to disable dns buckets.

Hope this helps,

Calvin

[zeng zhen]

Thanks, I will have a try.

2017年1月21日土曜日 3時43分58秒 UTC+9 Calvin Jia:

[zeng zhen]

I tried with the configurations in PR #4217, however it didn't work.

Still the same error:

com.amazonaws.services.s3.model.AmazonS3Exception: ' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 57428d0e-787c-4424-48fc-d319984b1435), S3 Extended Request ID: api1114

But I got some hint from your PR,

I downgrade AWS Signature V4 to V2, by adding some code in alluxio/underfs/s3a/src/main/java/alluxio/underfs/s3a/S3AUnderFileSystem.java

Before initialize Amazon s3 client, 

--------------

// TODO(zzeng) added temporary
clientConf.withSignerOverride("S3SignerType");      <----- added

AmazonS3Client amazonS3Client = new AmazonS3Client(credentials, clientConf);
// Set a custom endpoint.
if (Configuration.containsKey(PropertyKey.UNDERFS_S3_ENDPOINT)) {
  amazonS3Client.setEndpoint(Configuration.get(PropertyKey.UNDERFS_S3_ENDPOINT));
}

--------------

Then rebuilt jar file, now I can see my files in alluxio:

[sso@worker0406 ~/alluxio]$ ./bin/alluxio runTests
2017-01-23 13:29:08,114 INFO  type (MetricsSystem.java:startSinksFromConfig) - Starting sinks with config: {}.
2017-01-23 13:29:08,215 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.5.0-SNAPSHOT) is trying to connect with FileSystemMasterClient master @ localhost/127.0.0.1:19998
2017-01-23 13:29:08,236 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemMasterClient master @ localhost/127.0.0.1:19998
runTest Basic CACHE_PROMOTE MUST_CACHE
2017-01-23 13:29:08,556 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.5.0-SNAPSHOT) is trying to connect with BlockMasterClient master @ localhost/127.0.0.1:19998
2017-01-23 13:29:08,559 INFO  type (AbstractClient.java:connect) - Client registered with BlockMasterClient master @ localhost/127.0.0.1:19998
2017-01-23 13:29:08,628 INFO  type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.BlockWorkerClientService$Client@70beb599
2017-01-23 13:29:08,652 INFO  type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.BlockWorkerClientService$Client@646be2c3
2017-01-23 13:29:08,743 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 338 ms.
2017-01-23 13:29:08,832 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 89 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE MUST_CACHE
2017-01-23 13:29:08,975 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_MUST_CACHE took 11 ms.
2017-01-23 13:29:08,986 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_MUST_CACHE took 11 ms.
Passed the test!
runTest Basic CACHE_PROMOTE CACHE_THROUGH
2017-01-23 13:29:09,310 INFO  type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.FileSystemWorkerClientService$Client@4df50bcc
2017-01-23 13:29:09,327 INFO  type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.FileSystemWorkerClientService$Client@6ac13091
2017-01-23 13:29:09,526 INFO  type (NettyChannelPool.java:createNewResource) - Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true)}, handler: alluxio.client.netty.NettyClient$1@292b08d6, remoteAddress: worker0406.qe.kks.yahoo.co.jp/100.98.240.21:29999).
2017-01-23 13:29:09,687 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH took 700 ms.
2017-01-23 13:29:09,700 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH took 13 ms.
Passed the test!

Thanks

2017年1月21日土曜日 10時02分28秒 UTC+9 zeng zhen:

[Calvin Jia]

Hi,

Thanks for posting your solution! Is the reason because the backend you are connecting to does not support AWS V4? If so it makes sense to allow users to specify the signature type. Could you contribute your fix to the open source project?

Cheers,

Calvin

[zeng zhen]

OK, I will have a try!

2017年1月24日火曜日 4時19分29秒 UTC+9 Calvin Jia: