How to config Alluxio with using a non-Amazon service provider

673 views
Skip to first unread message

test520

unread,
Sep 22, 2016, 10:12:10 AM9/22/16
to Alluxio Users
Hello alluxio experts,

My environment setting :
=> Alluxio version : 1.2
=> Java version : java version "1.8.0_71"
=> OS version : CentOS release 6.5

I am trying to connect alluxio 1.2 with using a non-Amazon service provider(minio) through S3 connector.

My alluxio cluster  and non-Amazon service provider(minio) are deployed on different machines.

the S3_BUCKET named 'testBucket' and S3_DIRECTORY named 'testDirectory' are already created on minio server.

The necessary parameters already added into /conf/alluxio-site.properties

-------------------------------------------------------------------------------------------------------
alluxio.underfs.address=s3n://testBucket/testDirectory/
fs.s3n.awsAccessKeyId=Q0LOTBB9XV1TAJHUD85Z
fs.s3n.awsSecretAccessKey=OI7G1BO7kOAejGhTa9KEC1HuuIUc4xxg0PXQf9RR
alluxio.underfs.s3.disable.dns.buckets=true
alluxio.underfs.s3.endpoint=http://74.125.203.103
alluxio.underfs.s3.proxy.https.only=false
alluxio.underfs.s3.endpoint.http.port=9000
--------------------------------------------------------------------------------------------------------

Next, I start alluxio server and do not see any mount failed messages on master.log

./alluxio format
./alluxio-start.sh local

But, the alluxio web GUI shows that the underFS is unknown and I do not see anything when I typing ./alluxio fs ls / on alluxio master

--------------------------------------------------------------------------------------------------------
UnderFS Capacity : UNKNOWN
UnderFS Free / Used : UNKNOWN / UNKNOWN
--------------------------------------------------------------------------------------------------------

Also, I had tried to mount s3n://testBucket/testDirectory/ directly by "./bin/alluxio fs mount" and get following messages :

---------------------------------------------------------------------------------------------------------
ThriftIOException(message:Ufs path /testBuck/testDirectory does not exist)
---------------------------------------------------------------------------------------------------------

Can anyone share the experience of using a non-Amazon service provider through S3 connector ??

I will be grateful for any help you can provide.

Calvin Jia

unread,
Sep 22, 2016, 7:47:44 PM9/22/16
to Alluxio Users
Hi,

The capacity you see on the web ui is expected, since for an object store like S3 there is no real limit.

Do you already have files in your s3 bucket? You can run /alluxio fs ls -f / to load the metadata to Alluxio.

To verify if the connector is working, you can run basic I/O tests through /alluxio runTests.

Hope this helps,
Calvin

test520

unread,
Sep 22, 2016, 10:10:43 PM9/22/16
to Alluxio Users
Hi Calvin,

Yes, I already have some files in my s3 bucket. Nothing happened after running the commands

[root@alluxioMaster bin]# jps
16946 HistoryServer
17656 HistoryServer
22379 Jps
20603 AlluxioWorker
17070 HistoryServer
20527 AlluxioMaster
[root@alluxioMaster bin]# ./alluxio fs ls -f /
[root@alluxioMaster bin]# ./alluxio fs ls /
[root@alluxioMaster bin]#


I have run Tests through /alluxio and get some error messages :

runTest Basic CACHE_PROMOTE CACHE_THROUGH

2016-09-23 10:04:30,392 ERROR type (S3UnderFileSystem.java:mkdirsInternal) - Failed to create directory: s3n://testBucket/testDirectory
alluxio
.org.jets3t.service.S3ServiceException: Request Error: java.net.UnknownHostException: http PUT '//74.125.203.103:9000/testBucket/testDirectory_%24folder%24'
        at alluxio
.org.jets3t.service.S3Service.putObject(S3Service.java:2123)
        at alluxio
.underfs.s3.S3UnderFileSystem.mkdirsInternal(S3UnderFileSystem.java:651)
        at alluxio
.underfs.s3.S3UnderFileSystem.mkdirs(S3UnderFileSystem.java:309)
        at alluxio
.underfs.s3.S3UnderFileSystem.mkdirs(S3UnderFileSystem.java:282)
        at alluxio
.underfs.s3.S3UnderFileSystem.mkdirs(S3UnderFileSystem.java:313)
        at alluxio
.underfs.s3.S3UnderFileSystem.mkdirs(S3UnderFileSystem.java:282)
        at alluxio
.underfs.s3.S3UnderFileSystem.create(S3UnderFileSystem.java:169)
        at alluxio
.client.file.FileOutStream.<init>(FileOutStream.java:124)
        at alluxio
.client.file.BaseFileSystem.createFile(BaseFileSystem.java:102)
        at alluxio
.examples.BasicOperations.writeFile(BasicOperations.java:85)
        at alluxio
.examples.BasicOperations.call(BasicOperations.java:72)
        at alluxio
.examples.BasicOperations.call(BasicOperations.java:42)
        at alluxio
.cli.CliUtils.runExample(CliUtils.java:51)
        at alluxio
.cli.TestRunner.runTest(TestRunner.java:148)
        at alluxio
.cli.TestRunner.runTests(TestRunner.java:121)
        at alluxio
.cli.TestRunner.main(TestRunner.java:98)
Caused by: java.net.UnknownHostException: http
        at java
.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java
.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java
.net.Socket.connect(Socket.java:589)
        at java
.net.Socket.connect(Socket.java:538)
        at java
.net.Socket.<init>(Socket.java:434)
        at java
.net.Socket.<init>(Socket.java:286)
        at org
.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
        at org
.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$1.doit(ControllerThreadSocketFactory.java:91)
        at org
.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$SocketTask.run(ControllerThreadSocketFactory.java:158)
        at java
.lang.Thread.run(Thread.java:745)
2016-09-23 10:04:30,397 ERROR type (CliUtils.java:runExample) - Exception running test: alluxio.examples.BasicOperations@130d63be
java
.lang.NullPointerException
        at alluxio
.client.file.FileOutStream.write(FileOutStream.java:303)
        at alluxio
.client.file.FileOutStream.write(FileOutStream.java:270)
        at alluxio
.examples.BasicOperations.writeFile(BasicOperations.java:86)
        at alluxio
.examples.BasicOperations.call(BasicOperations.java:72)
        at alluxio
.examples.BasicOperations.call(BasicOperations.java:42)
        at alluxio
.cli.CliUtils.runExample(CliUtils.java:51)
        at alluxio
.cli.TestRunner.runTest(TestRunner.java:148)
        at alluxio
.cli.TestRunner.runTests(TestRunner.java:121)
        at alluxio
.cli.TestRunner.main(TestRunner.java:98)
Failed the test!

How to config s3 connector correctly if my s3 connector doesn't work.

Thanks for your reply.

Calvin Jia

unread,
Sep 27, 2016, 2:00:15 PM9/27/16
to Alluxio Users
Hi,

It seems like the endpoint you specified is malformed. Try setting the endpoint like alluxio.underfs.s3.endpoint=74.125.203.103 without the leading http://

Hope this helps,
Calvin

test520

unread,
Sep 28, 2016, 12:49:04 AM9/28/16
to Alluxio Users
Hi Calvin,

I cannot connect to non-Amazon service provider thru. s3 connector s3a, can you share the detailed steps for demonstration with using s3a?

alluxio-1.2.0 cch$ cat ./conf/alluxio-env.sh

#!/usr/bin/env bash
#
# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
# (the "License"). You may not use this work except in compliance with the License, which is
# available at www.apache.org/licenses/LICENSE-2.0
#

# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied, as more fully set forth in the License.
#
# See the NOTICE file distributed with this work for information regarding copyright ownership.
#
# This file is sourced to launch Alluxio servers or use Alluxio shell commands
# This file was generated by command "bin/alluxio bootstrapConf localhost s3"
# For a list of supported environment variables, see "conf/alluxio-env.sh.template"

ALLUXIO_MASTER_HOSTNAME
=${ALLUXIO_MASTER_HOSTNAME:-"localhost"}

ALLUXIO_WORKER_MEMORY_SIZE
=${ALLUXIO_WORKER_MEMORY_SIZE:-"10922MB"}

ALLUXIO_RAM_FOLDER
=${ALLUXIO_RAM_FOLDER:-"/Volumes/ramdisk"}

#ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-"s3a://bucket/directory/"}


alluxio-1.2.0 cch$ cat conf/alluxio-site.properties

alluxio.underfs.address=s3a://alluxio-quick-start
aws
.accessKeyId=xxxxxxxxxxxx
aws
.secretKey=xxxxxxxxxxxxxxxxxxx
alluxio
.underfs.s3.disable.dns.buckets=true
alluxio
.underfs.s3.endpoint=127.0.0.1
alluxio
.underfs.s3.proxy.https.only=false
alluxio
.underfs.s3.endpoint.http.port=9000


alluxio-1.2.0 cch$ ./bin/alluxio runTests

2016-09-28 12:36:08,032 ERROR type (CliUtils.java:runExample) - Exception running test: alluxio.examples.BasicOperations@246ae04d

 

java
.io.IOException: Failed to cache: alluxio.exception.BlockAlreadyExistsException: Temp blockId 16,777,216 is not available, because it is already committed

 at alluxio
.client.file.FileOutStream.handleCacheWriteException(FileOutStream.java:342)

 at alluxio
.client.file.FileOutStream.write(FileOutStream.java:298)

 at alluxio
.client.file.FileOutStream.write(FileOutStream.java:270)

 at alluxio
.examples.BasicOperations.writeFile(BasicOperations.java:86)

 at alluxio
.examples.BasicOperations.call(BasicOperations.java:72)

 at alluxio
.examples.BasicOperations.call(BasicOperations.java:42)

 at alluxio
.cli.CliUtils.runExample(CliUtils.java:51)

 at alluxio
.cli.TestRunner.runTest(TestRunner.java:148)

 at alluxio
.cli.TestRunner.runTests(TestRunner.java:121)

 at alluxio
.cli.TestRunner.main(TestRunner.java:98)

Caused by: java.io.IOException: alluxio.exception.BlockAlreadyExistsException: Temp blockId 16,777,216 is not available, because it is already committed

 at alluxio
.client.block.BlockWorkerClient.requestBlockLocation(BlockWorkerClient.java:358)

 at alluxio
.client.block.LocalBlockOutStream.<init>(LocalBlockOutStream.java:64)

 at alluxio
.client.block.AlluxioBlockStore.getOutStream(AlluxioBlockStore.java:170)

 at alluxio
.client.file.FileOutStream.getNextBlock(FileOutStream.java:320)

 at alluxio
.client.file.FileOutStream.write(FileOutStream.java:285)

 
... 8 more

Caused by: alluxio.exception.BlockAlreadyExistsException: Temp blockId 16,777,216 is not available, because it is already committed

 at sun
.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

 at sun
.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

 at sun
.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

 at java
.lang.reflect.Constructor.newInstance(Constructor.java:408)

 at alluxio
.exception.AlluxioException.fromThrift(AlluxioException.java:99)

 at alluxio
.AbstractClient.retryRPC(AbstractClient.java:326)

 at alluxio
.client.block.BlockWorkerClient.requestBlockLocation(BlockWorkerClient.java:348)

 
... 12 more

Failed the test!

2016-09-28 12:36:12,578 ERROR type (S3AUnderFileSystem.java:mkdirsInternal) - Failed to create directory: s3a://alluxio-quick-start/default_tests_files

com
.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 4DB686A5802EC6AE), S3 Extended Request ID: CO+BdtG2dv11Km8IG2JBAHoiJQIgSytPWecQ5qcN9Vg96gJx1YpzAhlcDUEmQZE+NTiQHkIQC0Q=

 at com
.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1305)

 at com
.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:852)

 at com
.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:630)

 at com
.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:405)

 at com
.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:367)

 at com
.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)

 at com
.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787)

 at com
.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1399)

 at alluxio
.underfs.s3a.S3AUnderFileSystem.mkdirsInternal(S3AUnderFileSystem.java:678)

 at alluxio
.underfs.s3a.S3AUnderFileSystem.mkdirs(S3AUnderFileSystem.java:323)

 at alluxio
.underfs.s3a.S3AUnderFileSystem.mkdirs(S3AUnderFileSystem.java:296)

 at alluxio
.underfs.s3a.S3AUnderFileSystem.create(S3AUnderFileSystem.java:177)

 at alluxio
.client.file.FileOutStream.<init>(FileOutStream.java:124)

 at alluxio
.client.file.BaseFileSystem.createFile(BaseFileSystem.java:102)

 at alluxio
.examples.BasicOperations.writeFile(BasicOperations.java:85)

 at alluxio
.examples.BasicOperations.call(BasicOperations.java:72)

 at alluxio
.examples.BasicOperations.call(BasicOperations.java:42)

 at alluxio
.cli.CliUtils.runExample(CliUtils.java:51)

 at alluxio
.cli.TestRunner.runTest(TestRunner.java:148)

 at alluxio
.cli.TestRunner.runTests(TestRunner.java:121)

 at alluxio
.cli.TestRunner.main(TestRunner.java:98)


alluxio-1.2.0 cch$ vim ./logs/master.log

2016-09-28 12:36:45,207 ERROR logger.type (S3AUnderFileSystem.java:mkdirsInternal) - Failed to create directory: s3a://alluxio-quick-start/default_tests_files
com
.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 6BDA56B40E8A6053), S3 Extended Request ID: 1My+SgBlvYRtjuqjHQtMKxvqWfnNUeUARRz5fv1femUUcXIFpz5AHku8tXj7jS0/aWbp7Ge5Zek=
        at com
.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1305)
        at com
.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:852)
        at com
.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:630)
        at com
.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:405)
        at com
.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:367)
        at com
.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)
        at com
.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787)
        at com
.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1399)
        at alluxio
.underfs.s3a.S3AUnderFileSystem.mkdirsInternal(S3AUnderFileSystem.java:678)
        at alluxio
.underfs.s3a.S3AUnderFileSystem.mkdirs(S3AUnderFileSystem.java:323)
        at alluxio
.master.file.meta.InodeTree.createPath(InodeTree.java:635)
        at alluxio
.master.file.FileSystemMaster.createFileInternal(FileSystemMaster.java:783)
        at alluxio
.master.file.FileSystemMaster.createFileAndJournal(FileSystemMaster.java:763)
        at alluxio
.master.file.FileSystemMaster.createFile(FileSystemMaster.java:737)
        at alluxio
.master.file.FileSystemMasterClientServiceHandler$3.call(FileSystemMasterClientServiceHandler.java:102)
        at alluxio
.master.file.FileSystemMasterClientServiceHandler$3.call(FileSystemMasterClientServiceHandler.java:99)
        at alluxio
.RpcUtils.call(RpcUtils.java:61)
        at alluxio
.master.file.FileSystemMasterClientServiceHandler.createFile(FileSystemMasterClientServiceHandler.java:99)
        at alluxio
.thrift.FileSystemMasterClientService$Processor$createFile.getResult(FileSystemMasterClientService.java:1386)
        at alluxio
.thrift.FileSystemMasterClientService$Processor$createFile.getResult(FileSystemMasterClientService.java:1370)
        at org
.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org
.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org
.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
        at org
.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java
.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java
.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java
.lang.Thread.run(Thread.java:745)



Calvin Jia於 2016年9月28日星期三 UTC+8上午2時00分15秒寫道:

Calvin Jia

unread,
Sep 28, 2016, 1:46:22 PM9/28/16
to Alluxio Users
Hi,

For your particular issue, it looks like your credentials are not correct, but the connector is successfully connecting to the non-amazon endpoint you specified. You can take a look at this documentation page for more information.

Hope this helps,
Calvin

test520

unread,
Oct 2, 2016, 11:58:47 PM10/2/16
to Alluxio Users
Hi Calvin,

I follow the guide but still cannot hook alluxio 1.2 with non-Amazon service provider.

Does s3a module of alluxio 1.2 compatible with non-Amazon service provider??



Calvin Jia於 2016年9月29日星期四 UTC+8上午1時46分22秒寫道:

Calvin Jia

unread,
Oct 3, 2016, 3:23:37 PM10/3/16
to Alluxio Users
Hi,

The s3a module should support non-Amazon service providers. However the support for each non-Amazon provider would be on a case by case basis, ie. s3a with Minio has not specifically been validated. What kind of error or unexpected behavior do you see? Were you able to resolve the security credential issue?

Thanks,
Calvin

test520

unread,
Oct 3, 2016, 10:49:18 PM10/3/16
to Alluxio Users
Hi Calvin,

 the security credential issue occurred when using s3n module to connect non-Amazon service providers.

 I prefer to use s3a module to connect minio if it can support non-Amazon service providers.

 However,
I still cannot connect to non-Amazon service provider thru. s3a module

 Why does alluxio still try to connect Amazon s3????


alluxio-1.2.0 cch$ cat conf/alluxio-site.properties

alluxio.underfs.address=s3a://myBucket
aws
.accessKeyId=xxxxxxxxxxxxxxxxxxxxxxxx
aws
.secretKey=xxxxxxxxxxxxxxxxxxxxxxx
alluxio
.underfs.s3.disable.dns.buckets=true
alluxio
.underfs.s3.endpoint=127.0.0.1
alluxio
.underfs.s3.proxy.https.only=false
alluxio
.underfs.s3.endpoint.http.port=9000


alluxio-1.2.0 cch$ ./bin/alluxio runTests

2016-10-04 10:30:58,113 INFO  AmazonHttpClient (AmazonHttpClient.java:executeHelper) - Unable to execute HTTP request: Connect to myBucket.s3.amazonaws.com:80 [myBucket.s3.amazonaws.com/54.231.81.104] failed: connect timed out
org
.apache.http.conn.ConnectTimeoutException: Connect to myBucket.s3.amazonaws.com:80 [myBucket.s3.amazonaws.com/54.231.81.104] failed: connect timed out
        at org
.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:143)
        at org
.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
        at sun
.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun
.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun
.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java
.lang.reflect.Method.invoke(Method.java:497)
        at com
.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
        at com
.amazonaws.http.conn.$Proxy3.connect(Unknown Source)
        at org
.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
        at org
.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org
.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org
.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org
.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org
.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at com
.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
        at com
.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:787)

        at com
.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:630)
        at com
.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:405)
        at com
.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:367)
        at com
.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)
        at com
.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787)

        at com
.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1063)
        at com
.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4200)
        at com
.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4180)
        at com
.amazonaws.services.s3.AmazonS3Client.resolveServiceEndpoint(AmazonS3Client.java:4155)
        at com
.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:997)
        at com
.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:982)
        at alluxio
.underfs.s3a.S3AUnderFileSystem.getFolderMetadata(S3AUnderFileSystem.java:517)
        at alluxio
.underfs.s3a.S3AUnderFileSystem.isFolder(S3AUnderFileSystem.java:578)
        at alluxio
.underfs.s3a.S3AUnderFileSystem.mkdirs(S3AUnderFileSystem.java:304)

        at alluxio
.underfs.s3a.S3AUnderFileSystem.mkdirs(S3AUnderFileSystem.java:296)
        at alluxio
.underfs.s3a.S3AUnderFileSystem.create(S3AUnderFileSystem.java:177)
        at alluxio
.client.file.FileOutStream.<init>(FileOutStream.java:124)
        at alluxio
.client.file.BaseFileSystem.createFile(BaseFileSystem.java:102)
        at alluxio
.examples.BasicOperations.writeFile(BasicOperations.java:85)
        at alluxio
.examples.BasicOperations.call(BasicOperations.java:72)
        at alluxio
.examples.BasicOperations.call(BasicOperations.java:42)
        at alluxio
.cli.CliUtils.runExample(CliUtils.java:51)
        at alluxio
.cli.TestRunner.runTest(TestRunner.java:148)
        at alluxio
.cli.TestRunner.runTests(TestRunner.java:121)
        at alluxio
.cli.TestRunner.main(TestRunner.java:98)
Caused by: java.net.SocketTimeoutException: connect timed out


Calvin Jia於 2016年10月4日星期二 UTC+8上午3時23分37秒寫道:

Calvin Jia

unread,
Oct 4, 2016, 6:29:17 PM10/4/16
to Alluxio Users
Hi,

Setting the endpoint for S3A is supported from 1.3 onwards, you can try using the 1.3 release candidate here: http://downloads.alluxio.org/downloads/files/1.3.0-RC1

However, the endpoint is intended for specific AWS regions and may not be compatible with non Amazon providers.

Hope this helps,
Calvin

test520

unread,
Oct 10, 2016, 3:00:54 AM10/10/16
to Alluxio Users
  Hi Calvin,

  Again, I try alluxio 1.3 for mounting non Amazon providers but still failed.

  Why does s3a connect to my s3 server thru. port 80 after I set the port to 9000 in alluxio conf ???


alluxio-1.3.0 cch$ vim /Users/cch/Downloads/alluxio-1.3.0/logs/master.log



 
2016-10-10 14:47:31,679 INFO  http.AmazonHttpClient (AmazonHttpClient.java:executeHelper) - Unable to execute HTTP request: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
        at com.amazonaws.http.conn.$Proxy3.connect(Unknown Source)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:787)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:630)
        at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:405)
        at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:367)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3729)
        at com.amazonaws.services.s3.AmazonS3Client.getS3AccountOwner(AmazonS3Client.java:689)
        at com.amazonaws.services.s3.AmazonS3Client.getS3AccountOwner(AmazonS3Client.java:681)
        at alluxio.underfs.s3a.S3AUnderFileSystem.createInstance(S3AUnderFileSystem.java:161)
        at alluxio.underfs.s3a.S3AUnderFileSystemFactory.create(S3AUnderFileSystemFactory.java:45)
        at alluxio.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:114)
        at alluxio.underfs.UnderFileSystem$Cache.get(UnderFileSystem.java:136)
        at alluxio.underfs.UnderFileSystem.get(UnderFileSystem.java:207)
        at alluxio.underfs.UnderFileSystem.get(UnderFileSystem.java:194)
        at alluxio.master.file.FileSystemMaster$Metrics.registerGauges(FileSystemMaster.java:2715)
        at alluxio.master.file.FileSystemMaster$Metrics.access$000(FileSystemMaster.java:2658)
        at alluxio.master.file.FileSystemMaster.<init>(FileSystemMaster.java:274)
        at alluxio.master.file.FileSystemMaster.<init>(FileSystemMaster.java:247)
        at alluxio.master.AlluxioMaster.<init>(AlluxioMaster.java:258)
        at alluxio.master.AlluxioMaster$Factory.create(AlluxioMaster.java:208)
        at alluxio.master.AlluxioMaster.main(AlluxioMaster.java:86)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
        ... 36 more
2016-10-10 14:47:31,920 ERROR logger.type (AlluxioMaster.java:<init>) - All eligible Under File Systems were unable to create an instance for the given path: s3a:/
/alluxio-quick-start/data
com
.amazonaws.AmazonClientException: Unable to execute HTTP request: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3a:/
/alluxio-quick-start/data
com
.amazonaws.AmazonClientException: Unable to execute HTTP request: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
        at alluxio
.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:130)
        at alluxio
.underfs.UnderFileSystem$Cache.get(UnderFileSystem.java:136)
        at alluxio
.underfs.UnderFileSystem.get(UnderFileSystem.java:207)
        at alluxio
.underfs.UnderFileSystem.get(UnderFileSystem.java:194)
        at alluxio
.master.file.FileSystemMaster$Metrics.registerGauges(FileSystemMaster.java:2715)
        at alluxio
.master.file.FileSystemMaster$Metrics.access$000(FileSystemMaster.java:2658)
        at alluxio
.master.file.FileSystemMaster.<init>(FileSystemMaster.java:274)
        at alluxio
.master.file.FileSystemMaster.<init>(FileSystemMaster.java:247)
        at alluxio
.master.AlluxioMaster.<init>(AlluxioMaster.java:258)
        at alluxio
.master.AlluxioMaster$Factory.create(AlluxioMaster.java:208)
        at alluxio
.master.AlluxioMaster.main(AlluxioMaster.java:86)



alluxio-1.3.0 cch$ vim conf/alluxio-site.properties

alluxio.underfs.address=s3a://alluxio-quick-start/data
aws
.accessKeyId=xxxxxxxx
aws
.secretKey=xxxxxxxxxx
alluxio
.underfs.s3.endpoint=127.0.0.1
alluxio
.underfs.s3.proxy.https.only=false
alluxio
.underfs.s3.endpoint.http.port=9000


alluxio-1.3.0 cch$ vim conf/alluxio-env.sh

ALLUXIO_MASTER_HOSTNAME=${ALLUXIO_MASTER_HOSTNAME:-"localhost"}
ALLUXIO_WORKER_MEMORY_SIZE
=${ALLUXIO_WORKER_MEMORY_SIZE:-"10922MB"}
ALLUXIO_RAM_FOLDER
=${ALLUXIO_RAM_FOLDER:-"/Volumes/ramdisk"}


alluxio-1.3.0 cch$ java -version

java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)




Calvin Jia於 2016年10月5日星期三 UTC+8上午6時29分17秒寫道:

Calvin Jia

unread,
Oct 10, 2016, 2:02:47 PM10/10/16
to Alluxio Users
Hi,

If you are using the S3A connector, you should specify the entire uri in the s3 endpoint configuration (ie. alluxio.underfs.s3.endpoint=127.0.0.1:9000)

Only s3n uses the port configuration as of now. Feel free to open a JIRA ticket if you have a strong preference for how the configuration should work.

Hope this helps,
Calvin

test520

unread,
Oct 10, 2016, 10:35:27 PM10/10/16
to Alluxio Users
Hi Calvin,

 Thanks for the suggestion, I re-configure the properties file but get another error message as below, 

 Do I config the right parameter for using s3a connector with non Amazon providers??

alluxio-1.3.0 cch$ vim conf/alluxio-site.properties
alluxio.underfs.address=s3a://alluxio-quick-start/data
aws
.accessKeyId=xxxxxxxxxxxxxxxxx
aws
.secretKey=xxxxxxxxxxxxxxxxxxx
alluxio
.underfs.s3.endpoint=127.0.0.1:9000

alluxio-1.3.0 cch$ vim /Users/cch/Downloads/alluxio-1.3.0/logs/master.log

2016-10-11 10:16:17,942 ERROR logger.type (S3AUnderFileSystemFactory.java:create) - Failed to create S3AUnderFileSystem.
com
.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented (Service: Amazon S3; Status Code: 501; Error Code: NotImplemented; Request ID: 3L137), S3 Extended Request ID: 3L137

    at com
.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1305)
    at com
.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:852)

    at com
.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:630)
    at com
.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:405)
    at com
.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:367)
    at com
.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)
    at com
.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787)

    at com
.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3008)
    at com
.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:915)
    at com
.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:906)
    at alluxio
.underfs.s3a.S3AUnderFileSystem.createInstance(S3AUnderFileSystem.java:173)

    at alluxio
.underfs.s3a.S3AUnderFileSystemFactory.create(S3AUnderFileSystemFactory.java:45)
    at alluxio
.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:114)
    at alluxio
.underfs.UnderFileSystem$Cache.get(UnderFileSystem.java:136)
    at alluxio
.underfs.UnderFileSystem.get(UnderFileSystem.java:207)
    at alluxio
.underfs.UnderFileSystem.get(UnderFileSystem.java:194)
    at alluxio
.master.file.FileSystemMaster$Metrics.registerGauges(FileSystemMaster.java:2715)
    at alluxio
.master.file.FileSystemMaster$Metrics.access$000(FileSystemMaster.java:2658)
    at alluxio
.master.file.FileSystemMaster.<init>(FileSystemMaster.java:274)
    at alluxio
.master.file.FileSystemMaster.<init>(FileSystemMaster.java:247)
    at alluxio
.master.AlluxioMaster.<init>(AlluxioMaster.java:258)
    at alluxio
.master.AlluxioMaster$Factory.create(AlluxioMaster.java:208)
    at alluxio
.master.AlluxioMaster.main(AlluxioMaster.java:86)
2016-10-11 10:16:17,945 ERROR logger.type (AlluxioMaster.java:<init>) - All eligible Under File Systems were unable to create an instance for the given path: s3a://alluxio-quick-start/data
com.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented (Service: Amazon S3; Status Code: 501; Error Code: NotImplemented; Request ID: 3L137), S3 Extended Request ID: 3L137
 java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3a://alluxio-quick-start/data
 com.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented (Service: Amazon S3; Status Code: 501; Error Code: NotImplemented; Request ID: 3L137), S3 Extended Request ID: 3L137

Calvin Jia

unread,
Oct 13, 2016, 2:01:25 PM10/13/16
to Alluxio Users
Hi,

It seems like the back end is not fully compatible with the S3A connector. One thing you may want to try is disabling HTTPS by setting alluxio.underfs.s3a.secure.http.enabled=false.

Hope this helps,
Calvin

test520

unread,
Oct 14, 2016, 12:23:35 PM10/14/16
to Alluxio Users
  Hi Calvin,

  Thanks for the help.

  We solved the issue by modifying the s3a source code and upgrade aws sdk version.

Calvin Jia於 2016年10月14日星期五 UTC+8上午2時01分25秒寫道:

Calvin Jia

unread,
Oct 14, 2016, 2:24:50 PM10/14/16
to Alluxio Users
Great to hear! Could you share which version of aws-sdk you needed to use (or do you mean a custom version you guys rebuilt)? As far as I know the version Alluxio uses should have the latests updates for S3.

Thanks,
Calvin

test520

unread,
Oct 17, 2016, 9:53:37 PM10/17/16
to Alluxio Users
Hi Calvin,


Add short bucketMode = (short) 0700; @ line 175

Change aws-java-sdk to 1.11.40

<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk -->
<dependency>
   
<groupId>com.amazonaws</groupId>
   
<artifactId>aws-java-sdk</artifactId>
   
<version>1.11.40</version>
</dependency>





Calvin Jia於 2016年10月15日星期六 UTC+8上午2時24分50秒寫道:

Calvin Jia

unread,
Oct 18, 2016, 3:55:12 PM10/18/16
to Alluxio Users
Thanks for sharing! Do you know what change was necessary to have in the upgraded java-sdk version?

Cheers,
Calvin

krishna chaitanya Sasubilli

unread,
Apr 9, 2018, 5:41:20 PM4/9/18
to Alluxio Users
I am still facing the same issue with the latest version of alluxio 1.7.1.
my config looks like this:


alluxio.underfs.address=s3a://hello/
alluxio.underfs.s3.endpoint=10.242.102.102:9000
alluxio.underfs.s3.proxy.https.only=false
alluxio.underfs.s3.endpoint.http.port=9000
alluxio.underfs.s3.disable.dns.buckets=true
alluxio.underfs.s3a.inherit_acl=false
alluxio.underfs.s3a.secure.http.enabled=false
aws.accessKeyId=%%
aws.secretKey=%%

krishna chaitanya Sasubilli

unread,
Apr 9, 2018, 5:44:25 PM4/9/18
to Alluxio Users
Exception:

2018-04-09 17:43:03,896 WARN  UnderFileSystem$Factory - Failed to create UnderFileSystem by factory alluxio.underfs.s3a.S3AUnderFileSystemFactory@800df0a: javax/xml/bind/JAXBException
2018-04-09 17:43:03,898 ERROR AlluxioMaster - Failed to create master process
java.lang.RuntimeException: java.lang.RuntimeException: Failed to start masters
        at alluxio.master.AlluxioMasterProcess.<init>(AlluxioMasterProcess.java:156)
        at alluxio.master.MasterProcess$Factory.create(MasterProcess.java:45)
        at alluxio.master.AlluxioMaster.main(AlluxioMaster.java:45)
Caused by: java.lang.RuntimeException: Failed to start masters
        at alluxio.master.MasterUtils.createMasters(MasterUtils.java:52)
        at alluxio.master.AlluxioMasterProcess.<init>(AlluxioMasterProcess.java:154)
        ... 2 more

Calvin Jia

unread,
Apr 10, 2018, 9:13:55 PM4/10/18
to Alluxio Users
Hi,

Which version of Java are you using?

Thanks,
Calvin

Calvin Jia

unread,
Apr 19, 2018, 8:13:14 PM4/19/18
to Alluxio Users
Were you able to resolve the issue? I think Java 9 may cause this kind of problem.
Reply all
Reply to author
Forward
0 new messages