Hadoop S3 Error

397 views
Skip to first unread message

Amran Chen

unread,
Jul 10, 2016, 12:06:12 PM7/10/16
to Alluxio Users
Hi, I found this error:

 hadoop fs -ls "s3n://my-new-bucket"
16/07/08 21:08:42 WARN util.NativeCodeLoader: Unable to load native-hadoop libra         ry for your platform... using builtin-java classes where applicable
16/07/08 21:08:47 WARN utils.RestUtils: Unable to calculate value of time offset          between the local machine and AWS server
16/07/08 21:08:47 WARN httpclient.RestStorageService: Adjusted time offset in re         sponse to RequestTimeTooSkewed error. Local machine and S3 server disagree on th         e time by approximately 0 seconds. Retrying connection.
16/07/08 21:08:47 WARN httpclient.RestStorageService: Error Response: GET '/?del         imiter=/&max-keys=1&prefix=user/userceph/' -- ResponseCode: 403, ResponseStatus:          Forbidden, Request Headers: [Date: Fri, 08 Jul 2016 13:08:45 GMT, Authorization         : AWS QINHSA7JD2MBEKN97F4A:cSmg+13FhOE6L1+KgOGGzD0r+6U=], Response Headers: [x-a         mz-bucket-region: eu-west-1, x-amz-request-id: 141CDBB578E2A92A, x-amz-id-2: hzM         3nlJ9Z0fSUm0QTmI5oL+g4rWE5ncSk2rw6hlKy65lYBcNh4WEwUYgUMSqRlTMf6fpaW74gRY=, Conte         nt-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 10 Jul 2016 16:         02:35 GMT, Server: AmazonS3]
16/07/08 21:08:48 WARN httpclient.RestStorageService: Adjusted time offset in re         sponse to RequestTimeTooSkewed error. Local machine and S3 server disagree on th         e time by approximately 183228 seconds. Retrying connection.
16/07/08 21:08:48 WARN httpclient.RestStorageService: Error Response: GET '/?del         imiter=/&max-keys=1&prefix=user/userceph/' -- ResponseCode: 403, ResponseStatus:          Forbidden, Request Headers: [Date: Fri, 08 Jul 2016 13:08:47 GMT, Authorization         : AWS QINHSA7JD2MBEKN97F4A:lOciu/E5NIRFR9eXZkcdQcaKZzM=], Response Headers: [x-a         mz-bucket-region: eu-west-1, x-amz-request-id: 158E2A8226ED0749, x-amz-id-2: 4Ht         efWcVBc5kPHVZ6v9IiiEN16DfJ5YDKdC0ur7AcEyS9aDxV8LF0XY/ilvHpbHe3QhPMY/TQHI=, Conte         nt-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 10 Jul 2016 16:         02:36 GMT, Server: AmazonS3]
-ls: Fatal internal error
java.lang.NullPointerException
        at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(Native         S3FileSystem.java:433)
        at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
        at org.apache.hadoop.fs.Globber.glob(Globber.java:238)
        at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1624)
        at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
        at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
        at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
        at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:1         90)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)

anyone can help me?

Amran Chen

unread,
Jul 10, 2016, 1:18:13 PM7/10/16
to Alluxio Users
Hello all,
I want to update after running Hadoop 2.3.0 because of alluxio reference using hadoop 2.2.0, 2.3.0 or 2.4.0 I got an error like this

userceph@hadoop:~$ hadoop fs -ls "s3n://my-new-bucket"
16/07/08 22:21:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: Permission denied: s3n://my-new-bucket/user/userceph

I am sure that i input S3 AccessKey and SecretKey are correct. Why? pls help

Gene Pang

unread,
Jul 11, 2016, 10:02:31 AM7/11/16
to Alluxio Users
Hi,

Are you using Alluxio? It looks like you are using the 'hadoop' command, and there is no mention of Alluxio in the stack trace.

Could you describe your environment and use case?

Thanks,
Gene
Message has been deleted

Gene Pang

unread,
Jul 11, 2016, 10:19:45 AM7/11/16
to Alluxio Users
Hi,

From the stack trace, hadoop is still trying to access the s3 bucket, not Alluxio.

Since you already used your s3 bucket as the ALLUXIO_UNDERFS_ADDRESS for Alluxio, you can access the s3 bucket through Alluxio, with the transparent naming feature (http://www.alluxio.org/docs/master/en/Unified-and-Transparent-Namespace.html).

Therefore, the s3 bucket should be replaced by the an Alluxio path, to where you want to write the data. For example:

hadoop jar ... alluxio://host:port/wordcount/myfile alluxio://host:port/test

That will read and write through Alluxio, and Alluxio can take care of the s3 interactions.

Thanks,
Gene

Amran Chen

unread,
Jul 11, 2016, 10:31:10 AM7/11/16
to Alluxio Users
Hi Gene,
Thank you very much for your replied.

I tried to change it, I found this error after running:


userceph@hadoop:~$ hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount -libjars alluxio/core/client/target/alluxio-core-client-1.2.0-SNAPSHOT-jar-with-dependencies.jar alluxio://192.168.244.144:19998/wordcount/myfile alluxio://192.168.244.141:7480/my-new-bucket
16/07/11 22:27:45 INFO logger.type: initialize(alluxio://192.168.244.144:19998/wordcount/myfile, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Alluxio: alluxio://192.168.244.144:19998/wordcount/myfile
16/07/11 22:27:45 INFO logger.type: alluxio://192.168.244.144:19998 alluxio://192.168.244.144:19998
16/07/11 22:27:45 INFO logger.type: Loading Alluxio properties from Hadoop configuration: {fs.s3n.awsAccessKeyId=QINHSA7JD2MBEKN97F4A, fs.s3n.awsSecretAccessKey=mbjd9H80AokLYFP2Yx44K64Jgy5JdXZIlrWRrcPn}
16/07/11 22:27:45 INFO logger.type: getWorkingDirectory: /
16/07/11 22:27:45 INFO logger.type: initialize(alluxio://192.168.244.141:7480/my-new-bucket, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Alluxio: alluxio://192.168.244.141:7480/my-new-bucket
16/07/11 22:27:45 INFO logger.type: alluxio://192.168.244.141:7480 alluxio://192.168.244.141:7480
16/07/11 22:27:45 INFO logger.type: getWorkingDirectory: /
16/07/11 22:27:46 INFO client.RMProxy: Connecting to ResourceManager at hadoop/192.168.244.144:8032
16/07/11 22:27:46 INFO logger.type: getFileStatus(alluxio://192.168.244.141:7480/my-new-bucket)
16/07/11 22:27:46 INFO logger.type: Alluxio client (version 1.2.0-SNAPSHOT) is trying to connect with FileSystemMasterClient master @ /192.168.244.144:19998
16/07/11 22:27:46 INFO logger.type: Client registered with FileSystemMasterClient master @ /192.168.244.144:19998
16/07/11 22:27:47 INFO logger.type: getFileStatus(alluxio://192.168.244.144:19998/wordcount/myfile)
16/07/11 22:27:47 INFO input.FileInputFormat: Total input paths to process : 1
16/07/11 22:27:47 INFO mapreduce.JobSubmitter: number of splits:1
16/07/11 22:27:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468230529370_0003
16/07/11 22:27:48 INFO impl.YarnClientImpl: Submitted application application_1468230529370_0003
16/07/11 22:27:48 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1468230529370_0003/
16/07/11 22:27:48 INFO mapreduce.Job: Running job: job_1468230529370_0003
16/07/11 22:28:02 INFO mapreduce.Job: Job job_1468230529370_0003 running in uber mode : false
16/07/11 22:28:02 INFO mapreduce.Job:  map 0% reduce 0%
16/07/11 22:28:04 INFO mapreduce.Job: Job job_1468230529370_0003 failed with state FAILED due to: Job setup failed : java.io.IOException
        at alluxio.AbstractClient.checkVersion(AbstractClient.java:112)
        at alluxio.AbstractClient.connect(AbstractClient.java:175)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
        at alluxio.client.file.FileSystemMasterClient.createDirectory(FileSystemMasterClient.java:90)
        at alluxio.client.file.BaseFileSystem.createDirectory(BaseFileSystem.java:79)
        at alluxio.hadoop.AbstractFileSystem.mkdirs(AbstractFileSystem.java:491)
        at alluxio.hadoop.FileSystem.mkdirs(FileSystem.java:25)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1817)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:291)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:254)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:234)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


16/07/11 22:28:05 INFO mapreduce.Job: Counters: 2
        Job Counters
                Total time spent by all maps in occupied slots (ms)=0
                Total time spent by all reduces in occupied slots (ms)=0
userceph@hadoop:~$

What happened with this, please help..


Gene Pang於 2016年7月11日星期一 UTC+8下午10時19分45秒寫道：

Gene Pang

unread,
Jul 11, 2016, 10:34:55 AM7/11/16
to Alluxio Users
Are you running different versions of Alluxio for the client and master? It looks like checking the version failed?

16/07/11 22:28:04 INFO mapreduce.Job: Job job_1468230529370_0003 failed with state FAILED due to: Job setup failed : java.io.IOException
        at alluxio.AbstractClient.checkVersion(AbstractClient.java:112)

Thanks,
Gene
Message has been deleted

Gene Pang

unread,
Jul 11, 2016, 10:50:57 AM7/11/16
to Alluxio Users

Hi,

Could you check your Alluxio master logs, to see if there is any information there?

Also, it seems like your Alluxio client is from the master branch (not 1.1.0 release), given by the name 1.2.0-SNAPSHOT.

... -libjars alluxio/core/client/target/alluxio-core-client-1.2.0-SNAPSHOT-jar-with-dependencies.jar ...

Thanks,
Gene

Amran Chen

unread,
Jul 11, 2016, 11:00:16 AM7/11/16
to Alluxio Users
Hello Gene,
in my alluxio/logs/master.log



userceph@hadoop:~/alluxio/logs$ tail master.log
2016-07-11 22:33:51,755 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:33:51 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:J3qtrHnLKklQkyRGgMemoDyxL1c=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005e2-005783ae4f-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:33:51 GMT]
2016-07-11 22:33:51,801 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:33:51 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:lqp3gDV1tEkYecbxp+P1438c3iA=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005e4-005783ae4f-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:33:51 GMT]
2016-07-11 22:42:14,459 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:42:14 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:t5zqWM22vkatJ9bYJTZQ/AIgf+I=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005e5-005783b046-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:42:14 GMT]
2016-07-11 22:42:14,529 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:42:14 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:ohNzARc8bSR/Wqv59qllkieqbfc=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005e7-005783b046-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:42:14 GMT]
2016-07-11 22:43:31,393 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:43:31 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:GZpxidq64IASF/FCENmTZnDMhmQ=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005e8-005783b093-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:43:31 GMT]
2016-07-11 22:43:31,469 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:43:31 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:rEUOwGRegCIBl9iT63WmvaC1XjI=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005ea-005783b093-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:43:31 GMT]
2016-07-11 22:49:38,760 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket%2Ftest_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:49:38 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:lHbFlZ/TXUsMm5VTgPgkKRVfBQ0=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005eb-005783b202-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:49:38 GMT]
2016-07-11 22:49:38,825 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket%2Ftest' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:49:38 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:yMmcN+kJFwU+PPs61n0PgPo0OCM=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005ed-005783b202-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:49:38 GMT]
2016-07-11 22:57:22,605 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:57:22 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:mms3AUGfaHzeEFRkTpfKg46SHv4=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005ee-005783b3d2-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:57:22 GMT]
2016-07-11 22:57:22,653 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/my-new-bucket/my-new-bucket' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Mon, 11 Jul 2016 14:57:22 GMT, Authorization: AWS QINHSA7JD2MBEKN97F4A:J86Ge5qAVuCCNClTYv7uXAUiDhY=, User-Agent: JetS3t/0.8.1 (Linux/4.2.0-27-generic; amd64; en; JVM 1.7.0_101), Host: mon1:7480], Response Headers: [x-amz-request-id: tx0000000000000000005f0-005783b3d2-ac41-default, Content-Length: 145, Accept-Ranges: bytes, Content-Type: application/xml, Date: Mon, 11 Jul 2016 14:57:22 GMT]


in the core/client/target I have alluxio-core-client-1.2.0-SNAPSHOT.jar

userceph@hadoop:~/alluxio/core/client/target$ ls
alluxio-core-client-1.2.0-SNAPSHOT.jar                        checkstyle-checker.xml       generated-sources
alluxio-core-client-1.2.0-SNAPSHOT-jar-with-dependencies.jar  checkstyle-result.xml        generated-test-sources
alluxio-core-client-1.2.0-SNAPSHOT-javadoc.jar                checkstyle-suppressions.xml  javadoc-bundle-options
alluxio-core-client-1.2.0-SNAPSHOT-sources.jar                classes                      maven-archiver
alluxio-core-client-1.2.0-SNAPSHOT-tests.jar                  findbugs-exclude.xml         maven-status
apidocs                                                       findbugs.xml                 test-classes
checkstyle-cachefile                                          findbugsXml.xml
userceph@hadoop:~/alluxio/core/client/target$


what's this mean Gene...


Gene Pang於 2016年7月11日星期一 UTC+8下午10時50分57秒寫道：

Amran Chen

unread,
Jul 11, 2016, 12:28:39 PM7/11/16
to Alluxio Users
Hello Gene,
I have change to Alluxio 1.1.0 version

userceph@hadoop:~/alluxio-1.1.0/core/client/target$ ls
alluxio-core-client-1.1.0.jar                        classes                 maven-status
alluxio-core-client-1.1.0-jar-with-dependencies.jar  generated-sources       test-classes
alluxio-core-client-1.1.0-sources.jar                generated-test-sources
alluxio-core-client-1.1.0-tests.jar                  maven-archiver
userceph@hadoop:~/alluxio-1.1.0/core/client/target$

userceph@hadoop:~/alluxio-1.1.0$ bin/alluxio version
Alluxio version: 1.1.0
userceph@hadoop:~/alluxio-1.1.0$

it's like the same error :

hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount -libjars alluxio-1.1.0/core/client/target/alluxio-core-client-1.1.0-jar-with-dependencies.jar alluxio://192.168.244.144:19998/wordcount/myfile alluxio://192.168.244.141:7480/my-new-bucket/test
16/07/12 00:27:35 INFO logger.type: initialize(alluxio://192.168.244.144:19998/wordcount/myfile, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Alluxio: alluxio://192.168.244.144:19998/wordcount/myfile
16/07/12 00:27:35 INFO logger.type: alluxio://192.168.244.144:19998 alluxio://192.168.244.144:19998
16/07/12 00:27:35 INFO logger.type: Loading Alluxio properties from Hadoop configuration: {fs.s3n.awsAccessKeyId=QINHSA7JD2MBEKN97F4A, fs.s3n.awsSecretAccessKey=mbjd9H80AokLYFP2Yx44K64Jgy5JdXZIlrWRrcPn}
16/07/12 00:27:35 INFO logger.type: getWorkingDirectory: /
16/07/12 00:27:35 INFO logger.type: initialize(alluxio://192.168.244.141:7480/my-new-bucket/test, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Alluxio: alluxio://192.168.244.141:7480/my-new-bucket/test
16/07/12 00:27:35 INFO logger.type: alluxio://192.168.244.141:7480 alluxio://192.168.244.141:7480
16/07/12 00:27:35 INFO logger.type: getWorkingDirectory: /
16/07/12 00:27:35 INFO client.RMProxy: Connecting to ResourceManager at hadoop/192.168.244.144:8032
16/07/12 00:27:35 INFO logger.type: getFileStatus(alluxio://192.168.244.141:7480/my-new-bucket/test)
16/07/12 00:27:35 INFO logger.type: Alluxio client (version 1.1.0) is trying to connect with FileSystemMasterClient master @ /192.168.244.144:19998
16/07/12 00:27:35 INFO logger.type: Client registered with FileSystemMasterClient master @ /192.168.244.144:19998
16/07/12 00:27:37 INFO logger.type: getFileStatus(alluxio://192.168.244.144:19998/wordcount/myfile)
16/07/12 00:27:37 INFO input.FileInputFormat: Total input paths to process : 1
16/07/12 00:27:38 INFO mapreduce.JobSubmitter: number of splits:1
16/07/12 00:27:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468230529370_0017
16/07/12 00:27:38 INFO impl.YarnClientImpl: Submitted application application_1468230529370_0017
16/07/12 00:27:38 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1468230529370_0017/
16/07/12 00:27:38 INFO mapreduce.Job: Running job: job_1468230529370_0017
16/07/12 00:27:44 INFO mapreduce.Job: Job job_1468230529370_0017 running in uber mode : false
16/07/12 00:27:44 INFO mapreduce.Job:  map 0% reduce 0%
16/07/12 00:27:45 INFO mapreduce.Job: Job job_1468230529370_0017 failed with state FAILED due to: Job setup failed : java.io.IOException
        at alluxio.AbstractClient.checkVersion(AbstractClient.java:115)
        at alluxio.AbstractClient.connect(AbstractClient.java:178)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:325)
        at alluxio.client.file.FileSystemMasterClient.createDirectory(FileSystemMasterClient.java:92)
        at alluxio.client.file.BaseFileSystem.createDirectory(BaseFileSystem.java:79)
        at alluxio.hadoop.AbstractFileSystem.mkdirs(AbstractFileSystem.java:494)
        at alluxio.hadoop.FileSystem.mkdirs(FileSystem.java:25)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1817)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:291)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:254)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:234)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


16/07/12 00:27:45 INFO mapreduce.Job: Counters: 2
        Job Counters
                Total time spent by all maps in occupied slots (ms)=0
                Total time spent by all reduces in occupied slots (ms)=0
userceph@hadoop:~$


Help me please...

Thank you Gene 





Gene Pang於 2016年7月11日星期一 UTC+8下午10時50分57秒寫道：

Amran Chen

unread,
Jul 12, 2016, 2:11:10 AM7/12/16
to Alluxio Users



Amran Chen於 2016年7月12日星期二 UTC+8上午12時28分39秒寫道：

Amran Chen

unread,
Jul 12, 2016, 2:12:46 AM7/12/16
to Alluxio Users
I found Alluxio-1.1.0 and Alluxio-1.2.0-RC1 are the same error like this:



Amran Chen於 2016年7月12日星期二 UTC+8下午2時11分10秒寫道：

Gene Pang

unread,
Jul 12, 2016, 12:30:38 PM7/12/16
to Alluxio Users
Hi,

What version of Alluxio are you running for the master? That information can be found near the beginning of the master log.

Does a simple use case work without running hadoop mapreduce? Can you try simple Alluxio commands like:

./bin/alluxio fs ls /
./bin/alluxio fs copyFromLocal LICENSE /LICENSE

Thanks,
Gene
Message has been deleted
Message has been deleted

Amran Chen

unread,
Jul 12, 2016, 2:44:02 PM7/12/16
to Alluxio Users
I tried to run it again with no mounted:

 hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount -libjars alluxio-1.2.0-RC1/core/client/target/alluxio-core-client-1.2.0-RC1-jar-with-dependencies.jar alluxio://192.168.244.146:19998/wordcount/myfile alluxio://192.168.244.141:7480/my-new-bucket/


192.168.244.146 : hadoop + alluxio
192.168.244.141 : monitor + radosgw + s3 (Ceph)



the output:

userceph@hadoop:~$ hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount -libjars alluxio-1.2.0-RC1/core/client/target/alluxio-core-client-1.2.0-RC1-jar-with-dependencies.jar alluxio://192.168.244.146:19998/wordcount/myfile alluxio://192.168.244.141:7480/my-new-bucket/
16/07/13 02:40:10 INFO logger.type: initialize(alluxio://192.168.244.146:19998/wordcount/myfile, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Alluxio: alluxio://192.168.244.146:19998/wordcount/myfile
16/07/13 02:40:10 INFO logger.type: alluxio://192.168.244.146:19998 alluxio://192.168.244.146:19998
16/07/13 02:40:10 INFO logger.type: Loading Alluxio properties from Hadoop configuration: {fs.s3n.awsAccessKeyId=QINHSA7JD2MBEKN97F4A, fs.s3n.awsSecretAccessKey=mbjd9H80AokLYFP2Yx44K64Jgy5JdXZIlrWRrcPn}
16/07/13 02:40:10 INFO logger.type: getWorkingDirectory: /
16/07/13 02:40:10 INFO logger.type: initialize(alluxio://192.168.244.141:7480/my-new-bucket, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Alluxio: alluxio://192.168.244.141:7480/my-new-bucket
16/07/13 02:40:10 INFO logger.type: alluxio://192.168.244.141:7480 alluxio://192.168.244.141:7480
16/07/13 02:40:10 INFO logger.type: getWorkingDirectory: /
16/07/13 02:40:10 INFO client.RMProxy: Connecting to ResourceManager at hadoop/192.168.244.146:8032
16/07/13 02:40:11 INFO logger.type: getFileStatus(alluxio://192.168.244.141:7480/my-new-bucket)
16/07/13 02:40:11 INFO logger.type: Alluxio client (version 1.2.0-RC1) is trying to connect with FileSystemMasterClient master @ /192.168.244.146:19998
16/07/13 02:40:11 INFO logger.type: Client registered with FileSystemMasterClient master @ /192.168.244.146:19998
16/07/13 02:40:13 INFO logger.type: getFileStatus(alluxio://192.168.244.146:19998/wordcount/myfile)
16/07/13 02:40:13 INFO input.FileInputFormat: Total input paths to process : 1
16/07/13 02:40:13 INFO mapreduce.JobSubmitter: number of splits:1
16/07/13 02:40:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468337876410_0012
16/07/13 02:40:14 INFO impl.YarnClientImpl: Submitted application application_1468337876410_0012
16/07/13 02:40:14 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1468337876410_0012/
16/07/13 02:40:14 INFO mapreduce.Job: Running job: job_1468337876410_0012
16/07/13 02:40:22 INFO mapreduce.Job: Job job_1468337876410_0012 running in uber mode : false
16/07/13 02:40:22 INFO mapreduce.Job:  map 0% reduce 0%
16/07/13 02:40:22 INFO mapreduce.Job: Job job_1468337876410_0012 failed with state FAILED due to: Job setup failed : java.io.IOException
        at alluxio.AbstractClient.checkVersion(AbstractClient.java:112)
        at alluxio.AbstractClient.connect(AbstractClient.java:175)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
        at alluxio.client.file.FileSystemMasterClient.createDirectory(FileSystemMasterClient.java:90)
        at alluxio.client.file.BaseFileSystem.createDirectory(BaseFileSystem.java:79)
        at alluxio.hadoop.AbstractFileSystem.mkdirs(AbstractFileSystem.java:491)
        at alluxio.hadoop.FileSystem.mkdirs(FileSystem.java:25)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1817)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:291)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:254)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:234)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


16/07/13 02:40:22 INFO mapreduce.Job: Counters: 2
        Job Counters
                Total time spent by all maps in occupied slots (ms)=0
                Total time spent by all reduces in occupied slots (ms)=0
userceph@hadoop:~$





Amran Chen於 2016年7月13日星期三 UTC+8上午2時11分41秒寫道：
Hi Gene,
I tried both of version : Alluxio-1.1.0 and Alluxio-1.2.0-RC1

Both of versions can do these :

userceph@hadoop:~/alluxio-1.2.0-RC1$ ./bin/alluxio fs ls /                               24.00B    07-12-2016 23:44:06:961  Directory      /default_tests_files
2.00B     07-13-2016 00:55:24:781  Directory      /wordcount
1.00B     07-13-2016 01:51:17:432  Directory      /s3
userceph@hadoop:~/alluxio-1.2.0-RC1$

userceph@hadoop:~/alluxio-1.2.0-RC1$ ./bin/alluxio fs copyFromLocal LICENSE /LICENSE
Copied LICENSE to /LICENSE
userceph@hadoop:~/alluxio-1.2.0-RC1$

I tried two ways to mounted s3

./bin/alluxio fs mkdir /s3/data

./bin/alluxio fs chown -R userceph /s3/data

First way mounted:
 ./bin/alluxio fs mount "alluxio://192.168.244.146:19998/s3/data/" "s3n://192.168.244.141:7480/my-new-bucket/"

the output : Mount point alluxio://192.168.244.146:19998/s3/data already exists (What does this mean? it's already exists, this is first time execution)

Execute: 
hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount -libjars alluxio-1.2.0-RC1/core/client/target/alluxio-core-client-1.2.0-RC1-jar-with-dependencies.jar alluxio://192.168.244.146:19998/wordcount/myfile alluxio://192.168.244.146:19998/s3/data/result

Output: The output can be printed on alluxio, but I print out in my bucket there is empty.

and I tried to unmouted:
./bin/alluxio fs unmount "alluxio://192.168.244.146:19998/s3/data/"

and then do the second way

Second way mounted:
 ./bin/alluxio fs mount "alluxio://192.168.244.146:19998/s3/data/" "s3n://my-new-bucket/"

it the same way with first one step anymore

the result was the same, in my bucket is empty

any help...



Gene Pang於 2016年7月13日星期三 UTC+8上午12時30分38秒寫道：

Gene Pang

unread,
Jul 13, 2016, 3:40:37 PM7/13/16
to Alluxio Users
Hi,

It looks like you are using two different ips for the Alluxio host. The second one also has a non-standard Alluxio port number. Is that intentional?
Thanks,
Gene
Message has been deleted

Amran Chen

unread,
Jul 14, 2016, 3:47:19 AM7/14/16
to Alluxio Users
I think the status file is NOT_PERSISTED
How to change to be PERSISTED




Amran Chen於 2016年7月14日星期四 UTC+8下午1時30分46秒寫道：
Hello Gene,

I tried to use alluxio IP be:


There was no error, myfile was proceed by hadoop environment and the result it put out in the alluxio environment

my-new-bucketNOT_PERSISTEDNO07-14-2016 13:15:04:21707-14-2016 13:15:04:217
 testNOT_PERSISTEDNO07-14-2016 13:15:04:21707-14-2016 13:15:19:560
 _SUCCESS0.00B512.00MB 100%NOT_PERSISTEDNO07-14-2016 13:15:19:56007-14-2016 13:15:19:588
 part-r-000003408.00B512.00MB 100%NOT_PERSISTEDNO07-14-2016 13:15:18:71807-14-2016 13:15:18:887

But, in my Ceph Bucket, I didn't see any file. The bucket is empty. It was stored in Alluxio environment only.

For alluxio-env.sh configuration :

ALLUXIO_MASTER_HOSTNAME=${ALLUXIO_MASTER_HOSTNAME:-"localhost"}
ALLUXIO_WORKER_MEMORY_SIZE=${ALLUXIO_WORKER_MEMORY_SIZE:-"1322MB"}
ALLUXIO_RAM_FOLDER=${ALLUXIO_RAM_FOLDER:-"/mnt/ramdisk"}
ALLUXIO_UNDERFS_ADDRESS=s3n://my-new-bucket/test/

#ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-"${ALLUXIO_HOME}/underFSStorage/"}

export ALLUXIO_WORKER_MEMORY_SIZE=${ALLUXIO_WORKER_MEMORY_SIZE:-1GB}

export ALLUXIO_JAVA_OPTS+="
  -Dlog4j.configuration=file:${CONF_DIR}/log4j.properties
  -Dalluxio.worker.tieredstore.levels=1
  -Dalluxio.worker.tieredstore.level0.alias=MEM
  -Dalluxio.worker.tieredstore.level0.dirs.path=${ALLUXIO_RAM_FOLDER}
  -Dalluxio.worker.tieredstore.level0.dirs.quota=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.underfs.address=${ALLUXIO_UNDERFS_ADDRESS}
  -Dalluxio.worker.memory.size=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.master.hostname=${ALLUXIO_MASTER_ADDRESS}
  -Dorg.apache.jasper.compiler.disablejsr199=true
  -Djava.net.preferIPv4Stack=true
  -Dfs.s3n.awsAccessKeyId=QINHSA7JD2MBEKN97F4A
  -Dfs.s3n.awsSecretAccessKey=mbjd9H80AokLYFP2Yx44K64Jgy5JdXZIlrWRrcPn
  -Dalluxio.underfs.s3.disable.dns.buckets=true
  -Dalluxio.underfs.s3.endpoint=mon1
  -Dalluxio.underfs.s3.proxy.https.only=false
  -Dalluxio.underfs.s3.endpoint.http.port=7480
"
export ALLUXIO_MASTER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

export ALLUXIO_WORKER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"

Why Alluxio couldn't automatic interactive with Ceph S3 API? The bucket is empty.
But I have try to run alluxio runTests the default_test_files could transfer to Ceph S3 API, this one only for testing..



Gene Pang於 2016年7月14日星期四 UTC+8上午3時40分37秒寫道：

Amran Chen

unread,
Jul 14, 2016, 5:39:22 AM7/14/16
to Alluxio Users
Hello Gene,
After try many times, 
The final error status that I found is here:

16/07/14 17:35:57 INFO mapreduce.Job:  map 0% reduce 0%
16/07/14 17:36:08 INFO mapreduce.Job: Task Id : attempt_1468430146706_0016_m_000000_0, Status : FAILED
Error: java.lang.NullPointerException
        at alluxio.client.block.UnderStoreBlockInStream.read(UnderStoreBlockInStream.java:134)
        at alluxio.client.file.FileInStream.read(FileInStream.java:208)
        at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:185)
        at java.io.DataInputStream.read(DataInputStream.java:100)
        at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
        at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:143)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:183)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
        at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
        at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Is it related with what Gene? Thank you for your good respond..




Amran Chen於 2016年7月14日星期四 UTC+8下午3時47分19秒寫道：

Gene Pang

unread,
Jul 14, 2016, 9:16:59 AM7/14/16
to Alluxio Users
If you want a file to be persisted, there are several ways to do it. You can call the persist command in the shell.


./bin/alluxio fs persist /alluxio/path/to/file

Or, you can change the write type to CACHE_THROUGH when you create the file. http://www.alluxio.org/docs/master/en/File-System-API.html#io-options

Thanks,
Gene

Amran Chen

unread,
Jul 15, 2016, 2:10:09 PM7/15/16
to Alluxio Users
Hello Gene,
I found that in Hadoop MapReduce job failed because of "Cannot complete a file without all the blocks committed" when closing a file

16/07/16 02:03:48 INFO mapreduce.Job: Job job_1468598609668_0010 running in uber mode : false
16/07/16 02:03:48 INFO mapreduce.Job:  map 0% reduce 0%
16/07/16 02:04:00 INFO mapreduce.Job:  map 100% reduce 0%
16/07/16 02:04:21 INFO mapreduce.Job:  map 100% reduce 67%
16/07/16 02:04:44 INFO mapreduce.Job: Task Id : attempt_1468598609668_0010_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException
        at alluxio.client.file.FileOutStream.close(FileOutStream.java:190)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
        at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:111)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:550)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:629)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

16/07/16 02:04:45 INFO mapreduce.Job:  map 100% reduce 0%
16/07/16 02:04:56 INFO mapreduce.Job:  map 100% reduce 67%
^Cuserceph@hadoop:~$


I couldn't add CACHE_THROUGH after jar file in running, they would take error


Gene Pang於 2016年7月14日星期四 UTC+8下午9時16分59秒寫道：

Gene Pang

unread,
Jul 18, 2016, 9:28:47 AM7/18/16
to Alluxio Users
Hi,

How large is the file the map reduce job is trying to write? How much MEM is Alluxio configured with? Which version of Alluxio are you using?

Thanks,
Gene

Gene Pang

unread,
Jul 25, 2016, 11:24:55 AM7/25/16
to Alluxio Users
Hi,

Do you have a simple use case with which we could try to reproduce the error?

Thanks,
Gene

Gene Pang

unread,
Aug 1, 2016, 11:14:41 AM8/1/16
to Alluxio Users
Hi,

Were you able to resolve your issue?

Thanks,
Gene

test520

unread,
Sep 23, 2016, 6:14:03 AM9/23/16
to Alluxio Users
Hi Amran,

I follow your configuration and cannot mount my object storage with  Alluxio 1.2

Could you share how you mount ceph with Alluxio 1.2 ??? 

# vim /conf/alluxio-env.sh

# Hostname of the master.

export ALLUXIO_MASTER_HOSTNAME=test01



# Address of the under filesystem address. (Default: ${ALLUXIO_HOME}/underFSStorage)
export ALLUXIO_UNDERFS_ADDRESS=s3n://testBucket/testDirectory/

# How much memory to use per worker. (Default: 1GB)
export ALLUXIO_WORKER_MEMORY_SIZE=22GB


export ALLUXIO_JAVA_OPTS="$ALLUXIO_JAVA_OPTS -Dalluxio.user.file.readtype.default=CACHE_PROMOTE -Dalluxio.user.file.writetype.default=CACHE_THROUGH -Dalluxio.user.file.write.location.policy.class=alluxio.client.file.policy.RoundRobinPolicy -Dalluxio.user.file.cache.partially.read.block=true -Dalluxio.user.block.size.bytes.default=32MB -Dalluxio.underfs.address=${ALLUXIO_UNDERFS_ADDRESS} -Dfs.s3n.awsAccessKeyId=Q0LOTBB9XV1TAJHUD85Z -Dfs.s3n.awsSecretAccessKey=OI7G1BO7kOAejGhTa9KEC1HuuIUc4xxg0PXQf9RR -Dalluxio.underfs.s3.disable.dns.buckets=true -Dalluxio.underfs.s3.endpoint=http://174.125.203.103 -Dalluxio.underfs.s3.proxy.https.only=false -Dalluxio.underfs.s3.endpoint.http.port=9000"


export ALLUXIO_MASTER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"


export ALLUXIO_WORKER_JAVA_OPTS="${ALLUXIO_JAVA_OPTS}"


export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_71


# ./alluxio format
# ./alluxio-start.sh local

# ./alluxio runTests
runTest Basic CACHE_PROMOTE CACHE_THROUGH
2016-09-23 18:06:14,997 ERROR type (S3UnderFileSystem.java:mkdirsInternal) - Failed to create directory: s3n://testBucket/testDirectory
alluxio.org.jets3t.service.S3ServiceException: Request Error: java.net.UnknownHostException: http PUT '//74.125.203.103:9000/testBucket/testDirectory_%24folder%24'
        at alluxio.org.jets3t.service.S3Service.putObject(S3Service.java:2123)
        at alluxio.underfs.s3.S3UnderFileSystem.mkdirsInternal(S3UnderFileSystem.java:651)
        at alluxio.underfs.s3.S3UnderFileSystem.mkdirs(S3UnderFileSystem.java:309)
        at alluxio.underfs.s3.S3UnderFileSystem.mkdirs(S3UnderFileSystem.java:282)
        at alluxio.underfs.s3.S3UnderFileSystem.mkdirs(S3UnderFileSystem.java:313)
        at alluxio.underfs.s3.S3UnderFileSystem.mkdirs(S3UnderFileSystem.java:282)
        at alluxio.underfs.s3.S3UnderFileSystem.create(S3UnderFileSystem.java:169)
        at alluxio.client.file.FileOutStream.<init>(FileOutStream.java:124)
        at alluxio.client.file.BaseFileSystem.createFile(BaseFileSystem.java:102)
        at alluxio.examples.BasicOperations.writeFile(BasicOperations.java:85)
        at alluxio.examples.BasicOperations.call(BasicOperations.java:72)
        at alluxio.examples.BasicOperations.call(BasicOperations.java:42)
        at alluxio.cli.CliUtils.runExample(CliUtils.java:51)
        at alluxio.cli.TestRunner.runTest(TestRunner.java:148)
        at alluxio.cli.TestRunner.runTests(TestRunner.java:121)
        at alluxio.cli.TestRunner.main(TestRunner.java:98)
Caused by: java.net.UnknownHostException: http
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at java.net.Socket.<init>(Socket.java:434)
        at java.net.Socket.<init>(Socket.java:286)
        at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
        at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$1.doit(ControllerThreadSocketFactory.java:91)
        at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$SocketTask.run(ControllerThreadSocketFactory.java:158)
        at java.lang.Thread.run(Thread.java:745)
2016-09-23 18:06:15,002 ERROR type (CliUtils.java:runExample) - Exception running test: alluxio.examples.BasicOperations@130d63be
java.lang.NullPointerException
        at alluxio.client.file.FileOutStream.write(FileOutStream.java:303)
        at alluxio.client.file.FileOutStream.write(FileOutStream.java:270)
        at alluxio.examples.BasicOperations.writeFile(BasicOperations.java:86)
        at alluxio.examples.BasicOperations.call(BasicOperations.java:72)
        at alluxio.examples.BasicOperations.call(BasicOperations.java:42)
        at alluxio.cli.CliUtils.runExample(CliUtils.java:51)
        at alluxio.cli.TestRunner.runTest(TestRunner.java:148)
        at alluxio.cli.TestRunner.runTests(TestRunner.java:121)
        at alluxio.cli.TestRunner.main(TestRunner.java:98)
Failed the test!



# cat /alluxio-1.2.0/logs/master.log

2016-09-23 17:39:54,567 INFO  logger.type (AbstractMaster.java:start) - BlockMaster: Starting leader master.
2016-09-23 17:39:54,568 INFO  logger.type (JournalWriter.java:completeAllLogs) - Marking all logs as complete.
2016-09-23 17:39:54,570 INFO  logger.type (AbstractMaster.java:start) - BlockMaster: journal checkpoint does not exist, nothing to process.
2016-09-23 17:39:54,570 INFO  logger.type (JournalWriter.java:getCheckpointOutputStream) - Creating tmp checkpoint file: /userap/alluxio-1.2.0/journal/BlockMaster/checkpoint.data.tmp
2016-09-23 17:39:54,570 INFO  logger.type (JournalWriter.java:getCheckpointOutputStream) - Latest journal sequence number: 0 Next journal sequence number: 1
2016-09-23 17:39:54,610 INFO  logger.type (JournalWriter.java:close) - Successfully created tmp checkpoint file: /userap/alluxio-1.2.0/journal/BlockMaster/checkpoint.data.tmp
2016-09-23 17:39:54,610 INFO  logger.type (JournalWriter.java:close) - Renamed checkpoint file /userap/alluxio-1.2.0/journal/BlockMaster/checkpoint.data.tmp to /userap/alluxio-1.2.0/journal/BlockMaster/checkpoint.data
2016-09-23 17:39:54,610 INFO  logger.type (JournalWriter.java:deleteCompletedLogs) - Deleting all completed log files...
2016-09-23 17:39:54,610 INFO  logger.type (JournalWriter.java:deleteCompletedLogs) - Finished deleting all completed log files.
2016-09-23 17:39:54,616 INFO  logger.type (JournalWriter.java:openCurrentLog) - Opened current log file: /userap/alluxio-1.2.0/journal/BlockMaster/log.out
2016-09-23 17:39:54,619 INFO  logger.type (MountTable.java:add) - Mounting s3n://testBucket/testDirectory at /
2016-09-23 17:39:54,619 INFO  logger.type (AbstractMaster.java:start) - FileSystemMaster: Starting leader master.
2016-09-23 17:39:54,620 INFO  logger.type (JournalWriter.java:completeAllLogs) - Marking all logs as complete.
2016-09-23 17:39:54,620 INFO  logger.type (AbstractMaster.java:start) - FileSystemMaster: journal checkpoint does not exist, nothing to process.
2016-09-23 17:39:54,620 INFO  logger.type (JournalWriter.java:getCheckpointOutputStream) - Creating tmp checkpoint file: /userap/alluxio-1.2.0/journal/FileSystemMaster/checkpoint.data.tmp
2016-09-23 17:39:54,621 INFO  logger.type (JournalWriter.java:getCheckpointOutputStream) - Latest journal sequence number: 0 Next journal sequence number: 1
2016-09-23 17:39:54,627 INFO  logger.type (JournalWriter.java:close) - Successfully created tmp checkpoint file: /userap/alluxio-1.2.0/journal/FileSystemMaster/checkpoint.data.tmp
2016-09-23 17:39:54,627 INFO  logger.type (JournalWriter.java:close) - Renamed checkpoint file /userap/alluxio-1.2.0/journal/FileSystemMaster/checkpoint.data.tmp to /userap/alluxio-1.2.0/journal/FileSystemMaster/checkpoint.data
2016-09-23 17:39:54,627 INFO  logger.type (JournalWriter.java:deleteCompletedLogs) - Deleting all completed log files...
2016-09-23 17:39:54,627 INFO  logger.type (JournalWriter.java:deleteCompletedLogs) - Finished deleting all completed log files.
2016-09-23 17:39:54,783 INFO  server.Server (Server.java:doStart) - jetty-7.x.y-SNAPSHOT
2016-09-23 17:39:54,798 INFO  handler.ContextHandler (ContextHandler.java:startContext) - started o.e.j.s.ServletContextHandler{/metrics/json,null}
2016-09-23 17:39:54,866 INFO  handler.ContextHandler (ContextHandler.java:startContext) - started o.e.j.w.WebAppContext{/,file:/userap/alluxio-1.2.0/core/server/src/main/webapp/},/userap/alluxio-1.2.0/core/server/src/main/webapp
2016-09-23 17:40:03,003 INFO  server.AbstractConnector (AbstractConnector.java:doStart) - Started SelectChannelConnector@0.0.0.0:19999
2016-09-23 17:40:03,003 INFO  logger.type (UIWebServer.java:startWebServer) - Alluxio Master Web service started @ 0.0.0.0/0.0.0.0:19999
2016-09-23 17:40:03,004 INFO  logger.type (AlluxioMaster.java:startServing) - Alluxio master version 1.2.0 started @ test01/127.0.0.1:19998
2016-09-23 17:40:03,048 INFO  logger.type (BlockMaster.java:getWorkerId) - getWorkerId(): WorkerNetAddress: WorkerNetAddress{host=test01, rpcPort=29998, dataPort=29999, webPort=30000} id: 1
2016-09-23 17:40:03,058 INFO  logger.type (BlockMaster.java:workerRegister) - registerWorker(): MasterWorkerInfo{id=1, workerAddress=WorkerNetAddress{host=test01, rpcPort=29998, dataPort=29999, webPort=30000}, capacityBytes=23622320128, usedBytes=0, lastUpdatedTimeMs=1474623603058, blocks=[]}




Amran Chen於 2016年7月14日星期四 UTC+8下午3時47分19秒寫道：

Gene Pang

unread,
Sep 26, 2016, 11:39:28 PM9/26/16
to Alluxio Users
Hi,

Could you try changing the parameter: "alluxio.underfs.s3.endpoint" to not have the "http" prefix? Something like: "-Dalluxio.underfs.s3.endpoint=174.125.203.103"

Thanks,
Gene

test520

unread,
Sep 27, 2016, 3:03:37 AM9/27/16
to Alluxio Users
Hi Gene,

I simplify my configuration for using Alluxio 1.2 with S3, and get some error after running Tests.

why there are some test files in Amazon S3 buckets after I run Alluxio Tests if the Tests are failed?

May you share your experience for using Alluxio 1.2 with S3 correctly?

Many thanks.

alluxio-1.2.0 cch$ vim conf/alluxio-site.properties

alluxio.underfs.address=s3n://myBucket/

fs.s3n.awsAccessKeyId=xxxxxxxxxxxx

fs.s3n.awsSecretAccessKey=xxxxxxxxxxxxxx



alluxio-1.2.0 cch$ vim conf/alluxio-env.sh


#!/usr/bin/env bash
#
# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
# (the "License"). You may not use this work except in compliance with the License, which is
# available at www.apache.org/licenses/LICENSE-2.0
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied, as more fully set forth in the License.

#
# See the NOTICE file distributed with this work for information regarding copyright ownership.
#

# This file is sourced to launch Alluxio servers or use Alluxio shell commands
# This file was generated by command "bin/alluxio bootstrapConf localhost local"
# For a list of supported environment variables, see "conf/alluxio-env.sh.template"

ALLUXIO_MASTER_HOSTNAME=${ALLUXIO_MASTER_HOSTNAME:-"localhost"}

ALLUXIO_WORKER_MEMORY_SIZE=${ALLUXIO_WORKER_MEMORY_SIZE:-"10922MB"}

ALLUXIO_RAM_FOLDER=${ALLUXIO_RAM_FOLDER:-"/Volumes/ramdisk"}


#ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-"${ALLUXIO_HOME}/underFSStorage/"}

#export ALLUXIO_UNDERFS_ADDRESS=/tmp



alluxio-1.2.0 cch$ ./bin/alluxio runTests


runTest BasicNonByteBuffer CACHE_PROMOTE MUST_CACHE

 

2016-09-27 14:37:34,110 ERROR type (CliUtils.java:runExample) - Exception running test: alluxio.examples.BasicNonByteBufferOperations@215be6bb

java.io.IOException: Failed to cache: alluxio.exception.BlockAlreadyExistsException: Temp blockId 33,554,432 is not available, because it is already committed

 at alluxio.client.file.FileOutStream.handleCacheWriteException(FileOutStream.java:342)

 at alluxio.client.file.FileOutStream.write(FileOutStream.java:256)

 at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)

 at alluxio.examples.BasicNonByteBufferOperations.write(BasicNonByteBufferOperations.java:85)

 at alluxio.examples.BasicNonByteBufferOperations.call(BasicNonByteBufferOperations.java:78)

 at alluxio.examples.BasicNonByteBufferOperations.call(BasicNonByteBufferOperations.java:46)

 at alluxio.cli.CliUtils.runExample(CliUtils.java:51)

 at alluxio.cli.TestRunner.runTest(TestRunner.java:152)

 at alluxio.cli.TestRunner.runTests(TestRunner.java:121)

 at alluxio.cli.TestRunner.main(TestRunner.java:98)

 Suppressed: java.io.IOException: alluxio.exception.BlockInfoException: Cannot complete a file without all the blocks committed

 at alluxio.client.file.FileOutStream.close(FileOutStream.java:227)

 at java.io.FilterOutputStream.close(FilterOutputStream.java:159)

 at alluxio.examples.BasicNonByteBufferOperations.write(BasicNonByteBufferOperations.java:89)

 ... 6 more

 Caused by: alluxio.exception.BlockInfoException: Cannot complete a file without all the blocks committed

 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

 at java.lang.reflect.Constructor.newInstance(Constructor.java:408)

 at alluxio.exception.AlluxioException.fromThrift(AlluxioException.java:99)

 at alluxio.AbstractClient.retryRPC(AbstractClient.java:326)

 at alluxio.client.file.FileSystemMasterClient.completeFile(FileSystemMasterClient.java:128)

 at alluxio.client.file.FileOutStream.close(FileOutStream.java:225)

 ... 8 more

Caused by: java.io.IOException: alluxio.exception.BlockAlreadyExistsException: Temp blockId 33,554,432 is not available, because it is already committed

 at alluxio.client.block.BlockWorkerClient.requestBlockLocation(BlockWorkerClient.java:358)

 at alluxio.client.block.LocalBlockOutStream.<init>(LocalBlockOutStream.java:64)

 at alluxio.client.block.AlluxioBlockStore.getOutStream(AlluxioBlockStore.java:170)

 at alluxio.client.file.FileOutStream.getNextBlock(FileOutStream.java:320)

 at alluxio.client.file.FileOutStream.write(FileOutStream.java:252)

 ... 8 more

Caused by: alluxio.exception.BlockAlreadyExistsException: Temp blockId 33,554,432 is not available, because it is already committed

 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

 at java.lang.reflect.Constructor.newInstance(Constructor.java:408)

 at alluxio.exception.AlluxioException.fromThrift(AlluxioException.java:99)

 at alluxio.AbstractClient.retryRPC(AbstractClient.java:326)

 at alluxio.client.block.BlockWorkerClient.requestBlockLocation(BlockWorkerClient.java:348)

 ... 12 more

Failed the test!





Gene Pang於 2016年9月27日星期二 UTC+8上午11時39分28秒寫道：

Gene Pang

unread,
Oct 10, 2016, 10:02:19 AM10/10/16
to Alluxio Users
Hi,

Could you try restarting and formatting Alluxio and starting from a fresh state?

Thanks,
Gene

Gene Pang

unread,
Oct 18, 2016, 6:16:40 PM10/18/16
to Alluxio Users
Hi,

Do you have any updates on this issue?

Thanks,
Gene

test520

unread,
Oct 23, 2016, 1:12:55 AM10/23/16
to Alluxio Users
Hi Gene,


 Thanks for your suggestions! :D

Gene Pang於 2016年10月19日星期三 UTC+8上午6時16分40秒寫道：

Gene Pang

unread,
Oct 24, 2016, 9:10:05 AM10/24/16
to Alluxio Users
Thanks for confirming!
Reply all
Reply to author
Forward
0 new messages