AccessControlException: ould not setMode for UFS file

181 views
Skip to first unread message

shuy...@gmail.com

unread,
Aug 10, 2016, 10:15:00 PM8/10/16
to Alluxio Users
Hi,

Hadoop version: 2.6.0(HDP version: 2.2.0)
Alluxio: 1.2.0
JDK: 1.8.0

When I run mapreduce through Alluxio, I got exceptons as belowing:

16/08/11 10:07:35 INFO input.FileInputFormat: Total input paths to process : 1
16/08/11 10:07:35 INFO logger.type: create(/user/hduser/.staging/job_1470024122519_5219/job.split, rw-r--r--, true, 131072, 1, 268435456, null)
16/08/11 10:07:35 INFO logger.type: setMode(/user/hduser/.staging/job_1470024122519_5219/job.split,rw-r--r--)
16/08/11 10:07:35 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/hduser/.staging/job_1470024122519_5219
16/08/11 10:07:35 INFO logger.type: delete(/user/hduser/.staging/job_1470024122519_5219, true)
Exception in thread "main" java.io.IOException: alluxio.exception.AccessControlException: Could not setMode for UFS file hdfs://hstore/user/hduser/.staging/job_1470024122519_5219/job.split
at alluxio.hadoop.AbstractFileSystem.setPermission(AbstractFileSystem.java:355)
at alluxio.hadoop.FileSystem.setPermission(FileSystem.java:25)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:578)
at org.apache.hadoop.mapreduce.split.JobSplitWriter.createFile(JobSplitWriter.java:101)
at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:77)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:603)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
at com.antfact.laundry.shuqi.job.WebExporter.run(WebExporter.java:84)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.antfact.laundry.shuqi.job.WebExporter.main(WebExporter.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: alluxio.exception.AccessControlException: Could not setMode for UFS file hdfs://hstore/user/hduser/.staging/job_1470024122519_5219/job.split
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at alluxio.exception.AlluxioException.fromThrift(AlluxioException.java:99)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:326)
at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:299)
at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:298)
at alluxio.hadoop.AbstractFileSystem.setPermission(AbstractFileSystem.java:353)
... 23 more
16/08/11 10:07:35 ERROR hdfs.DFSClient: Failed to close inode 22041272
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hduser/.staging/job_1470024122519_5219/job.split.alluxio.0x1F18898819E4516E.tmp (inode 22041272): File does not exist. Holder DFSClient_NONMAPREDUCE_703010429_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3798)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3886)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3856)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:725)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:528)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

at org.apache.hadoop.ipc.Client.call(Client.java:1469)
at org.apache.hadoop.ipc.Client.call(Client.java:1400)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy28.complete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:459)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy29.complete(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2251)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2235)
at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:938)
at org.apache.hadoop.hdfs.DFSClient.closeOutputStreams(DFSClient.java:976)
at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:917)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2710)
at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2727)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

Thanks.

Chaomin Yu

unread,
Aug 10, 2016, 10:29:51 PM8/10/16
to shuy...@gmail.com, Alluxio Users
Hi Shuyun,

In general, "Could not setMode for UFS file " Exception happens when the setPermission op to HDFS file failed.

Let me see if I can help you troubleshoot the problem. 
Firstly, can you please check whether the file to setPermission still exists? If the file was somehow gone/deleted, setPermission could fail.
If it still exists, can you try to manually set permission to that HDFS file, with the same UserGroupInformation as you were using via Alluxio? It's possibly due to some permission denied errors for some user to setPermission on HDFS files.

Hope this helps,
Chaomin 

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

shuy...@gmail.com

unread,
Aug 10, 2016, 11:39:44 PM8/10/16
to Alluxio Users
Hi,

I checked the log when committing a mapreduce job and alluxio master, here is the thing:

1. when committing job, the job dir under ".staging" is created and all the jars are copied









2. But the job dir is "clean up", I don't know why ,it should be deleted after the job is done. When I run the same job without Alluxio, everything is fine , the job dir exists until the job is done.

3. Alluxio master log got exceptions as belowing:










Thanks.


在 2016年8月11日星期四 UTC+8上午10:15:00，shuy...@gmail.com写道：

Chaomin Yu

unread,
Aug 11, 2016, 1:47:35 PM8/11/16
to shuy...@gmail.com, Alluxio Users
Hi,

Thanks for the more detailed logs. It's confirmed that the previous setPermission error was due to missing file.
Looks like the ".staging" files were deleted by mapreduce.JobSubmitter, maybe not via Alluxio?

What is your Mapreduce and HDFS configuration and setup, related with Alluxio?
You might find this Running-Hadoop-MapReduce-on-Alluxio doc helpful.
One thing to notice in particular: you should make sure the Alluxio client jar is distributed to all the nodes in the cluster and added to "HADOOP_CLASSPATH". 

Hope this helps,
Chaomin


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

shuy...@gmail.com

unread,
Aug 17, 2016, 4:29:18 AM8/17/16
to Alluxio Users, shuy...@gmail.com
HI,

After I kept tracing in "JobSubmitter" and "JobSplitWriter", I found that job's failure was caused by creating job.split file, due to this file needs to set permission mode to 644, and then threw AccessControlException as Follows.

16/08/17 16:04:19 INFO split.JobSplitWriter: ***************alluxio://localhost:19998/user/hduser/.staging/job_1470024122519_8114, owner: hduser, group: hduser, permission: rwx------
16/08/17 16:04:19 INFO split.JobSplitWriter: ***************target permission of job.split: rw-r--r--
16/08/17 16:04:19 INFO logger.type: create(/user/hduser/.staging/job_1470024122519_8114/job.split, rw-r--r--, true, 131072, 1, 268435456, null)
16/08/17 16:04:19 INFO logger.type: setMode(/user/hduser/.staging/job_1470024122519_8114/job.split,rw-r--r--)
16/08/17 16:04:19 ERROR mapreduce.JobSubmitter: Error when writing splits
java.io.IOException: alluxio.exception.AccessControlException: Could not setMode for UFS file hdfs://hstore/user/hduser/.staging/job_1470024122519_8114/job.split
        at alluxio.hadoop.AbstractFileSystem.setPermission(AbstractFileSystem.java:355)
        at alluxio.hadoop.FileSystem.setPermission(FileSystem.java:25)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:578)
        at org.apache.hadoop.mapreduce.split.JobSplitWriter.createFile(JobSplitWriter.java:107)
        at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:78)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:650)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:661)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:514)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at com.antfact.laundry.shuqi.job.WebExporter.run(WebExporter.java:84)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at com.antfact.laundry.shuqi.job.WebExporter.main(WebExporter.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: alluxio.exception.AccessControlException: Could not setMode for UFS file hdfs://hstore/user/hduser/.staging/job_1470024122519_8114/job.split
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at alluxio.exception.AlluxioException.fromThrift(AlluxioException.java:99)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:326)
        at alluxio.client.file.FileSystemMasterClient.setAttribute(FileSystemMasterClient.java:299)
        at alluxio.client.file.BaseFileSystem.setAttribute(BaseFileSystem.java:298)
        at alluxio.hadoop.AbstractFileSystem.setPermission(AbstractFileSystem.java:353)
        ... 23 more

So the cause is permission problem, my hdfs cluster was enabled "dfs.permissions.enabled" and set "hadoop.security.authentication=simple", and alluxio is running under linux user "hduser" and its' security properties :

alluxio.security.authentication.type=SIMPLE
alluxio.security.authorization.permission.enabled=true
alluxio.security.authorization.permission.supergroup=hdfs

1st question: Can Alluxio work with hdfs with permission enabled and how?
2nd question: How can I fix this problem?

Thanks

在 2016年8月12日星期五 UTC+8上午1:47:35，Chaomin Yu写道：
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Chaomin Yu

unread,
Aug 17, 2016, 7:02:58 PM8/17/16
to shuy...@gmail.com, Alluxio Users
Hi,

Thanks for the detailed information.

To answer your questions:
1. Alluxio works with HDFS with permission enabled, for basic HDFS operations. Alluxio inherits HDFS under storage owner/group/mode to Alluxio namespace, and propagates any chown/chgrp/chmod op to underlying HDFS. There are some limitations for the permission sync between Alluxio and UFS, such as any permission change directly on HDFS would not be reflected to the mounted Alluxio namespace.

2. Taking a closer into the log and the code of JobSplitWriter, the root cause of the setMode failure is due to an issue in Alluxio permission propagation logic to the mapped HDFS file, when the Alluxio file has not been completed yet.

https://alluxio.atlassian.net/browse/ALLUXIO-2198 is created to track this issue, with more details explained.
I have a quick work-around here: https://github.com/Alluxio/alluxio/pull/3833

Could you please patch it and recompile your Alluxio binary, to see whether this fixes your setMode problem? 

Thanks!
Chaomin

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

shuy...@gmail.com

unread,
Aug 18, 2016, 1:17:46 AM8/18/16
to Alluxio Users, shuy...@gmail.com
Hi,

Thanks for the patch, after I patched in my Alluxio cluster, the permission problem is gone.

But new exception occurred:

16/08/18 13:11:41 INFO impl.YarnClientImpl: Submitted application application_1470024122519_8543
16/08/18 13:11:41 INFO mapreduce.Job: The url to track the job: http://store33.antfact.com:8088/proxy/application_1470024122519_8543/
16/08/18 13:11:41 INFO mapreduce.Job: Running job: job_1470024122519_8543
16/08/18 13:11:45 INFO mapreduce.Job: Job job_1470024122519_8543 running in uber mode : false
16/08/18 13:11:45 INFO mapreduce.Job:  map 0% reduce 0%
16/08/18 13:11:45 INFO mapreduce.Job: Job job_1470024122519_8543 failed with state FAILED due to: Application application_1470024122519_8543 failed 2 times due to AM Container for appattempt_1470024122519_8543_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://store33.antfact.com:8088/proxy/application_1470024122519_8543/Then, click on links to logs of each attempt.
Diagnostics: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2076)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2578)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:249)
        at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2074)
        ... 22 more
Caused by: Class alluxio.hadoop.FileSystem not found
java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2074)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2578)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:249)
        at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Failing this attempt. Failing the application.

I have the 3 properties added to core-site.xml and hdfs-site.xml already.













I also use libjars for the dependencies, and the jar does exist in hdfs.
Do I miss any step, Thanks.






在 2016年8月18日星期四 UTC+8上午7:02:58，Chaomin Yu写道：

Chaomin Yu

unread,
Aug 18, 2016, 1:15:42 PM8/18/16
to shuy...@gmail.com, Alluxio Users
Hi,

Seems like the dependency jar path references to a HDFS path?
Can you please try to put the "alluxio-core-client-1.2.0-jar-with-dependencies.jar" on the local file system, instead of HDFS? 

Also, when you re-compile Alluxio, please make sure you have the "-Dhadoop.version=" specified.

Hope this helps,
Chaomin

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

shuy...@gmail.com

unread,
Aug 19, 2016, 1:26:06 AM8/19/16
to Alluxio Users, shuy...@gmail.com
Hi,

I use "-Dhadoop.version=2.6.0" when building Alluxio and distributed cache in hadoop to resolve the dependency jar. The jars of the job was all uploaded to hdfs already. I think that when yarn start AM and needs to download the jars from hdfs to local, it may also through Alluxio API, but now the alluxio jar was still on hdfs, not locally, the the class were not found.

The exception is belowing, please note the part of greeen:

16/08/19 12:58:11 INFO impl.YarnClientImpl: Submitted application application_1471510548319_0461
16/08/19 12:58:11 INFO mapreduce.Job: The url to track the job: http://store33.antfact.com:8088/proxy/application_1471510548319_0461/
16/08/19 12:58:11 INFO mapreduce.Job: Running job: job_1471510548319_0461
16/08/19 12:58:18 INFO mapreduce.Job: Job job_1471510548319_0461 running in uber mode : false
16/08/19 12:58:18 INFO mapreduce.Job:  map 0% reduce 0%
16/08/19 12:58:18 INFO mapreduce.Job: Job job_1471510548319_0461 failed with state FAILED due to: Application application_1471510548319_0461 failed 2 times due to AM Container for appattempt_1471510548319_0461_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://store33.antfact.com:8088/proxy/application_1471510548319_0461/Then, click on links to logs of each attempt.

Here is the code of FSDownload.java, it get the FileSystem through Path, is it possible that the path is with "alluxio://" prefix, so it find Alluxio FileSystem to download the jars.












Thanks.


在 2016年8月19日星期五 UTC+8上午1:15:42，Chaomin Yu写道：

Chaomin Yu

unread,
Aug 19, 2016, 1:40:31 PM8/19/16
to shuy...@gmail.com, Alluxio Users
Hi,

I'm not sure whether YARN would be able to download the jars from hdfs to some local path that is recognized by HADOOP_CLASSPATH ?

To verify this, per the doc here, can you please distribute the Alluxio client jar manually on all the nodes, or use "-libjars" specifying /<PATH_TO_ALLUXIO>/core/client/target/alluxio-core-client-1.2.0-jar-with-dependencies.jar as the argument?
Please make sure <PATH_TO_ALLUXIO> is a local path, not a HDFS path.

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Message has been deleted
Message has been deleted
Message has been deleted

Chaomin Yu

unread,
Aug 24, 2016, 2:34:33 PM8/24/16
to shuy...@gmail.com, Alluxio Users
Hi,

Thank you for providing the detailed command. The way you set Alluxio client jar using -libjars looks right to me.

Seems like the issue is due to some Hadoop configuration that you should set to make it recognize Alluxio FS.
Just to double check, as you've already pasted your "core-site.xml", do you have the "fs.alluxio.impl" property set to "alluxio.hadoop.FileSystem" ?

Also, a dumb question: when you re-compiled Alluxio, did the compilation finish successfully?


On Sun, Aug 21, 2016 at 8:11 PM, <shuy...@gmail.com> wrote:
Hi,

I found that the error may cause by command options, now the command I used to submit mapreduce job as follows:

yarn jar shuqi-1.0-SNAPSHOT.jar com.antfact.laundry.shuqi.job.WebExporter -D alluxio.security.authentication.type=SIMPLE -D alluxio.user.block.size.bytes.default=256MB -D alluxio.user.file.writetype.default=CACHE_THROUGH -libjars hstore-common-1.0.1.jar,hive-exec-eefung-2.0.1.jar,weibo-common-1.3.3.jar,hstore-batch-1.0.1.jar,alluxio-core-client-1.2.0-jar-with-dependencies.jar,ahocorasick-0.3.0.jar alluxio://172.19.121.51:19998/crawled/webDocument/20160809120147_0 alluxio://172.19.121.51:19998/eefung/shuqi/test 1 &

And the command will fail due to submit job:

16/08/22 11:00:29 INFO logger.type: initialize(alluxio://172.19.121.51:19998/crawled/webDocument/20160809120147_0, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Alluxio: alluxio://172.19.121.51:19998/crawled/webDocument/20160809120147_0
16/08/22 11:00:29 INFO logger.type: alluxio://172.19.121.51:19998 alluxio://172.19.121.51:19998
16/08/22 11:00:29 INFO logger.type: Loading Alluxio properties from Hadoop configuration: {alluxio.security.authentication.type=SIMPLE, alluxio.user.block.size.bytes.default=256MB, alluxio.user.file.writetype.default=CACHE_THROUGH}
16/08/22 11:00:29 INFO logger.type: getWorkingDirectory: /
16/08/22 11:00:29 INFO logger.type: getWorkingDirectory: /
16/08/22 11:00:30 INFO impl.TimelineClientImpl: Timeline service address: http://store108.antfact.com:8188/ws/v1/timeline/
16/08/22 11:00:31 INFO logger.type: getFileStatus(alluxio://172.19.121.51:19998/eefung/shuqi/test)
16/08/22 11:00:31 INFO logger.type: Alluxio client (version 1.2.0) is trying to connect with FileSystemMasterClient master @ /172.19.121.51:19998
16/08/22 11:00:31 INFO logger.type: Client registered with FileSystemMasterClient master @ /172.19.121.51:19998
16/08/22 11:00:33 INFO logger.type: getFileStatus(alluxio://172.19.121.51:19998/crawled/webDocument/20160809120147_0)
16/08/22 11:00:33 ERROR input.InputPathFilter: Error when call isFile
java.lang.IllegalArgumentException: Wrong FS: alluxio://172.19.121.51:19998/crawled/webDocument/20160809120147_0, expected: hdfs://hstore
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:193)
        at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:105)
        at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1136)
        at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1132)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1132)
        at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1449)
        at com.antfact.hstore.batch.mr.input.InputPathFilter.accept(InputPathFilter.java:57)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat$MultiPathFilter.accept(FileInputFormat.java:110)
        at org.apache.hadoop.fs.Globber.glob(Globber.java:270)
        at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1667)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:292)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:264)
        at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:385)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:597)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at com.antfact.laundry.shuqi.job.WebExporter.run(WebExporter.java:84)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at com.antfact.laundry.shuqi.job.WebExporter.main(WebExporter.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
16/08/22 11:00:33 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/hduser/.staging/job_1471510548319_1784
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: alluxio://172.19.121.51:19998/crawled/webDocument/20160809120147_0
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:321)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:264)
        at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:385)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:597)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at com.antfact.laundry.shuqi.job.WebExporter.run(WebExporter.java:84)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at com.antfact.laundry.shuqi.job.WebExporter.main(WebExporter.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

The command I used before is plus another params: -D fs.defaultFS=alluxio://172.19.121.51:19998，with this param the job can be submitted to yarn, but got CalssNotFound Exception as mentioned above.

Could you give me some suggestions about correct command options, thanks.

在 2016年8月22日星期一 UTC+8上午9:47:04，shuy...@gmail.com写道：
Hi,

I exactly used local path for alluxio jar when using "libjars", so that's why the jars can be uploaded to hdfs dir [.staging]。

Thanks。

在 2016年8月20日星期六 UTC+8上午1:40:31，Chaomin Yu写道：

Chaomin Yu

unread,
Sep 12, 2016, 11:30:31 AM9/12/16
to shuy...@gmail.com, Alluxio Users
Hi,

Were you able to resolve the configuration setup?

Best,
Chaomin
Reply all
Reply to author
Forward
0 new messages