java.lang.IllegalArgumentException: Wrong FS: alluxio://ip, expected: hdfs://ip

[Sambhu R]

Hello guys,

Could you please help me with this....

I can run wordcount on alluxio with an example jar provided by cloudera, using :  sudo -u hdfs hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar wordcount -libjars /home/nn1/alluxio-1.2.0/core/client/target/alluxio-core-client-1.2.0-jar-with-dependencies.jar alluxio://nn1:19998/wordcount alluxio://nn1:19998/wc1

and its success
16/12/12 16:03:29 INFO mapreduce.Job: Job job_1480580864224_0014 running in uber mode : false
16/12/12 16:03:29 INFO mapreduce.Job:  map 0% reduce 0%
16/12/12 16:03:35 INFO mapreduce.Job:  map 100% reduce 0%
16/12/12 16:03:40 INFO mapreduce.Job:  map 100% reduce 100%
16/12/12 16:03:41 INFO mapreduce.Job: Job job_1480580864224_0014 completed successfully..


But I can't when i use the jar created with the ATTACHED CODE,  cmd:  sudo -u hdfs hadoop jar /home/nn1/HadoopWordCount-0.0.1-SNAPSHOT-jar-with-dependencies.jar edu.am.bigdata.C45TreeModel.C45DecisionDriver -libjars /home/nn1/alluxio-1.2.0/core/client/target/alluxio-core-client-1.2.0-jar-with-dependencies.jar alluxio://10.30.60.45:19998/abdf alluxio://10.30.60.45:19998/outabdf


Above code is build using maven
Pom.xml file contains
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>2.6.0-mr1-cdh5.4.5</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.6.0-cdh5.4.5</version>
</dependency>
 
Could you please help me in running my wordcount program in alluxio cluster. Hope no extra configurations are added into pom file for running the same.
 
I am getting the following error after running my jar :

java.lang.IllegalArgumentException: Wrong FS: alluxio://10.30.60.45:19998/outabdf, expected: hdfs://10.30.60.45:8020
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
    at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
    at edu.WordCount.run(WordCount.java:47)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at edu.WordCount.main(WordCount.java:23)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)


OS: Ubuntu Trusty 14.04
Hadoop : Cloudera with Yarn



Thank You.....

[Bin Fan]

Hi Sambhu,

don't know if this thread helps

https://groups.google.com/forum/#!topic/alluxio-users/fYqiTlwQ5g4

Also, it looks like you are running CDH

could you try and follow instruction here to ensure Hadoop related Alluxio configuration is correctly set up

https://www.alluxio.com/docs/community/1.3/en/Running-CDH-Compute-On-Alluxio.html

[Sambhu R]

Hi Bin,

I did like what you suggested but the I'm still getting the same error...

Thank You....

[Sambhu R]

Do i need to be more specific do you want to see any config files ??
Thank You...

[Sambhu R]

Hello BIN,

error: java.lang.IllegalArgumentException: Wrong FS: alluxio://10.30.60.45:19998/abdfout/C4.5_Mean, expected: hdfs://10.30.60.45:8020

I ran the program, i used HDFS instead of ALLUXIO  like this:

 sudo -u hdfs hadoop jar /home/nn1/ABDFMRAlgos.jar edu.am.bigdata.C45TreeModel.C45DecisionDriver -libjars /home/nn1/alluxio-1.2.0/core/client/target/alluxio-core-client-1.2.0-jar-with-dependencies.jar  alluxio://10.30.60.45:19998/abdf hdfs://10.30.60.45:8020/abdfout , 9 1,8 Col0,Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8,Col9 k C45model

But there is not much difference in TIME

*****
without alluxio: sudo -u hdfs hadoop jar /home/nn1/ABDFMRAlgos_TachyonTest.jar  edu.am.bigdata.C45TreeModel.C45DecisionDriver /ABDFin /ABDFout , 9 1,8 Col0,Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8,Col9 k C45model

Time Taken:  3hrs 30min

with Alluxio: sudo -u hdfs hadoop jar /home/nn1/ABDFMRAlgos_TachyonTest.jar edu.am.bigdata.C45TreeModel.C45DecisionDriver -libjars /home/nn1/alluxio-1.2.0/core/client/target/alluxio-core-client-1.2.0-jar-with-dependencies.jar alluxio://10.30.60.45:19998/abdf hdfs://10.30.60.45:8020/abdfOUTTT  , 9 1,8 Col0,Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8,Col9 k C45model

time taken: 3hrs 27mins
*****

please help, Why am i Not seeing any difference in time??

On Thursday, December 15, 2016 at 11:20:49 AM UTC+5:30, Bin Fan wrote:

[Haoyuan Li]

Hi Sambhu, 

Seems you have some setting issue. Did you try this successfully? http://www.alluxio.org/docs/1.2/en/Running-Hadoop-MapReduce-on-Alluxio.html

Suggest you to use Alluxio 1.3 instead of 1.2. 

Two blogs you may have interests: 

https://alluxio.com/blog/effective-spark-dataframes-with-alluxio

https://alluxio.com/blog/effective-spark-rdds-with-alluxio

Best regards,

Haoyuan

Alluxio, Inc. 

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sambhu R]

HI Li,

Thanks for the reply i tried to upgrade alluxio to version 1.3 like you suggested but while running wordcount I'm getting an error like this:

 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:SIMPLE) cause:java.io.IOException: Frame size (67108864) larger than max length (16777216)!
java.io.IOException: Frame size (67108864) larger than max length (16777216)!
    at alluxio.AbstractClient.checkVersion(AbstractClient.java:112)
    at alluxio.AbstractClient.connect(AbstractClient.java:175)
    at alluxio.AbstractClient.retryRPC(AbstractClient.java:322)
    at alluxio.client.file.FileSystemMasterClient.getStatus(FileSystemMasterClient.java:183)
    at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:175)
    at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:167)
    at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:293)
    at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:25)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:270)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)

    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Please Help and Thanks :)

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

[Haoyuan Li]

Seems that your Alluxio client side and Alluxio server sides are in versions. Please double check.

Best regards,

Haoyuan

Alluxio, Inc. 

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.

[Bin Fan]

Hi 

This error is typically seen when the security setting mismatches between client and server side.

See explanation and solution here:

http://www.alluxio.org/docs/master/en/Debugging-Guide.html#q-im-seeing-error-messages-like-frame-size-67108864-larger-than-max-length-16777216-what-is-wrong

Bin

[Sambhu R]

Hi Bin,  Where am I supposed to make these changes,  is that in alluxio-properties file?  If so, I haven't made any changes in that file but still why am I getting this error?  I didn't face this error in alluxio version 1.2 so its kind of wierd one.....or i suppose that the newer version is using a different security feature?

Please walk me through..

 Also Kindly find my conf folder from attachments....

Thank You..

[Bin Fan]

Hi Sambhu,

is this problem deprecated by what you reported in https://groups.google.com/forum/#!topic/alluxio-users/L0XAQZfwjLA?

- Bin

[Sambhu R]

Hi Bin,

There I was using alluxio 1.2 version, but 1.3v is having security features enabled so i couldn't run until now. I didn't look into that much. And also facing the same problem  when I am running the job inside alluxio environment.

here is the full output of the command   sudo hadoop jar /home/nn1/ABDFMRAlgos_TachyonTest.jar  edu.am.bigdata.C45TreeModel.C45DecisionDriver -libjars /home/nn1/alluxio-1.2.0/core/client/target/alluxio-core-client-1.2.0-jar-with-dependencies.jar alluxio://10.30.60.45:19998/abdf alluxio://10.30.60.45:19998/abdfalluxioFS  , 9 1,8 Col0,Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8,Col9 k C45model


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/nn1/alluxio-1.2.0/core/client/target/alluxio-core-client-1.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/01/06 10:18:15 INFO logger.type: initialize(alluxio://10.30.60.45:19998/abdf, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Alluxio: alluxio://10.30.60.45:19998/abdf
17/01/06 10:18:15 INFO logger.type: alluxio://10.30.60.45:19998 alluxio://10.30.60.45:19998
17/01/06 10:18:15 INFO logger.type: Loading Alluxio properties from Hadoop configuration: {}
17/01/06 10:18:16 INFO logger.type: getWorkingDirectory: /
17/01/06 10:18:16 INFO client.RMProxy: Connecting to ResourceManager at /10.30.60.45:8032
17/01/06 10:18:16 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/01/06 10:18:17 INFO logger.type: getFileStatus(alluxio://10.30.60.45:19998/abdf)
17/01/06 10:18:17 INFO logger.type: Alluxio client (version 1.2.0) is trying to connect with FileSystemMasterClient master @ /10.30.60.45:19998
17/01/06 10:18:17 INFO logger.type: Client registered with FileSystemMasterClient master @ /10.30.60.45:19998
17/01/06 10:18:17 INFO input.FileInputFormat: Total input paths to process : 1
17/01/06 10:18:17 INFO mapreduce.JobSubmitter: number of splits:2
17/01/06 10:18:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1483165765526_0030
17/01/06 10:18:17 INFO impl.YarnClientImpl: Submitted application application_1483165765526_0030
17/01/06 10:18:17 INFO mapreduce.Job: The url to track the job: http://nn1:8088/proxy/application_1483165765526_0030/
17/01/06 10:18:17 INFO mapreduce.Job: Running job: job_1483165765526_0030
17/01/06 10:18:23 INFO mapreduce.Job: Job job_1483165765526_0030 running in uber mode : false
17/01/06 10:18:23 INFO mapreduce.Job:  map 0% reduce 0%
17/01/06 10:18:34 INFO mapreduce.Job:  map 3% reduce 0%
17/01/06 10:18:35 INFO mapreduce.Job:  map 6% reduce 0%
17/01/06 10:18:37 INFO mapreduce.Job:  map 7% reduce 0%
17/01/06 10:18:38 INFO mapreduce.Job:  map 9% reduce 0%
17/01/06 10:18:40 INFO mapreduce.Job:  map 11% reduce 0%
17/01/06 10:18:41 INFO mapreduce.Job:  map 13% reduce 0%
17/01/06 10:18:43 INFO mapreduce.Job:  map 15% reduce 0%
17/01/06 10:18:44 INFO mapreduce.Job:  map 16% reduce 0%
17/01/06 10:18:46 INFO mapreduce.Job:  map 18% reduce 0%
17/01/06 10:18:47 INFO mapreduce.Job:  map 19% reduce 0%
17/01/06 10:18:49 INFO mapreduce.Job:  map 21% reduce 0%
17/01/06 10:18:50 INFO mapreduce.Job:  map 23% reduce 0%
17/01/06 10:18:52 INFO mapreduce.Job:  map 25% reduce 0%
17/01/06 10:18:53 INFO mapreduce.Job:  map 27% reduce 0%
17/01/06 10:18:55 INFO mapreduce.Job:  map 28% reduce 0%
17/01/06 10:18:56 INFO mapreduce.Job:  map 30% reduce 0%
17/01/06 10:18:58 INFO mapreduce.Job:  map 31% reduce 0%
17/01/06 10:18:59 INFO mapreduce.Job:  map 33% reduce 0%
17/01/06 10:19:01 INFO mapreduce.Job:  map 34% reduce 0%
17/01/06 10:19:02 INFO mapreduce.Job:  map 36% reduce 0%
17/01/06 10:19:04 INFO mapreduce.Job:  map 37% reduce 0%
17/01/06 10:19:05 INFO mapreduce.Job:  map 38% reduce 0%
17/01/06 10:19:07 INFO mapreduce.Job:  map 40% reduce 0%
17/01/06 10:19:08 INFO mapreduce.Job:  map 42% reduce 0%
17/01/06 10:19:10 INFO mapreduce.Job:  map 44% reduce 0%
17/01/06 10:19:11 INFO mapreduce.Job:  map 45% reduce 0%
17/01/06 10:19:13 INFO mapreduce.Job:  map 47% reduce 0%
17/01/06 10:19:14 INFO mapreduce.Job:  map 49% reduce 0%
17/01/06 10:19:16 INFO mapreduce.Job:  map 51% reduce 0%
17/01/06 10:19:17 INFO mapreduce.Job:  map 52% reduce 0%
17/01/06 10:19:19 INFO mapreduce.Job:  map 54% reduce 0%
17/01/06 10:19:20 INFO mapreduce.Job:  map 55% reduce 0%
17/01/06 10:19:22 INFO mapreduce.Job:  map 56% reduce 0%
17/01/06 10:19:23 INFO mapreduce.Job:  map 57% reduce 0%
17/01/06 10:19:25 INFO mapreduce.Job:  map 59% reduce 0%
17/01/06 10:19:26 INFO mapreduce.Job:  map 60% reduce 0%
17/01/06 10:19:28 INFO mapreduce.Job:  map 61% reduce 0%
17/01/06 10:19:29 INFO mapreduce.Job:  map 62% reduce 0%
17/01/06 10:19:31 INFO mapreduce.Job:  map 64% reduce 0%
17/01/06 10:19:32 INFO mapreduce.Job:  map 82% reduce 0%
17/01/06 10:19:33 INFO mapreduce.Job:  map 100% reduce 0%
17/01/06 10:19:38 INFO mapreduce.Job:  map 100% reduce 100%
17/01/06 10:19:38 INFO mapreduce.Job: Job job_1483165765526_0030 completed successfully
17/01/06 10:19:38 INFO mapreduce.Job: Counters: 54
    File System Counters
        ALLUXIO: Number of bytes read=1094003158
        ALLUXIO: Number of bytes written=0
        ALLUXIO: Number of read operations=2
        ALLUXIO: Number of large read operations=0
        ALLUXIO: Number of write operations=0
        FILE: Number of bytes read=1066
        FILE: Number of bytes written=358588
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=194
        HDFS: Number of bytes written=558
        HDFS: Number of read operations=7
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters
        Launched map tasks=2
        Launched reduce tasks=1
        Data-local map tasks=2
        Total time spent by all maps in occupied slots (ms)=268856
        Total time spent by all reduces in occupied slots (ms)=5728
        Total time spent by all map tasks (ms)=134428
        Total time spent by all reduce tasks (ms)=2864
        Total vcore-seconds taken by all map tasks=134428
        Total vcore-seconds taken by all reduce tasks=2864
        Total megabyte-seconds taken by all map tasks=137654272
        Total megabyte-seconds taken by all reduce tasks=2932736
    Map-Reduce Framework
        Map input records=37198200
        Map output records=106
        Map output bytes=848
        Map output materialized bytes=1072
        Input split bytes=194
        Combine input records=0
        Combine output records=0
        Reduce input groups=9
        Reduce shuffle bytes=1072
        Reduce input records=106
        Reduce output records=1
        Spilled Records=212
        Shuffled Maps =2
        Failed Shuffles=0
        Merged Map outputs=2
        GC time elapsed (ms)=2016
        CPU time spent (ms)=133110
        Physical memory (bytes) snapshot=1261678592
        Virtual memory (bytes) snapshot=4022956032
        Total committed heap usage (bytes)=1146028032
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters
        Bytes Read=1094003158
    File Output Format Counters
        Bytes Written=558
17/01/06 10:19:38 INFO C45TreeModel.C45DecisionDriver: Numeric index
17/01/06 10:19:38 INFO C45TreeModel.C45DecisionDriver: Finding mean
java.lang.IllegalArgumentException: Wrong FS: alluxio://10.30.60.45:19998/abdfalluxioFS/C4.5_Mean, expected: hdfs://10.30.60.45:8020

    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
    at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)

    at edu.am.bigdata.C45TreeModel.C45DecisionDriver.getMean(C45DecisionDriver.java:984)
    at edu.am.bigdata.C45TreeModel.C45DecisionDriver.run(C45DecisionDriver.java:277)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at edu.am.bigdata.C45TreeModel.C45DecisionDriver.main(C45DecisionDriver.java:89)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Please Help,
Thank You.

[Bin Fan]

Hi Sambhu,

I have a theory: by having -libjar /path/to/alluxio-core-client-1.2.0-jar-with-dependencies.jar , tasks of your MapReduce job is able to find Alluxio jar and thus able to communicate with Alluxio file system when seeing "alluxio://" prefix. However, in C45DecisionDriver.java:98 where you call FileSystem.exists() in your jar, is it already outside of the MapReduce? If yes, it is possible that your code outside MapReduce is not able to find alluxio client jar (even you have specified -libjar indicating the jar to MR). So could you also add alluxio client jar to $HADOOP_CLASSPATH (which will be picked when you run "hadoop jar")?

[Sambhu R]

Hi Bin,

Thanks for the reply. May I ask a question? while running "hadoop classpath" command I can see multiple destinations like:

nn1@nn1:/usr/lib/hadoop-0.20-mapreduce/lib$ hadoop classpath
/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:/home/nn1/alluxio-1.2.0/core/client/target/alluxio-core-client-1.2.0-jar-with-dependencies.jar:

So I am kinda stuck here on where exactly should i add the alluxio jar, you can also see the compiled jar file of alluxio.
I have already added my classpath in hadoop-env.sh file, and also in yarn-site.xml file. so please walk me through on how to do? I'm new to hadoop that's why.
 
I'm using Cloudera Hadoop with MapReduce Version 2.

Thank You. :)

[Bin Fan]

Hi Sambhu,

my suggestion is still very similar to the setting in http://www.alluxio.org/docs/master/en/Running-Hadoop-MapReduce-on-Alluxio.html

First you need to update core-site.xml of your Hadoop distribution to have:

<property>

  <name>fs.alluxio.impl</name>

  <value>alluxio.hadoop.FileSystem</value>

  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>

</property>

<property>

  <name>fs.alluxio-ft.impl</name>

  <value>alluxio.hadoop.FaultTolerantFileSystem</value>

  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x) with fault tolerant support</description>

</property>

<property>

  <name>fs.AbstractFileSystem.alluxio.impl</name>

  <value>alluxio.hadoop.AlluxioFileSystem</value>

  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>

</property>

This step helps your application to recognize the URL prefixing with "alluxio://" when using Hadoop API,

so it knows URL like "alluxio://foo" is legit and tries to look for the corresponding implementation classes.

Second, you need to add alluxio jar (in your case
/home/nn1/alluxio-1.2.0/core/client/target/alluxio-core-client-1.2.0-jar-with-dependencies.jar)
on your $HADOOP_CLASSPATH, so when the Hadoop API is looking for the implementation classes, the 

binaries is there ready for use.

It looks to me you have problem in the first step, as your application still calls 

org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)

from the trace. Therefore, could you double check the first step?

thanks,

- Bin

[黄志]

I think you should check that your Alluxio client version is the same as Alluxio server!

在 2017年1月7日星期六 UTC+8下午4:07:28，Sambhu R写道：

[Sambhu R]

HI Bin,

Here is my 'core.xml' file

<configuration>
<property>
 <name>fs.defaultFS</name>
 <value>hdfs://10.30.60.45:8020</value>
</property>

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
  <name>fs.alluxio-ft.impl</name>
  <value>alluxio.hadoop.FaultTolerantFileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x) with fault tolerant support</description>
</property>
<property>
  <name>fs.AbstractFileSystem.alluxio.impl</name>
  <value>alluxio.hadoop.AlluxioFileSystem</value>
  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>

</configuration>

I've also added this in 'hdfs-site.xml' file as well:

<configuration>
<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>
<property>
     <name>dfs.namenode.name.dir</name>
     <value>file:///var/lib/hadoop-hdfs/cache/hdfs/dfs/name</value>
</property>
<property>
 <name>dfs.permissions.superusergroup</name>
 <value>hadoop</value>
</property>
<property>
 <name>dfs.datanode.data.dir</name>
 <value>file:///data/1/dfs/dn,file:///data/2/dfs/dn,file:///data/3/dfs/dn,file:///data/4/dfs/dn</value>
</property>
<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>

<property>
  <name>fs.alluxio.impl</name>
  <value>alluxio.hadoop.FileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x)</description>
</property>
<property>
  <name>fs.alluxio-ft.impl</name>
  <value>alluxio.hadoop.FaultTolerantFileSystem</value>
  <description>The Alluxio FileSystem (Hadoop 1.x and 2.x) with fault tolerant support</description>
</property>
<property>
  <name>fs.AbstractFileSystem.alluxio.impl</name>
  <value>alluxio.hadoop.AlluxioFileSystem</value>
  <description>The Alluxio AbstractFileSystem (Hadoop 2.x)</description>
</property>

</configuration>

But still facing the error.

Thank You............

[Bin Fan]

Hi Sambhu,

the configuration looks correct to me.

One more question regarding the tracestack:

```

at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
at edu.am.bigdata.C45TreeModel.C45DecisionDriver.getMean(C45DecisionDriver.java:984)

```

It looks to me in C45DecisionDriver.java:984 where you call FileSystem.exists 

do you have code like 

```

DistributedFileSystem fs = new DistributedFileSystem();

fs.exists(new Path("alluxio://10.30.60.45:19998/abdfalluxioFS/C4.5_Mean"));

```

or do you have

```

final Configuration conf = new Configuration();

final FileSystem fs = FileSystem.get(conf);

fs.exists(new Path("alluxio://10.30.60.45:19998/abdfalluxioFS/C4.5_Mean"));

```

For the former one, it will not work as it always use hadoop fs (DistributedFileSystem),

while the latter one will be smart to pick a fs implementation based on the prefix (here alluxio)

[unmesha sreeveni]

Yes Bin, you are right. 

We use hdfs path instead of alluxio as our concentration was on hdfs.The job that we tried to execute was a chaining job where output of the first job is fed into second job's input.so in our case it is hdfs path. 

So if we need to switch to alluxio I think we will need some sort of modifications in our current code.

[Bin Fan]

Hi, 

just to clarify, it is not about the path but about how you create the fs handler.

with code like `DistributedFileSystem fs = new DistributedFileSystem();`, the HDFS implementation of FileSystem is always used because DistributedFileSystem is the class provided by Hadoop

while with code like `FileSystem fs = FileSystem.get(new Configuration());`, it creates a FileSystem handler and the implementation will be chosen based on the path prefix

- Bin

[unmesha sreeveni]

We are using 

final Configuration conf = new Configuration();

final FileSystem fs = FileSystem.get(conf);

fs.exists(new Path("hdfs://ip:port/hdfspath/C4.5_Mean"));

So it might not be a problem i guess. 

For sure the path doesnt matter right?

While we run the code in alluxio -  hdfs path will change to alluxio. 

[Bin Fan]

If the data is already saved in Alluxio, you should use the following to read it.

```

final Configuration conf = new Configuration();

final FileSystem fs = FileSystem.get(conf);

fs.exists(new Path("alluxio://ip:port/hdfspath/C4.5_Mean"));

```

But I am still confused by the error message 

Wrong FS: alluxio://10.30.60.45:19998/abdfalluxioFS/C4.5_Mean, expected: hdfs://10.30.60.45:8020

[unmesha sreeveni]

Ok thanks Bin

--
You received this message because you are subscribed to a topic in the Google Groups "Alluxio Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alluxio-users/S0N_djlquf4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to alluxio-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks & Regards

Unmesha Sreeveni U.B

Hadoop, Bigdata Developer

Centre for Cyber Security | Amrita Vishwa Vidyapeetham

http://www.unmeshasreeveni.blogspot.in/

[Bin Fan]

Were you able to resolve the error?

- Bin

[Sambhu R]

Hi Bin,

The problem is that we have to make changes on our current code, So once we do it, then its good to go, Hope so. We have to plan for making code changes, and it will take time. We will follow up with the same, if we  are going ahead.

And thanks for the help... :)
Cheers..

Ok thanks Bin

To unsubscribe from this group and all its topics, send an email to alluxio-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.