Re: problem when reading from spark

[Jiří Šimša]

You need to specify the data port in the Alluxio URL. The default one is 19998, so you should use sc.textFile("alluxio://192.168.30.128:19998/data/hosts").

On Mon, Feb 29, 2016 at 11:53 PM, Nguyen khac tu <tub...@gmail.com> wrote:

i 'm trying to integrate Alluxio with Spark but when i try to read an text file from Alluxio it throws me some errors :  ( alluxio build from master and Spark is 1.6 ) 

sc.hadoopConfiguration.set("fs.alluxio.impl", "alluxio.hadoop.FileSystem") 
sc.textFile("alluxio://192.168.30.128/data/hosts")
res0.first

error: 

ava.lang.IllegalArgumentException: port out of range:-1

at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)

at java.net.InetSocketAddress.<init>(InetSocketAddress.java:224)

at alluxio.client.ClientContext.init(ClientContext.java:63)

at alluxio.hadoop.AbstractFileSystem.initialize(AbstractFileSystem.java:433)

at alluxio.hadoop.FileSystem.initialize(FileSystem.java:25)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)

at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:256)

at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)

at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:304)

at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

at scala.Option.getOrElse(Option.scala:120)

.........................

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jiří Šimša

Alluxio, Inc.

[Nguyen khac tu]

yep , you 're right , it 's a my mistake !! sorry to bother you :(