Incomplete HDFS URI, no host: hdfs:///spark-logs... error while starting Spark shell

Ophir Cohen

unread,

Jan 20, 2015, 8:54:46 AM1/20/15

to tachyo...@googlegroups.com

Hi Guys,
I set up Tachyon on my EMR Spark cluster.
I manged to start Tachyon cluster and its up and running but when starting Spark shell I get the below error and the shell starts only as scala shell (i.e. without SparkContext).
It looks that it misses host for the spark logs but I couldn't understand how to set it up...
Any help will be welcomed!
Thanks,
Ophir

2015-01-20 13:44:59,800 INFO [main] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Registered BlockManager
java.io.IOException: Incomplete HDFS URI, no host: hdfs:///spark-logs/application_1421752461071_0007
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:365)
    at org.apache.spark.util.FileLogger.<init>(FileLogger.scala:90)
    at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:63)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:352)
    at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:986)
    at $iwC$$iwC.<init>(<console>:9)
    at $iwC.<init>(<console>:18)
    at <init>(<console>:20)
    at .<init>(<console>:24)
    at .<clinit>(<console>)

Ophir Cohen

unread,

Jan 20, 2015, 10:27:49 AM1/20/15

to tachyo...@googlegroups.com

One more piece of information:

When removing the core-site.xml file and starting the shell without it I can manage to use Tachyon as follows:
sc.hadoopConfiguration.set("fs.tachyon.impl", "tachyon.hadoop.TFS")
var tFile = sc.textFile("tachyon://10.181.43.174:19998/tach.test")

This time it works...

--
You received this message because you are subscribed to a topic in the Google Groups "Tachyon Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tachyon-users/vHeuDElFVLY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tachyon-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Haoyuan Li

unread,

Jan 21, 2015, 12:10:19 PM1/21/15

to Ophir Cohen, tachyo...@googlegroups.com

Ophir,

This issue seems unrelated to Tachyon.

Best,

Haoyuan

--
You received this message because you are subscribed to the Google Groups "Tachyon Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tachyon-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Haoyuan Li

AMPLab, EECS, UC Berkeley

http://www.cs.berkeley.edu/~haoyuan/

Ophir Cohen

unread,

Jan 21, 2015, 2:03:00 PM1/21/15

to tachyo...@googlegroups.com, oph...@gmail.com

OK, thanks.
I'll search my luck in the Spark group...
Ophir

Henry Saputra

unread,

Jan 21, 2015, 2:09:39 PM1/21/15

to Ophir Cohen, tachyo...@googlegroups.com

It looks like it cannot resolve the URL hdfs:///spark-logs. Could you
replace it with IP and try it?

On Tue, Jan 20, 2015 at 5:54 AM, Ophir Cohen <oph...@gmail.com> wrote:

Ophir Cohen

unread,

Jan 21, 2015, 2:15:23 PM1/21/15

to Henry Saputra, tachyo...@googlegroups.com

It's inside the Spark shell so I don't have access to that.
BTW

The exact same path works when I remove the core-site.xml file that adds the Tachyon configuration.