Failed to create directory ...spark-0.7.0/work/app-20130423160011-0001/19

[Håvard Wahl Kongsgård]

Hi, after starting a small cluster I get this error in the shell

3160011-0001/19 removed: class java.io.IOException: Failed to create directory /home/havard/source/spark-0.7.0/work/app-20130423160011-0001/19

scala 2.9.2

java sun 1.6

my config

export JAVA_HOME=/usr/lib/jvm/java-6-sun

export SCALA_HOME=/home/havard/source/scala-2.9.2

export SPARK_MASTER_IP=kongs7.medisin.ntnu.no

-Håvard

[Patrick Wendell]

Is there another more specific exception in the trace as well? Usually this means

- There is a permissions error and the user running spark does nto have access to $SPARK_HOME/work

- The disk drive is out of space

or for some other reason it can't create the file. 

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Håvard Wahl Kongsgård]

Thanks, I solved it my removing 

"export SPARK_JAVA_OPTS="-Dspark.local.dir=...." from the conf

However, I don't know that the issue was, the permission for work was 777 (with same owner as spark user), HD space is not an issue.

Another question: is it ok to use hostnames with spark or should I stick to IP's only?

-Håvard

[Patrick Wendell]

"export SPARK_JAVA_OPTS="-Dspark.local.dir=...." from the conf

When you remove this from the config it will just change the working directory to /tmp. In your case it seems like /tmp does not have the same filesystem issue, whatever it was.

Another question: is it ok to use hostnames with spark or should I stick to IP's only?

In general hostnames are okay - for instance in the slaves file. When connecting to the master (e.g. spark://my_host:7077), make sure you use the exact same hostname that appears in the WebUI.

[Håvard Wahl Kongsgård]

... I still get this error(but not always... ), so it might be a bug. But spark seems to be working anyway.

13/04/25 15:42:40 INFO cluster.SparkDeploySchedulerBackend: Executor app-20130425154240-0002/18 removed: class java.io.IOException: Failed to create directory /home/havard/source/spark-0.7.0/work/app-20130425154240-0002/18

CDH 4

ubuntu 12.04 LTS

-Håvard

[Patrick Wendell]

If you go look in the logs on the slave node (in the logs/ folder inside of $SPARK_HOME) it might give you a finer grained error message. You should be able to look further up in the logs and figure out which node has that executor on it.

The reason why the jobs are still succeeding is because spark sees that the particular slave has failed and it just reschedules work on other nodes.

- Patrick

--