Standalone mode, move "work" directory or cleanup after job?

[Aki Matsukawa]

Hi,

I am running Spark in standalone mode. I am creating a SparkContext with a jar to ship to all workers. This jar is fairly big (100MB) and I am observing that each time I run a job, this jar is getting shipped to each machine, under a new folder under $SPARK_HOME/work/<app_id>/. 

As you can imagine, running many jobs quickly causes disks on the worker machines to fill up. Is there a way to automatically delete these jars after the job is done, and/or move where the "work" directory is?

Thanks!

[Matei Zaharia]

You can move the location of the work directory by adding the following to your conf/spark-env.sh:

export SPARK_WORKER_DIR=/path/to/dir

Unfortunately, right now there's no automatic cleanup, but the directories are numbered in order of execution. You could create your own periodic cleanup script that deletes old ones.

Matei

> --
> You received this message because you are subscribed to the Google Groups "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Dmitriy Lyubimov]

is this still the case with 0.7.2? there's still no automatic cleanup?

[Dmitriy Lyubimov]

On Mon, Jun 24, 2013 at 7:56 PM, Dmitriy Lyubimov <dli...@gmail.com> wrote:

is this still the case with 0.7.2? there's still no automatic cleanup?

Nobody knows this??

[Matei Zaharia]

Yup, this is still the case in 0.7.2. Actually this might be a good feature to add in 0.7.3 -- thanks for bringing it up. Will definitely add it for 0.8.

Matei

[Dmitriy Lyubimov]

Thank you.