spark.local.dir getting full

[Dilip Joseph]

Hello,

Is there any support in spark/mesos to periodically auto-cleanup spark.local.dir?  Currently, my local dir fills up the disk every few days.

Thanks,

Dilip

--
_________________________________________
Dilip Antony Joseph
http://csgrad.blogspot.com
http://www.marydilip.info

[Matei Zaharia]

Unfortunately there's no perfect way to do this, but I'm going to add some calls to File.deleteOnExit() on the temporary directories. The problem with deleteOnExit is that it only works when the JVM exits normally, not if gets killed from outside.

Maybe the best way to solve this would be to let Mesos manage the temp directories, because the mesos-slave is a long-lived process that knows for sure when our JVM crashes. We've discussed adding support for that in Mesos but it's not there yet.

Matei

[Matei Zaharia]

Hi Dilip,

I've committed a fix to do this via a Java shutdown hook: https://github.com/mesos/spark/commit/8f5ed51234a06a086d75d5ed2c289baa51fc094b. However, we will add the foolproof version of this feature to Mesos. The following issue tracks that: https://issues.apache.org/jira/browse/MESOS-149.

Matei