Zeppelin Cron job out of memory

[kevin...@gmail.com]

Hi, all

The zeppelin cron job throw "OutOfMemoryError" after a while. It looks like a lot of new classes are loaded (never got Gced) every time a cron job is run.

java.lang.OutOfMemoryError: PermGen space

        at java.lang.ClassLoader.defineClass1(Native Method)

        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

        at org.apache.spark.repl.SparkIMain.logAndDiscard(SparkIMain.scala:283)

        at org.apache.spark.repl.SparkIMain.org$apache$spark$repl$SparkIMain$$withLastExceptionLock(SparkIMain.scala:297)

        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.bindError(SparkIMain.scala:829)

        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1126)

        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669)

        at com.nflabs.zeppelin.spark.SparkInterpreter.interpretInput(SparkInterpreter.java:395)

        at com.nflabs.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:369)

        at com.nflabs.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:363)

        at com.nflabs.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:52)

        at com.nflabs.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:56)

        at com.nflabs.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:183)

        at com.nflabs.zeppelin.scheduler.Job.run(Job.java:147)

        at com.nflabs.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:85)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

Best Regards,

Kevin.

[moon soo Lee]

Hi Kevin,

Because of scala repl compile code into java classes and load them into class loader, OutOfMemoryError PermGen space is highly expected error from iterative run.

According to http://stackoverflow.com/questions/3334911/what-does-jvm-flag-cmsclassunloadingenabled-actually-do and http://community.cloudera.com/t5/Data-Science-and-Machine/spark-shell-java-lang-OutOfMemoryError-PermGen-space/td-p/8772, CMSClassUnloadingEnabled and  UseConcMarkSweepGC JVM flags can solve the problem. (but not tested with zeppelin)

Hope this helps

Thanks,

moon

[kevin...@gmail.com]

Ooops, it doesn't work

在 2015年1月4日星期日 UTC+8下午1:35:29，moon soo Lee写道：