|
I met the same problem. if I don't set SPARK_MEM, I'll get
Launching java with command /usr/java/latest/bin/java -Xmx512m -cp '/usr/lib64/R/library/SparkR/sparkr-assembly-0.1.jar::/etc/hadoop/conf' edu.berkeley.cs.amplab.sparkr.SparkRBackend /tmp/RtmpAofpSE/backend_port29666fc2259 ....... Exception in thread "writer for R" java.lang.OutOfMemoryError: Java heap space at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:266) at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:242) at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$1.next(RRDD.scala:66) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$1.foreach(RRDD.scala:62) at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$2.run(RRDD.scala:166) Error in readBin(con, raw(), as.integer(dataLen), endian = "big") : ........
But after I set SPARK_MEM=4g, I got the following message:
Launching java with command /usr/java/latest/bin/java -Xmx4g -cp '/usr/lib64/R/library/SparkR/sparkr-assembly-0.1.jar::/etc/hadoop/conf' edu.berkeley.cs.amplab.sparkr.SparkRBackend /tmp/RtmpdgSWw8/backend_port43159b4cd7e
R Writer thread got an exception org.apache.spark.SparkException: R worker exited unexpectedly (crashed) org.apache.spark.SparkException: R worker exited unexpectedly (crashed) at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:273) at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:242) at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$1.next(RRDD.scala:66) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$1.foreach(RRDD.scala:62) at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$2.run(RRDD.scala:166) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:262) ... 5 more
|