[JIRA] (SPARKR-212) Backended Runs out of heap space -- adjust backend RAM

61 views
Skip to first unread message

Amos (JIRA)

unread,
Feb 25, 2015, 11:25:02 PM2/25/15
to spark...@googlegroups.com
Amos created an issue
 
SparkR / Bug SPARKR-212
Backended Runs out of heap space -- adjust backend RAM
Issue Type: Bug Bug
Assignee: Unassigned
Components: Internals
Created: 25/Feb/15 8:24 PM
Priority: Major Major
Reporter: Amos

sparkR always seems to launch the backend with 512MB. I don't see a documented option for adjusting that. I get a Java heap space error with the stack trace below every time I run my app with a large enough amount of data.

Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876)
at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:164)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
at org.apache.spark.SparkContext.clean(SparkContext.scala:1478)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1319)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1339)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1353)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1367)
at org.apache.spark.rdd.RDD.collect(RDD.scala:797)
at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:309)
at org.apache.spark.api.java.JavaRDD.collect(JavaRDD.scala:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.handleMethodCall(SparkRBackendHandler.scala:107)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:60)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:22)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Feb 25, 2015, 11:28:01 PM2/25/15
to spark...@googlegroups.com
Shivaram Venkataraman commented on Bug SPARKR-212
 
Re: Backended Runs out of heap space -- adjust backend RAM

You can set the environment variable SPARK_MEM and it should get picked up if you are directly including the package. If you are using sparkR-submit then you'll need to set the driver memory flag used by spark-submit.

Max Lin (JIRA)

unread,
Mar 4, 2015, 8:51:02 PM3/4/15
to spark...@googlegroups.com
Max Lin commented on Bug SPARKR-212

I met the same problem. if I don't set SPARK_MEM, I'll get

Launching java with command /usr/java/latest/bin/java -Xmx512m -cp '/usr/lib64/R/library/SparkR/sparkr-assembly-0.1.jar::/etc/hadoop/conf' edu.berkeley.cs.amplab.sparkr.SparkRBackend /tmp/RtmpAofpSE/backend_port29666fc2259
.......
Exception in thread "writer for R" java.lang.OutOfMemoryError: Java heap space
at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:266)
at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:242)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$1.next(RRDD.scala:66)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$1.foreach(RRDD.scala:62)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$2.run(RRDD.scala:166)
Error in readBin(con, raw(), as.integer(dataLen), endian = "big") :
........

But after I set SPARK_MEM=4g, I got the following message:

Launching java with command /usr/java/latest/bin/java -Xmx4g -cp '/usr/lib64/R/library/SparkR/sparkr-assembly-0.1.jar::/etc/hadoop/conf' edu.berkeley.cs.amplab.sparkr.SparkRBackend /tmp/RtmpdgSWw8/backend_port43159b4cd7e

R Writer thread got an exception org.apache.spark.SparkException: R worker exited unexpectedly (crashed)
org.apache.spark.SparkException: R worker exited unexpectedly (crashed)
at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:273)
at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:242)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$1.next(RRDD.scala:66)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$1.foreach(RRDD.scala:62)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD$$anon$2.run(RRDD.scala:166)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at edu.berkeley.cs.amplab.sparkr.RRDD.read(RRDD.scala:262)
... 5 more

Max Lin (JIRA)

unread,
Mar 17, 2015, 9:15:02 PM3/17/15
to spark...@googlegroups.com
Max Lin commented on Bug SPARKR-212

The problem I met during unserialization was solved after I update my SparkR package.(to amplab-extras-SparkR-pkg-d984983.tar.gz)

Shivaram Venkataraman (JIRA)

unread,
Apr 10, 2015, 1:18:02 AM4/10/15
to spark...@googlegroups.com

With the merge into Apache Spark we use spark-submit to configure the memory of the driver and executors. So I think this problem should not occur there. I'm marking this as fixed here and we can open a new issue at https://issues.apache.org/jira/browse/SPARK in case this comes up again.

This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014-sha1:e47e3fa)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Apr 10, 2015, 1:18:02 AM4/10/15
to spark...@googlegroups.com
Shivaram Venkataraman resolved as Fixed
 
Change By: Shivaram Venkataraman
Resolution: Fixed
Status: Open Resolved
Reply all
Reply to author
Forward
0 new messages