[JIRA] (SPARKR-219) fail to start and stop SparkContext multiple times in R process

13 views
Skip to first unread message

Davies (JIRA)

unread,
Mar 5, 2015, 4:41:02 PM3/5/15
to spark...@googlegroups.com
Davies assigned an issue to Shivaram Venkataraman
 
SparkR / Improvement SPARKR-219
fail to start and stop SparkContext multiple times in R process
Change By: Davies
Assignee: Shivaram Venkataraman
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Davies (JIRA)

unread,
Mar 5, 2015, 4:41:02 PM3/5/15
to spark...@googlegroups.com
Davies created an issue
Issue Type: Improvement Improvement
Assignee: Unassigned
Created: 05/Mar/15 1:40 PM
Priority: Major Major
Reporter: Davies

If we have a sparkR.stop() in any unit test, then the later tests will fail as:

collect on 44 failed with java.lang.IllegalArgumentException: Object not found 44
java.lang.IllegalArgumentException: Object not found 44
	at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.handleMethodCall(SparkRBackendHandler.scala:91)
	at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:60)
	at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:22)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)
2

The reason is that we track all the reference of Java objects in two global environments. All new SparkContext will count the Java objects from 0, they could be confusing, for example, the reference for Java object in previous JVM, will remove the Java object with same ID in current JVM, then you will fail to access this Java object.

Shivaram Venkataraman (JIRA)

unread,
Mar 6, 2015, 2:28:03 PM3/6/15
to spark...@googlegroups.com
Shivaram Venkataraman commented on Improvement SPARKR-219
 
Re: fail to start and stop SparkContext multiple times in R process

Thanks for reporting this - I think the main issue is that we don't clear the JVM object ids on the R side when we stop the JVM. I'll work on a fix for this.

Davies (JIRA)

unread,
Mar 6, 2015, 2:44:01 PM3/6/15
to spark...@googlegroups.com
Davies commented on Improvement SPARKR-219

I had tried to clear all the ID in .validJobj and .toRemoveId, they help a little bit, but still did not work properly.

There are some jobj still hold a ID, the finalizer will be called once they are freed.

Maybe we should create a uuid for each SparkContext, put it in every jobj, check that in the finalizers.

Shivaram Venkataraman (JIRA)

unread,
Mar 16, 2015, 3:04:01 PM3/16/15
to spark...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages