[JIRA] (SPARKR-214) Launch R worker by a daemon

6 views
Skip to first unread message

Davies (JIRA)

unread,
Feb 26, 2015, 3:05:02 PM2/26/15
to spark...@googlegroups.com
Davies created an issue
 
SparkR / Improvement SPARKR-214
Launch R worker by a daemon
Issue Type: Improvement Improvement
Assignee: Davies
Created: 26/Feb/15 12:04 PM
Priority: Critical Critical
Reporter: Davies

The Spark executor could use lots of memory to cache some data, then it's very expensive to fork a R process (or easy to fail).

The solution is fork a lightweight daemon (in R) process in the beginning (when needed), call the daemon to launch R worker when needed.

This is only works in Linux/Mac, In Windows, it will still fork a R worker from JVM.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Mar 13, 2015, 2:38:01 AM3/13/15
to spark...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages