|
The Spark executor could use lots of memory to cache some data, then it's very expensive to fork a R process (or easy to fail).
The solution is fork a lightweight daemon (in R) process in the beginning (when needed), call the daemon to launch R worker when needed.
This is only works in Linux/Mac, In Windows, it will still fork a R worker from JVM.
|