Hi all
As some of you might be aware we recently submitted a proposal to merge SparkR with the mainline Apache Spark project [1]. Given the developments around new DataFrame APIs and the stabilization of other components like the RDD API, spark-submit and JVM backend, I think this is a good time to create a pull request for the merge.
The merge process will cause some changes to our development workflow and I have detailed a plan for that below. I plan to start this in a week or so after merging the PRs that are currently open. Do let me know if you have any feedback or questions.
Thanks
SparkR Merge Steps
1. Our development right now happens on two branches `master` and `sparkr-sql`. The latter is used for developing the DataFrame API which ships with Spark 1.3. To support existing users, the convention we will use is that the master branch will support Spark versions 0.9 to 1.2. We will rename the `sparkr-sql` branch to `branch-1.3` and this will support Spark 1.3.
After the merge we will release SparkR as a part of Spark 1.4
2. The master branch and `sparkr-sql` branch have diverged a bit and we will port relevant changes to make them share a common baseline. The only changes in `sparkr-sql` that will not be in `master` will be those related to DataFrames. We will also merge all the PRs currently open on the SparkR repository before starting a code move.
3. We will create a new repository at
https://github.com/amplab-extras/spark as a fork of the Apache Spark repository. This repository will be used to stage our merge PR and we will continue development on this repository for any bug fixes, documentation changes etc.
4. After the merge, development will continue on the Apache Spark repository. We will add a SparkR developers guide to the Spark contributors wiki.
5. We will also transition the JIRA issues and related documentation to the Spark repository after the code merge. I will post another message before we do that.