SparkR branch updates, merge plans

17 views
Skip to first unread message

Shivaram Venkataraman

unread,
Mar 12, 2015, 6:42:52 PM3/12/15
to spark...@googlegroups.com, Patrick Wendell, Davies Liu, Chris Freeman, Dan Putler, rui...@intel.com
Hi all

As some of you might be aware we recently submitted a proposal to merge SparkR with the mainline Apache Spark project [1]. Given the developments around new DataFrame APIs and the stabilization of other components like the RDD API, spark-submit and JVM backend, I think this is a good time to create a pull request for the merge.

The merge process will cause some changes to our development workflow and I have detailed a plan for that below. I plan to start this in a week or so after merging the PRs that are currently open. Do let me know if you have any feedback or questions.

Thanks
Shivaram

SparkR Merge Steps

1.  Our development right now happens on two branches `master` and `sparkr-sql`. The latter is used for developing the DataFrame API which ships with Spark 1.3. To support existing users, the convention we will use is that the master branch will support Spark versions 0.9 to 1.2. We will rename the `sparkr-sql` branch to `branch-1.3` and this will support Spark 1.3. 
After the merge we will release SparkR as a part of Spark 1.4

2. The master branch and `sparkr-sql` branch have diverged a bit and we will port relevant changes to make them share a common baseline. The only changes in `sparkr-sql` that will not be in `master` will be those related to DataFrames. We will also merge all the PRs currently open on the SparkR repository before starting a code move.

3. We will create a new repository at https://github.com/amplab-extras/spark as a fork of the Apache Spark repository. This repository will be used to stage our merge PR and we will continue development on this repository for any bug fixes, documentation changes etc. 

4. After the merge, development will continue on the Apache Spark repository. We will add a SparkR developers guide to the Spark contributors wiki.

5. We will also transition the JIRA issues and related documentation to the Spark repository after the code merge. I will post another message before we do that.

Shivaram Venkataraman

unread,
Mar 12, 2015, 10:30:51 PM3/12/15
to Patrick Wendell, Sun, Rui, Davies Liu, spark...@googlegroups.com, Chris Freeman, Dan Putler
[Adding some replies back to the dev list]

On Thu, Mar 12, 2015 at 7:20 PM, Patrick Wendell <pat...@databricks.com> wrote:
We will have to merge this as a single patch using our existing infrastructure - this is what we've done in the past for Spark SQL and other large components. Unfortunately, there is a "one-time" cost that you lose the history.

Spark as a policy does not use git merge commits (this happened in early 2014, IIRC) and there are downstream packagers, tools, etc, that assume that Spark's history is linear. For instance, this is a key assumption when we perform bisections that identify release regressions.

Yeah its unfortunate we will lose some of this history. One thing I was thinking about is if we can break the PR into say 5 or 10 parts (or those many PRs) -- We have only had around 200 PRs merged in SparkR overall and we could do some logical grouping by say squashing all the DataFrame issues into one and all the SparkR JVM changes into one etc.  Or we could also do some timeline-based grouping which might help for some debugging scenarios.
 

- Patrick

On Thu, Mar 12, 2015 at 7:13 PM, Sun, Rui <rui...@intel.com> wrote:
Yes, I have the same concern. Spark now uses rebase way with all changes for a patch squashed into a single commit.  Could we use merge way for SparkR's first integration to keep history, and later we can use rebase way?

-----Original Message-----
From: Davies Liu [mailto:dav...@databricks.com]
Sent: Friday, March 13, 2015 7:44 AM
To: Shivaram Venkataraman
Cc: spark...@googlegroups.com; Patrick Wendell; Chris Freeman; Dan Putler; Sun, Rui
Subject: Re: SparkR branch updates, merge plans

Thanks for the update, the plan looks good to me. I can work on merge master into sql branch.

For step 3, how can we merge the current repository into an existing spark repository? It will be good to keep some history.

Patrick Wendell

unread,
Mar 12, 2015, 11:22:51 PM3/12/15
to Dan Putler, Shivaram Venkataraman, Sun, Rui, Davies Liu, spark...@googlegroups.com, Chris Freeman
People's github cred won't go disappear on the current SparkR repo, but you are right that they won't be inherited into Spark. Only new work people do will show up in Spark's history. It's definitely one downside of this approach, but with Spark SQL it didn't seem to matter much, since most of the same people just continued contributing to it as part of the upstream project.

Overall, my advice is for SparkR to focus on making sure it can be merged in a way that minimizes the impact to the existing Spark project. This is the best way to make sure everyone is enthusiastic about the merge. Unfortunately, a naive "git merge" would pollute the git history in a way that has adverse consequences for many people in the Spark ecosystem, even those who do not use or intend to use SparkR.

- Patrick

Patrick Wendell

unread,
Mar 12, 2015, 11:26:39 PM3/12/15
to Dan Putler, Shivaram Venkataraman, Sun, Rui, Davies Liu, spark...@googlegroups.com, Chris Freeman
When other thing - if this does go in as planned we'll definitely list the contributors to SparkR specifically when doing our own "credit's".

- Patrick
Reply all
Reply to author
Forward
0 new messages