Re: [blink-dev] Merging the Blink and Chromium repositories

[Ben Goodger (Google)]

I feel obliged to loop in chromium-dev and John here too since the discussion now affects a huge swath of Chromium code :-)

I'll repeat a high level point made to Darin via IM:

IMO having a separate repo and pulling that repo via DEPS is mostly only useful for third party modules where the API is pretty stable and doesn't change that much. Many if not most of our third party dependencies fit this model.

This is definitely not the case for Content and its dependencies (the ones closest to my heart being Aura and Views, but also other stuff that other people care about), and so hoisting them into a blink repo (let's just call it "platform" at that point) will make a lot more people's lives harder.

But it sounds like it's also not the case for Blink either.

Really, we're one huge happy family now, and we sink or swim together. We probably shouldn't be planning now who gets the kids or who gets the dog. It seems like many of the issues with cohabitation are just opportunities to improve our house (tools). As far as history goes, resets are not without precedent in the Chrome project history. I think we have at least two old internal source control systems you can poke around on still for historic curiosity. If history is important we can just keep a clone of the pre-merged state around in perpetuity.

-Ben

On Sat, Apr 6, 2013 at 11:02 PM, Ben Goodger (Google) <b...@chromium.org> wrote:

It's not just API changes. It's implementation changes. You're just shifting the burden up the stack by doing this. For example in this world a bugfix for WebContentsViewAura now requires > 1 step before it's picked up by Chrome. API changes are worse, as Alec notes. And then like you say, you end up hosting other stuff (e.g. Aura and Views) whose APIs is constantly changing.

-Ben

On Sat, Apr 6, 2013 at 10:58 PM, Darin Fisher <da...@chromium.org> wrote:

On Sat, Apr 6, 2013 at 10:47 PM, Adam Barth <aba...@chromium.org> wrote:

For features like IndexedDB, I'd like to get to the point where they can be self-contained modules that interface with the core rendering code in an easy-to-understand way (e.g., by describing themselves with WebIDL).  If we do a good job, these modules could live in the Chromium repository rather than the Blink repository.

My main concern is the one that Dimitri raised.  The current approach is painful for folks working on features that span the boundary between Blink and Content but merging the repositories would be painful for folks working on features solely contained within Blink (and would tax all the other developers working in the main Chromium repo).  It seems like there's a balance to strike.

We should also think about ways of reducing these costs, for example by making it possible for modules to live in other repositories.  Another approach is to move the repository boundary.  For example, perhaps Content and it's dependencies should be in the "Blink" repository and all the code that depends on content should be in the "Main" repository.

^^^ I'm really curious to explore this idea further.  I think we have a lot of data to look at.  We can study how frequently API changes were made at the WebKit boundary versus the Content boundary.  We just have to be careful to realize that the Content boundary actually includes the API surface area of every module that Content depends on, which Chrome and other Content embedders consume.  Concretely, I'm referring to things like src/base and src/net.

-Darin

 

At a higher level, I think these are good questions, but likely ones that we'll be in a better position to answer in a month or two.  Maybe we'll have such a green and wonderful tree that we can close the Blink tree and land repository spanning patches atomically.

Adam

On Sat, Apr 6, 2013 at 10:05 PM, Alec Flett <alec...@chromium.org> wrote:

This is simliar to Dominic's comparison, but I feel Alexandre's pain every day. I typically spend 1-2 days working on a "patch" that spans both trees that requires a function to change a parameter or two.Then I spend 3-6 days landing it - 

If you've never experienced this pain, or wonder if I'm just exaggerating, this is the typical workflow:

1) land stubs interfaces in WebKit (in the WebKit API)

2) wire up chromium <-> webkit stub code in chromium

3) replace webkit stubs with real implementation, leaving old implementation around so chromium still builds

4) remove plumbing for old implementation from chromium

5) remove old implementation from WebKit

And you have to wait for a webkit roll between steps 1/2, and steps 3/4, which means this is guaranteed to take 2 days. Add in a slow commit queue and it's easily 4+.

I had thought that being able to expose more in Platform/chromium would help this, but I realize it just means less code, not less steps.

I have been working at what feels like half-speed for the better part of a year. My particular project (IndexedDB) is going to do our best to lift ourselves out of Blink proper, and live much of our lives in the Chromium tree, mainly to address this issue.. all *we* need is a hook into the DOM for a window, and after that we can more or less divorce ourselves from Blink.. so that's likely what we'll do.

I just went through a 3-month refactoring of IDB that feels like I could have done in maybe 2-3 weeks if it weren't for the above. And then I'd already be working on other features like IDB Mutation Observers :)

It seems like Blink is really two things:

1) an HTML layout engine - a parser, a renderer, CSS engine, and a DOM.

2)  a container for all the other HTML5 non-html features, such as storage subsystems (indexedDB/filesystem/etc) non-layout rendering (WebGL, Canvas, etc)

Perhaps there is a better way to divide this such that all the people in the 2nd bucket are not doing 3 or 5 sided patches. If our embedding layer is really chromium/src/content, then lets divide the tree there, not at Blink. 

That said I'm fully in favor of just rolling it together into one project, assuming all the source control issues can be sorted out. I don't believe there are N+M engineers that get blocked because there are huge overlaps between N and M.

Alec

On Sat, Apr 6, 2013 at 8:29 PM, Dimitri Glazkov <dgla...@chromium.org> wrote:

One con that I see and find pretty significant is inability of Blink
and Chromium to move independently. For example: if the Blink tree is
closed, N engineers twiddle their thumbs. If Chromium tree is closed,
M engineers take a coffee break. If Blink+Chromium tree is closed, N+M
engineers are affected.

:DG<

[John Abd-El-Malek]

I completely agree that moving more stuff into a repo that's rolled through DEPs would be a large pain and not feasible in practice. The amount of changes that span chrome and net/base/media/content/ui etc is very large. If they are moved into a separate repo, 3-sided patches would be required often. At least with WebKit repo, from early on there was a proper API put in place to separate the two. The other modules listed above don't have that, and the class declarations are definitely not stable.

I'm sympathetic to the argument that it's more productive for subteams to work in a separate repo. On the other hand, it also seems that the reason subteams prefer working on separate repos is because they view the chromium tree as being closed too often. In the past, this often would happen because of a mist-match between what the CQ/trybots tested and what closed the tree. This is now infrequent because the infrastructure team have done an impressive job of adding a lot more configs with faster throughput, and more improvements are on the way which should speed things up. In my experience, the biggest reason the tree closes now is because too many patches land without CQ or green try runs. This is simply unacceptable given the size of the team, since any brokenness affect too many people. Whether only non-blink devs are affected, or all devs, seems besides the point for the overall team since this is a problem that we need to fix either way (although I can see that from the pov of blink team, this is paramount :) ).

Personally, I think having blink in the same repo would be better. As mentioned above, there would be no gardening needed, the same CQ that tests the bling patches would also run chromium tests (i.e. fine-grained tracking of which bling change breaks a chrome test, instead of gardeners having to figure this out manually), no 3-sided patches, and easier bisecting. Also, right now only a small number of devs work in both chromium and bling repo. The process to contribute is different, the base libraries are different and so on. If it was easier to jump between codebases, I imagine more people would be inclined to fix bugs in the other codebase. That would seem like another very positive benefit of the fork.

I'm curious, what are the objections? The chromium tree is closed too much compared to the bling tree? The chrome try bots take too long? If these are the reasons, what numbers would the team be happy with before they're ok with a single repo?

I'm also curious what the relative number of commits in each tree is. I have heard previous numbers from the WebKit tree, but I assume that'll be less now since it's just the chromium port.

[Dirk Pranke]

On Sun, Apr 7, 2013 at 4:14 PM, John Abd-El-Malek <j...@chromium.org> wrote:

I completely agree that moving more stuff into a repo that's rolled through DEPs would be a large pain and not feasible in practice. The amount of changes that span chrome and net/base/media/content/ui etc is very large. If they are moved into a separate repo, 3-sided patches would be required often. At least with WebKit repo, from early on there was a proper API put in place to separate the two. The other modules listed above don't have that, and the class declarations are definitely not stable.

I'm sympathetic to the argument that it's more productive for subteams to work in a separate repo. On the other hand, it also seems that the reason subteams prefer working on separate repos is because they view the chromium tree as being closed too often. In the past, this often would happen because of a mist-match between what the CQ/trybots tested and what closed the tree. This is now infrequent because the infrastructure team have done an impressive job of adding a lot more configs with faster throughput, and more improvements are on the way which should speed things up. In my experience, the biggest reason the tree closes now is because too many patches land without CQ or green try runs. This is simply unacceptable given the size of the team, since any brokenness affect too many people. Whether only non-blink devs are affected, or all devs, seems besides the point for the overall team since this is a problem that we need to fix either way (although I can see that from the pov of blink team, this is paramount :) ).

Personally, I think having blink in the same repo would be better. As mentioned above, there would be no gardening needed, the same CQ that tests the bling patches would also run chromium tests (i.e. fine-grained tracking of which bling change breaks a chrome test, instead of gardeners having to figure this out manually), no 3-sided patches, and easier bisecting. Also, right now only a small number of devs work in both chromium and bling repo. The process to contribute is different, the base libraries are different and so on. If it was easier to jump between codebases, I imagine more people would be inclined to fix bugs in the other codebase. That would seem like another very positive benefit of the fork.

I'm curious, what are the objections? The chromium tree is closed too much compared to the bling tree? The chrome try bots take too long? If these are the reasons, what numbers would the team be happy with before they're ok with a single repo?

"blink", not "bling" :)

Also, note that gardening in some form doesn't go away any time soon. At the very least, it gets merged w/ being a chromium sheriff, which is already a difficult job. Beyond that, we need to get to a world where updating test baselines is not done via a tree being red :).

Those two things said, having the code in one repository is almost orthogonal to how we deal with tree closures; obviously, we could just stop worrying about "closing the tree", for example (not that I'd prefer this).

The good reasons for having multiple repos are:

1) the version control software will probably be happier with multiple smaller repos - every large company or software project I've worked on has hit this wall sooner or later.

2) it's possible to isolate changes in one repo from changes in others (so that you only pull in "stable" versions periodically).

I think most of us prefer "keep tip-of-tree stable" as the alternative to (2). (1) is the big concern for me, though I think it is something we should be able to surmount; I don't think Chromium + Blink is *that* big.

 

I'm also curious what the relative number of commits in each tree is. I have heard previous numbers from the WebKit tree, but I assume that'll be less now since it's just the chromium port.

I would roughly guess that Blink will have about half the commits per day that the rest of Chromium has, and a fair amount of those are new test results. It is also possible that we'll start getting more if we are able to get a higher velocity of changes now that we don't have to keep so many ports in sync.

-- Dirk

[Darin Fisher]

I wouldn't be too quick to eliminate the API boundary around Blink.  Blink is still a well defined module.

In the early days of Chrome, we did not have a WebKit API, and that led to a lot of messy usage of WebCore classes and functions from the guts of Chrome.  The WebKit API not only insulated Chrome from the rapidly evolving WebCore, but also helped us rationalize a lot of Chrome's usage of WebCore.  The code became easier to understand.  A similar thing happened when we introduced the Content API.

The main advantages in my mind of merging repositories would be:

1)  Ability to change the Blink API atomically with its consumers.  No 3-sided patches!

2)  Ability to use src/base from Blink.

3)  Easier to merge in new versions of V8, Skia, and other dependencies that are currently shared by Chromium and Blink.

-Darin

On Mon, Apr 8, 2013 at 8:33 AM, Greg Billock <gbil...@chromium.org> wrote:

+1

I can't imagine not regretting the decision to keep them separate. There's simply too much potential code reduction we'll want to take advantage of, landing changes across two repos is too hard, and I don't think anyone is arguing the existing API boundary is in the right place, anyway.

[Ben Goodger (Google)]

+1... good to have a clearly defined API even if you're all living at ToT. For example it has been useful for content to have a public API, it prevents bad dependencies from src/chrome into implementation details etc.

-Ben

[Jói Sigurðsson]

> +1... good to have a clearly defined API even if you're all living at ToT.
> For example it has been useful for content to have a public API, it prevents
> bad dependencies from src/chrome into implementation details etc.

+1 that maintaining the clealry defined API is good.

However, this feels like a weak reason to keep a
source-control-system-imposed separation of the codebases. We have
ways of enforcement (DEPS files) that can provide the same benefit
with none of the downsides of needing to bring Blink in via DEPS. We
might need to improve the DEPS mechanism a little to make it just as
strong, but that is a minor thing.

Cheers,
Jói

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
>
>
>

[Darin Fisher]

Blink is in a separate repository because we wanted a minimal step #1 to

implement the fork.  The long term strategy is undecided.

Blink as a separate repository was super easy to implement. We didn't

have to do anything funky to preserve SVN history, etc.

-Darin

[Greg Billock]

Just FTR, I do agree that maintaining the api boundary is a good plan, but does anyone imagine it not migrating, or other architectural considerations (i.e. /net) not becoming an irresistible pull? Those desires are going to be more painful to navigate in a decoupled repo.

There are existing boundaries within the chromium repo and within the blink repo, and of course we want those (and likely more). What I'm arguing is that creating and maintaining those boundaries is harder cross-repo. For example, doing something like creating the content boundary is difficult enough within a single repo.

[Aaron Colwell]

While I too would love the day when we had a single repo, I'd like to figure out how we can make the current situation a little more bearable. Having to split changes up into 2-5 pieces is annoying, but the fact that it takes a week to land them is demoralizing. How can we get to a place where we are able to do 3-5 DEPS rolls in Chromium & Blink per day? This would significantly decrease the amount of time it takes to land these types of patches which would definitely make any dev that has to regularly do them much happier. 

Here are some potential not completely baked ideas:

- Automatic tree closure if DEPS gets more then N revs beyond ToT in either project.

- Automatic DEPS roll every N revs.

- Add more builders to make identifing DEPS roll candidates faster?

I realize that most people primarily focus on one side or the other of the Chromium <-> Blink divide, but I would really like both camps to be willing to pitch in if tree redness is preventing one side from advancing DEPS.

Aaron

[Dominic Mazzoni]

So, I thought that the plan was to put more responsibility onto the patch author, and revert Blink changes that break *any* tests. That was already basically the policy for Googlers contributing to WebKit, as it is for all Chromium contributors.

Isn't one of the main reasons gardening was so difficult was that non-Googlers landed changes that broke lots of tests, especially Chromium tests that they didn't have any visibility into? Won't that problem be basically solved now, can't we just make WebKit gardeners the same as Chrome sheriffs and make DEPS rolls relatively trivial?

- Dominic

[Dirk Pranke]

That is the plan, but it's still difficult (practically implausible) to get all of the baselines you need updated before committing a patch, since we don't have fully try server coverage for all platforms, the try servers don't accept binary patches, and you can't easily pull the results from them :).

Once those things are fixed (bugs are currently filed for all of them), we should be *much* closer to keeping the tree green all the time, and at that point DEPS rolls should be closer to routine.

-- Dirk

[Rick Byers]

I suggest we avoid coupling the concerns of isolation (tree closures, etc.) with this discussion of repository structure. 

If we were in a single repo, we could still have a web platform team branch with its own builders and gardening rotation for merging into trunk, while avoiding the multi-sided patch pain. IMHO, using separate repositories is a poor way to achieve this isolation - for one, branching strategies can be adapted over time to meet team needs much easier than repo structure can.

Rick