Proposal: New directory structure for Blink

[Adam Barth]

We're making good progress removing unused code from Blink.  We should think some about how we'd like to organize the code we're actually using.  Below is one proposal:

Directories

public/
 - platform/ (previously Source/Platform/chromium/public)
 - client/ (previously Source/WebKit/chromium/public)
wtf/ (previously Source/WTF/wtf)
yarr/ (previously Source/JavaScriptCore/yarr, to be removed)
platform/ (previously Source/WebCore/platform)
core/ (previously Source/WebCore)
modules/ (previously Source/WebCore/Modules)
 - [...] (many sub-directories)

bindings/ (previously Source/WebCore/bindings/v8)
client/ (previously Source/WebKit/chromium/src)
testing/ (previously Source/WebCore/testing)
shell/ (previously Tools/DumpRenderTree/chromium, to be replaced with content_shell)
tests/
 - unit/ (previously Source/WebKit/chromium/tests and Tools/TestWebKitAPI/Tests)

 - web/ (previously LayoutTests)
 - perf/ (previously PerformanceTests)
 - manual/ (previously ManualTests)
tools/ (previously Tools)

Discussion

Each directory can depend on the directory above it in the list, with the exception of public/client.  The only directories that can depend on public/client are client, shell, tests, and tools.  Additionally, sub-directories of modules cannot depend on each other.

Each top-level directory should be buildable as a separate component in the component build.  Similarly, each subdirectory of modules should be buildable as a separate component.  However, we’ll likely build client and everything it depends on as one giant component for the time being.  Over time, we can clean up the dependencies and actually build the components separately.

Feedback welcome!

Adam

[Eric Seidel]

Could you post this in a chromium.org doc so that we could make inline
comments? :) I worry email is a difficult medium for this.

[Adam Barth]

Sure:

https://docs.google.com/document/d/1BCRkA5ERZtBMXkXB1aMFKKLl5LR_Wq5LpAHtlEf1j_c/edit?usp=sharing

Adam

[Alexis Menard]

This change means that any possible sharing with WebKit is gone.
People working on some parts of WebCore where there is interest to
have them both in WebKit and Blink will have to duplicate work already
but the directory structure change will make it harder. I know with
the time probably both trees will diverge too much so it will happen
anyway. I just want to reinforce it.

--
Software Engineer @
Intel Open Source Technology Center

[PhistucK]

That makes me sad as well.

☆PhistucK

[Thiago Farina]

On Fri, Apr 5, 2013 at 8:32 AM, Alexis Menard <ale...@webkit.org> wrote:
> This change means that any possible sharing with WebKit is gone.

May that isn't one of the first goals? I don't see why Blink should
share code with WebKit in the first place though, it's a separate
(new) project with its own guidelines and they diverge from the WebKit
project hence the split, no?


--
Thiago

[Thiago Farina]

On Fri, Apr 5, 2013 at 8:37 AM, PhistucK <phis...@gmail.com> wrote:
> That makes me sad as well.
>

What makes you sad? The lack of code sharing or the new directory structure?

--
Thiago

[PhistucK]

The lack of code sharing.

The goal is indeed to maintain a rendering engine that fit Chrome in a better way and without compromises.

However, as long as it is easy to share code, bug fixes can be shared between the two.

Currently, WebKit has more contributors (companies or individuals) than Blink.

I am not sure how much bug fixes, feature polishes and similar were contributed by engineers other than Google, but losing that so soon is sad, mainly due to loss of bug fixes in Blink (but also in WebKit. It is still being used in lots of other places, including places where Blink would probably never (or not now, at least) be used, like iOS).

☆PhistucK

[Jarred Nicholls]

On Fri, Apr 5, 2013 at 9:02 AM, PhistucK <phis...@gmail.com> wrote:

The lack of code sharing.

The goal is indeed to maintain a rendering engine that fit Chrome in a better way and without compromises.

However, as long as it is easy to share code, bug fixes can be shared between the two.

Currently, WebKit has more contributors (companies or individuals) than Blink.

I am not sure how much bug fixes, feature polishes and similar were contributed by engineers other than Google, but losing that so soon is sad, mainly due to loss of bug fixes in Blink (but also in WebKit. It is still being used in lots of other places, including places where Blink would probably never (or not now, at least) be used, like iOS).

A lot of bugs were usually in a specific platform implementation anyways, and not in the core layers; but that's a blanketed statement with no real data on hand to support it.

[PhistucK]

Even not "a lot" core bugs are a win. Any bug fix is a win for a platform.

(Good refactoring and reordering are also a win, of course, but it is still sad and a bit scary with such consequences)

☆PhistucK

[PhistucK]

*"a lot" of

☆PhistucK

[Dominic Mazzoni]

On Fri, Apr 5, 2013 at 4:32 AM, Alexis Menard <ale...@webkit.org> wrote:

This change means that any possible sharing with WebKit is gone.
People working on some parts of WebCore where there is interest to
have them both in WebKit and Blink will have to duplicate work already
but the directory structure change will make it harder.

I don't buy this argument at all. It would take someone less than an hour's work to write a script that does most of the work of rewriting WebKit patches into Blink patches and vice-versa, and I'm sure we'd happily accept such a script in the codebase. A naive regex would probably get you 80% of the way there, but since Blink carries the entire WebKit history, a git-based tool that does a 3-way merge could potentially get you 99% of the way there and handle other refactoring, not just easy stuff like directory renaming.

I'm sure that if this were to become an actual burden, someone would write this script.

- Dominic

[PhistucK]

Then I wish that it would not prevent sharing and that sharing would happen.

☆PhistucK

[Jonathan Dixon]

I have written such a tool (well, sed script) before and it worked pretty well (that was for java though - bit simpler)

However -

If the fix is small it can probably be ported over manually just as easily as via patch.

If the fix is large, the removal of ifdef all over the code will pretty quickly stop the patch applying cleanly anyway.

YMMV..

[Jarred Nicholls]

On Fri, Apr 5, 2013 at 11:40 AM, PhistucK <phis...@gmail.com> wrote:

Even not "a lot" core bugs are a win. Any bug fix is a win for a platform.

I agree.  However, I share Adam's prediction that code in all layers - include core - will start to diverge quite significantly over time and the difference in directory structure won't matter so much.  There are plans on the table to do full include pathing from "src", overtime adopt Chromium coding styles, etc.  I think it's short-sighted to disagree with chromium-consistent directory restructuring when it is arguably best for the project long term.

[PhistucK]

I never disagreed. I just said it is sad. :(

☆PhistucK

[Dimitri Glazkov]

On Fri, Apr 5, 2013 at 9:02 AM, PhistucK <phis...@gmail.com> wrote:
> I never disagreed. I just said it is sad. :(

Don't be sad. A big step is often scary, but if you don't make them,
you're stuck in the past.

:DG<

[Nayan K]

I like the layout of new directory structure, however, it would be nice if we start moving to new directory structure after a week or two. This would help WebKit/blink contributors to finish their currently *in-progress* feature implementation or bug fixes in both WebKit and blink with ease.

Thanks,

[James Robinson]

I think that'll just be a matter of an easy git rebase.  I recently did a directory move of a smaller scale (~110k lines) in chromium and I think it was fairly painless for developers to resolve their in-progress changes across the directory move.

- James

[Adam Barth]

I'm not in a huge rush to move the directories around.  Certainly cleaning up the code itself is a higher priority.  A week or two sounds like a reasonable time frame.

Note: We'll probably need to set up forwarding headers for the public API headers.  We might want to wait to remove the forwarding headers until we move to src/blink so that we only need to rewrite all the includes in the main Chromium repo once.

Adam

[Chris Rogers]

On Fri, Apr 5, 2013 at 9:00 AM, Jarred Nicholls <jarred....@gmail.com> wrote:

On Fri, Apr 5, 2013 at 11:40 AM, PhistucK <phis...@gmail.com> wrote:

Even not "a lot" core bugs are a win. Any bug fix is a win for a platform.

I agree.  However, I share Adam's prediction that code in all layers - include core - will start to diverge quite significantly over time and the difference in directory structure won't matter so much.  There are plans on the table to do full include pathing from "src", overtime adopt Chromium coding styles, etc.  I think it's short-sighted to disagree with chromium-consistent directory restructuring when it is arguably best for the project long term.

"overtime adopt Chromium coding styles, etc" -- Really?

As others have pointed out, the directory structure is not that much of an issue because scripts can be written...  Full include pathnames can also be dealt with, but systematically converting to chromium style would be a very bad thing in my opinion.

[Thiago Farina]

On Sat, Apr 6, 2013 at 4:03 PM, Chris Rogers <cro...@google.com> wrote:
> "overtime adopt Chromium coding styles, etc" -- Really?
>
> As others have pointed out, the directory structure is not that much of an
> issue because scripts can be written... Full include pathnames can also be
> dealt with, but systematically converting to chromium style would be a very
> bad thing in my opinion.
>

Why? If/when the repo is merged into chromium trunk I'd expect it to
follow our coding style (see the cc/ example when they move from
WebKit into chromium) and not diverge from the rest of the code base.
Otherwise it's better to have it separately into its own repository.

With clang-format -style=Chromium, that shouldn't a big problem to fix.

--
Thiago

[Daniel Cheng]

Certainly, it's fixable. But that doesn't mean we should. Doing something like that would make git blame useless.

Daniel

[Thiago Farina]

On Sat, Apr 6, 2013 at 4:07 PM, Daniel Cheng <dch...@chromium.org> wrote:
> Certainly, it's fixable. But that doesn't mean we should. Doing something
> like that would make git blame useless.
>

I'm not saying we should (decision is not mine ;)), although I'd like
it if it happens. I don't care too much though :)

--
Thiago

[Eric Seidel]

The blink coding style is posted here:
http://www.chromium.org/blink/coding-style

And will be enforced (shortly) by check-webkit-style as part of PRESUBMIT.py:
https://code.google.com/p/chromium/issues/detail?id=227256

I think we have an opportunity to improve on WebKit's style
enforcement and consistency, but we have no plans at this time to
convert the whole codebase to a separate style. :)

[John Yani]

> If/when the repo is merged into chromium trunk

Are you talking about merging blink into chromium repo?

[Eric Seidel]

No. :)

To recap:

- We plan to reorganize Blink's internal directory structure to better
match how the code is factored. (Please see/comment-on Adam's doc.)
This will take a while, but is likely to start in the next couple
weeks.
- We plan to move Blink's checkout location in a Chromium checkout
from third_party/WebKit to /blink. This will be pre-announced on this
list, and may not be for a few weeks due to the disruptive potential
of such a large move (both for other Chromium projects and Chromium's
infrastructure team).
- We have no plans to ever check Blink's sources directly into
Chromium's repository. (This would be hugely disruptive to both Blink
and Chromium. Both repositories each have over 100k commits and
neither would like to lose/move that history.)

I hope that clears things up. I'm also happy to answer questions in
#blink, but we should probably leave this thread for Adam's proposed
directory restructuring.

[Peter Kasting]

On Sat, Apr 6, 2013 at 12:07 PM, Daniel Cheng <dch...@chromium.org> wrote:

Certainly, it's fixable. But that doesn't mean we should. Doing something like that would make git blame useless.

Mass style conversion isn't on the table right now, so this is sort of moot.  But for reference, I hate this argument.

If we do change style at some point, it will be because we perceive there to be some benefit.  The magnitude of that benefit becomes very large if we assume the life of the project is long.  Whereas the importance of git blame against the old code will steadily diminish over time.  Much like refactoring, this is an argument that one-time costs for long-term gains is a good tradeoff.

Furthermore, it's already the case that you have to hop past multiple stops most of the time when spelunking code's history, due to other changes or cleanups that occur over time.  Adding one more is not really a big deal.

In the code I own in Chromium I've encouraged aggressive cleanups, refactorings, etc. and I've never regretted that decision.

PK

[Adam Barth]

On Sat, Apr 6, 2013 at 12:37 PM, Eric Seidel <ese...@chromium.org> wrote:

No. :)

To recap:

- We plan to reorganize Blink's internal directory structure to better
match how the code is factored.  (Please see/comment-on Adam's doc.)
This will take a while, but is likely to start in the next couple
weeks.
- We plan to move Blink's checkout location in a Chromium checkout
from third_party/WebKit to /blink.  This will be pre-announced on this
list, and may not be for a few weeks due to the disruptive potential
of such a large move (both for other Chromium projects and Chromium's
infrastructure team).
- We have no plans to ever check Blink's sources directly into
Chromium's repository.  (This would be hugely disruptive to both Blink
and Chromium.  Both repositories each have over 100k commits and
neither would like to lose/move that history.)

I'm not sure we want to commit to never merging the repos, but it's not something we're currently planning to do.

Adam

[John Yani]

> I'm not sure we want to commit to never merging the repos, but it's not something we're currently planning to do.
>
> Adam

What about going in the opposite direction? I.e. separating parts of blink into separate projects/repos?
Is one the goals of blink to squash things, e.g. making v8 and WebCore more coupled?

[Thiago Farina]

On Sat, Apr 6, 2013 at 6:21 PM, John Yani <van...@gmail.com> wrote:
> What about going in the opposite direction? I.e. separating parts of blink
> into separate projects/repos?

How that would be helpful? instead of hacking in a single repository,
now you would have to checkout a bunch, this doesn't sound an
improvement.


--
Thiago

[John Yani]

> > What about going in the opposite direction? I.e. separating parts of blink
> > into separate projects/repos?
> How that would be helpful? instead of hacking in a single repository,
> now you would have to checkout a bunch, this doesn't sound an
> improvement.

I think one of the great reasons is that people could reuse parts of it in other projects.

For example they use v8 in node, a great project for server-side applications. And (theoretically) it would be easier for them to track changes in v8 if it remained in a separate repo.

Who knows what projects could emerge if parts of WebKit were developed out of the main repo.

I understand that it's obviously not Google's policy to make things easier for projects that compete with theirs ones (probably Apple-Google competition was not the last reason to fork WebKit), but I'm still naive to believe in Google's "don't be evil" slogan.

[Adam Barth]

On Sat, Apr 6, 2013 at 2:21 PM, John Yani <van...@gmail.com> wrote:

> I'm not sure we want to commit to never merging the repos, but it's not something we're currently planning to do.
>
> Adam

What about going in the opposite direction? I.e. separating parts of blink into separate projects/repos?

That's certainly something that's worth discussing on this mailing list.  One split that folks have talked about in the past is separating the LayoutTests from the actual code because the LayoutTests are large.  One downside of that split is that we'd lose the ability to make changes to code and tests atomically, which is something we do frequently.

A middle ground that might be worth more the PNG results files to a separate repo and then to refer to them by their hash value.  That way you could change the results atomically, assuming the PNG was already in the PNG repo.

Is one the goals of blink to squash things, e.g. making v8 and WebCore more coupled?

We're hoping to better integrate WebCore with V8 while maintaining the V8 API.  The V8 API has served us well for a while, and it's let other projects (like node.js) use V8.

On Sat, Apr 6, 2013 at 3:09 PM, John Yani <van...@gmail.com> wrote:

I think one of the great reasons is that people could reuse parts of it in other projects.

For example they use v8 in node, a great project for server-side applications. And (theoretically) it would be easier for them to track changes in v8 if it remained in a separate repo.

Who knows what projects could emerge if parts of WebKit were developed out of the main repo.

I'm hopeful that folks will be able to develop Blink modules outside of the Blink repository.  The dependency structure between the code in the "modules" directory and the rest of the code is meant to encourage that.  We'll probably start off by having all the modules in the main repo while we iterate on the modules system, however.

I understand that it's obviously not Google's policy to make things easier for projects that compete with theirs ones (probably Apple-Google competition was not the last reason to fork WebKit), but I'm still naive to believe in Google's "don't be evil" slogan.

Please refrain from raising non-technical issues on this mailing list.  This mailing list is about the technical nuts and bolts of developing Blink.

Adam

[William Chan (陈智昌)]

On Sat, Apr 6, 2013 at 12:37 PM, Eric Seidel <ese...@chromium.org> wrote:

No. :)

To recap:

- We plan to reorganize Blink's internal directory structure to better
match how the code is factored.  (Please see/comment-on Adam's doc.)
This will take a while, but is likely to start in the next couple
weeks.
- We plan to move Blink's checkout location in a Chromium checkout
from third_party/WebKit to /blink.  This will be pre-announced on this
list, and may not be for a few weeks due to the disruptive potential
of such a large move (both for other Chromium projects and Chromium's
infrastructure team).

Are we planning to discuss this at some point? Or it it already decided that it'll move to src/blink? It reminds me of the old skia discussion: https://groups.google.com/forum/?fromgroups=#!topic/chromium-dev/s-ay9CrpJ-M. I don't believe we ever achieved consensus there.

Will there be discussions at some point about reuse of src/base and other such libraries from within blink? Or, since blink is a separate repo (and presumably separately buildable from the chromium repo), is the plan not to use src/base at all?

Not voicing any opinions. Just curious since I haven't seen these topics discussed anywhere (but I'm likely to have missed it somewhere in the swarm of emails and docs).

[Jarred Nicholls]

On Sat, Apr 6, 2013 at 9:03 PM, William Chan (陈智昌) <will...@chromium.org> wrote:

On Sat, Apr 6, 2013 at 12:37 PM, Eric Seidel <ese...@chromium.org> wrote:

No. :)

To recap:

- We plan to reorganize Blink's internal directory structure to better
match how the code is factored.  (Please see/comment-on Adam's doc.)
This will take a while, but is likely to start in the next couple
weeks.
- We plan to move Blink's checkout location in a Chromium checkout
from third_party/WebKit to /blink.  This will be pre-announced on this
list, and may not be for a few weeks due to the disruptive potential
of such a large move (both for other Chromium projects and Chromium's
infrastructure team).

Are we planning to discuss this at some point? Or it it already decided that it'll move to src/blink? It reminds me of the old skia discussion: https://groups.google.com/forum/?fromgroups=#!topic/chromium-dev/s-ay9CrpJ-M. I don't believe we ever achieved consensus there.

Will there be discussions at some point about reuse of src/base and other such libraries from within blink? Or, since blink is a separate repo (and presumably separately buildable from the chromium repo), is the plan not to use src/base at all?

I'd love to see this discussion occur (on a separate thread) to see what others think about src/base as a dependency layer.  I think there are obvious pros and cons to such coupling, but it'd be good to sit around the camp fire and talk it out sometime soon.

[Adam Barth]

On Sat, Apr 6, 2013 at 6:03 PM, William Chan (陈智昌) <will...@chromium.org> wrote:

On Sat, Apr 6, 2013 at 12:37 PM, Eric Seidel <ese...@chromium.org> wrote:

No. :)

To recap:

- We plan to reorganize Blink's internal directory structure to better
match how the code is factored.  (Please see/comment-on Adam's doc.)
This will take a while, but is likely to start in the next couple
weeks.
- We plan to move Blink's checkout location in a Chromium checkout
from third_party/WebKit to /blink.  This will be pre-announced on this
list, and may not be for a few weeks due to the disruptive potential
of such a large move (both for other Chromium projects and Chromium's
infrastructure team).

Are we planning to discuss this at some point? Or it it already decided that it'll move to src/blink? It reminds me of the old skia discussion: https://groups.google.com/forum/?fromgroups=#!topic/chromium-dev/s-ay9CrpJ-M. I don't believe we ever achieved consensus there.

I think it's clear we need to move the repo somewhere other than it is currently (mostly because it's currently in a directory called "WebKit" and it isn't WebKit).  Presumably we'll call the new directory "blink" because that's the name of the project.  The remaining question is whether to put that directory in src/third_party/blink or in src/blink.  My feeling is that we want to treat the codebase more as part of the Chromium project than we have in the past and putting the code in src/blink rather than src/third_party/blink will make that easier psychologically.

That said, I don't think we've really talked it though completely, especially not in a forum like this mailing list where everyone can participate.  If you're interested in starting a thread on this topic, I'm more than happy to discuss it in more detail.

 

Will there be discussions at some point about reuse of src/base and other such libraries from within blink? Or, since blink is a separate repo (and presumably separately buildable from the chromium repo), is the plan not to use src/base at all?

That's also a discussion worth having on this mailing list.  Our current plan is not to use base, but I also don't think we've talked it through thoroughly.

Not voicing any opinions. Just curious since I haven't seen these topics discussed anywhere (but I'm likely to have missed it somewhere in the swarm of emails and docs).

We had to made some decisions about what to do on the first day in private because we wanted to have a clear idea of what the first day would look like before we were ready to announce.  These longer-term topics haven't been discussed as thoroughly because we realized we didn't have to figure them out for the first day.

 

Adam

[Dirk Pranke]

On Sat, Apr 6, 2013 at 12:37 PM, Eric Seidel <ese...@chromium.org> wrote:

> On Sat, Apr 6, 2013 at 12:22 PM, John Yani <van...@gmail.com> wrote:
>> If/when the repo is merged into chromium trunk
>
> Are you talking about merging blink into chromium repo? 

- We have no plans to ever check Blink's sources directly into
Chromium's repository.  (This would be hugely disruptive to both Blink
and Chromium.  Both repositories each have over 100k commits and
neither would like to lose/move that history.)

I hope that clears things up.  I'm also happy to answer questions in
#blink, but we should probably leave this thread for Adam's proposed
directory restructuring.

Clearly there is some talk and some desire to have things be in a single repo, so saying we have no plans to ever do this is too strong. This has been talked about at some length on Alexander's thread, and I don't think we know the answer yet. I'll just add that it's probably unlike that we'd merge repos before moving one or both of them to Git (where, it is believed, we can keep both sets of history without things being nearly as disruptive).

-- Dirk 

[Silvia Pfeiffer]

I am happy that we're getting more diversity in rendering engines again! It will make for a better Web and more competition in features - I can already see that some deadlocks on new features have been removed on both sides. Good work all around!

Also: I like a simpler directory structure. I would prefer the directory names to be less generic names and more obvious to newcomers, though - e.g. "v8bindings" instad of "bindings" alone. It's more verbose, but easier to understand.

Finally: I am planning to write patches that can be applied to both code bases and I'd be happy for any script that can help make that easier. Something with git cherry-pick but some automated path conversion would likely help.

Silvia.