Merge of Chromium and Blink repositories

[Jochen Eisinger]

tl;dr

• Blink will be merged into chromium in //src/third_party/WebKit

• The merge will happen Wednesday, September 23, sometime in the GMT+1 morning / midnight PST / late afternoon in TOK

• During that time, all trees will be closed, and the chromium and blink repositories will be read only.

• The merge commit will include the entire Blink history with the exception of the history of pixel results

• Existing branches and codereviews in Blink will have to be manually transferred

• There’s a FAQ here: http://dev.chromium.org/blink/blink-post-merge-faq

Hey all,

the Blink merge is finally close: Wednesday next week, September 23, during the EMEA morning hours (PST midnight, late afternoon in TOK). Expect major disruptions during that time. All trees will be closed, automated bots will be taken offline, and the chromium and blink repositories will be read-only.

After the merge, checkouts created with either fetch chromium or fetch blink will work, however, new checkouts should be created using fetch chromium going forward.

At this point, large parts of the developer workflow are already changed to the post-merge world, e.g., Blink CLs already now go through the chromium commit-queue, the Blink tree already uses the Chromium tree status. However, some things will change, most notably, local branches and in-flight code reviews will not be automatically migrated. Instead, you’ll have to recreate those branches in your chromium checkout, and upload new code reviews.

Also, all upstream branches but the current stable and beta branch will not be merged. If you check out an earlier branch, you’ll get a split checkout as before. Other branches, like the dart branch will have to be recreated.

Please let us know if you have any concerns!

best

-jochen, paweł, and primiano

[Юрий Соловьёв]

Sometimes I look at https://blink.lc/blink/log/?showmsg=1 and https://blink.lc/chromium/log/?showmsg=1 for some interesting commits, what will happen to them?

пятница, 18 сентября 2015 г., 12:19:31 UTC+3 пользователь Jochen Eisinger написал:

[Primiano Tucci]

The first one will remain stale and you'll look only at the second one, as all the commits will happen in chromium

[Jochen Eisinger]

https://blink.lc/chromium/log/third_party/WebKit?showmsg=1 will work after the merge

[Anthony Berent]

How will merges to the Blink M46 release branch work after this?

[Jochen Eisinger]

Since M46 will also be updated, you can use the same process as for chromium (git drover)

best

-jochen

[Daniel Bratell]

On Fri, 18 Sep 2015 11:19:17 +0200, Jochen Eisinger <joc...@chromium.org> wrote:

the Blink merge is finally close: Wednesday next week, September 23, during the EMEA morning hours (PST midnight, late afternoon in TOK). Expect major disruptions during that time. All trees will be closed, automated bots will be taken offline, and the chromium and blink repositories will be read-only.

Is updating the omahaproxy.appspot.com scripts on the list? That one has seemed a bit unmaintained (right now the true_branch field is broken).

And I am, like everyone else, really looking forward to a world post-merge so many thanks for pushing this project all the way!

/Daniel

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

[Jochen Eisinger]

Filed crbug.com/533323 for omahaproxy, thanks

[Primiano Tucci]

On Fri, Sep 18, 2015 at 10:38 AM, Anthony Berent <abe...@chromium.org> wrote:

How will merges to the Blink M46 release branch work after this?

I've just added a section to the FAQs for this. Thanks for pointing out. 

[Charlie Andrews]

I for one am pretty darn excited about this. Thanks to everyone who helped move it forward!

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

--

Charlie Andrews |

Software Engineer |

char...@google.com

 

[Mikhail Naganov]

That's exciting!

What will happen to webkit_version.h after the merge? Will WEBKIT_SVN_REVISION switch to Chromium's commit number, or will  it just be stuck at the last Blink revision?

[Jochen Eisinger]

it will point at Chromium's commit position. In the future, we might want to drop it

[Elliott Sprehn]

Will we be generating a commit position for every webkit change that's merged?

[Jochen Eisinger]

New commits to third_party/WebKit after the merge will get the same kind of commit numbers as chromium does. Old commits will still have their svn number, but won't get a new commit position.

[Primiano Tucci]

also, there will be just one merge commit ever in total (and that one will get a consistent commit position tag, that is chromium ToT + 1)

[Elliott Sprehn]

So if I'm using git blame and git log will WebKit's history still be there?

[Primiano Tucci]

yes, that was the major point if the merge + history rewrite

[Primiano Tucci]

Heads up.

We are gradually pushing the blink git history into chromium's branch ignore/blink_merge_cache.

You should expect the chromium repo size to gradually grow in the course of the next 3 days of about +1.5 GB

This is to allow the build infrastructure and developers repos to gradually digest the upcoming flood of blink objects. 

This branch will not affect your workflow (well, until the actual merge day) and, as the name suggest, you should completely ignore it.
It serves purely as a temporary git GC root and will be deleted after the merge. 

Please shout loudly if you are sheriffing / trooping see unexpected timeouts or git errors during the bot_update steps.

Primiano

[Dale Curtis]

As a data point, this morning git pull took ~20 minutes before failing with the following:

error: RPC failed; result=18, HTTP code = 200

fatal: The remote end hung up unexpectedly

fatal: protocol error: bad pack header

An immediate retry took a little while longer (~10 minutes) and completed gclient sync w/o issue.

- dale

[Jochen Eisinger]

Yeah, if downloading objects takes unusually long, aborting and restarting will probably help.

However, once the local operations start ("resolving deltas") aborting does not help - please wait until it's completed. Note that this can take especially long on Windows.

best

-jochen

[Scott Graham]

Yeah, resolving deltas was ~3 hours for me yesterday, but I didn't encounter any errors. Looks like today's might run longer based on current progress.

It's only downloading objects at around 50KiB/s too, which is << than my downstream connection. Is that expected capping? Alternatively, do we think a `fetch chromium` to a new tree on Thursday might be a better tactic overall?

(Exciting anyway! :)

[Jochen Eisinger]

On Tue, Sep 22, 2015 at 9:01 PM Scott Graham <sco...@chromium.org> wrote:

Yeah, resolving deltas was ~3 hours for me yesterday, but I didn't encounter any errors. Looks like today's might run longer based on current progress.

It's only downloading objects at around 50KiB/s too, which is << than my downstream connection. Is that expected capping? Alternatively, do we think a `fetch chromium` to a new tree on Thursday might be a better tactic overall?

no, retrying might give you a better download rate.

the more you update, the faster the individual updates will be. so running fetch chromium after the merge will probably not save you a lot. 

[John Abd-El-Malek]

FWIW, as a datapoint I tried this today. I synced a checkout that I hadn't synced in a week on Windows. It took about 3 minutes for "Receiving objects" and then "Resolving deltas" took about 15 minutes.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CANHK6RaVzHvh-7NYShag5EqXVXCkUSBFTMm1N8cs3zqMDnaXFw%40mail.gmail.com.

[Brandon Jones]

Adding my datapoint: Similar to John I synced a Windows machine that hadn't been updated in a week, but it took significantly longer. I was using "git pull --rebase" to pull the code. The first attempted spent ~20 minutes int "Receiving Objects", then failed. This is what the console output looked like:

remote: Counting objects: 1664622, done

remote: Finding sources: 100% (2051851/2051851)

efrror: RPC failed; result=18, HTTP code = 200.52 MiB | 381.00 KiB/s

atal: The remote end hung up unexpectedly

fatal: early EOF

fatal: index-pack failed

Re-running succeeded, though again it took ~20 minutes to Receive Objects, and an unknown amount of time after that to Resolve Deltas (I was in a meeting when it completed).

[Primiano Tucci]

Interesting. Unfortunately some of the windows bots took very long time after the last chunk I pushed: ~1-2h as opposite to ~11m of Linux bots. I wonder if that is due to the git version (jam@ what is your git --version?).

I'm really sorry for the inconvenience, I did not expect git on windows to be *that* slower compared to linux and was hoping this gradual push to go much more unnoticed.

On the good side, right now we are at 90% push state (% of history of blink pushed into chromium). I'll wait another hour before giving a final kick as the last chunks caused the waterfall/CQ to be throttled in the past hour.

After that, there will be the only final push tomorrow EMEA morning during the tree closure to recover the last 24h of blink.git activity.

[John Abd-El-Malek]

On Tue, Sep 22, 2015 at 3:22 PM, Primiano Tucci <prim...@chromium.org> wrote:

Interesting. Unfortunately some of the windows bots took very long time after the last chunk I pushed: ~1-2h as opposite to ~11m of Linux bots. I wonder if that is due to the git version (jam@ what is your git --version?).

D:\src\chrome3\src>git --version

git version 1.9.5.chromium.6 

This is on a Z840 with a PCIe SSD (with things that slow it down not running).

[Scott Graham]

Wow, mine's still resolving deltas (at 41%) after starting git fetch origin around noon. It only downloaded 780M of objects, so I have no idea what it's up to.

git version 1.9.5.chromium.6, Z620, Samsung 840 Pro SSD. Time for an upgrade I guess!

One time is no big deal, but hopefully it doesn't presage extraordinarily slow operations on src.git post-merge.

[Stefan Zager]

Scott, would you mind posting the output from these commands, when run your repository:

> git config core.deltaBaseCacheLimit

> git config pack.threads

The performance of "resolving deltas" is highly sensitive to those variables.

[Scott Graham]

d:\>git config core.deltaBaseCacheLimit

1G

d:\>git config pack.threads

d:\>

(Who needs all those cores?)

[Stefan Zager]

On Tue, Sep 22, 2015 at 4:44 PM Scott Graham <sco...@chromium.org> wrote:

d:\>git config core.deltaBaseCacheLimit

1G

d:\>git config pack.threads

d:\>

(Who needs all those cores?)

Hmm, that looks legit.  According to the git docs for pack.threads, "specifying 0 will cause Git to auto-detect the number of CPU’s and set the number of threads accordingly."  Maybe check the task manager, see if it's using multiple threads?

[Scott Graham]

It's only using 3 threads in one process, and they're all running at about 30% of a core, so effectively single threaded.

I tried forcing pack.threads to 32, but alas when I restarted the fetch I went back to pulling (the rest of the?) objects at 50KiB/s so I won't know if that helped for a few hours.

[Primiano Tucci]

If fetching from the remote is slow, killing it and restarting has good chances to make it restart faster.

[Stefan Zager]

If I remember correctly, there's some empirical evidence that increasing pack.threads above 3 or 4 has rapidly diminishing returns, so I don't think that's your problem.  30% is a low number, but I don't think that adds up to 4+ hours running time.

It's possible that the number of existing pack files in your .git\objects\ will affect unpack time; maybe it has to process everything you have in addition to everything it receives?  Actually, now that the words are on the page, I do believe that's true.  If you have an excessive number of pack files ("excessive" being defined in the git code as >50), I'm pretty sure that will slow things down.

[Scott Graham]

Thanks. I have 3.5G in .git\objects\pack in 11 files, so I guess that's reasonable.

[Bruce]

Why is the "Receiving objects" stage CPU bound? I would expect that this stage would be download bandwidth limited but it seems pretty clear that the 300-500 KiB/s I'm getting is because git.exe is using an entire core (source: procexp.exe).

I grabbed an ETW trace but without symbols that's not going to be much use. Anybody know if they are available, or do I need to build my own version?

Oh - it just spiked to 10 MiB/s. Still using an entire core, but now I'm not as annoyed.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.

[Primiano Tucci]

The truth is that git is lying to you :)

the output makes you believe that it first downloads the packs and then indexes them. 

The truth is that there is no reason why you should wait for the full pack before indexing it. In fact, the pack file IS being indexed while it is downloaded. This is the reason while you see high cpu usage even during download (you should see an index-pack thread burning cpu power even during the download).

I think that this is just to avoid the awkward output where the upper bound of the "resolving delta X/Y" grows over time (you cannot know the upper bound until you finish download all the pack)

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

[Yoav Weiss]

By "git drover" did you mean following the steps in the online doc which AFAICT mean landing changes manually, or is there a better way? 

On Fri, Sep 18, 2015 at 11:40 AM, Jochen Eisinger <joc...@chromium.org> wrote:

Since M46 will also be updated, you can use the same process as for chromium (git drover)

best

-jochen

On Fri, Sep 18, 2015 at 11:38 AM Anthony Berent <abe...@chromium.org> wrote:

How will merges to the Blink M46 release branch work after this?

On Fri, 18 Sep 2015 at 10:35 Jochen Eisinger <joc...@chromium.org> wrote:

https://blink.lc/chromium/log/third_party/WebKit?showmsg=1 will work after the merge

On Fri, Sep 18, 2015 at 11:32 AM Primiano Tucci <prim...@chromium.org> wrote:

The first one will remain stale and you'll look only at the second one, as all the commits will happen in chromium

On Fri, Sep 18, 2015 at 10:27 AM, Юрий Соловьёв <biohaz...@gmail.com> wrote:

Sometimes I look at https://blink.lc/blink/log/?showmsg=1 and https://blink.lc/chromium/log/?showmsg=1 for some interesting commits, what will happen to them?

пятница, 18 сентября 2015 г., 12:19:31 UTC+3 пользователь Jochen Eisinger написал:

tl;dr

• Blink will be merged into chromium in //src/third_party/WebKit

• The merge will happen Wednesday, September 23, sometime in the GMT+1 morning / midnight PST / late afternoon in TOK

• During that time, all trees will be closed, and the chromium and blink repositories will be read only.

• The merge commit will include the entire Blink history with the exception of the history of pixel results

• Existing branches and codereviews in Blink will have to be manually transferred

• There’s a FAQ here: http://dev.chromium.org/blink/blink-post-merge-faq

Hey all,

the Blink merge is finally close: Wednesday next week, September 23, during the EMEA morning hours (PST midnight, late afternoon in TOK). Expect major disruptions during that time. All trees will be closed, automated bots will be taken offline, and the chromium and blink repositories will be read-only.

After the merge, checkouts created with either fetch chromium or fetch blink will work, however, new checkouts should be created using fetch chromium going forward.

At this point, large parts of the developer workflow are already changed to the post-merge world, e.g., Blink CLs already now go through the chromium commit-queue, the Blink tree already uses the Chromium tree status. However, some things will change, most notably, local branches and in-flight code reviews will not be automatically migrated. Instead, you’ll have to recreate those branches in your chromium checkout, and upload new code reviews.

Also, all upstream branches but the current stable and beta branch will not be merged. If you check out an earlier branch, you’ll get a split checkout as before. Other branches, like the dart branch will have to be recreated.

Please let us know if you have any concerns!

best

-jochen, paweł, and primiano

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Jochen Eisinger]

Right, it's a number of git commands you'll need to enter manually

[Steven Bennetts]

I've noticed extremely slow 'git rebase' performance since syncing this morning (~30 minutes on a z620). I assume it's related to the Blink merge.

Any suggestions for speeding up 'git rebase'? I have about a dozen branches I'll need to update at some point. I guess I can just cherry-pick the changes instead.

It might also be nice to send a PSA to warn people.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Bruce Dawson]

I briefly looked at the git trace that I captured and, for the ~20 s or so that I looked at more than half of the CPU time is in KernelBase.dll!FindNextFileW and most of the rest of the CPU time was in KernelBase.dll!GetFileAttributesW and KernelBase.dll!GetFileAttributesExW.

It's hard to tell whether this is the fault of Windows (why does DirEnum take so much CPU time) or git (why is git enumerating D:\src\chromium\src\.git\objects\pack 91,480 times in 20 seconds?). Yes, git really is enumerating the same directory that many times, and no other directories.

It seems clear that fixing this would make this particular git operation orders of magnitude faster, but I'm not sure if it's worth trying to fix.

[Primiano Tucci]

30 mins sounds much more like you replaying the history of blink on your branches.

> It might also be nice to send a PSA to warn people.

The truth is that this is unexpected. 

The best solution that comes to my mind is:

• git log and find the SHA1 of the first commit in master (i.e. the first non-yours commit)
  
• git rebase SHA1_of_previous_point your_branch_name --onto origin/master

That rebases your branch in few seconds.

If you have only one commit on a branch, that becomes

git rebase branch_name~1 branch_name --onto origin/master

[Scott Graham]

There have been efforts and patches towards improving those two operations (stating and dir enumeration) on the msysgit list [0], but I'm not sure if they're all active and/or effective at the moment in Chromium's git or "git for Windows". It wasn't too too painful to build git last time I tried following https://github.com/git-for-windows/build-extra/releases/tag/git-sdk-1.0.1 if you're enthused enough to poke around a bit.

iannucci@ and zturner@ and others also did some work a while back on improving performance on src.git pulls, which is one reason why we have a custom build.

("wit" is clearly the perfect name for a rewrite that's ground-up for Windows, in case anyone is inclined. :)

[0] https://github.com/msysgit/git/commit/864390171e1d09aebfe195f2ebfb491b07f840c2 is the only one I immediately found, but I believe there were a bunch more.

[Nico Weber]

On Wed, Sep 23, 2015 at 2:09 PM, Scott Graham <sco...@chromium.org> wrote:

There have been efforts and patches towards improving those two operations (stating and dir enumeration) on the msysgit list [0], but I'm not sure if they're all active and/or effective at the moment in Chromium's git or "git for Windows". It wasn't too too painful to build git last time I tried following https://github.com/git-for-windows/build-extra/releases/tag/git-sdk-1.0.1 if you're enthused enough to poke around a bit.

iannucci@ and zturner@ and others also did some work a while back on improving performance on src.git pulls, which is one reason why we have a custom build.

("wit" is clearly the perfect name for a rewrite that's ground-up for Windows, in case anyone is inclined. :)

[0] https://github.com/msysgit/git/commit/864390171e1d09aebfe195f2ebfb491b07f840c2 is the only one I immediately found, but I believe there were a bunch more.

https://github.com/msysgit/git/commit/ac03519b145e9ba56e973595293362f06c41de61 pulls most of these commits together I think (it contains the stuff in your link and a bit more.)

[Mikhail Naganov]

I'm glad this has happened, congratulations, guys!

Returning to 'webkit_version.h', now it looks like this:

#define WEBKIT_VERSION_MAJOR 537

#define WEBKIT_VERSION_MINOR 36

#define WEBKIT_SVN_REVISION "@c2aa2ce2da1b8feec53ea6a859981659eeec5525-refs/heads/master@{#350327}"

Is this how you plan to leave the SVN_REVISION, or do you have any plans to change this? This is important for DevTools on mobile, as they use this value to fetch the right version of the frontend.

On Fri, Sep 18, 2015 at 8:44 AM, Jochen Eisinger <joc...@chromium.org> wrote:

it will point at Chromium's commit position. In the future, we might want to drop it

On Fri, Sep 18, 2015, 5:34 PM Mikhail Naganov <mnag...@chromium.org> wrote:

That's exciting!

What will happen to webkit_version.h after the merge? Will WEBKIT_SVN_REVISION switch to Chromium's commit number, or will  it just be stuck at the last Blink revision?

[Primiano Tucci]

Blink is merged. Tree is open. CQ is pumping commits. Committer ACLs have been restored.

I guess that's a good definition for "a day".

The situation looks generally green % some test failures. Sheriffs should be on that.

Conventional activity resumes as usual.

It has a been a long merge day in EMEA.

Enjoy a merged repo on the other side of the world :)

[Dirk Pranke]

Woo! Congrats to everyone involved and thanks for all the hard work. It's been a long road getting here.

-- Dirk

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CA%2ByH71dwLTqc0SsjGqiU%3D2CS89cpYAdN%2BRtrBdKqx7e1_JJxqg%40mail.gmail.com.

[Andrey Lushnikov]

I experience the following error when I try to upload a patch which resides in third_party/WebKit directory:

** Presubmit ERRORS **

check-webkit-style failed

  Total errors found: 0 in 0 files

Is it a known issue?

[Nico Weber]

On Wed, Sep 23, 2015 at 2:30 PM, Mikhail Naganov <mnag...@chromium.org> wrote:

I'm glad this has happened, congratulations, guys!

Returning to 'webkit_version.h', now it looks like this:

#define WEBKIT_VERSION_MAJOR 537

#define WEBKIT_VERSION_MINOR 36

#define WEBKIT_SVN_REVISION "@c2aa2ce2da1b8feec53ea6a859981659eeec5525-refs/heads/master@{#350327}"

Is this how you plan to leave the SVN_REVISION, or do you have any plans to change this? This is important for DevTools on mobile, as they use this value to fetch the right version of the frontend.

Is this why a bunch of devtools browser_tests are broken on most main waterfall testers?

[Kenneth Russell]

Awesome work everybody! This repo merge will make life hugely better for all Chromium developers. Thank you for your persistence in getting us here!

-Ken

On Wed, Sep 23, 2015 at 11:32 AM, Primiano Tucci <prim...@chromium.org> wrote:

[Bruce Dawson]

I grabbed a couple of traces of the Resolving deltas stage on Windows and they showed basically the same distribution of CPU time - that is, all of it inside directory enumeration and getting file attributes.

It would be great to get that fscache change into our version of git on Windows. It is quite clear that it either isn't in there now, or the git pull/rebase tasks aren't using it. In extreme cases like this the speedup looks like it could be 20:1. The impedance mismatch between git and Windows is quite severe.

[Mikhail Naganov]

Looks like WEBKIT_SVN_REVISION is fed to GURL in order to make an URL. I guess, in the above form, this can build a malformed URL.

[Dirk Pranke]

Hi all,

I've filed crbug.com/535269 to track any bugs that might be fallout from the merge. Feel free to file issues and block 535269 on them (I have 3 so far).

-- Dirk

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAB6az6HO-H7T_vaSAZt7mqnOYQp7Yb0gWTm-k%2BQz7R7N_AVvXA%40mail.gmail.com.

[Anton Vayvod]

Yep. One of those that Dirk filed. https://code.google.com/p/chromium/issues/detail?id=535272

[Dirk Pranke]

A fix for this is in the CQ.

-- Dirk

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAOjek6r4mKHDtjeNgMo-g7arJy9Qy%2BfY16TRbeNkyA7pbaqXUA%40mail.gmail.com.

[Avi Drissman]

First CL to cross chromium/blink lines! https://codereview.chromium.org/1360223002/

Whoo! Awesome job, team!

Avi

[John Abd-El-Malek]

+1, congrats everyone on this major accomplishment!

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CACWgwAa_xzN_x8Sfb3N37%3DQku8nKbzbEhsP7H7NjZEDt%2BebBdg%40mail.gmail.com.

[Kinuko Yasuda]

+100!

2015/09/24 午前10:23 "John Abd-El-Malek" <j...@chromium.org>:

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Alexandre Elias]

Big thanks to the team who did all the grunge work for this.  From my point of view this merge went really smoothly: I just synced and got it, and it even kept my old Blink branches around in that "old_" directory.  And I'm chomping at the bit to start doing some cross-repo-spanning cleanups I've been putting off because it was too annoying.

[John Bauman]

Based on the merge commit that Nico mentioned, it looks like the cache is disabled unless the git config core.fscache is set to 1

[Morten Stenshorne]

Good job!

Everything is working fine here too, and this change will simplify my
workflow.

One thing that seems to have got slightly worse is git performance
(rebase, for instance), but I can certainly live with it.

>>>>>>>>>>> -

>>>>>>>>>>>
>>>>>>>>>>> Blink will be merged into chromium in //src/third_party/WebKit

>>>>>>>>>>> -

>>>>>>>>>>>
>>>>>>>>>>> The merge will happen Wednesday, September 23, sometime in
>>>>>>>>>>> the GMT+1 morning / midnight PST / late afternoon in TOK

>>>>>>>>>>> -

>>>>>>>>>>>
>>>>>>>>>>> During that time, all trees will be closed, and the chromium
>>>>>>>>>>> and blink repositories will be read only.

>>>>>>>>>>> -

>>>>>>>>>>>
>>>>>>>>>>> The merge commit will include the entire Blink history with
>>>>>>>>>>> the exception of the history of pixel results

>>>>>>>>>>> -

>>>>>>>>>>>
>>>>>>>>>>> Existing branches and codereviews in Blink will have to be
>>>>>>>>>>> manually transferred

>>>>>>>>>>> -

>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAOjek6r4mKHDtjeNgMo-g7arJy9Qy%2BfY16TRbeNkyA7pbaqXUA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .

>>>>>>
>>>>>
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "infra-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to infra-dev+...@chromium.org.
>>>> To post to this group, send email to infr...@chromium.org.
>>>> To view this discussion on the web visit

>>>> https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CACWgwAa_xzN_x8Sfb3N37%3DQku8nKbzbEhsP7H7NjZEDt%2BebBdg%40mail.gmail.com
>>>> <https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CACWgwAa_xzN_x8Sfb3N37%3DQku8nKbzbEhsP7H7NjZEDt%2BebBdg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .

>>>>
>>>
>>> --
>>> --
>>> Chromium Developers mailing list: chromi...@chromium.org
>>> View archives, change email options, or unsubscribe:
>>> http://groups.google.com/a/chromium.org/group/chromium-dev
>>>
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to chromium-dev...@chromium.org.
>>>
>>
>

> To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

---- Morten Stenshorne, developer, Opera Software ASA ----
------------------ http://www.opera.com/ -----------------

[Primiano Tucci]

> One thing that seems to have got slightly worse is git performance
> (rebase, for instance), but I can certainly live with it.

I would expect that rebase times should go back to normal once you start having again CL based after the branch point.

If you are experiencing amazingly slow rebases, see my other post on this ML

[Morten Stenshorne]

Primiano Tucci <prim...@chromium.org> writes:

>> One thing that seems to have got slightly worse is git performance
>> (rebase, for instance), but I can certainly live with it.
> I would expect that rebase times should go back to normal once you start
> having again CL based after the branch point.

This was after the branch point. I replayed a branch from the old WebKit
repo on top of an up-to-date Chromium repo.

> If you are experiencing amazingly slow rebases, see my other post on this ML

> <https://groups.google.com/a/chromium.org/d/msg/chromium-dev/mtfzuRp_vLk/SfdQrN1AAwAJ

It's not amazingly slow, just slower. Maybe twice as slow or so. But I'm
not amazed. :) More commits, bigger repo, etc.

Chromium repo + Webkit repo - PNG layout test expectations > WebKit repo

[Primiano Tucci]

You could try experimenting leaving a :

git repack -f -a -d --window=50 --depth=100

running overnight.

That is likely to take O(full night) and O(GB of ram), but breaking all the delta chains and rebuilding them might speed up a lot.

I'll try to do myself one of these evenings to see if it makes any actual difference.

[Ruud van Asseldonk]

I wanted to do a repack, but it is prohibitively slow. The following worked for me and is much faster:

Do a fresh clone. This will ensure that the repo has one big packfile. (And the

server packs it for you so no need to do it locally.)

    $ git clone https://chromium.googlesource.com/chromium/src.git chromium_src

Restore your local work to this repo (you will lose your stashes):

    $ cd chromium_src

    $ git remote add premerge file:///path/to/chromium/src

    $ git fetch premerge --tags

Swap the slow fragmented objects with the fresh tidy packfiles:

    $ cd /path/to/chromium/src/.git

    $ mv objects objects.bak

    $ mv /path/to/fresh/chromium_src/.git/objects objects

Stop Git from complaining about missing stashes:

    $ git stash clear

If you are convinced that you did not lose any local work, remove `chromium_src`

and `.git/objects.bak`.

You should now have two packfiles, a big one (~4.5 GiB) with the Chromium (and

Blink) history, and a small one with your local work. There should be no loose

objects.

[Primiano Tucci]

On Thu, Sep 24, 2015 at 12:27 PM, 'Ruud van Asseldonk' via infra-dev <infr...@chromium.org> wrote:

I wanted to do a repack, but it is prohibitively slow.

That's why I said "overnight" :)

 

Do a fresh clone. This will ensure that the repo has one big packfile. (And the

server packs it for you so no need to do it locally.)

This is the problem. Your performance now depend on the choice of delta-chains-length that the server does. If the server decides to optimize for packfile size it will have longer delta chain -> slower performances. If you repack locally you can have shorter chains -> bigger packs but better performances.

 

[Morten Stenshorne]

Primiano Tucci <prim...@chromium.org> writes:

> You could try experimenting leaving a :
> git repack -f -a -d --window=50 --depth=100
> running overnight.
>
> That is likely to take O(full night) and O(GB of ram), but breaking all the
> delta chains and rebuilding them might speed up a lot.
> I'll try to do myself one of these evenings to see if it makes any actual
> difference.

Thanks for the tip. This only took 4 hours, but it seems to be as slow
as before.

[Stefan Zager]

This may not be the most opportune time to mention this, but using a local git cache (as all of the buildbot machines do) makes all of this pain go away.  You can keep your local git cache up-to-date in the background.  The git-cache code will notice when you have too many pack files, and rather than repacking (which takes forever and a day), it will automatically re-bootstrap by downloading a zipped repository bundle from Google Storage.  The bundles for all repositories are re-created every 24 hours, and -- this is the best part -- your local machine doesn't have to run the lengthy 'Resolving deltas' step.

We've been doing this to great effect on the buildbot machines since before the chromium->git migration last year.  We always intended to evangelize git-cache for developer use, but never got around to writing up the docs, etc.  I've been filing off a few of the rough edges, but it still probably needs some work.

Anyways, if this sounds appealing to you, please let the infra team know :)

[Will Harris]

d:\src\gclient\src>git repack -f -a -d --window=50 --depth=100

Counting objects: 7024156, done.

Delta compression using up to 32 threads.

fatal: Out of memory? mmap failed: No error

:(

Any solution? Are there plans to ship a 64-bit version of git on Windows?

Will

[Primiano Tucci]

but using a local git cache (as all of the buildbot machines do) makes all of this pain go away. 

How is this related with git performances on status/rebase? AFAIK git cache can solve the problem of fetching from the remote, and AFAICT that has been a pain just in the day of crossing the merge point because they had to get 1.2 GB of data. In the next days people will just get the delta packs as usual.

 

> Any solution? Are there plans to ship a 64-bit version of git on Windows?

Lower down pack.threads and/or pack.windowMemory (see man git-config)

[Scott Graham]

On Thu, Sep 24, 2015 at 10:44 AM, Will Harris <w...@chromium.org> wrote:

d:\src\gclient\src>git repack -f -a -d --window=50 --depth=100

Counting objects: 7024156, done.

Delta compression using up to 32 threads.

fatal: Out of memory? mmap failed: No error

Is that sort of like `git gc` or something completely different? Mine survived a plain git gc, at a peak of ~1G fwiw.

git repack seems to have a --window-memory option, maybe that could help.

(I just tried your command line and OOM'd too though.)

[Primiano Tucci]

I might be terribly wrong, but afaik git gc does not break the existing delta chains in packs, just appends new stuff to that.

If they got borked over time a repack -f is, afaik, the only way to force git to rebuild the full packs.

An alternative could be re-fetching from scratch, but I have no idea how much GoB packs are optimized for size vs client speed.

[Stefan Zager]

On Thu, Sep 24, 2015 at 11:19 AM Primiano Tucci <prim...@chromium.org> wrote:

but using a local git cache (as all of the buildbot machines do) makes all of this pain go away. 

 

How is this related with git performances on status/rebase? AFAIK git cache can solve the problem of fetching from the remote, and AFAICT that has been a pain just in the day of crossing the merge point because they had to get 1.2 GB of data. In the next days people will just get the delta packs as usual.

It doesn't directly help rebase, but it does the equivalent of git-gc and git-repack in a way that doesn't crush your machine or produce OOM errors, and runs *much* faster than gc/repack, which is pretty high value, IMO.

[Stefan Zager]

To be a bit more explicit than this: when we were experimenting with ways to speed up the 'update' step on the bots, it became evident that the bots couldn't get away with disabling gc/repack forever; git operations would just get slower and slower over time.  It also became clear that running gc/repack locally on each bot could take a really, really long time.  So, the solution we came up with was: when a local checkout reaches the threshold of pack files that would normally trigger an automatic gc, we instead delete the checkout and re-seed it from the Google Storage bundle.  That turned out to be *much* faster and more reliable.