is there a way to swtich branches without fully rebuilding each time?

[Demetrios Tsillas]

I have a repo where I need to switch between two dev branches. Each time I switch I need to

gclient sync and rebuild. The build takes hours.

Is there some way to avoid this? Can I save my out/Release or out/Debug some way and restore

it when I switch branches?

thanks!!!

[Anton Vayvod]

Generally speaking you don't need to do gclient sync and full rebuild when switching branches. You might only need to run gclient runhooks if .gyp/.gn files are different between the branches. Other than that there should be no difference to manually editing all source files in one branch so they have the same contents as in the other branch. Are you changes that significant/close to the root of the dependency tree (e.g. changing files in base/ would cause many dependent components to recompile/relink)?

For what you want, can't you simply use different out directories for the branches?

E.g. some "ninja -C out_branch1/Debug" when on branch1 and "ninja -C out_branch2/Debug" when on branch 2? Some clever scripting could automate it for you.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Demetrios Tsillas]

Yes I'm going to try some script. I just wanted to know if there was some support already there in depot_tools for this.

[Robert Flack]

For using a different output dir:

https://code.google.com/p/chromium/wiki/CommonBuildTasks#Change_Build_Output_Directory

For example:

GYP_GENERATOR_FLAGS="output_dir=out_branch1" gclient sync (or runhooks)

But if the changes between the branches are only code then you likely don't need to run gclient at all, and if they do touch .gyp* files but don't change DEPS then you only need to gclient runhooks when switching between them.

[Demetrios Tsillas]

The problem is that the branches have different dependency versions (.DEPS.git)

[Demetrios Tsillas]

Exactly what I was looking for! I should be able to put that in some script and do exactly what I need.

On Thursday, March 19, 2015 at 11:07:47 AM UTC-4, Robert Flack wrote:

[Joel Weinberger]

I am a big fan of and make extensive use of git-new-workdir, which allows you to share a common git repo (branches, trees, etc) between multiple working directories. It should be available in the contrib/ folder of your git install, although you can also find it on the GitHub Git repo. Additionally, here are some instructions on it as well.

--Joel

[Primiano Tucci]

I'll be the crazy guy in town here, but if you want that kind of workflow you should definitely try out btrfs.

I can write a one pager doc on how to set it up and use reasonably for chromium workflows if there is any interest.

I've been using in the last 1+ years and it has incredibly speeded up my workflow.

The TL;DR of my workflow is: I keep a chrome-vanilla/ tree where I never do any changes, just git fetch and gclient sync.

Whenever I need to do any work, or whenever somebody stops at my desk asking for any random experiment i just "btrfs subvolume snapshot chrome-vanilla/ chrome-feature/". That creates in < 1 second a full copy of the tree (with subprojects and everything) which is copied-on-write and on which I can do actual work.

Furthermore, if I am working on featureA and want to give a shot playing with some weird compiler flags (Which will clobber the state of out/), I just btrfs subvolume snapshot chrome-featureA/ chrome-featureA_crazyflags. That gives a full snapshot of my current tree + out folder on which I can immediately do experiments (without clobbering the chrome-featureA out state if I want to go back), without having to wait for any git checkout or gclient operation.

The beauty of CoW, beyond being zero-delay, is that I can have 7-8 full chrome checkouts and 4-5 full android trees at the same time on a 512 SSD, which is unthinkable in a conventional filesystem.

Where is the catch? Well, honestly stability. There is a non-zero risk of compromising the filesystem. I personally never lost any data, but ended up (with older kernels on Lucid) in a state where I couldn't build anymore and I had to blow the partition away and re-create it. It never happened in the last 6 months which makes me pretty happy about that.

In essence, I wouldn't personally endorse btrfs as a general solution for every situation. But, IMHO, if you reached the state where you know and care about git-new-workdir you should probably start taking a serious look at btrfs.

[Demetrios Tsillas]

Looks like a good solution to many situations but the problem I can see with git-new-workdir is that doing a gclient sync in the resulting workdir causes it to fetch all the dependency repos from scratch. This takes a long time.

[Scott Hess]

Yeah, I've been meaning to poke around and see how hard it would be to
git-new-workdir all the other gits. I suspect it would be insane, so
I'm thinking of dialing back to only doing third_party/WebKit.

If it's a one-off case, what you can do is use cp -a or rsync to
schlep over all the other stuff from your original checkout.

My main learning with git-new-workdir is to always label the branches
I'm working on in the non-main repo so that I know where they are. If
you do branch rebasing or something on those branches in the main
repo, it will entirely mess up the workdir repo. Don't do that!

-scott

[Demetrios Tsillas]

OK so it sounds like git-new-workdir is the workflow I am looking for.

One question: how do you sync non-main dirs to your main repo dir?

Do you use git pull/push?

[Rob Wu]

I'm using ccache to speed up rebuild, exactly for the use case of switching between branches without having to do a complete rebuild.

ccache skips the cache if unrecognized options are passed, so you need a wrapper around ccache that inserts --cache-skip in front of it. I've written a c program to do that (see attachment, compile it using gcc ccache-wrapper.c -o ccache-wrapper).

After compiling the wrapper and putting it at /path/to/ccache-wrapper, set the following environment variables:
export CC="/path/to/ccache-wrapper clang -Qunused-arguments"
export CXX="/path/to/ccache-wrapper clang++ -Qunused-arguments"
export CCACHE_CPP2=yes
export CCACHE_SLOPPINESS=time_macros

# For clang++

export PATH=/path/to/chromium/src/third_party/llvm-build/Release+Asserts/bin:$PATH

And the following in my ~/.gyp/include.gypi

{
    "variables": {
        "clang": 1, # I use my own ccache+clang wrapper. Set clang=1 to avoid GetVersion error in compiler_version.py
    },
}

After doing that, run "gclient runhooks" or "build/gyp_chromium", and from then on, compiled objects will be cached and re-used if possible.

This setup usually works well for me. I once had a problem with this setup after a grd change (I'm not even sure if the issue was caused by my setup), but to solve that I just wiped the outdir (like a landmine), ran "gclient runhooks" and started ninja with a flag to bypass the existing cache: CCACHE_RECACHE=1 ninja -c out/Debug chrome

Kind regards,

 Rob

 https://robwu.nl

[Jiang Jiang]

On Fri, Mar 20, 2015 at 2:35 PM, Rob Wu <r...@robwu.nl> wrote:
> I'm using ccache to speed up rebuild, exactly for the use case of switching
> between branches without having to do a complete rebuild.
> ccache skips the cache if unrecognized options are passed, so you need a
> wrapper around ccache that inserts --cache-skip in front of it. I've written
> a c program to do that (see attachment, compile it using gcc
> ccache-wrapper.c -o ccache-wrapper).

Out of curiosity, do you still need this wrapper now with the latest
ccache released? Which unrecognized clang argument is causing this?

- Jiang

[Rob Wu]

I haven't tested, but I think that the wrapper is still necessary. I wrote the wrapper a few months ago (in January 2015) because back then it was still needed. I extracted the affected flags from ccache's source code (as you can see in the comments in my c program), which are still present in the latest version. My current ccache version is 3.2.1.

Kind regards,

 Rob

 https://robwu.nl

[Jiang Jiang]

On Fri, Mar 20, 2015 at 3:11 PM, Rob Wu <r...@robwu.nl> wrote:
> I haven't tested, but I think that the wrapper is still necessary. I wrote
> the wrapper a few months ago (in January 2015) because back then it was
> still needed. I extracted the affected flags from ccache's source code (as
> you can see in the comments in my c program), which are still present in the
> latest version. My current ccache version is 3.2.1.

Does it mean that you still get a lot of cache misses without this
wrapper? How many would that be? We use ccache at Opera for all
Linux/Mac buildbots and never needed such a wrapper, I just checked
our Mac buildbot's best case ccache stats for running one Opera build
(closer to running one Chrome build out of Chromium tree), it's like:

cache hit (direct) 15241
cache hit (preprocessed) 554
cache miss 12
called for link 56
unsupported source language 17

- Jiang

[Rob Wu]

I have just tried without my wrapper, and almost every ccache attempt fails: ccache -s shows a steadily increasing "unsupported compiler option" count. After about 1000 tasks, I have had zero cache hits and zero cache misses, and almost 2k "unsupported compiler option". After enabling logging (CCACHE_LOGFILE=/tmp/clog ninja -C out_branch1/Debug chrome), I see logs like:

[2015-03-20T15:40:45.332856 16961] Working directory: /cr/src/out_branch1/Debug
[2015-03-20T15:40:45.332989 16961] Compiler option -gsplit-dwarf is unsupported
[2015-03-20T15:40:45.333003 16961] Failed; falling back to running the real compiler
[2015-03-20T15:40:45.333010 16961] Executing /cr/src/third_party/llvm-build/Release+Asserts/bin/clang++ -Qunused-arguments (...)

I'm compiling on Linux, x64. I guess that you (at Opera) have disabled the linux_use_debug_fission flag, which is responsible for the -gsplit-dwarf option: https://cs.chromium.org/%22linux_use_debug_fission:%20whether%20to%20use%22.

Kind regards,

 Rob

 https://robwu.nl

[Jiang Jiang]

On Fri, Mar 20, 2015 at 3:50 PM, Rob Wu <r...@robwu.nl> wrote:
> I have just tried without my wrapper, and almost every ccache attempt fails:
> ccache -s shows a steadily increasing "unsupported compiler option" count.
> After about 1000 tasks, I have had zero cache hits and zero cache misses,
> and almost 2k "unsupported compiler option". After enabling logging
> (CCACHE_LOGFILE=/tmp/clog ninja -C out_branch1/Debug chrome), I see logs
> like:
>
> [2015-03-20T15:40:45.332856 16961] Working directory:
> /cr/src/out_branch1/Debug
> [2015-03-20T15:40:45.332989 16961] Compiler option -gsplit-dwarf is
> unsupported
> [2015-03-20T15:40:45.333003 16961] Failed; falling back to running the real
> compiler
> [2015-03-20T15:40:45.333010 16961] Executing
> /cr/src/third_party/llvm-build/Release+Asserts/bin/clang++
> -Qunused-arguments (...)
>
> I'm compiling on Linux, x64. I guess that you (at Opera) have disabled the
> linux_use_debug_fission flag, which is responsible for the -gsplit-dwarf
> option:
> https://cs.chromium.org/%22linux_use_debug_fission:%20whether%20to%20use%22.

Right, I just realized that you might be using Linux, so either that or use
https://bitbucket.org/zephyrus00jp/ccache-gsplit-dwarf-support instead.

- Jiang

[Rob Wu]

Cool, thanks for the link! I'll try to drop my wrapper when that patch gets merged upstream.

 

- Jiang

[Joel Weinberger]

On Thu, Mar 19, 2015 at 3:14 PM Demetrios Tsillas <jtsi...@gmail.com> wrote:

OK so it sounds like git-new-workdir is the workflow I am looking for.

One question: how do you sync non-main dirs to your main repo dir?

Do you use git pull/push?

Hm, I don't think I understand the question. Let me just explain my workflow.

• I checkout the Chromium repo into work/
• Using git-new-workdir, I setup work1/ and work2/ based off of work/. git-new-workdir setsup all of the hard links for the main Chromium repo only (but not V8, anything in third_party/, etc).
• Now I can go into work/, work1/, or work/, do a git fetch in any of them, and they'll all have access to the latest Chromium remote branches.
• I do a gclient sync in all of the directories to get all of the dependencies.
• But since, like Scott, I work on Blink, I want my Blink repos to also be connected. So in work1/ and work2/, I blow away third_party/WebKit and then git-new-workdir WebKit from work/.
  
• Now, gclient sync again in work1/ and work2/, and you should be setup!

Note that this is almost entirely a one-time cost, and I haven't had to do this since the last time I got a new hard drive.

--Joel

[Demetrios Tsillas]

Right after asking that question I realized you don't really sync with the repo dir.