Consistent errors running gclient sync for stable mac code: tags/48.0.2564.109

[Bobo B.]

Hello,

I am running into the same issue repeatedly (five times). Machine is a macbook pro, 16GB ram, running OSX El Capitan 10.11.2

Workflow is as follows:

1) Have an up to date copy of the Chromium source code
2) git checkout -b new_release tags/48.0.2564.109
3) gclient sync

Step 3 causes the following error, which now happens to me consistently. It hangs. ctrl+c shows the following output:

$ gclient sync
^C
Allowed parallel jobs: 8
# queued: 0
Ran:
Running: 1

src (interrupted)
----------------------------------------
[0:00:00] Started.
[0:00:21] _____ src : Attempting rebase onto origin...
----------------------------------------
interrupted


Please advise. I would like to build the stable version of Chromium for mac. My estimated workflow (following https://groups.google.com/a/chromium.org/d/msg/chromium-dev/VTOniO05UDc/nG3F2e67_4sJ) is as follows (in /src/):

1) On master branch: Have up to date version of Chromium source code (or use : gclient sync --nohooks --no-history --with_tags --verbose --verbose --verbose)
2) git checkout -b new_release tags/$TARGET (where $TARGET is latest stable version, like 48.0.2564.109)
3) gclient sync --no-history --verbose --verbose --verbose (I am under the impression this step is necessary from above link. What happens if I add --nohooks here?)
4) build/gyp_chromium -Dfastbuild=1 -Dmac_strip_release=1 -Dbuildtype=Official
5) gclient runhooks
6) ninja -C out/Release chrome

Thanks,
Bobo

[Christian Biesinger]

I don't really understand your set of steps. gclient sync includes
runhooks (unless, as you suggest, you add --nohooks), so you don't
need to do both. runhooks includes gyp_chromium, so you don't need to
run it manually, *except* that you pass the defines in the
commandline. I would instead set GYP_DEFINES="fastbuild=1
mac_strip_release=1 buildtype=Official" and skip the manual running of
gyp. As it is, your runhooks command after gyp_chromium means that
your defines are ignored (unless you separately set them in
GYP_DEFINES or another mechanism already).

Anyway, that's not your actual issue. It looks like you are using
managed mode for gclient, because it is trying to rebase src. That's
deprecated -- please see
https://www.chromium.org/developers/how-tos/get-the-code/gclient-managed-mode.
I would suggest turning off managed mode (as described in that link)
and doing a manual "git fetch --tags origin" before your checkout -b
command.

(I don't actually know why your rebase command hangs. You could try
running gclient with -j 1 and see if that provides a better output.
But I would do the above things either way)

-Christian

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev

[Primiano Tucci]

Out of curiosity do you happen to have a somewhat older checkout which has a managed=True in the .gclient file?

It is extremely suspicious that src gets rebased. gclient should never touch your main project... unless you have managed=True.

Do the following:

1) add an explicit   'managed': False, under solutions (example below)

2) git checkout -b new_release tags/$TARGET   # this looks good, as a general practice I'd also add -t but it's irrelevant here.

3) gclient sync --with_branch_heads --force --reset

Note, 3. with the force and reset optiosn will clobber any local state. Generally it's not needed but given the fact that you had some unsuccessful experiments it is more likely that some projects are in an inconsistent state, and --force --reset will give you more chances to get to the right place. Just be sure to not have changes leaving around (i.e. not in a branch)

* here's how .gclient should look like

-----

solutions = [

{

  'managed': False,

  'name': 'src',

  'url': 'https://chromium.googlesource.com/chromium/src.git',

  'deps_file': '.DEPS.git',

  'safesync_url': u''

}]

target_os = ['mac']

-----

[Thiago Farina]

On Thu, Feb 18, 2016 at 8:39 AM, Primiano Tucci <prim...@chromium.org> wrote:

* here's how .gclient should look like

-----

solutions = [

{

  'managed': False,

  'name': 'src',

  'url': 'https://chromium.googlesource.com/chromium/src.git',

  'deps_file': '.DEPS.git',

  'safesync_url': u''

}]

target_os = ['mac']

target_os is probably not necessary, right? As it should be detected automatically, no?

I have only used this for android and ios. Where the target device is different from the

host machine.

--

Thiago Farina

[Primiano Tucci]

Oh I think you are right. I just did s/android/mac/ while copy pasting, but android is special as that doesn't match the host os.

So, yes, that might be superfluous.

[Bobo B.]

@Primiano
What's the difference between gclient sync --with_tags and gclient sync --with_branch_heads? Which do I want to use when?

Build Order
@Christian - thank you so much, this is very helpful. Does the following make sense? Not sure how to set the GYP_DEFINES (are they environment variables?) and do I need to set them again when I switch branches?

1) On master branch (set GYP_DEFINES):
export GYP_DEFINES="fastbuild=1 mac_strip_release=1 buildtype=Official"

2) On master branch (ensure source is up to date):
gclient sync --no-history --with_tags --verbose --verbose --verbose

3) Switch to latest stable:

git checkout -b new_release tags/$TARGET (where $TARGET is latest stable version, like 48.0.2564.109)

4) Sync the latest stable (is this necessary?):

gclient sync --no-history --verbose --verbose --verbose

5) Build latest stable:
ninja -C out/Release chrome

Managed Mode
@Christian -
Why does gclient config https://chromium.googlesource.com/chromium/src.git result in a managed mode .gclient if it's deprecated?

gclient --version returns: gclient.py 0.7

@Primiano
Please see here for longer output concerning rebasing src: https://groups.google.com/a/chromium.org/d/msg/chromium-dev/a-UHhvZ2SP0/UfmF4Q-2AgAJ

[Torne (Richard Coles)]

More than one project uses gclient, so the gclient behaviour hasn't changed. We expect people to get chromium checkouts with "fetch chromium", not by manually configuring gclient, and fetch will give you an unmanaged config.

--

[Bobo B.]

@Torne -
Custom .gclient with fetch
understood, thanks. how can I edit a custom .gclient and use that with fetch? If I already have a .gclient in the directory fetch errors out saying:

Running: /usr/bin/python /Users/bobo/Code/betatest/depot_tools/gclient.py root
Your current directory appears to already contain, or be part of,
a checkout. "fetch" is used only to get new checkouts. Use
"gclient sync" to update existing checkouts.

Fetch also does not yet deal with partial checkouts, so if fetch
failed, delete the checkout and start over (crbug.com/230691).

Second, deps_file:  what should deps_file be? If I use gclient sync it is

"deps_file"   : "DEPS",

If I use fetch it is

"deps_file": ".DEPS.git",

Is this related to managed mode / unmanaged mode?

FWIW, I tried to use fetch initially but could not get it to work, didn't write down the error but I'm suspecting it's because I was implementing a custom .gclient. My belief is that the custom .gclient below will result in less code being pulled and/or faster compile times, please correct if wrong (i also note it has an extra line for cache_dir missing from the fetch one, would love any clarification on that :) ):

solutions = [
  { "name"        : "src",
    "url"         : "https://chromium.googlesource.com/chromium/src.git",
    "deps_file"   : "DEPS",
    "managed"     : True,
    "custom_deps" : {
    "src/third_party/WebKit/LayoutTests": None,
            "src/chrome/tools/test/reference_build/chrome": None,
            "src/chrome_frame/tools/test/reference_build/chrome": None,
            "src/chrome/tools/test/reference_build/chrome_linux": None,
            "src/chrome/tools/test/reference_build/chrome_mac": None,
            "src/third_party/hunspell_dictionaries": None,
    },
    "safesync_url": "",
  },
]
cache_dir = None

Thanks,
Bobo

[Torne (Richard Coles)]

Those custom deps entries are all irrelevant now and don't work, I think. It isn't possible to exclude specific directories like layout tests in git, this only worked when we used svn.

If you want a smaller download pass --nohistory to fetch, to only download the current tip of tree version and not the entire history. Other than that there are not many improvements to be had.

[Christian Biesinger]

See the other responses for .gclient and gclient config -- don't
bother with a custom .gclient

Yes, GYP_DEFINES is an environment variable.
https://www.chromium.org/developers/gyp-environment-variables has some
more details but basically all you have to do is ensure the variable
is set at the time you run gclient sync (or runhooks / gyp_chromium if
you were to run them directly).

There is no point in running your step 2; replace it with "git fetch origin"

-Christian

[Thiago Farina]

On Thu, Feb 18, 2016 at 6:59 PM, Bobo B. <the....@gmail.com> wrote:

Build Order
@Christian - thank you so much, this is very helpful. Does the following make sense? Not sure how to set the GYP_DEFINES (are they environment variables?) and do I need to set them again when I switch branches?

The variables you set? The variables I don't think they are environment variables, but I could be wrong, they are defined in GYP language in .gyp or .gypi files. GYP_DEFINES itself as below with 'export' is an environment variable :)

There are some ways to set it, the ones I remember are: like you do, export GYP_DEFINES="..." (could be added to a .bashrc), or .gyp/include.gypi, or even through chromium.gyp_env.

If you have it in one of these permanent places, then when you switch to a different branch, if you run 'gclient runhooks', you don't need to set them again as it will be picked automatically.

Hope that helps,

Regards,

--

Thiago Farina

[Bobo B.]

Hitting a wall - no matter how I try to sync the specific tag I've checked out, I end up with this type of "not a tree" error (my .gclient is Managed Mode False):

Error: Command 'git checkout --quiet 3ba3ca22ec610fe95683f6bfdeea9d90c768abd7' returned non-zero exit status 128 in /Users/bobo/Code/betatest/src/buildtools
fatal: reference is not a tree: 3ba3ca22ec610fe95683f6bfdeea9d90c768abd7

The hash in question changes from run to run.

Here is the workflow I am using:

1) git fetch --tags origin
2) export GYP_DEFINES="fastbuild=1 mac_strip_release=1 buildtype=Official"
3) git checkout -b new_release tags/48.0.2564.109
4) try to sync this branch
5) ninja -C out/Release chrome

I keep getting stuck on step 4.

For step 4 i've tried the following (each time on a new checkout of tags/48.0.2564.109):

gclient sync --no-history --verbose --verbose --verbose

gclient sync --no-history --with_branch_heads --force --reset --verbose --verbose --verbose

gclient sync

Here is my .gclient:

solutions = [
  {
    "managed": False,
    "name": "src",
    "url": "https://chromium.googlesource.com/chromium/src.git",

    "custom_deps": {},
    "deps_file": ".DEPS.git",
    "safesync_url": "",
  },
]

This is frustrating, thanks for any help!

[Primiano Tucci]

That reference IS definitely a valid three: here

The only possible option is that --no-history doesn't play good with switching to release branches. That option was meant to check out ToT faster.

Try to unshallow your projects, specifically buildtools (rm .git/shallow and re- git fetch) and re-run gclient sync without --no-history.

--

[Bobo B.]

Hey Primiano,

Want to make sure I understand you. Suggestion is to:
1) rm .git/shallow from the /src/ directory
2) git fetch --tags origin

3) git checkout -b new_release tags/48.0.2564.109

4) gclient sync

Do I understand correctly? How much disk space will this require compared to the no-history version?

Thanks for the reply, I've tried 15 different ways of trying to accomplish this workflow (lots of minor variations) and it's been tough. Thank you for your help!

[Primiano Tucci]

> rm .git/shallow from the /src/ directory

No, from the src/buildtools directory (and all the other ones that will fail)

Then just do gclient sync --with_branch_heads --force --reset

[Bobo B.]

Still getting an error, here is the console output:

bobo@bobos-MacBook-Pro:~/Code/gammatest/src$ rm ./buildtools/.git/shallow
bobo@bobos-MacBook-Pro:~/Code/gammatest/src$ ../depot_tools/gclient sync --with_branch_heads --force --reset --jobs 16
Syncing projects:  15% (10/65) src/native_client                              

src/buildtools (ERROR)

----------------------------------------
[0:00:00] Started.

----------------------------------------
Error: Command 'git rev-list -n 1 HEAD' returned non-zero exit status 128 in /Users/bobo/Code/gammatest/src/buildtools
error: Could not read 81863fe70639e85606b541d9d36e9e98c96b957e
fatal: Failed to traverse parents of commit 0f8e6e4b126ee88137930a0ae4776c4741808740

[Primiano Tucci]

My bad, in order to unshallow you have to:

git -C src/buildtools/ fetch --unshallow

not just rm .git/shallow

[Bobo]

Unfortunately it does not allow me to fetch --unshallow.

Also, I tried a new fetch of the source code with --no-history and have confirmed that I cannot build the latest stable version of Chromium on a fetch with --no-history.

Here is the --unshallow error:
$ git -C buildtools/ fetch --unshallow
fatal: --unshallow on a complete repository does not make sense

[Torne (Richard Coles)]

If you deleted the shallow file already your repository is now broken and you won't be able to unshallow it. You'll have to download a fresh copy.

[Torne (Richard Coles)]

Also, yes, if you have a shallow checkout you will not be able to sync with branch heads unless you first unshallow it. The shallow config basically says "never ever download any commit that is earlier in history than this point", and --no-history sets that point to " the current latest master branch commit". All the release branches are earlier in history than that, so cannot be downloaded. If you want to use release branches you will have to use a full checkout.

[Bobo]

Thanks Torne, working on getting a full checkout now.

1) Documentation - Could a line about needing a full checkout be added to the Get the Code (https://www.chromium.org/developers/how-tos/get-the-code) and Working with Release Branches (https://www.chromium.org/developers/how-tos/get-the-code/working-with-release-branches) documentation?

2) git fetch tags - Instead of git fetch --tags origin after downloading the full checkout, can I just do git fetch origin 48.0.2564.116:48.0.2564.116 and then git checkout -b new_release tags/48.0.2564.116 instead? This is the latest stable version for Mac according to https://omahaproxy.appspot.com/all.json?os=mac&channel=stable

3) progress indicator for fetch? - can i pass --verbose to fetch to get more progress reporting? like fetch chromium --verbose --verbose --verbose (i know gclient takes up to three --verbose flags)

4) gclient sync for the checked out tag - once i've checked out the tag i want to build, do i need to pass anything special to gclient sync to sync it (like --with_branch_heads) or will a simple gclient sync work?

Thanks so much,

Bobo