Having trouble checking out Chromium with Git

[Matt Giuca]

Hi Chromium-dev,

I'm trying to check out Chromium on a new Linux machine with Git. I haven't done this for awhile, but I don't remember having this much trouble last time. The machine is running Ubuntu 13.10 (saucy) which *may* be causing trouble, or it may be something else.

I followed the Linux instructions here: http://dev.chromium.org/developers/how-tos/get-the-code

I ran gclient config, then customized .gclient with managed = False and custom_deps to reduce the checkout time. Here is my .gclient file:

solutions = [

  { "name"        : "src",

    "url"         : "https://git.chromium.org/chromium/src.git",

    "deps_file"   : "DEPS",

    "managed"     : False,

    "custom_deps" : {

        "src/content/test/data/layout_tests/LayoutTests": None,

        "src/chrome/tools/test/reference_build/chrome_win": None,

        "src/chrome_frame/tools/test/reference_build/chrome_win": None,

        "src/chrome/tools/test/reference_build/chrome_linux": None,

        "src/chrome/tools/test/reference_build/chrome_mac": None,

        "src/third_party/hunspell_dictionaries": None,

    },

    "safesync_url": "",

  },

]

cache_dir = None

Then I ran "gclient sync", as instructed there. The Chromium checkout completed successfully, but then I ran into all kinds of trouble checking out the submodules.

First, it wrote this: ".gclient file in parent directory /home/matt/src-pkg/chrome might not be the file you want to use". Not sure what that means or if it's serious, but it sure is confusing.

Second, I had a bunch of empty directories that complained like this:

Error: 14> Can't update/checkout /home/matt/src-pkg/chrome/src/sdch/open-vcdiff if an unversioned directory is present. Delete the directory and try again.

Then, I manually deleted those (deleting each one as it came up). Many of these directories then successfully checked out (with Subversion), but I ultimately ran into this problem:

________ running 'svn checkout https://git.chromium.org/chromium/trunk/deps/frame_rate/content@93671 /home/matt/src-pkg/chrome/src/chrome/test/data/perf/frame_rate/content --revision 93671 --non-interactive --ignore-externals' in '/home/matt/src-pkg/chrome'

svn: E175002: Unable to connect to a repository at URL 'https://git.chromium.org/chromium/trunk/deps/frame_rate/content'

svn: E175002: The OPTIONS request returned invalid XML in the response: XML parse error at line 1: Extra content at the end of the document

 (https://git.chromium.org/chromium/trunk/deps/frame_rate/content)

Is it supposed to be using SVN to check out the submodules, or Git? It seems very strange that it is trying to use Subversion at all. But at least the first few directories it used Subversion to check out an SVN URL. However, in the case of frame_rate/content, it is trying to use "svn checkout" with a git URL. That can't be right!

It's also trying to do:

svn checkout https://git.chromium.org/chromium/trunk/deps/canvas_bench@122605

I have tried heaps of things. First, I tried deleting all the files except .git, and starting again. This gave the same problem.

Then, I found this page with a conflicting set of instructions to the above: https://code.google.com/p/chromium/wiki/UsingGit

Note that it says to use "fetch chromium" instead of "gclient sync" for the first checkout. It's possible that the reason I'm having so much trouble is that I didn't do "fetch chromium". But I'm not really willing to try, because it's telling me I have to delete my checkout and start over (which takes several hours), and there's no guarantee it will help (it looks as though it's just manually doing gclient sync).

Can somebody please advise if I am doing something wrong, or whether there is actually a problem here? I have been trying for several hours now to check out Chromium and am becoming quite frustrated.

Thanks

Matt

[Daniel Connelly]

From the right address this time.

On Sat, Oct 26, 2013 at 5:20 PM, Daniel Connelly <dcon...@google.com> wrote:

Some people were having trouble with checkouts and updates on Friday. Maybe something changed or broke with the old workflow?

Anyway, I can confirm that `fetch chromium` works and is the "right way" to get the code from scratch. For subsequent updates do `git pull --rebase && gclient sync`. The "Using Git" page you found (https://code.google.com/p/chromium/wiki/UsingGit) is indeed the one that's up-to-date.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--

Daniel Connelly | Software Engineer – Chrome | dcon...@google.com | 

Google Germany GmbH

Dienerstr. 12

80331 München

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Graham Law, Christine Elizabeth Flores

[Dirk Pranke]

I don't see anything obviously wrong with your .gclient spec. Daniel's response (infra woes) sounds plausible, but it wouldn't explain the weird svn checkout issues you're seeing. Maybe one of the infra folks has an idea. I would either wait for them to chime in, or just do a 'fetch chromium' and see (since there's no guarantee that we'll have any bright ideas in the less than the time it'll take for the fetch to complete).

-- Dirk

[Isaac Levy]

Your gclient config is non standard.  The entry should read:

"deps_file"   : ".DEPS.git"

Recommended workflow these days is to use 'fetch' script in depot_tools. i.e. start from scratch, add depot_tools to path and type 'fetch chromium'.

[Matt Giuca]

Thanks for the advice everybody. Especially Isaac: You are right, my "deps_file" was wrong for Git.

I've changed it to ".DEPS.git" and it is now checking out the third-party code nicely. Thank you!

To go over a little post-mortem:

• gclient config needs to be run with --git-deps when using Git, or it doesn't work properly (my bad .gclient file was given to me by gclient). The page I linked to gives direct gclient commands for Git which do not feature --git-deps, so they are always wrong. The documentation should be updated to include this flag wherever gclient config is used with Git.
• gclient config itself could be more helpful. Perhaps look for URLs that end in ".git" or special-case the domain "git.chromium.org" and then say "It looks like you are configuring a Git URL, but gclient is running in Subversion mode. Did you mean to run with --git-deps?"
• I can confirm that "fetch chromium" would have worked. It doesn't use gclient config; it writes its own config file with the correct deps_file.
• Lastly, the Get The Code page is the main entry point to checking out Chrome. While it does link to UsingGit, it's easy to miss and if you have the mentality of "just give me the command lines to run" you just scroll down to your operating system and run the four commands (which include a "Or, to download via git" section). If "fetch chromium" is indeed the preferred way to check out, it should be on that page.
• However, I wouldn't want the documentation on using gclient config and gclient sync to go away. It's still useful for advanced usages (such as when fetch chromium fails). The "fetch chromium" command seems a bit magic, monolithic and brittle to me, so I think I would probably prefer the old way.

In general, I think the documentation for checking out Chromium is way too hard to follow. It is distributed over many pages, some of which give overlapping and conflicting advice (as in this case). There are lots of different flows (SVN vs Git, bootstrapping using the tarball (which doesn't seem worthwhile), checking out with write access, LKGR vs HEAD, managed vs unmanaged, and modifying .gclient to exclude a bunch of libraries vs not doing that). It seems like most developers these days are expected to use Git, will not need write access, should be in unmanaged mode, and don't need all those libraries. So could there be a little section at the top of "Get the Code" that is basically this:

Normal checkout

Use this advice if you just want to do a normal checkout of Chromium with Git, and do not need direct Subversion write access. (Most developers who are just submitting patches do not need Subversion write access.)

1. Set up depot_tools.
   
2. mkdir ~/chromium
   
3. cd ~/chromium
4. fetch chromium --nosvn=True

If the above step does not work, or you want a little more control over the checkout, follow these steps:

1. Set up depot_tools.
   
2. mkdir ~/chromium
   
3. cd ~/chromium
4. gclient config --git-deps https://git.chromium.org/chromium/src.git
   
5. Modify .gclient as necessary (see below).
6. gclient sync

And in the below sections, we can discuss the customizations to .gclient such as LKGR vs HEAD, managed vs unmanaged, excluding unnecessary libraries, etc. And a separate section for using SVN. Basically I'm not proposing to delete any information, just to shuffle it around so that the common path is described up front. I think this would make the onboarding process a lot smoother.

I'm happy to help out with editing the docs if people think this is worthwhile.

Anyway, thank you Isaac and others!

Matt

[Isaac Levy]

Thanks for the great writeup, Matt, and glad you were able to fix your checkout.

The recommended git host is https://chromium.googlesource.com/chromium/src.git, and this will sync faster, especially for those outside of the US.  I agree we should try to consolidate our docs and make sure they're accurate; this one (which is the main one!) has issues.

[Robert Iannucci]

I took the liberty of fixing the exclusion of '--git-deps' and switching the URL to https://chromium.googlesource.com/chromium/src.git 

There is a sizable amount of work to consolidate all the docs though. Hopefully we can consolidate them into code-used-on-a-daily-basis-by-everyone. Then they may actually stay up to date.

[Robert Iannucci]

(though these instructions should probably be just replaced with a pointer to the fetch instructions as Matt mentions, or deleted altogether.)