Trying to get latest source code fails when fetching

[Kevin Luty]

First I followed the directions here (non-Cygwin): http://dev.chromium.org/developers/how-tos/install-depot-tools

and continued here: http://dev.chromium.org/developers/how-tos/get-the-code

and I'm trying to complete this process: http://dev.chromium.org/developers/how-tos/build-instructions-windows#TOC-Setting-up-the-environment-for-Visual-Studio-2013

Then I get to the fetch chromium --nosvn=True and that takes about 1 hour complete.  In the middle of the process, at some point (I'm guessing about 15 minutes in), _bad_scm folder was created but there was no output in the cmd window.  It seemed like its trying to autorecover because a new _gclient_src_5jr22d folder was created. Regardless, each folders contents ends up being src/.git with nothing else in it.  The size of the src folder is 2.41GB where src/.git/objects/pack/___.pack is ~2460MB.  The .idx file is 81MB.

Eventually I get an error after an hour:

[0:50:16] Resolving deltas:  99% (2333766/2339844)

[0:50:18] Resolving deltas:  99% (2339797/2339844)

[0:50:18] Resolving deltas: 100% (2339844/2339844)

[0:50:26] remote: Total 2969933 (delta 2338260), reused 2966448 (delta 2338260)

[0:50:27] Checking connectivity... done.

----------------------------------------

Error: Command git checkout --quiet refs/remotes/origin/master returned non-zero

 exit status 128 in C:\googlecode\src

fatal: Not a git repository (or any of the parent directories): .git

Warnings:

Conflicting directory C:\googlecode\src moved to C:\googlecode\_bad_scm\srcxq6xd

e.

Traceback (most recent call last):

  File "C:\googlecode\depot_tools\\fetch.py", line 302, in <module>

    sys.exit(main())

  File "C:\googlecode\depot_tools\\fetch.py", line 298, in main

    return run(options, spec, root)

  File "C:\googlecode\depot_tools\\fetch.py", line 292, in run

    return checkout.init()

  File "C:\googlecode\depot_tools\\fetch.py", line 119, in init

    self.run_gclient(*sync_cmd)

  File "C:\googlecode\depot_tools\\fetch.py", line 75, in run_gclient

    return self.run(cmd_prefix + cmd, **kwargs)

  File "C:\googlecode\depot_tools\\fetch.py", line 65, in run

    return subprocess.check_call(cmd, **kwargs)

  File "C:\googlecode\depot_tools\python276_bin\lib\subprocess.py", line 540, in

 check_call

    raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '('C:\\googlecode\\depot_tools\\python276

_bin\\python.exe', 'C:\\googlecode\\depot_tools\\gclient.py', 'sync')' returned

non-zero exit status 1

Some things to note are:

   - I'm new to chromium-dev and trying to build

   - I'm not in a clean environemnt.  I have GitHub and GitShell previously installed as well as a version of SVN. I have Python 3.4 which was previously installed (to C:\Python34).

   - I've installed depot_tools and tried installing from the beginning with no luck, multiple times

   - I have Cygwin on my machine but haven't used the Cygwin-based instructions because I'm not very familiar with it

   - Speedtest.net shows ~37Mbps DL/UL speeds on my current internet connection

Here is my .gclient file inside C:\googlecode (depot_tools folder is along side of it):

solutions = [{u'managed': False, u'name': u'src', u'url': u'https://chromium.googlesource.com/chromium/src.git', u'custom_deps': {}, u'deps_file': u'.DEPS.git', u'safesync_url': u''}]

Any help would be greatly appreciated!

[Kevin Luty]

Also, after the error I have a C:\googlecode\src with a .git folder.  The .git folder has config, HEAD, description, packed-refs files along with branches, hooks, info, logs, objects, refs folders.  

I also get this error:

C:\googlecode>gclient sync

src (ERROR)

----------------------------------------

[0:00:00] Started.

[0:00:00] ________ unmanaged solution; skipping src

----------------------------------------

Error: Command git rev-parse --verify HEAD returned non-zero exit status 128 in

[Primiano Tucci]

Kevin, you seem to have a wrong gclient. Did you just "fetch chromium --nosvn=true" ?

I've just tried on my to fetch on my home PC (outside of Google, running on a DSL). If you follow the instructions, I can assure you It works fine. it just takes a long time.

Probably you had some connectivity issue and eneded up in a bad state. Try to remove your folder and fetch again.

It takes 1.5 hours to fetch everything on a 70Mbps DSL, but hopefully you need to wait that long only the first time ever.

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Kevin Luty]

How do I then get the correct gclient?  I deleted it the first time I tried and tried using gclient config but I didn't know the arguments to pass it so I didn't know how to properly regenerate the .gclient file and thats when I started trying to download/build out of C:\googlecode.

 

I did use fetch chromium --nosvn=True so I don't know how it could have gone wrong.  I have a very reliable connection so I don't see where connectivity would be an issue.  In both cases during the fetch, the pack files are 2.41GB, it just seems like the source code isn't getting unpacked.  As seen here in my recycle bin:

 

I've tried this a few times and all folder sizes are about the same.  my PATH right now is C:\googlecode\depot_tools.  

When I tried the first time in C:\, I tried what was suggested in the last post here: https://code.google.com/p/chromium/issues/detail?id=230691 

about running git clone https://chromium.googlesource.com/chromium/src in PowerShell and I saw the progress of Receiving objects and Resolving deltas all complete without error, but for some reason it still wasn't being recognized as a git repo.  I also was getting the ________ unmanaged solution; skipping src error when trying to run gclient sync, IIRC.  src seemed to have the contents that was supposed to be in there, however there still was no out\Debug folder so I couldn't run the ninja command to build.  

If there anymore details I can provide that might be of use towards figuring this issue out, let me know and I'll be sure to get it for you.  

 

[Christian Biesinger]

Hmm, what's your PATH variable? Is your depot_tools directory in there before the one from your github?

-christian

--

[Kevin Luty]

Sorry Christian, I accidentally hit "Reply to author" my last reply so it didn't get posted on this thread rather it sent straight to you.  If you could, could you copy/paste my response below or e-mail it to me so I can submit it to this thread?  Thanks.

[Kevin Luty]

Ahha! I hit Ctrl+C when I saw the "autorecover" take place again and this is the info I saw output to cmd:

[0:26:57] Resolving deltas:  98% (2296746/2340060)

[0:26:59] Resolving deltas:  98% (2298937/2340060)

[0:27:00] Resolving deltas:  98% (2301052/2340060)

[0:27:02] Resolving deltas:  98% (2304107/2340060)

[0:27:04] Resolving deltas:  98% (2306736/2340060)

[0:27:05] Resolving deltas:  98% (2310933/2340060)

[0:27:07] Resolving deltas:  98% (2314522/2340060)

[0:27:09] Resolving deltas:  98% (2315068/2340060)

[0:27:10] Resolving deltas:  98% (2315354/2340060)

[0:27:11] Resolving deltas:  98% (2316585/2340060)

[0:27:13] Resolving deltas:  99% (2319021/2340060)

[0:27:14] Resolving deltas:  99% (2320079/2340060)

[0:27:15] Resolving deltas:  99% (2320444/2340060)

[0:27:17] Resolving deltas:  99% (2320486/2340060)

[0:27:19] Resolving deltas:  99% (2327864/2340060)

[0:27:21] Resolving deltas:  99% (2333448/2340060)

[0:27:22] Resolving deltas:  99% (2334403/2340060)

[0:27:24] Resolving deltas: 100% (2340060/2340060)

[0:27:33] remote: Total 2970462 (delta 2338476), reused 2966839 (delta 2338476)

[0:27:33] Checking connectivity... done.

[0:27:34] _____ Conflicting directory found in C:\googlecode\src. Moving to C:\g

ooglecode\_bad_scm\srcmmksai.

[0:27:34]

[0:27:34] running 'git -c core.deltaBaseCacheLimit=512m clone --no-checkout --pr

ogress https://chromium.googlesource.com/chromium/src.git C:\googlecode\_gclient

_src_xauil1' in 'C:\googlecode'

[0:27:34] Cloning into 'C:\googlecode\_gclient_src_xauil1'...

[0:29:07] remote: Sending approximately 2.21 GiB ...

[0:29:07] remote: Counting objects: 1

[0:29:07] remote: Counting objects: 2857

[0:29:07] remote: Counting objects: 3666

[0:29:07] remote: Counting objects: 4608

[0:29:07] remote: Counting objects: 5536

[0:29:07] remote: Counting objects: 6238

[0:29:07] remote: Counting objects: 6821

[0:29:07] remote: Counting objects: 7528

How am I getting that error when I'm running as administrator and there is no other src folder in C:\googlecode?   Even after it started downloading again all I see is a _bad_scm folder, so its not like a C:\googlecode\src was created because it would still be there unless the scripts are doing something funny and removing the folder before I see it happen.  

This is at the end of my command prompt:

[0:30:30] Receiving objects:  15% (462969/2970508), 220.84 MiB | 3.82 MiB/s

[0:30:31] Receiving objects:  16% (475638/2970508), 225.17 MiB | 4.03 MiB/s

[0:30:32] Receiving objects:  16% (490643/2970508), 229.37 MiB | 4.11 MiB/s

[0:30:33] Receiving objects:  17% (514187/2970508), 233.37 MiB | 4.13 MiB/s

[0:30:34] Receiving objects:  18% (534692/2970508), 236.74 MiB | 3.91 MiB/s

[0:30:36] Receiving objects:  19% (564397/2970508), 242.61 MiB | 3.79 MiB/s

[0:30:37] Receiving objects:  19% (579966/2970508), 246.16 MiB | 3.22 MiB/s

----------------------------------------

Warnings:

Conflicting directory C:\googlecode\src moved to C:\googlecode\_bad_scm\srcmmksa

i.

Traceback (most recent call last):

  File "C:\googlecode\depot_tools\gclient.py", line 2014, in <module>

    sys.exit(Main(sys.argv[1:]))

  File "C:\googlecode\depot_tools\gclient.py", line 2002, in Main

    return dispatcher.execute(OptionParser(), argv)

  File "C:\googlecode\depot_tools\subcommand.py", line 245, in execute

    return command(parser, args[1:])

  File "C:\googlecode\depot_tools\gclient.py", line 1780, in CMDsync

    ret = client.RunOnDeps('update', args)

  File "C:\googlecode\depot_tools\gclient.py", line 1297, in RunOnDeps

    work_queue.flush(revision_overrides, command, args, options=self._options)

  File "C:\googlecode\depot_tools\gclient_utils.py", line 850, in flush

    self.ready_cond.wait(10)

  File "C:\googlecode\depot_tools\python276_bin\lib\threading.py", line 358, in

wait

    _sleep(delay)

KeyboardInterrupt

Traceback (most recent call last):

  File "C:\googlecode\depot_tools\\fetch.py", line 302, in <module>

    sys.exit(main())

  File "C:\googlecode\depot_tools\\fetch.py", line 298, in main

    return run(options, spec, root)

  File "C:\googlecode\depot_tools\\fetch.py", line 292, in run

    return checkout.init()

  File "C:\googlecode\depot_tools\\fetch.py", line 119, in init

    self.run_gclient(*sync_cmd)

  File "C:\googlecode\depot_tools\\fetch.py", line 75, in run_gclient

    return self.run(cmd_prefix + cmd, **kwargs)

  File "C:\googlecode\depot_tools\\fetch.py", line 65, in run

    return subprocess.check_call(cmd, **kwargs)

  File "C:\googlecode\depot_tools\python276_bin\lib\subprocess.py", line 535, in

 check_call

    retcode = call(*popenargs, **kwargs)

  File "C:\googlecode\depot_tools\python276_bin\lib\subprocess.py", line 522, in

 call

    return Popen(*popenargs, **kwargs).wait()

  File "C:\googlecode\depot_tools\python276_bin\lib\subprocess.py", line 1006, i

n wait

    _subprocess.INFINITE)

KeyboardInterrupt

Terminate batch job (Y/N)?

Hopefully this helps. 

On Tuesday, June 24, 2014 10:42:21 AM UTC-5, Christian Biesinger wrote:

[Christian Biesinger]

Oh interesting that you mention this, yesterday evening I also saw some errors when updating my existing checkout that made it sound like gclient was running *two* git fetches on the same repository. This may well be the same problem!

I should file a bug...

-christian

[Christian Biesinger]

There may be a command-line flag to get verbose output? Personally I'm happy that the output is now a lot less verbose than it used to be!

Your path does look right.

-christian

On Tue, Jun 24, 2014 at 12:27 PM, Kevin Luty <kevi...@gmail.com> wrote:

I don't think so, currently running fetch chromium --nosvn=True once again in C:\googlecode (next to .gclient file and depot_tools folder).  

Path=%ProgramFiles%\Java\jdk1.7.0_45\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDO

WS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Mi

crosoft Windows Performance Toolkit\;C:\Program Files\Microsoft\Web Platform Ins

taller\;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\;C:\Prog

ram Files (x86)\Windows Kits\8.0\Windows Performance Toolkit\;C:\Program Files\M

icrosoft SQL Server\110\Tools\Binn\;C:\Program Files\TortoiseSVN\bin;C:\Program

Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Python34\;C:\Progra

m Files (x86)\Hyland\Web ActiveX\;C:\googlecode\depot_tools\

to confirm locations:

C:\>where git

C:\googlecode\depot_tools\git.bat

C:\>where svn

C:\googlecode\depot_tools\svn.bat

C:\>where gclient

C:\googlecode\depot_tools\gclient

C:\googlecode\depot_tools\gclient.bat

C:\googlecode\depot_tools\gclient.py

 

Possible issue in gclient?: 

C:\googlecode>gclient hookinfo

Please fix your script, having invalid --revision flags will soon considered an

error.

Also, running fetch chromium --nosvn=True gives me the bland "Still working on: src" messages while downloading.  Was anything ever implemented to provide verbose output?  When I was trying to download at first I found that I can hit Ctrl+C and get info--like Retrieving objects (xxx/nnnnnnn), but then I got "Continue running batch [Y/N]" (or something similar) and choosing Y/N wouldn't tell me if gclient is still working or not.  I was wondering if there was a way to run fetch so that I can see progress updates like what is shown when I run "git clone https://chromium.googlesource.com/chromium/src" in PowerShell.  If something is happening during download currently I have no way of seeing because I get the same "Still working on: src" message and nothing else.

Thanks for your reply.

On Tuesday, June 24, 2014 10:42:21 AM UTC-5, Christian Biesinger wrote:

[Kevin Luty]

That's what it seems like it happening so I would agree with you that you should file a bug.  It happens every time I run fetch so it must be something with the current repo of depot_tools or something.  Instead of running fetch chromium --nosvn=True I'm now running gclient sync --verbose --verbose and it looks like it is pulling the repo in.  We'll see what the outcome is.  I believe I did gclient sync last time and got a src directory but it didn't have a out\Debug folder so I couldn't use the ninja -C command.  

[Kevin Luty]

Below is my original response that I accidentally sent directly to Christian instead of posting on the thread.

I don't think so, currently running fetch chromium --nosvn=True once again in C:\googlecode (next to .gclient file and depot_tools folder).  

Path=%ProgramFiles%\Java\jdk1.7.0_45\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDO

WS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Mi

crosoft Windows Performance Toolkit\;C:\Program Files\Microsoft\Web Platform Ins

taller\;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\;C:\Prog

ram Files (x86)\Windows Kits\8.0\Windows Performance Toolkit\;C:\Program Files\M

icrosoft SQL Server\110\Tools\Binn\;C:\Program Files\TortoiseSVN\bin;C:\Program

Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Python34\;C:\Progra

m Files (x86)\Hyland\Web ActiveX\;C:\googlecode\depot_tools\

to confirm locations:

C:\>where git

C:\googlecode\depot_tools\git.bat

C:\>where svn

C:\googlecode\depot_tools\svn.bat

C:\>where gclient

C:\googlecode\depot_tools\gclient

C:\googlecode\depot_tools\gclient.bat

C:\googlecode\depot_tools\gclient.py

 

Possible issue in gclient?: 

C:\googlecode>gclient hookinfo

Please fix your script, having invalid --revision flags will soon considered an

error.

Also, running fetch chromium --nosvn=True gives me the bland "Still working on: src" messages while downloading.  Was anything ever implemented to provide verbose output?  When I was trying to download at first I found that I can hit Ctrl+C and get info--like Retrieving objects (xxx/nnnnnnn), but then I got "Continue running batch [Y/N]" (or something similar) and choosing Y/N wouldn't tell me if gclient is still working or not.  I was wondering if there was a way to run fetch so that I can see progress updates like what is shown when I run "git clone https://chromium.googlesource.com/chromium/src" in PowerShell.  If something is happening during download currently I have no way of seeing because I get the same "Still working on: src" message and nothing else.

On Tuesday, June 24, 2014 10:42:21 AM UTC-5, Christian Biesinger wrote:

[Kevin Luty]

So I ended up taking a "failed" src download from recycle bin and did git reset --hard HEAD and everything in src populated.  I think did git pull to make sure everything was update to date, which it was.

I then went to finish #2 here: http://dev.chromium.org/developers/how-tos/build-instructions-windows#TOC-Setting-up-the-environment-for-Visual-Studio-2013 and ran gclient runhooks at C:\googlecode

This is my output:

________ running 'C:\googlecode\depot_tools\python276_bin\python.exe src/build/download_nacl_toolchains.py --exclude arm_trusted' in 'C:\googlecode'

C:\googlecode\depot_tools\python276_bin\python.exe: can't open file 'src/build/download_nacl_toolchains.py': [Errno 2] No such file or directory

Error: Command C:\googlecode\depot_tools\python276_bin\python.exe src/build/download_nacl_toolchains.py --exclude arm_trusted returned non-zero exit status 2 in C:\googlecode

so I tried running it manually and it seemed to work:

C:\googlecode>C:\googlecode\depot_tools\python276_bin\python.exe src/build/download_nacl_toolchains.py --exclude arm_trusted

Can't find 'C:\googlecode\src\native_client\build\package_version\package_version.py'

Presumably you are intentionally building without NativeClient.

Skipping NativeClient toolchain download.

I noticed the "src/build/download....py" is using forward slashes instead of back slashes.  Might that be causing the problem?  I don't want to manually run the hooks myself.

Also, after the hooks run, will out\Debug be created?  I still don't have those folders nor build.ninja file so I wouldn't be able to run ninja -C out\Debug.

[Christian Biesinger]

src/ is not enough, there are lot of other repositories that need to be pulled. Try gclient sync?

But yes, once gclient runhooks runs successfully, it will create out/Debug.

-christian

[Kevin Luty]

I decided to install depot_tools and fetch using Cygwin and everything seems to be installing fine.  Time will tell.

[Kevin Luty]

This is my output.  Seems to be failing but successfully retried?

 I removed timestamps in between. Are these WARNINGs expected do you think?

$ fetch chromium --nosvn=True

Running: gclient config --spec 'solutions = [{u'"'"'managed'"'"': False, u'"'"'n ame'"'"': u'"'"'src'"'"', u'"'"'url'"'"': u'"'"'https://chromium.googlesource.com/chromium/src.git'"'"', u'"'"'custom_deps'"'"': {}, u'"'"'deps_file'"'"': u'"'"'.DEPS.git'"'"', u'"'"'safesync_url'"'"': u'"'"''"'"'}]'

Running: gclient sync

[0:01:00] Still working on:

[0:01:00]   src

[0:05:10] Still working on:

[0:05:10]   src

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/src.git" "/home/kevlut/_gclient_src_pHBKxp"' in /home/kevlut failed; will retry after a short nap                                                            ...

[0:05:20] Still working on:

[0:05:20]   src

[0:09:30] Still working on:

[0:09:30]   src

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/src.git" "/home/kevlut/_gclient_src_pHBKxp"' in /home/kevlut failed; will retry after a short nap                                                            ...

[0:09:40] Still working on:

[0:09:40]   src

[0:15:00] Still working on:

[0:15:00]   src

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/src.git" "/home/kevlut/_gclient_src_pHBKxp"' in /home/kevlut failed; will retry after a short nap                                                            ...

[0:15:10] Still working on:

[0:15:10]   src

[0:55:11] Still working on:

[0:55:11]   src

Syncing projects:   0% ( 0/ 3)

[0:55:21] Still working on:

[0:55:21]   src

Syncing projects:  11% (10/88) src/chrome/test/data/extensions/api_test/permissi

Syncing projects:  12% (11/88) src/third_party/brotli/src

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/native_client/src/native_c lient.git" "/home/kevlut/src/_gclient_native_client_yKFO2x"' in /home/kevlut failed; will retry after a short nap...

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/deps/lighttpd.git" "/home/kevlut/src/third_party/_gclient_lighttpd_qheJMU"' in /home/kevlut failed; will retry after a short nap...

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/external/pywebsocket/src.git" "/home/kevlut/src/third_party/pywebsocket/_gclient_src_kUkBAv"' in /home/kevlut failed; will retry after a short nap...

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/native_client/src/third_party/scons-2.0.1.git" "/home/kevlut/src/third_party/_gclient_scons-2.0.1_3ahqTJ"'  in /home/kevlut failed; will retry after a short nap...

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/external/usrsctplib.git" "/home/kevlut/src/third_party/usrsctp/_gclient_usrsctplib_swN4je"' in /home/kevlut failed; will retry after a short nap...

Syncing projects:  75% (66/88) src/third_party/webdriver/pylib

[1:06:43] Still working on:

[1:06:43]   src/chrome/tools/test/reference_build/chrome_win

[1:06:43]   src/native_client

[1:06:43]   src/third_party/WebKit

[1:06:43]   src/third_party/ffmpeg

[1:06:43]   src/third_party/perl

[1:06:43]   src/third_party/skia

[1:06:43]   src/third_party/trace-viewer

[1:06:43]   src/third_party/webgl/src

[1:07:49] Still working on:

[1:07:49]   src/chrome/tools/test/reference_build/chrome_win

[1:07:49]   src/native_client

[1:07:49]   src/third_party/WebKit

[1:07:49]   src/third_party/ffmpeg

[1:07:49]   src/third_party/perl

[1:07:49]   src/third_party/skia

[1:07:49]   src/third_party/trace-viewer

[1:07:49]   src/third_party/webgl/src

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/tools/deps2git.git" "/home/kevlut/src/tools/_gclient_deps2git_xFpe3Q"' in /home/kevlut failed; will retry after a short nap...

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "-c" "pack.threads=1" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/reference_builds/chrome_win.git" "/home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9"' in /home/kevlut failed; will retry after a short nap...

Syncing projects:  89% (79/88) src/tools/gyp

[1:11:13] Still working on:

[1:11:13]   src/chrome/tools/test/reference_build/chrome_win

[1:11:13]   src/third_party/WebKit

[1:11:13]   src/third_party/ffmpeg

[1:11:13]   src/third_party/perl

[1:11:13]   src/third_party/skia

[1:11:13]   src/v8

[1:12:53] Still working on:

[1:12:53]   src/chrome/tools/test/reference_build/chrome_win

[1:12:53]   src/third_party/WebKit

[1:12:53]   src/third_party/ffmpeg

[1:12:53]   src/third_party/perl

[1:12:53]   src/third_party/skia

[1:12:53]   src/v8

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/external/v8.git" "/home/kevlut/src/_gclient_v8_Nxka0B"' in /home/kevlut failed; will retry after a short nap...

[1:13:11] Still working on:

[1:13:11]   src/chrome/tools/test/reference_build/chrome_win

[1:13:11]   src/third_party/WebKit

[1:13:11]   src/third_party/ffmpeg

[1:13:11]   src/third_party/perl

[1:13:11]   src/third_party/skia

[1:13:11]   src/v8

Syncing projects:  90% (80/88) src/third_party/perl

[1:15:25] Still working on:

[1:15:25]   src/chrome/tools/test/reference_build/chrome_win

[1:15:25]   src/third_party/WebKit

[1:15:25]   src/third_party/ffmpeg

[1:15:25]   src/third_party/skia

[1:15:25]   src/v8

Syncing projects:  93% (82/88) src/third_party/ffmpeg

[1:17:57] Still working on:

[1:17:57]   src/chrome/tools/test/reference_build/chrome_win

[1:17:57]   src/third_party/WebKit

[1:17:57]   src/v8

Syncing projects:  94% (83/88) src/v8

[1:20:18] Still working on:

[1:20:18]   src/chrome/tools/test/reference_build/chrome_win

[1:20:18]   src/third_party/WebKit

[1:30:09] Still working on:

[1:30:09]   src/chrome/tools/test/reference_build/chrome_win

[1:30:09]   src/third_party/WebKit

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "-c" "pack.threads=1" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/reference_builds/chrome_win.git" "/home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9"' in /home/kevlut failed; will retry after a short nap...

[1:30:19] Still working on:

[1:30:19]   src/chrome/tools/test/reference_build/chrome_win

[1:30:19]   src/third_party/WebKit

[1:39:49] Still working on:

[1:39:49]   src/chrome/tools/test/reference_build/chrome_win

[1:39:49]   src/third_party/WebKit

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "-c" "pack.threads=1" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/reference_builds/chrome_win.git" "/home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9"' in /home/kevlut failed; will retry after a short nap...

[1:39:59] Still working on:

[1:39:59]   src/chrome/tools/test/reference_build/chrome_win

[1:39:59]   src/third_party/WebKit

(I'm letting it continue still, it's going on 2 hours right now).

[arunoday sarkar]

I have had a similar issue last week during an initial checkout, took over 7 hours. Looks like it take pretty long to sync WebKit folder.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Kevin Luty]

So I'm considering this thread "done."  I am doing the instructions with Cygwin and am seeing better results.  I am experiencing WebKit taking > 1 hour so far as well.  I'm not sure what the issue is, but it's separate from this one I guess.  

And depot_tools and src are in my ~ so I don't think src has to be in its own directory.  That might be the case for using CMD though.

[Torne (Richard Coles)]

The WebKit repo is very big both in terms of the raw number of bytes to download from the server, and also the amount of processing required once it's received to validate the downloaded pack file. git doesn't scale particularly well for such a large repository full of horrible binary deltas (layout tests).

[Kevin Luty]

Is it misleading to output this:

Syncing projects:  95% (84/88) src/chrome/tools/test/reference_build/chrome_win

[2:02:58] Still working on:

[2:02:58]   src/third_party/WebKit

It looks just about done, and if there is only 5% left, but its taking > 2 times longer to finish the remaining 5% than the 95%, it kill your hopes of finishing soon!  Then again it says syncing and now downloading so it could go both ways, but that's irrelevant to the purpose of this thread.  

In short, don't ever get your hopes up! ;)

[Torne (Richard Coles)]

It has synced 95% of the total number of projects. gclient has no way to know how long a specific git clone is going to take - we have a 10000:1 or greater size ratio between different projects ;)

[Kevin Luty]

Latest news.  Its seems to have failed in Cygwin.  You can see the time previous to 2:01:49 in the quoted text to map the errors in git.  Anyone have advice as what I should do next?

[2:01:49] Still working on:

[2:01:49]   src/chrome/tools/test/reference_build/chrome_win

[2:01:49]   src/third_party/WebKit

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "-c" "pack.threads=1" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/reference_builds/chrome_win.git" "/home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9"' in /home/kevlut failed; will retry after a short nap...

[2:01:58] Still working on:

[2:01:58]   src/chrome/tools/test/reference_build/chrome_win

[2:01:58]   src/third_party/WebKit

Syncing projects:  95% (84/88) src/chrome/tools/test/reference_build/chrome_win

[2:02:58] Still working on:

[2:02:58]   src/third_party/WebKit

[2:30:29] Still working on:

[2:30:29]   src/third_party/WebKit

[2:30:39] Still working on:

[2:30:39]   src/third_party/WebKit

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/blink.git" "/home/kevlut/src/third_party/_gclient_WebKit_mrYiPx"' in /home/kevlut failed; will retry after a short nap...

[2:30:49] Still working on:

[2:30:49]   src/third_party/WebKit

[3:48:19] Still working on:

[3:48:19]   src/third_party/WebKit

[3:48:29] Still working on:

[3:48:29]   src/third_party/WebKit

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromium/blink.git" "/home/kevlut/src/third_party/_gclient_WebKit_mrYiPx"' in /home/kevlut failed; will retry after a short nap...

[3:48:39] Still working on:

[3:48:39]   src/third_party/WebKit

[3:52:49] Still working on:

[3:52:49]   src/third_party/WebKit

[3:52:59] Still working on:

[3:52:59]   src/third_party/WebKit

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromi                                                                                                 um/blink.git" "/home/kevlut/src/third_party/_gclient_WebKit_mrYiPx"' in /home/kevlut failed; will retry after a short nap...

[3:53:09] Still working on:

[3:53:09]   src/third_party/WebKit

[3:53:19] Still working on:

[3:53:19]   src/third_party/WebKit

[3:53:29] Still working on:

[3:53:29]   src/third_party/WebKit

WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "clone" "--no-checkout" "--progress" "https://chromium.googlesource.com/chromi                                                                                                 um/blink.git" "/home/kevlut/src/third_party/_gclient_WebKit_mrYiPx"' in /home/kevlut failed; will retry after a short nap...

[3:53:36] Still working on:

[3:53:36]   src/third_party/WebKit

Syncing projects:  96% (85/88) src/third_party/WebKit

src/chrome/tools/test/reference_build/chrome_win (ERROR)

----------------------------------------

[0:55:22] Started.

[0:55:22]

[0:55:22] running 'git -c core.deltaBaseCacheLimit=2g -c pack.threads=1 clone --no-checkout --progress https://chromium.googlesource.com/chr                                                                                                 omium/reference_builds/chrome_win.git /home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9' in '/home/kevlut'

[0:55:23] Cloning into '/home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9'...

[0:55:28] remote: Sending approximately 2.30 GiB ...

[0:55:29] Receiving objects:   0% (1/888)

[0:55:30] Receiving objects:   5% (45/888), 1.94 MiB | 1.80 MiB/s

[0:55:31] Receiving objects:   5% (50/888), 3.25 MiB | 1.52 MiB/s

...

[1:09:48] Receiving objects:  48% (427/888), 933.64 MiB | 727 KiB/s

[1:09:50] Receiving objects:  48% (428/888), 936.45 MiB | 1.10 MiB/s

[1:09:51] error: inflate: data stream error (incorrect data check)

[1:09:51] fatal: pack has bad object at offset 979550569: inflate returned -3

[1:10:00] fatal:

[1:10:01] Cloning into '/home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9'...

[1:10:04] remote: Sending approximately 2.30 GiB ...

[1:10:04] Receiving objects:   0% (1/888)

[1:10:05] Receiving objects:   4% (44/888), 451.87 KiB | 393 KiB/s

[1:10:06] Receiving objects:   5% (45/888), 643.81 KiB | 356 KiB/s

....

[1:30:07] Receiving objects:  70% (628/888), 1.75 GiB | 1.66 MiB/s

[1:30:08] Receiving objects:  70% (628/888), 1.75 GiB | 1.67 MiB/s

[1:30:09] Receiving objects:  70% (628/888), 1.75 GiB | 1.51 MiB/s

[1:30:10] Receiving objects:  70% (628/888), 1.75 GiB | 1.49 MiB/s

[1:30:11] Receiving objects:  70% (628/888), 1.76 GiB | 1.57 MiB/s

[1:30:12] fatal: write error: Broken pipe

[1:30:16] Cloning into '/home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9'...

[1:30:18] remote: Sending approximately 2.30 GiB ...

[1:30:18] Receiving objects:   0% (1/888)

[1:30:19] Receiving objects:   5% (45/888), 1.82 MiB | 1.75 MiB/s

[1:30:20] Receiving objects:   5% (50/888), 4.32 MiB | 2.08 MiB/s

[1:30:21] Receiving objects:   5% (50/888), 5.69 MiB | 1.83 MiB/s

...

[1:39:49] Receiving objects:  46% (416/888), 906.28 MiB | 1.77 MiB/s

[1:39:50] Receiving objects:  46% (416/888), 907.90 MiB | 1.68 MiB/s

[1:39:51] Receiving objects:  46% (416/888), 908.96 MiB | 1.58 MiB/s

[1:39:52] fatal: pack has bad object at offset 870463194: inflate returned -5

[1:39:56] Cloning into '/home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9'...

[1:39:57] remote: Sending approximately 2.30 GiB ...

[1:39:58] Receiving objects:   0% (1/888)

[1:39:59] Receiving objects:   5% (50/888), 2.38 MiB | 2.36 MiB/s

[1:40:00] Receiving objects:   5% (50/888), 4.32 MiB | 2.09 MiB/s

[1:40:01] Receiving objects:   5% (50/888), 5.81 MiB | 1.89 MiB/s

...

[1:58:42] Receiving objects: 100% (888/888), 2.29 GiB | 3.89 MiB/s

[1:58:42] remote: Total 888 (delta 536), reused 888 (delta 536)

[1:58:42] Resolving deltas:   0% (0/537)

...

[2:01:51] Resolving deltas:  86% (462/537)

[2:01:53] error: index-pack died of signal 11

[2:01:53] fatal: index-pack failed

Traceback (most recent call last):

  File "/cygdrive/c/googlecode/depot_tools/gclient_scm.py", line 833, in _Clone

    self._Run(clone_cmd, options, cwd=self._root_dir, retry=True)

  File "/cygdrive/c/googlecode/depot_tools/gclient_scm.py", line 1077, in _Run

    return gclient_utils.CheckCallAndFilter(cmd, env=env, **kwargs)

  File "/cygdrive/c/googlecode/depot_tools/gclient_utils.py", line 526, in CheckCallAndFilter

    rv, args, kwargs.get('cwd', None), None, None)

CalledProcessError: Command git -c core.deltaBaseCacheLimit=2g -c pack.threads=1 clone --no-checkout --progress https://chromium.googlesource.com/chromium/reference_builds/chrome_win.git /home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9 returned non-zero exit status 128 in /home/kevlut

----------------------------------------

Traceback (most recent call last):

  File "/cygdrive/c/googlecode/depot_tools/gclient.py", line 2014, in <module>

    sys.exit(Main(sys.argv[1:]))

  File "/cygdrive/c/googlecode/depot_tools/gclient.py", line 2002, in Main

    return dispatcher.execute(OptionParser(), argv)

  File "/cygdrive/c/googlecode/depot_tools/subcommand.py", line 245, in execute

    return command(parser, args[1:])

  File "/cygdrive/c/googlecode/depot_tools/gclient.py", line 1780, in CMDsync

    ret = client.RunOnDeps('update', args)

  File "/cygdrive/c/googlecode/depot_tools/gclient.py", line 1297, in RunOnDeps

    work_queue.flush(revision_overrides, command, args, options=self._options)

  File "/cygdrive/c/googlecode/depot_tools/gclient_utils.py", line 967, in run

    self.item.run(*self.args, **self.kwargs)

  File "/cygdrive/c/googlecode/depot_tools/gclient.py", line 699, in run

    file_list)

  File "/cygdrive/c/googlecode/depot_tools/gclient_scm.py", line 160, in RunCommand

    return getattr(self, command)(options, args, file_list)

  File "/cygdrive/c/googlecode/depot_tools/gclient_scm.py", line 389, in update

    self._Clone(revision, url, options)

  File "/cygdrive/c/googlecode/depot_tools/gclient_scm.py", line 841, in _Clone

    if os.listdir(tmp_dir):

OSError: [Errno 2] No such file or directory: '/home/kevlut/src/chrome/tools/test/reference_build/_gclient_chrome_win_rleNR9'

Traceback (most recent call last):

  File "/cygdrive/c/googlecode/depot_tools/fetch.py", line 302, in <module>

    sys.exit(main())

  File "/cygdrive/c/googlecode/depot_tools/fetch.py", line 298, in main

    return run(options, spec, root)

  File "/cygdrive/c/googlecode/depot_tools/fetch.py", line 292, in run

    return checkout.init()

  File "/cygdrive/c/googlecode/depot_tools/fetch.py", line 119, in init

    self.run_gclient(*sync_cmd)

  File "/cygdrive/c/googlecode/depot_tools/fetch.py", line 75, in run_gclient

    return self.run(cmd_prefix + cmd, **kwargs)

  File "/cygdrive/c/googlecode/depot_tools/fetch.py", line 65, in run

    return subprocess.check_call(cmd, **kwargs)

  File "/usr/lib/python2.7/subprocess.py", line 542, in check_call

    raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '('gclient', 'sync')' returned non-zero exit status 1

[Daniel Bratell]

On Wed, 25 Jun 2014 20:00:25 +0200, Kevin Luty <kevi...@gmail.com> wrote:

> Latest news. Its seems to have failed in Cygwin. You can see the time
> previous to 2:01:49 in the >quoted text to map the errors in git.
> Anyone have advice as what I should do next?

error: index-pack died of signal 11

Basically cygwin-git crashed.

I think cygwin in this case is a red herring. If you like cygwin and
prefer cygwin, by all means, use it, but otherwise stay in a cmd.exe which
is better supported. That it worked better in cygwin was more likely
because you made a new clean fetch than anything else.

Anyway, if you do crash/fail in gclient sync, it's possible to run it
again and again and hope for better luck. Use the --force flag if
necessary to remove old remains from earlier attempts.

And read through all the instructions and make sure you've not missed a
step. It's all too easy to do that.

/Daniel