Windows Chromium sync fail

[Binod Pant]

I keep running into something like this after about an hour of sync.

I do not wy the WebKit\layout stuff is syncing, althought my .gclient file looks like this


solutions = [
  { "name"        : "src",
    "url"         : "https://src.chromium.org/chrome/trunk/src",
    "deps_file"   : "DEPS",
    "managed"     : True,
  "custom_deps": {
      "src/third_party/WebKit/LayoutTests": None,
      "src/content/test/data/layout_tests/LayoutTests": None,
      "src/chrome/tools/test/reference_build/chrome_win": None,
      "src/chrome_frame/tools/test/reference_build/chrome_win": None,
      "src/chrome/tools/test/reference_build/chrome_linux": None,
      "src/chrome/tools/test/reference_build/chrome_mac": None,
      "src/third_party/hunspell_dictionaries": None,
    },
    "safesync_url": "",
  },
]




The command I ran was gclient sync --force

I used to be able to do this a few weeks ago, but when I tried this again, I am not able to get past this issue. I used an admin console just to be sure it was not a permissions issue.

After this, I do not see the src/chrome/chrome.sln file I expect. Probably means my sync did not complete.


Any ideas?




35>A    C:\Work\chromium_trunk\src\third_party\WebKit\LayoutTests\fast\xpath\4XPath\Core\test_node_test-expected.txt
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
35>Sleeping 5.0 seconds and retrying....
38>________ running 'svn update c:\Work\chromium_trunk\src\third_party\bidichecker --revision 4 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
38>At revision 4.
35>________ running 'svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
35>A    C:\Work\chromium_trunk\src\third_party\WebKit\LayoutTests\fast\xpath\4XPath\Core\test_core_functions-expected.txt
35>A    C:\Work\chromium_trunk\src\third_party\WebKit\LayoutTests\fast\xpath\4XPath\Core\test_literal_expr.html
35>A    C:\Work\chromium_trunk\src\third_party\WebKit\LayoutTests\fast\xpath\4XPath\Core\test_predicate_list.html
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
35>Sleeping 6.5 seconds and retrying....
35>________ running 'svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
35>Sleeping 8.5 seconds and retrying....
35>________ running 'svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
35>Sleeping 11.0 seconds and retrying....
35>________ running 'svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
35>Sleeping 14.3 seconds and retrying....
35>________ running 'svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
35>Sleeping 18.6 seconds and retrying....
35>________ running 'svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
35>Sleeping 24.1 seconds and retrying....
35>________ running 'svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
35>Sleeping 31.4 seconds and retrying....
35>________ running 'svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
35>Sleeping 40.8 seconds and retrying....
35>________ running 'svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals' in 'c:\Work\chromium_trunk'
35>svn: PROPFIND of '/blink/!svn/bc/19963/trunk/LayoutTests/fast/xpath/4XPath/Core/test.js': 207 Multi-Status (http://src.chromium.org)
Error: Command svn update c:\Work\chromium_trunk\src\third_party\WebKit --revision 150602 --non-interactive --force --accept theirs-conflict --ignore-externals returned non-zero exit status 1 in c:\Work\chromium_trunk

[Thiago Farina]

Are you using 'fetch chromium'? As far as I know WebKit, now Blink, is
required and it's necessary to fetch all of it. The exclude line you
added to .gclient

--
Thiago

[Dirk Pranke]

You're hitting a problem that seems to be affecting some combination of Windows svn +  our public SVN mirror (the 207 Multi-Status), which appears to be some sort of weird svn server bug that we don't really understand yet.

The only workaround I know of is to switch to Git. As Thiago suggests, 'fetch chromium' is the fastest way to get started with that.

-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
 
 
 

[Binod Pant]

Thanks Thiago and Dirk


So I tried to use the fetch chromium git method


Here is what i did

c:\Work\chromium>gclient

c:\Work\chromium>fetch chromium --nosvn=True
Cannot rebase: You have unstaged changes.
Please commit or stash them.
Running: svn ls --non-interactive svn://svn.chromium.org/chrome
svn: Can't get username or password
Please run `svn ls svn://svn.chromium.org/chrome`

So I tried to go to the sign up page for svn access, and that page does not seem to be working. How do I get access to be abel to run fetch chromium?

[Dirk Pranke]

On Tue, May 21, 2013 at 11:34 AM, Binod Pant <bin...@gmail.com> wrote:

Thanks Thiago and Dirk


So I tried to use the fetch chromium git method


Here is what i did

c:\Work\chromium>gclient

c:\Work\chromium>fetch chromium --nosvn=True
Cannot rebase: You have unstaged changes.
Please commit or stash them.
Running: svn ls --non-interactive svn://svn.chromium.org/chrome
svn: Can't get username or password
Please run `svn ls svn://svn.chromium.org/chrome`

So I tried to go to the sign up page for svn access, and that page does not seem to be working. How do I get access to be abel to run fetch chromium?

For reasons that aren't at all obvious to me, the --nosvn=True argument isn't doing the right thing for you. Note that 'fetch' only works for the initial checkout, so perhaps you're running this in an existing checkout and weird things are happening (we're really bad about that).

Try creating a new directory and running 'fetch chromium --nosvn=True' there. If in doubt, run 'fetch -n chromium --nosvn=True' to see what commands will be executed. If you're still getting the svn prompts, file a bug and we'll dig into it further.

You should not need svn access to get the code.

-- Dirk

[Binod Pant]

Thanks, after following the steps,

cd c:\temp
mkdir chromium
cd chromium
gclient
fetch -n chromium --noSvn=True


I get this

c:\Work\chromium>fetch -n chromium --noSvn=True

Cannot rebase: You have unstaged changes.
Please commit or stash them.
Running: svn ls --non-interactive svn://svn.chromium.org/chrome

Running: git config --global core.deltaBaseCacheLimit 1G
Running: gclient config --spec "solutions = [{u'managed': True, u'name': u'src', u'url': u'https://chromium.googlesource.com/chromium/src.git', u'custom_deps': {}, u'deps_file': u'.DEPS.git', u'safesync_url': u''}]"
Running: gclient sync
cd c:\Work\chromium\src
Running: git submodule foreach 'git config -f $toplevel/.git/config submodule.$name.ignore all'
Running: git config diff.ignoreSubmodules all
cd c:\Work\chromium\src
Running: git svn init --prefix=origin/ -T trunk/src svn://svn.chromium.org/chrome
Running: git config --replace svn-remote.svn.fetch trunk/src:refs/remotes/origin/git-svn
Running: git svn fetch


But nothing is synced or created where I am...

[Binod Pant]

Having not had sucess with the git workflow , I tried to use the gclient sync workflow again, and now I am able to get to this point


32>Updated to revision 150885.
32>Summary of conflicts:
32>  Skipped paths: 1
Syncing projects:  98% (100/102) src/third_party/WebKit
Allowed parallel jobs: 12
# queued: 0


But it stood stuck there for the whole night, and I had to interrupt the sync. After that, I don't see src/chrome/chrome.sln still! Is there a different way to do this? I have lost my ability to build on Windows now. Maybe i will try a different machine.

[Thiago Farina]

One thing is to checkout the source tree, the other is to actually
build the source code.

https://code.google.com/p/chromium/wiki/UsingGit
http://www.chromium.org/developers/how-tos/build-instructions-windows#TOC-Automatic-simplified-toolchain-setup

Are you following those instructions precisely?

--
Thiago

[Binod Pant]

I believe so.

Although this time i am not sure it is a sync issue. I look at this URL and I don't see a src/chrome/chrome.sln there either! My head is really spinning now, have the .sln files moved?  The doc does not indicate so. I know I must look like a fool in case I am missing something really basic here

http://src.chromium.org/chrome/trunk/src/chrome/

[Binod Pant]

actually, let me first try steps 4 and beyond of the second link, then see where I get. Thanks

[Binod Pant]

I know I must sound confusing based on the mixed messages I have posted so far, but there are two different things here. So let me clarify.


a. If i follow the 'simplified automated' stuff you pointed to, it says I cannot build using MSVS (it will be great if I can do that, since I have been doing so in the last round of work on this). I will try it anyway if it means I can build my projects.
b. Thus far in the lifespan of this thread, I have been trying to follow steps in the next section which I am pasting below. I am skipping steps 3 and 4, but I did 1 and 2 (and 2 is where I have been having problems I listed in this thread). But by the time I run step 2, there is no chrome.sln! That is the problem I am trying to get around

Building Chromium

1. Get the Chromium depot_tools.
2. Check out the source code using a direct svn or git checkout. Do not use a tarball since it is not compatible with Windows' svn client.
3. Set up the component build (or else creating chrome_dll.pdb may fail laterl) and then regenerate the build files by running "gclient runhooks --force" in a cygwin/cmd window.
4. Install API keys.
5. Open the chrome/chrome.sln solution file in Visual Studio and build the solution. This can take from 10 minutes to 2 hours. More likely 1 hour.
6. If you just want the Chromium browser, and none of the tests, you can speed up your build by right-clicking the chrome project in the solution explorer and selecting Build. You may want to make sure this project is the Startup project (which will display as bold) by right-clicking it and selecting Set as Startup Project. This will make Chromium (as opposed to some random test) build and run when you press F5.

So, first I need to get past step 2 (which is this thread), then I need to be able to do 5.

Thanks

[Jack]

That would be because the .sln file isn't on the repo. It's generated by the build\gyp_chromium 

python script, which is called at the end of the gclient runhooks procedure.

3. -> [...] and then regenerate the build files by running gclient runhooks --force [...]

Without completing step3, no build.

Also, without following the git workflow, the sync sometimes get stuck. Happened to me often.

In this case, interrupting the gclient sync (by ctrl-c'ing it or closing the shell window) does bad things to the tree.

The workaround I followed when it happened : kill the svn.exe processes instead. This will un-stuck your sync, and report a timeout error.

Then you clean the lock and log files in svn, and launch the sync again. It will resume.

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
 
 
 

--
Seriously, Jack

[Binod Pant]

Ya, no luck still

i have tried to sync in a clean directory like you said, still fails to get the WebKit stuff. This is with the gclient sync approach


If I switch to the git approach, i run fetch chrome --noSvn=True and I get the same error message about svn

Cannot rebase: You have unstaged changes.
Please commit or stash them.
Running: svn ls --non-interactive svn://svn.chromium.org/chrome
svn: Can't get username or password
Please run `svn ls svn://svn.chromium.org/chrome`

so, I am completely stuck now.

[Binod Pant]

And I can confirm that running gclient sync now consistently gives me this failure


Enabled Psyco JIT.
Updating projects from gyp files...
gyp: c:\Work\chromium_trunk\src\third_party\WebKit\Source\devtools\devtools.gyp not found (cwd: c:\Work\chromium_trunk) while loading dependencies of c:\Work\chromium_trunk\src\content\browser\devtools\devtools_resources.gyp while loading dependencies of c:\Work
romium_trunk\src\chrome\chrome_resources.gyp while loading dependencies of c:\Work\chromium_trunk\src\chrome\chrome.gyp while loading dependencies of c:\Work\chromium_trunk\src\build\all.gyp while trying to load c:\Work\chromium_trunk\src\build\all.gyp
Error: Command C:\installs\chromium_depot_tools\depot_tools\python_bin\python.exe src/build/gyp_chromium returned non-zero exit status 1 in c:\Work\chromium_trunk

[Binod Pant]

And it also kind of makes sense, since there is not third_party\WebKit\Source\devtools\devtools.gyp even in the trunk! Something is not right with my config, or with where I am syncing this from

Is my .gclient file even correct I wonder?

solutions = [
  { "name"        : "src",
    "url"         : "https://src.chromium.org/chrome/trunk/src",
    "deps_file"   : "DEPS",
    "managed"     : True,
  "custom_deps": {

      "src/content/test/data/layout_tests/LayoutTests": None,
      "src/chrome/tools/test/reference_build/chrome_win": None,
      "src/chrome_frame/tools/test/reference_build/chrome_win": None,
      "src/chrome/tools/test/reference_build/chrome_linux": None,
      "src/chrome/tools/test/reference_build/chrome_mac": None,
      "src/third_party/hunspell_dictionaries": None,
    },
    "safesync_url": "",
  },
]

[Thiago Farina]

On Thu, May 30, 2013 at 7:10 PM, Binod Pant <bin...@gmail.com> wrote:
> And I can confirm that running gclient sync now consistently gives me this
> failure
>
>
> Enabled Psyco JIT.
> Updating projects from gyp files...
> gyp:
> c:\Work\chromium_trunk\src\third_party\WebKit\Source\devtools\devtools.gyp
> not found (cwd: c:\Work\chromium_trunk) while loading dependencies of
> c:\Work\chromium_trunk\src\content\browser\devtools\devtools_resources.gyp
> while loading dependencies of c:\Work
> romium_trunk\src\chrome\chrome_resources.gyp while loading dependencies of
> c:\Work\chromium_trunk\src\chrome\chrome.gyp while loading dependencies of
> c:\Work\chromium_trunk\src\build\all.gyp while trying to load
> c:\Work\chromium_trunk\src\build\all.gyp
> Error: Command
> C:\installs\chromium_depot_tools\depot_tools\python_bin\python.exe
> src/build/gyp_chromium returned non-zero exit status 1 in
> c:\Work\chromium_trunk
>

This failure is not from gclient sync, it's from gclient runhooks
which is running build/gyp_chromium. See if in your checkout you have
third_party/WebKit/Source/devtools/devtools.gyp, if not you need to
sync again until you get all source tree synced correctly.

--
Thiago

[Vinit]

After running through exact same set of errors Binod reported. I am stuck on the same error, where updaring projects from gyp files fail consistently. I have been trying to solve this for two days now. Please keep me in loop for any resolution you reach.

[Daniel Cheng]

F

​rom what I can tell, it sounds like fetch failed somewhere and left an incomplete checkout. In this case, it looks like WebKit didn't get checked out completely. Here's what I'd do:

​Run fetch -n ​<your options of choice> to see what commands it normally runs to checkout WebKit.

rm -rf third_party/WebKit

Then run the git clone commands reported by fetch -n that create third_party/WebKit

Daniel

--

[Vinit]

I am not sure if you wanted me to start over or something else so I did this.

//in a completely new location I did

fetch chromium

//which throws this error in between
14>  No output for 30 seconds from command:
14>    git clone --progress https://chromium.googlesource.com/chromium/blink.git
 C:\Chrome\tree\src\third_party\WebKit;  cwd=C:\Chrome\tree

// I checked in WebKit location, there's nothing except ".git"

This message will appear for 180 seconds since that's the max_nag time in the script. after which it will try to kill the subprocess. I was seeing this initially, which is why I thought of doing "gclient sync" but neither works. As you mentioned it should not work with incomplete "fetch".

I don't know how to resolve this. also if I have misinterpreted your reply please feel free to correct me.

Thanks,
Vinit

--
Vinit

[Thiago Farina]

On Fri, May 31, 2013 at 8:28 PM, Vinit <shorte...@gmail.com> wrote:
> I am not sure if you wanted me to start over or something else so I did
> this.
>
> //in a completely new location I did
> fetch chromium
>

You need --nosvn=True here.

--
Thiago

[Scott Graham]

Some recent discussions here:

https://codereview.chromium.org/16004008/

https://code.google.com/p/chromium/issues/detail?id=241339

[Vinit]

I did it with (--nosvn=True) but forgot to mention it in the writeup. I am really sorry for that.

--
Vinit

[Vinit]

I have tried to raise nag_max to 10x but that does not help either it still fails after waiting 1800 seconds.

[Thiago Farina]

On Fri, May 31, 2013 at 8:53 PM, Vinit <shorte...@gmail.com> wrote:
> I have tried to raise nag_max to 10x but that does not help either it still
> fails after waiting 1800 seconds.
>

Is your connection fast enough?

--
Thiago

[Vinit]

Yes, see below. It downloads first 7-8 gig of source in less than 40 mins. It's only WebKit giving all these trouble.

14>Receiving objects:  90% (2043716/2270795), 3.60 GiB | 9.64 MiB/s
14>Receiving objects:  90% (2054590/2270795), 3.60 GiB | 9.88 MiB/s
14>Receiving objects:  90% (2065690/2270795), 3.61 GiB | 10.37 MiB/s
14>Receiving objects: 100% (2270795/2270795), 3.96 GiB | 8.90 MiB/s
14>Receiving objects: 100% (2270795/2270795), 3.97 GiB | 9.08 MiB/s, done

the recent fetch I did failed as well. I don't know why "fetch" is calling "sync" before fetch is complete.
 

here's the error I get :

Error: Command git clone --progress https://chromium.googlesource.com/chromium/b
link.git C:\Chrome\tree\src\third_party\WebKit returned non-zero exit status
1 in C:\Chrome\tree
Traceback (most recent call last):
  File "C:\Chrome\depot_tools\\fetch.py", line 278, in <module>
    sys.exit(main())
  File "C:\Chrome\depot_tools\\fetch.py", line 274, in main
    return run(dryrun, spec, root)
  File "C:\Chrome\depot_tools\\fetch.py", line 268, in run
    return checkout.init()
  File "C:\Chrome\depot_tools\\fetch.py", line 117, in init
    self.run_gclient('sync')
  File "C:\Chrome\depot_tools\\fetch.py", line 75, in run_gclient
    return self.run(cmd_prefix + cmd, **kwargs)
  File "C:\Chrome\depot_tools\\fetch.py", line 65, in run
    return subprocess.check_call(cmd, **kwargs)
  File "C:\Chrome\depot_tools\python_bin\lib\subprocess.py", line 462, in check_
call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('C:\\Chrome\\depot_tools\\python_bin\\p
ython.exe', 'C:\\Chrome\\depot_tools\\gclient.py', 'sync')' returned non-zero ex
it status 1

----------------------------------------------

I did "gclient sync" After this error, this is the error I get.

-----------------------------------------------------

Enabled Psyco JIT.
Updating projects from gyp files...

Warning: unrecognized setting VCLinkerTool/AdditionalDependencies_excluded while
 converting to MSBuild.
Warning: unrecognized setting VCLinkerTool/DelayLoadDLLs_excluded while converti
ng to MSBuild.
Warning: unrecognized setting VCLinkerTool/DelayLoadDLLs_excluded while converti
ng to MSBuild.
Warning: unrecognized setting VCLinkerTool/AdditionalDependencies_excluded while
 converting to MSBuild.
Warning: unrecognized setting VCLinkerTool/AdditionalDependencies_excluded while
 converting to MSBuild.
Warning: unrecognized setting VCLinkerTool/DelayLoadDLLs_excluded while converti
ng to MSBuild.
Warning: unrecognized setting VCLinkerTool/DelayLoadDLLs_excluded while converti
ng to MSBuild.
Warning: unrecognized setting VCLinkerTool/AdditionalDependencies_excluded while
 converting to MSBuild.

________ running 'C:\Chrome\depot_tools\python_bin\python.exe src/build/landmine
s.py' in 'C:\Chrome\tarball'

Although I got these errors  it generated all the project files so I can build now. I am not sure if this error I going to give me problems later but I am good for now.

Thanks,

Vinit

--
Vinit