win_rel try bots have been failing all day

[Lei Zhang]

I haven't been able to commit anything all day Monday (PDT) due to
win_rel try bot failures. Several other people have complained on IRC
as well.

I've filed a couple bugs:
https://code.google.com/p/chromium/issues/detail?id=129128
https://code.google.com/p/chromium/issues/detail?id=129129

[Wei, James]

Anybody has the experience to access chromium repo with svn:// prefix instead of http:// with proxy?

I met issue when I tried to submit a try job to the try server. It needs to use svn protocol instead of http protocol. Thanks

Best Regards

James

[Jochen Eisinger]

Only committers can use the tryservers currently, see http://dev.chromium.org/developers/testing/try-server-usage

Once you're a committer, you'll receive credentials for the svn://.. url

-jochen

James

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
   http://groups.google.com/a/chromium.org/group/chromium-dev

[Wei, James]

I already have the svn account and get the credentials from https://chromium-access.appspot.com/ , but have issue to access svn://… url. When submitting try job.

 

I suspect the root cause is the proxy.

 

Best Regards

 

James

[Jochen Eisinger]

what is the error message you get?

[Wei, James]

$ git try --user=jame...@intel.com -b linux_layout_rel -v -v -v -v

DEBUG trychange.py(669): ['--user=jame...@intel.com', '-b', 'linux_layout_rel', '-v', '-v', '-v', '-v', '--rietveld_url', 'https://chromiumcodereview.appspot.com']

INFO trychange.py(450): GuessVCS(/home/wistoch/workspace/chromium-gtk/src)

DEBUG subprocess2.py(171): git rev-parse --is-inside-work-tree;  cwd=/home/wistoch/workspace/chromium-gtk/src

DEBUG subprocess2.py(171): git rev-parse --show-cdup;  cwd=/home/wistoch/workspace/chromium-gtk/src

DEBUG subprocess2.py(171): git rev-parse --short=4 HEAD;  cwd=/home/wistoch/workspace/chromium-gtk/src

DEBUG subprocess2.py(171): git symbolic-ref HEAD;  cwd=/home/wistoch/workspace/chromium-gtk/src

DEBUG subprocess2.py(171): git config user.email;  cwd=/home/wistoch/workspace/chromium-gtk/src

DEBUG subprocess2.py(171): git symbolic-ref HEAD;  cwd=/home/wistoch/workspace/chromium-gtk/src

DEBUG subprocess2.py(171): git config branch.master.merge;  cwd=/home/wistoch/workspace/chromium-gtk/src

DEBUG subprocess2.py(171): git config branch.master.remote;  cwd=/home/wistoch/workspace/chromium-gtk/src

INFO trychange.py(266): GIT(/home/wistoch/workspace/chromium-gtk/src)

INFO trychange.py(153): Found .gclient at /home/wistoch/workspace/chromium-gtk

DEBUG subprocess2.py(171): svn info --xml;  cwd=/home/wistoch/workspace/chromium-gtk/src

INFO trychange.py(176): Found codereview.settings at /home/wistoch/workspace/chromium-gtk/src

INFO trychange.py(141): svn_repo: svn://svn.chromium.org/chrome-try/try

DEBUG subprocess2.py(171): git diff --name-status -r refs/remotes/origin/master...;  cwd=/home/wistoch/workspace/chromium-gtk/src

DEBUG subprocess2.py(171): git diff -p --no-color --no-prefix --no-ext-diff refs/remotes/origin/master...HEAD;  cwd=/home/wistoch/workspace/chromium-gtk/src

Results will be emailed to: jame...@intel.com

INFO trychange.py(380): Sending by SVN

INFO trychange.py(381): user=james.wei

name=master#8cd2ca

email=jame...@intel.com

root=src

bot=linux_layout_rel

 

INFO trychange.py(382): svn://svn.chromium.org/chrome-try/try

INFO trychange.py(383):

DEBUG subprocess2.py(171): svn --version

DEBUG subprocess2.py(171): svn import -q /tmp/tmptDs2_0 svn://svn.chromium.org/chrome-try/try --file /tmp/tmpQf9AWS --no-ignore

svn: Can't connect to host 'svn.chromium.org': Connection timed out

Command svn import -q /tmp/tmptDs2_0 svn://svn.chromium.org/chrome-try/try --file /tmp/tmpQf9AWS --no-ignore returned non-zero exit status 1

Sorry, Tryserver is not available.

 

 

Also I failed to run svn ls with svn:// but ok with http://

$ svn ls --username jame...@intel.com http://src.chromium.org/svn/trunk

deps/

o3d/

src/

tools/

 

$ svn ls --username jame...@intel.com svn://src.chromium.org/svn/trunk

svn: Can't connect to host 'src.chromium.org': Connection timed out

 

I tried to use tsocks with socks5 proxy to access it but failed.

$ tsocks svn ls --username jame...@intel.com svn://src.chromium.org/svn/trunk

17:59:06 libtsocks(7201): SOCKS V5 connect failed: 17:59:06 libtsocks(7201): General SOCKS server failure

svn: Can't connect to host 'src.chromium.org': Software caused connection abort

 

I used connect-tunnel to forward the connection:

$ connect-tunnel -P proxy-of-company.com:port -T 12345:src.chromium.org:3690

$ svn ls --username jame...@intel.com svn://localhost:12345/svn/trunk

svn: Network connection closed unexpectedly

 

I used svn+ssh but still failed.

.ssh/config

Host chromium.org

ProxyCommand /usr/bin/connect -S company-proxy-socks.com:1080 %h %p

$ svn ls --username jame...@intel.com svn+ssh://src.chromium.org/svn/trunk

ssh: connect to host src.chromium.org port 22: Connection timed out

svn: To better debug SSH connection problems, remove the -q option from 'ssh' in the [tunnels] section of your Subversion configuration file.

svn: Network connection closed unexpectedly

[Jochen Eisinger]

what about svn ls svn://svn.chromium.org/chrome-try ?

-jochen

[Wei, James]

Jochen,

 

Thanks for your help. here is the output of the command.

 

$ svn ls http://svn.chromium.org/chrome-try

svn: Server sent unexpected return value (503 Service Unavailable) in response to OPTIONS request for 'http://svn.chromium.org/chrome-try'

 

$ svn ls svn://svn.chromium.org/chrome-try

svn: Can't connect to host 'svn.chromium.org': Connection timed out

 

 

Best Regards

[Nicolas Sylvain]

On Tue, May 22, 2012 at 5:48 AM, Wei, James <jame...@intel.com> wrote:

Jochen,

 

Thanks for your help. here is the output of the command.

 

$ svn ls http://svn.chromium.org/chrome-try

svn: Server sent unexpected return value (503 Service Unavailable) in response to OPTIONS request for 'http://svn.chromium.org/chrome-try'

 

$ svn ls svn://svn.chromium.org/chrome-try

svn: Can't connect to host 'svn.chromium.org': Connection timed out

svn.chromium.org is not accessible over http. We are using the svn:// protocol, which is not easily proxyable.  You need to be able to open svn.chromium.org on port 3690 if you want to be able to manually commit or submit try jobs.

Nicolas 

[Lei Zhang]

I still see these failures on the win_rel trybots and everyone who has
committed recently has skipped the CQ.

Siggi has jumped on bug 129129 and said a fix is coming soon.
Bug 129128 has remained untouched...

[Nico Weber]

I guess this is triggered by the msvs 2008 -> 2010 migration? Maybe
that could be undone until the issues are worked out?

Nico

[Nicolas Sylvain]

On Tue, May 22, 2012 at 12:33 PM, Nico Weber <tha...@chromium.org> wrote:

I guess this is triggered by the msvs 2008 -> 2010 migration? Maybe
that could be undone until the issues are worked out?

Yeah, it looks like it.

I'm in favor of reverting, but I can also disable IB for vs2010.  I dont think we've seen those problems when IB is not running.  I have no idea what this will do to the cycle time though.

I'll start prepping the change, if someone disagree, let me know

Nicolas

[John Abd-El-Malek]

It also seems that win try jobs are triggering full builds each time, which takes ~40 minutes. Someone had pointed me towards a gyp change that caused this. Can we roll back gyp until that's fixed?

On Tue, May 22, 2012 at 12:33 PM, Nico Weber <tha...@chromium.org> wrote:

[Nico Weber]

On Tue, May 22, 2012 at 1:00 PM, John Abd-El-Malek <j...@chromium.org> wrote:
> It also seems that win try jobs are triggering full builds each time, which
> takes ~40 minutes. Someone had pointed me towards a gyp change that caused
> this. Can we roll back gyp until that's fixed?

The gyp change that caused this might've been rolled back in
http://code.google.com/p/gyp/source/detail?r=1383 , which has been
merged to chromium. Maybe there's another cause.

[oshima]

fwiw, win_aura trybot has been failing all day too. Filed crbug.com/129223.

- oshima

[Nicolas Sylvain]

btw when I suggested to disable IB I did not really mean to imply that it would fix the 2 bugs above.  Those 2 tests should probably be disabled if they are not fixed yet.

What I'm trying to fix is the problem causing most of the try jobs to fail to compile with :

18>------ Build started: Project: generate_supplemental_dependency, Configuration: Release Win32 ------

18>Build started 5/22/2012 12:59:17 PM.

18>InitializeBuildStatus:
18>  Creating "../../../../..\build\Release\obj\generate_supplemental_dependency\generate_supplemental_dependency.unsuccessfulbuild" because "AlwaysCreate" was specified.

18>CustomBuild:
18>  Resolving [Supplemental=XXX] dependencies in all IDL files
18>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppCommon.targets(151,5): error MSB6006: "cmd.exe" exited with code 512.

18>
18>Build FAILED.
18>
18>Time Elapsed 00:00:34.92

Anyone knows what the error means?


Nicolas

[Sigurður Ásgeirsson]

The fix [https://chromiumcodereview.appspot.com/10407098/] is in the CQ - maybe I should commit this by hand.

[Nicolas Sylvain]

On Tue, May 22, 2012 at 1:28 PM, Sigurður Ásgeirsson <si...@chromium.org> wrote:

The fix [https://chromiumcodereview.appspot.com/10407098/] is in the CQ - maybe I should commit this by hand.

Probably a good thing to do.

Do you know why the failure was not seen on the main waterfall?

Nicolas

[Sigurður Ásgeirsson]

On Tue, May 22, 2012 at 5:05 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:

On Tue, May 22, 2012 at 1:28 PM, Sigurður Ásgeirsson <si...@chromium.org> wrote:

The fix [https://chromiumcodereview.appspot.com/10407098/] is in the CQ - maybe I should commit this by hand.

Probably a good thing to do.

Do you know why the failure was not seen on the main waterfall?

The test was using uninitialized data, and only seems to fail if a GUID happens to be all zeros. Presumably something's changed, either it's slightly different layouts or allocation patterns under VS2010 or else something's been perturbed in that test.

[Sigurður Ásgeirsson]

On Tue, May 22, 2012 at 5:05 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:

On Tue, May 22, 2012 at 1:28 PM, Sigurður Ásgeirsson <si...@chromium.org> wrote:

The fix [https://chromiumcodereview.appspot.com/10407098/] is in the CQ - maybe I should commit this by hand.

Probably a good thing to do.

Looks like my checkout is in a wonky state - git cl dcommit started listing all kinds of files I don't want it to commit. Re-syncing :(.

[Wei, James]

Jochen, Nicolas,

 

I successfully enabled it with tsocks at last.

 

$ tsocks svn ls --username jame...@intel.com svn://svn.chromium.org/chrome-try

clean_tries.py

try/

try-nacl/

try-o3d/

 

thanks for your help.

 

Best Regards

 

James

[Greg Spencer]

On Tue, May 22, 2012 at 2:25 PM, Sigurður Ásgeirsson <si...@chromium.org> wrote:

On Tue, May 22, 2012 at 5:05 PM, Nicolas Sylvain <nsyl...@chromium.org> wrote:

Do you know why the failure was not seen on the main waterfall?

And any reason why the CQ is taking seven or more hours to fail?  If I want to be around when the jobs lands, I'm unable to use the CQ after 10am...

-Greg. 

[Nicolas Sylvain]

It's because the CQ retries when there are failed tests in case it's flaky.  The thing is, because of https://code.google.com/p/chromium/issues/detail?id=129128, all jobs were failing, so all jobs were being retried. Which means we had a lot more requests than usual on the CQ and there was a huge backlog.   Coupled with a few other problems caused the compile step to be much slower, this made everything really unusable.

Bug 129128 got fixed a few hours ago, and the fix made it to the trybots 30 minutes ago.  Things should start catching up now.

Nicolas

-Greg.