WK perf canaries ongoing failures on dromaeo_domcoremodify

[Andrew Wilson]

Looks like dromaeo_domcoremodify has been failing for months on Win7. Should we remove this test from the canary?

I dimly recall that dominicc was looking at this a few weeks back - did we ever figure out anything?

-atw

[Annie Sullivan]

+phajdan.jr

I think the problem is the reference build needs to be updated on Win7. See crbug.com/136110

[Paweł Hajdan, Jr.]

I had a huge trouble getting that reference build updated. With my update all reference perf tests started to fail with the browser failing to launch.

Help is welcome.

Paweł

[Annie Sullivan]

+cmp, tonyg

Tony, IIRC you needed to update the reference builds for Telemetry, once, right? Any ideas?

Chase, any thoughts on how to debug when the reference build is updated and the browser won't launch?

Thanks,

Annie

[Tony Gentilcore]

Yeah, I agree the ref builds just need to be updated for Dromaeo to pass.

I've never updated the reference builds. But I wonder whether the
build Paweł tried to update to was just a bad revision? Perhaps it is
worth another shot?

[Paweł Hajdan, Jr.]

Maybe someone else this time? I'm just not really enthusiastic about this, sorry. :(

Paweł

[Marcus Bulach]

Hey everyone,

Peter just asked me to update the ref build.

I tried to dig some docs, and found this:
http://src.chromium.org/viewvc/chrome/trunk/deps/reference_builds/README?revision=137801&view=markup

Questions:
1) Is this the right doc? :) I'll put a link to it from https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/perf-sheriffs
2) Any strong opposition if I codify that in python rather than in several manual steps?

3) I'm not quite sure about "For the reference build, just take a build off the waterfall's continuous builds."... do these correspond to http://build.chromium.org/p/chromium/builders/Win, Mac, Linux, Linux64? 

4) Are there any bots building from branch we should pick? Maybe we could have some better definition for which rev to pick up, say, update to builds from a given branch point?

Thanks,

Marcus

[Peter Kasting]

On Fri, Mar 22, 2013 at 2:55 AM, Marcus Bulach <bul...@chromium.org> wrote:

1) Is this the right doc? :) I'll put a link to it from https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/perf-sheriffs

I think so -- the infrastructure folks linked me to that too.

3) I'm not quite sure about "For the reference build, just take a build off the waterfall's continuous builds."... do these correspond to http://build.chromium.org/p/chromium/builders/Win, Mac, Linux, Linux64? 

The "continuous" link on the waterfall goes to http://commondatastorage.googleapis.com/chromium-browser-continuous/index.html .

4) Are there any bots building from branch we should pick? Maybe we could have some better definition for which rev to pick up, say, update to builds from a given branch point?

Maybe we should stick with LKGR builds?

PK 

[Paweł Hajdan, Jr.]

On Fri, Mar 22, 2013 at 2:55 AM, Marcus Bulach <bul...@chromium.org> wrote:

Peter just asked me to update the ref build.

I tried to dig some docs, and found this:
http://src.chromium.org/viewvc/chrome/trunk/deps/reference_builds/README?revision=137801&view=markup

Questions:
1) Is this the right doc? :) I'll put a link to it from https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/perf-sheriffs

SG

 

2) Any strong opposition if I codify that in python rather than in several manual steps?

SG

[Marcus Bulach]

On Fri, Mar 22, 2013 at 5:55 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

On Fri, Mar 22, 2013 at 2:55 AM, Marcus Bulach <bul...@chromium.org> wrote:

Peter just asked me to update the ref build.

I tried to dig some docs, and found this:
http://src.chromium.org/viewvc/chrome/trunk/deps/reference_builds/README?revision=137801&view=markup

Questions:
1) Is this the right doc? :) I'll put a link to it from https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/perf-sheriffs

SG

 

2) Any strong opposition if I codify that in python rather than in several manual steps?

SG

thanks both!

Looking for reviewer & dogfooder, it'd be nice if someone who has previously updated the ref build could give it a go:

https://codereview.chromium.org/12662037

This tool seem to be updating the "right" things, but a few questions:

1) I couldn't find a linux64 git mirror.

2) Actually, is it even possible to do this from git rather than directly via svn?

3) Mac seem to have a healthy mix of directories with version numbers, and a "un-numbered" directory, but I suppose this script should just update the un-numbered directory?

Thanks,

Marcus

[Paweł Hajdan, Jr.]

On Fri, Mar 22, 2013 at 12:06 PM, Marcus Bulach <bul...@chromium.org> wrote:

Looking for reviewer & dogfooder, it'd be nice if someone who has previously updated the ref build could give it a go:

https://codereview.chromium.org/12662037

Thank you for working on this!

Do we have a reference build successfully rolled using this tool? I think that'd be a prerequisite.

[Marcus Bulach]

On Mon, Mar 25, 2013 at 5:35 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

On Fri, Mar 22, 2013 at 12:06 PM, Marcus Bulach <bul...@chromium.org> wrote:

Looking for reviewer & dogfooder, it'd be nice if someone who has previously updated the ref build could give it a go:

https://codereview.chromium.org/12662037

Thank you for working on this!

Do we have a reference build successfully rolled using this tool? I think that'd be a prerequisite.

no, that's the part I'm asking for a dogfooder :)

I've never done any roll myself, so I'd rather mitigate the risk..

btw, whilst writing this tool, I wondered: do we really need to do this roundtrip, i.e., buildbot -> googlestorage -> local dev machine -> svn -> buildbot?

I don't know all the history and reasons behind it, but would it be safe / better to just let the bots pull from buildbot directly?

That would make "rolling the ref build" just a matter of updating one data file pointing to the googlestorage URL, and using this tool to fetch / unzip to the right place... wdyt?

[Chase Phillips]

On Mon, Mar 25, 2013 at 11:40 AM, Marcus Bulach <bul...@chromium.org> wrote:

On Mon, Mar 25, 2013 at 5:35 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

On Fri, Mar 22, 2013 at 12:06 PM, Marcus Bulach <bul...@chromium.org> wrote:

Looking for reviewer & dogfooder, it'd be nice if someone who has previously updated the ref build could give it a go:

https://codereview.chromium.org/12662037

Thank you for working on this!

Do we have a reference build successfully rolled using this tool? I think that'd be a prerequisite.

no, that's the part I'm asking for a dogfooder :)

I've never done any roll myself, so I'd rather mitigate the risk..

btw, whilst writing this tool, I wondered: do we really need to do this roundtrip, i.e., buildbot -> googlestorage -> local dev machine -> svn -> buildbot?

I don't know all the history and reasons behind it, but would it be safe / better to just let the bots pull from buildbot directly?

That would make "rolling the ref build" just a matter of updating one data file pointing to the googlestorage URL, and using this tool to fetch / unzip to the right place... wdyt?

Yes, this is where we're headed in general, but we're not going in this direction to simplify updating the ref build.  Our general goal is to avoid needing to check large binary files into SVN.  The future flow would look like:

buildbot -> googlestorage in bucket 1 -> local dev machine -> google storage in bucket 2 -> svn -> buildbot

If we make "bucket 2" be the same as "bucket 1" (aka pull ref builds from the same bucket we upload to), we can make the flow look like:

buildbot -> googlestorage

local dev machine -> svn -> buildbot

I agree this is simpler.  It will require some upfront work on someone's part to ensure the googlestorage bucket layout maps to the same format we'll use for our gclient sync bin file update.

Chase

[Marcus Bulach]

On Mon, Mar 25, 2013 at 6:57 PM, Chase Phillips <c...@google.com> wrote:

On Mon, Mar 25, 2013 at 11:40 AM, Marcus Bulach <bul...@chromium.org> wrote:

On Mon, Mar 25, 2013 at 5:35 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

On Fri, Mar 22, 2013 at 12:06 PM, Marcus Bulach <bul...@chromium.org> wrote:

Looking for reviewer & dogfooder, it'd be nice if someone who has previously updated the ref build could give it a go:

https://codereview.chromium.org/12662037

Thank you for working on this!

Do we have a reference build successfully rolled using this tool? I think that'd be a prerequisite.

no, that's the part I'm asking for a dogfooder :)

I've never done any roll myself, so I'd rather mitigate the risk..

btw, whilst writing this tool, I wondered: do we really need to do this roundtrip, i.e., buildbot -> googlestorage -> local dev machine -> svn -> buildbot?

I don't know all the history and reasons behind it, but would it be safe / better to just let the bots pull from buildbot directly?

That would make "rolling the ref build" just a matter of updating one data file pointing to the googlestorage URL, and using this tool to fetch / unzip to the right place... wdyt?

Yes, this is where we're headed in general, but we're not going in this direction to simplify updating the ref build.  Our general goal is to avoid needing to check large binary files into SVN.  The future flow would look like:

buildbot -> googlestorage in bucket 1 -> local dev machine -> google storage in bucket 2 -> svn -> buildbot

If we make "bucket 2" be the same as "bucket 1" (aka pull ref builds from the same bucket we upload to), we can make the flow look like:

buildbot -> googlestorage

local dev machine -> svn -> buildbot

I agree this is simpler.  It will require some upfront work on someone's part to ensure the googlestorage bucket layout maps to the same format we'll use for our gclient sync bin file update.

Since I've never updated the ref build, I don't quite understand what "bucket 1" and "bucket 2" mean, sorry. :)

For my own understanding, let me try to put in a different way, please correct me:

1) buildbot pushes zip to google storage

2) local dev downloads zip from google storage

3) local dev moves the files and put in a specific structure inside a given svn checkout

4) local dev then pushes to svn

5) perf bots pulls from svn

The patch that I just created above tries to takes care of (2, 3), but I wondered if I should go further, and change (5) to just use directly this tool.

I mean, the patch can be easily changed to become "download_refbuild.py --revision --platform", so we can shortcut pushing the binaries to svn....

would that work at all?

Thanks,

Marcus

[Chase Phillips]

I recommend you not go further here and instead checkpoint your current work.  This will be an improvement for most people.  If it means the reference build is easier to update with your current script, and then it happens more than once every year or two, that will be huge.

As for checking binaries into SVN, if you do nothing to improve this aspect, eventually Infrastructure will come up on this issue from another front.  We are working to remove all large blobs from our repos and instead store them in Google Storage with checksum references stored in src/DEPS.

Chase

[Marcus Bulach]

fantastic, thanks! the patch is available at https://codereview.chromium.org/12662037/

as I said, I'm a bit concerned of going ahead and actually pushing new ref builds from my timezone: I don't fully grasp all the side effects and would not know what to do if things went wrong..

I'd really appreciate if anyone could give this tool a shot, both in terms of reviewing the code and also using it to push new binaries! :)