PSA for WebKit Gardeners (WAS: Re: Windows bots producing corrupt binaries)

[Jeremy Orlow]

If you're a WebKit Gardener and see only the Windows bots start failing in weird and unexplained ways (especially crashes) you've probably hit upon this IB bug.

The solution is to clobber the Windows bot.  If you do this, there is a very strong chance you'll need to clobber all the Windows builders after your next roll as well.  If the clobber doesn't fix it, well then you have a real Windows-only regression.  :-)

J

On Fri, Aug 6, 2010 at 1:43 PM, Jeremy Orlow <jor...@chromium.org> wrote:

As a follow up: turns out it was a bug in IB.  We've submitted a reduced test case that reproduces the bug, so hopefully it's fixed soon.  In the mean time, we'll need to clobber a lot more.

Maybe we should re-consider moving the bots off of IB?  At least the WebKit canary given that this has affected us at least once a day for the last week?  (i.e. this is as bad if not worse to the canaries as the GRD bug was to build.chromium.org.)

J

On Mon, Aug 2, 2010 at 5:53 PM, Jochen Eisinger <joc...@chromium.org> wrote:

It's getting late and I'm heading home. If somebody wants to have a
look at it, here's another piece of the puzzle: when this happens,
there seem to be always SVG files involved.

-jochen

On Mon, Aug 2, 2010 at 4:09 PM, Jeremy Orlow <jor...@chromium.org> wrote:
> Thanks a lot, Jochen!
> J
>
> On Mon, Aug 2, 2010 at 3:07 PM, Jochen Eisinger <joc...@chromium.org> wrote:
>>
>> So indeed it looks like IB isn't tracking the dependencies correctly.
>> When I compile test_shell_tests with webkit at 64419 and the recompile
>> with webkit at 64420, msvs and ib differ on webcore_bindings.
>>
>> msvs compiles:
>>
>> 2>SVGElementFactory.cpp
>> 2>V8DerivedSources2.cpp
>> 2>V8DerivedSources1.cpp
>> 2>V8DerivedSources3.cpp
>> 2>V8HTMLElementWrapperFactory.cpp
>> 2>V8SVGElementWrapperFactory.cpp
>> 2>HTMLElementFactory.cpp
>>
>> while IB only compiles:
>>
>> HTMLElementFactory.cpp
>> V8DerivedSources2.cpp
>> V8DerivedSources3.cpp
>> V8HTMLElementWrapperFactory.cpp
>>
>> looks like it ignores rebuilds that should be triggered by updated
>> svg/ sources. I'll try to produce a minimal test case, so we can send
>> it to ib support.
>>
>> -jochen
>>
>>
>> On Mon, Aug 2, 2010 at 11:41 AM, Jeremy Orlow <jor...@chromium.org> wrote:
>> > So for the 4th (http://trac.webkit.org/changeset/64420) and 5th
>> > (http://trac.webkit.org/changeset/64440/) time during my gardening
>> > rotation,
>> > the windows canary produced a broken binary (that
>> > would subsequently crash
>> > on quite a few tests).  After a clobber, the bot always goes back to
>> > normal.
>> > But it's not just this one particular machine.  Each time I've rolled in
>> > one
>> > of these changes, I've seen all the main bots go crazy as well, until I
>> > clobber there too.
>> >
>> > When it was just a couple times, I shrugged it off, but now I'm starting
>> > to
>> > wonder if something is majorly wrong.  Is it possible this is linked to
>> > the
>> > change we pushed out to fix the GRD dependency tracking issues?
>> > J
>> > On Mon, Aug 2, 2010 at 10:31 AM, Jeremy Orlow <jor...@chromium.org>
>> > wrote:
>> >>
>> >> I clobbered the build and now it's happy.  In the last couple days,
>> >> I've
>> >> seen this bot create corrupt binaries that tend to crash several times
>> >> now.
>> >>  Not sure what's going on.
>> >> Anyway, Zimmermann, sorry for _yet another_ false alarm (generated by
>> >> one
>> >> of your patches).  Hopefully we can get this bot cleaned up soon.
>> >> J
>> >>
>> >> On Sat, Jul 31, 2010 at 11:47 PM, Drew Wilson <atwi...@chromium.org>
>> >> wrote:
>> >>>
>> >>> Hi there,
>> >>> The chromium webkit canaries are crashing running the
>> >>> style-reflect.html
>> >>> test on the canary bots with r64420 landed.
>> >>> Unfortunately there really isn't much of a stack trace or anything to
>> >>> help debug it:
>> >>>
>> >>> compilation error: file
>> >>>
>> >>> file:///C:/b/slave/webkit-rel-webkit-org/build/src/third_party/WebKit/LayoutTests/fast/xsl/resources/xhr-doc.xsl
>> >>> line 3 element import
>> >>> xsl:import : unable to load
>> >>>
>> >>> file:///C:/b/slave/webkit-rel-webkit-org/build/src/third_party/WebKit/LayoutTests/fast/xsl/resources/xhr-doc-imported.xsl
>> >>> compilation error: file
>> >>>
>> >>> file:///C:/b/slave/webkit-rel-webkit-org/build/src/third_party/WebKit/LayoutTests/fast/xsl/resources/xslt-bad-import-uri.xsl
>> >>> line 3 element import
>> >>> xsl:import : unable to load
>> >>>
>> >>> file:///C:/b/slave/webkit-rel-webkit-org/build/src/third_party/WebKit/LayoutTests/fast/xsl/resources/nosuchfileatall
>> >>> Backtrace:
>> >>>     (No symbol) [0x00878C84]
>> >>>     (No symbol) [0x35045800]
>> >>>
>> >>> Any ideas? Not sure if maybe this patch introduces some crashes on
>> >>> Windows that just aren't getting caught by the webkit bots, but I
>> >>> don't want
>> >>> to roll it out if you think that's unlikely.
>> >>> -atw
>> >
>> >
>
>

[Marc-Antoine Ruel]

In the interim, I've disabled IB on Webkit (webkit.org), Webkit.org
Builder and Webkit.org Reliability Builder. This had the side-effect
of disabling it for Chromium Builder too. (The one on FYI, not the one
of the chromium waterfall)

I just modified scripts\compile.py and clobbered the slaves manually.

I'll take a look at the slowliness later. In any case, this is a
temporary change as we're waiting on Xoreax for another version which
should take care of most issues.

M-A

> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
>

[Jeremy Orlow]

Great!  Thanks!

In that case, then this is just something to look out for when you're doing WebKit rolls.  If you see a bunch of unexplained failures (especially crashes), you should clobber the 4 Windows builders....and pray.  :-)

J

[Drew Wilson]

+1 to this advice. I spent a fretful 2-3 hours trying to figure out a slew of SVG errors on Wednesday afternoon, only to have them go away when I did a clobber build. In my case, I don't think it was crashing - I was getting pure black images - go figure.

-atw

[Jochen Eisinger]

If anyone sees build related failures on the waterfall, please forward
the build number and the logs to me, so I can make sure we report all
issues and have them fixed.

[Jian Li]

It seems to happen again.

  http://buildbot.jail.google.com/buildbot/chromium/builders/Webkit%20Builder%20(dbg)/builds/33540

  http://buildbot.jail.google.com/buildbot/chromium/builders/Webkit%20Builder/builds/33666

I've clobber the window build bots and hope it will solve the problems.

[Shinichiro Hamaji]

Are there any update on this issue? Marc said we disabled IB but
WebKit gardeners are still seeing crashes which can be fixed by
clobber build.

I've filed a bug to know any kind of updates on this issue.

http://code.google.com/p/chromium/issues/detail?id=52693

Thanks,

[Jochen Eisinger]

What about adding a step to the chromium/webkit builders that checks
for a webkit roll and kills the build directory in that case (as a
temporary work around)?

[John Gregg]

Might be a good idea, all the rolls have been needing clobbers this week and it adds a lot of work & tree redness while it gets sorted.

[Tony Chang]

An update on this.  The Webkit (webkit.org) builder was super slow w/o incredibuild (e.g., compiles take up to 2 hours), so Marc-Antoine changed the bot to use incredibuild, but always clobber.  Now builds consistently take about 10 minutes, which seems OK.

Of course, when rolling a new version of webkit into chrome, you need to remember to clobber, otherwise you can get a bad build.

I think Jochen's suggestion of trying to clobber when DEPS changes is reasonable, although I'm not sure what the best way to implement that is.  Alternately we could have the webkit builder always clobber like the webkit (webkit.org), but that seems less optimal but easier to implement.

[Jian Li]

Can we put some kind of  hint at description, like "CLOBBER=YES" after "TEST=NONE"? The builder can check for this and perform the action.

[Jochen Eisinger]

just a quick update. after trying another version and reporting
another round of issues, we'll hopefully be able to try another
version of IB soon.

-jochen