How to interpret buildbot orange status

[Shawn Singh]

(1) On the build bots, the legend ambiguously indicates that orange means "failed again".   Does this mean the *exact* same test cases are failing again?  Or does it mean that the entire run failed again, even if the exact test cases that failed have changed?

(2) Would the same answers most likely apply to webkit build bot also, since its the same software?

(3) on http://dev.chromium.org/developers/testing/chromium-build-infrastructure/tour-of-the-chromium-buildbot the status of orange is documented as indicating certain warnings.  Is this correct?  whats the relationship between these warnings and "failing again"?

Thanks in advance,

~Shawn

[Chase Phillips]

Hi Shawn,

On Wed, Mar 28, 2012 at 3:06 PM, Shawn Singh <shawn...@chromium.org> wrote:

(1) On the build bots, the legend ambiguously indicates that orange means "failed again".   Does this mean the *exact* same test cases are failing again?  Or does it mean that the entire run failed again, even if the exact test cases that failed have changed?

On the console, orange means "the whole build failed again."  For the box to be orange, there's no requirement that the same step be the one that failed again.

Here's an example in the same order the console would give:

- build 4: orange (failed because of unit_tests)

- build 3: orange (failed because of ui_tests)

- build 2: red (failed because of browser_tests)

- build 1: green

 

(2) Would the same answers most likely apply to webkit build bot also, since its the same software?

I believe the same answers apply, as long as you're comparing one's console to another's console.  It's not exactly the same software, though, so don't assume too much based on that alone.

 

(3) on http://dev.chromium.org/developers/testing/chromium-build-infrastructure/tour-of-the-chromium-buildbot the status of orange is documented as indicating certain warnings.  Is this correct?  whats the relationship between these warnings and "failing again"?

That page is correct.  There is no relationship between those warnings and "failing again" in the waterfall.  Orange on the waterfall means exclusively "exited with warning."

Hope that helps,

Chase

 

Thanks in advance,

~Shawn

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Shawn Singh]

very helpful, thanks!

On Wed, Mar 28, 2012 at 4:01 PM, Shawn Singh <shawn...@google.com> wrote:

Very helpful, thanks!