chromium.perf tryjobs for commit queue?

[Andrew Grieve]

We've had a few recent breakages where builds failed only when is_official_build=true target_os="android". 

It looks like tryserver.chromium.perf/android_perf_bisect_builder would allow testing this (it's what's been breaking), but it doesn't show up in the "Choose trybots" interface.

Any reason why? Can it be added?

[Paweł Hajdan, Jr.]

+infra-dev

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[John Budorick]

I don't think you should use the perf bisect builders for a try job that isn't related to either perf or bisection. If we want trybots that cover is_official_build=true target_os="android", let's have that discussion.

+sullivan +dtu

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPZ7ECMRzv2AdapqPNsStj5327zAguMQ5X9x-Djm%3DJFyGA%40mail.gmail.com.

[Dirk Pranke]

Having tryjobs for the official Chrome build configs is problematic. 

On the one hand, you can't see the results of the jobs unless you're a Googler, and so it's difficult for non-Googlers to be able to deal w/ failures (and, hence, difficult for people to figure out why their changes get reverted when they land).

On the other hand, they're our official build configs :).

I would like to get to a point where we actually have two solutions for this:

1) We should have internal/private builders that googlers can test things against if need be prior to landing.

2) We should really strive to reduce the differences between what runs on the internal official builders and

what runs on public ones. While some things do actually need to stay private, I've seen plenty of changes 

for things that broke internally only on the official builders, but could've been tested elsewhere (this was a

real problem during the last couple months of the GN migration).

But, there's a fair amount of work involved for both of these things.

Separate from those points, I agree that we shouldn't reuse perf builders for functional tests if we can avoid it.

-- Dirk 

On Wed, Aug 24, 2016 at 8:01 AM, John Budorick <jbud...@chromium.org> wrote:

I don't think you should use the perf bisect builders for a try job that isn't related to either perf or bisection. If we want trybots that cover is_official_build=true target_os="android", let's have that discussion.

+sullivan +dtu

On Wed, Aug 24, 2016 at 1:56 AM Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

+infra-dev

On Wed, Aug 24, 2016 at 5:57 AM, Andrew Grieve <agr...@chromium.org> wrote:

We've had a few recent breakages where builds failed only when is_official_build=true target_os="android". 

It looks like tryserver.chromium.perf/android_perf_bisect_builder would allow testing this (it's what's been breaking), but it doesn't show up in the "Choose trybots" interface.

Any reason why? Can it be added?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPZ7ECMRzv2AdapqPNsStj5327zAguMQ5X9x-Djm%3DJFyGA%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAOee4m77%3DfvS%2BMzgh4K2tS%3Dp75%3Dxhnd%2BQSUvHF6vHwM1_e56UA%40mail.gmail.com.

[Andrew Grieve]

On Wed, Aug 24, 2016 at 12:18 PM, Dirk Pranke <dpr...@chromium.org> wrote:

Having tryjobs for the official Chrome build configs is problematic. 

On the one hand, you can't see the results of the jobs unless you're a Googler, and so it's difficult for non-Googlers to be able to deal w/ failures (and, hence, difficult for people to figure out why their changes get reverted when they land).

On the other hand, they're our official build configs :).

I would like to get to a point where we actually have two solutions for this:

1) We should have internal/private builders that googlers can test things against if need be prior to landing.

I don't think we need any private code being tested here, just a bot that sets is_official_build=true and ensures that chrome compiles.

 

2) We should really strive to reduce the differences between what runs on the internal official builders and

what runs on public ones. While some things do actually need to stay private, I've seen plenty of changes 

for things that broke internally only on the official builders, but could've been tested elsewhere (this was a

real problem during the last couple months of the GN migration).

But, there's a fair amount of work involved for both of these things.

Separate from those points, I agree that we shouldn't reuse perf builders for functional tests if we can avoid it.

I don't think our unit tests are even meant to work with is_official_build=true. I strictly want something that will do a compile. Worth it?

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAEoffTCqAUx5p5HdpK7999sV88LHKRvLOaSG3OYePH_7pAZr-Q%40mail.gmail.com.

[John Budorick]

On Wed, Aug 24, 2016 at 9:36 AM Andrew Grieve <agr...@chromium.org> wrote:

On Wed, Aug 24, 2016 at 12:18 PM, Dirk Pranke <dpr...@chromium.org> wrote:

Having tryjobs for the official Chrome build configs is problematic. 

On the one hand, you can't see the results of the jobs unless you're a Googler, and so it's difficult for non-Googlers to be able to deal w/ failures (and, hence, difficult for people to figure out why their changes get reverted when they land).

On the other hand, they're our official build configs :).

I would like to get to a point where we actually have two solutions for this:

1) We should have internal/private builders that googlers can test things against if need be prior to landing.

I don't think we need any private code being tested here, just a bot that sets is_official_build=true and ensures that chrome compiles.

 

2) We should really strive to reduce the differences between what runs on the internal official builders and

what runs on public ones. While some things do actually need to stay private, I've seen plenty of changes 

for things that broke internally only on the official builders, but could've been tested elsewhere (this was a

real problem during the last couple months of the GN migration).

But, there's a fair amount of work involved for both of these things.

Separate from those points, I agree that we shouldn't reuse perf builders for functional tests if we can avoid it.

I don't think our unit tests are even meant to work with is_official_build=true. I strictly want something that will do a compile. Worth it?

This still seems like it's abusing both the intent and the capacity of the perf bisect builders.

 

 

-- Dirk 

On Wed, Aug 24, 2016 at 8:01 AM, John Budorick <jbud...@chromium.org> wrote:

I don't think you should use the perf bisect builders for a try job that isn't related to either perf or bisection. If we want trybots that cover is_official_build=true target_os="android", let's have that discussion.

+sullivan +dtu

On Wed, Aug 24, 2016 at 1:56 AM Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

+infra-dev

On Wed, Aug 24, 2016 at 5:57 AM, Andrew Grieve <agr...@chromium.org> wrote:

We've had a few recent breakages where builds failed only when is_official_build=true target_os="android". 

It looks like tryserver.chromium.perf/android_perf_bisect_builder would allow testing this (it's what's been breaking), but it doesn't show up in the "Choose trybots" interface.

Any reason why? Can it be added?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAATLsPZ7ECMRzv2AdapqPNsStj5327zAguMQ5X9x-Djm%3DJFyGA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAOee4m77%3DfvS%2BMzgh4K2tS%3Dp75%3Dxhnd%2BQSUvHF6vHwM1_e56UA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

[Dirk Pranke]

On Wed, Aug 24, 2016 at 9:35 AM, Andrew Grieve <agr...@chromium.org> wrote:

On Wed, Aug 24, 2016 at 12:18 PM, Dirk Pranke <dpr...@chromium.org> wrote:

Having tryjobs for the official Chrome build configs is problematic. 

On the one hand, you can't see the results of the jobs unless you're a Googler, and so it's difficult for non-Googlers to be able to deal w/ failures (and, hence, difficult for people to figure out why their changes get reverted when they land).

On the other hand, they're our official build configs :).

I would like to get to a point where we actually have two solutions for this:

1) We should have internal/private builders that googlers can test things against if need be prior to landing.

I don't think we need any private code being tested here, just a bot that sets is_official_build=true and ensures that chrome compiles.

In the past, this sometimes hasn't worked without also setting is_chrome_branded=true, which is the thing that requires internal code.

However, I think this should generally work and we should probably just standardize on expecting it to work. And maybe change

the /p/chromium builders to do so and set up tryserver mirrors.

-- Dirk

[Torne (Richard Coles)]

We've already concluded in a bunch of different threads/bugs in the past that is_official_build=true *must* be maintained to work with is_chrome_branded=false, because we have been explicitly telling downstream consumers of chromium that they should use is_official_build=true for *their* shipping-to-user builds instead of expecting just a release build to be okay.

[Dirk Pranke]

Hmm. I have no memory of such threads, and I would think that downstream consumers are free to do whatever they want, no? Why would we require someone else to use official? I mean, they should, because it's likely a faster and smaller build, but ...

(Of course, the requirements that we impose on our own internal downstream builds are different).

-- Dirk

[Torne (Richard Coles)]

Because they kept asking for us to add other config flags to change things in the release build to work the way they do in official, and we told them they should just use official where things already work that way. :)

[Dirk Pranke]

That's certainly a fair answer, but that is somewhat different than saying that they're *required* to use official.

-- Dirk

[Torne (Richard Coles)]

I didn't say they are required to. I said that we should be making sure that it works, because we tell people it's the right thing for them to do.

[Dirk Pranke]

Fair enough, you did say "should", not "must". My mistake :).

At any rate, it's not like we're disagreeing that we need to support this config and have builders for it ...

-- Dirk

[Dirk Pranke]

... at any rate, I will also mention that you can control the GN args used on any builder in a try job by changing the entries for the builder in //tools/mb/mb_config.pyl and including that change as part of your CL. Obviously, don't commit such a change, but this can be a useful option when we don't seem to have any existing tryservers that meet your needs.

-- Dirk

[Andrew Grieve]

Filed: https://bugs.chromium.org/p/chromium/issues/detail?id=642458

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAEoffTCQNksCibtOsKoH%3D3NkRzxtF1u8c2bRK9tJnzpWBj8r9g%40mail.gmail.com.

[Andrew Grieve]

also - editing mb_config.pyl is a great tip!