PSA: I will remove x86 32 bit linux builders used for chromium project

159 views
Skip to first unread message

Takuto Ikuta

unread,
Jun 10, 2019, 5:38:48 AM6/10/19
to Chromium-dev, Dirk Pranke, infra-dev, Nico Weber
Hi, chromium devs,

TL:DR;
I will remove following 4 builders next week.

More explanation:
For now, chromium does not support 32 bit Linux
https://support.google.com/chrome/a/answer/7100626?hl=en
and all 32 bit x86 devices for chromeos is EOL too.

Considering that, I was not able to find any reason we have builders for not supported platform now.
If you have any comments about this builder removal, please let me know.
I will start removing process of the builders next week if there is no concern from you.

Note: This removal does not include 32 bit android/windows/libfuzzer or other than chromium builders.

Thanks,
Takuto

--
Takuto Ikuta
Software Engineer in Tokyo
Chrome Ops (goma team)

Jakob Kummerow

unread,
Jun 11, 2019, 8:07:19 AM6/11/19
to tik...@google.com, Chromium-dev, Dirk Pranke, infra-dev, Nico Weber
For the record: some bugs are specific to 32-bit, as opposed to 32-bit-on-Linux, and it is much easier to debug a 32-bit Linux x86 build than it is to debug an arm32 Android build (or a 32-bit Windows build, for that matter, for developers who don't have a Windows machine). For that reason, I do appreciate it if we ensure that the 32-bit Linux build doesn't bit-rot, as long as we care about any 32-bit platforms at all.

That said, I haven't recently had to go through this exercise, so if the cost of keeping these builders is deemed too high, I would reluctantly agree to turn them down. Or maybe a compromise is possible, where we keep one of them? 


--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CALNjmMp7-s9AhKqxiMB%2ByoLJZH%2BrGkET9CfsE28%2B%3DVpkOHDrcQ%40mail.gmail.com.

Takuto Ikuta

unread,
Jun 11, 2019, 9:23:27 AM6/11/19
to Jakob Kummerow, Chromium-dev, Dirk Pranke, infra-dev, Nico Weber
Hi,

Thank you for response!

This is list of failure happened on Linux Builder (dbg)(32) in recent 3 months, but all of these failure were not 32bit linux only.
So I think that probability of we break 32bit linux build only is very low.

1
2019-06-10 18:22:50.497974 UTC
2
2019-05-29 20:13:32.773355 UTC
3
2019-05-29 20:13:32.773355 UTC
4
2019-05-29 20:03:13.944131 UTC
5
2019-05-29 19:59:08.249830 UTC
6
2019-05-21 00:02:11.416646 UTC
7
2019-05-20 23:57:27.685605 UTC
8
2019-05-20 19:00:55.737826 UTC
9
2019-05-17 12:57:32.186618 UTC
10
2019-05-17 12:53:01.921649 UTC
11
2019-05-10 00:54:47.716807 UTC
12
2019-05-07 07:31:58.044902 UTC
13
2019-05-07 06:53:59.627882 UTC
14
2019-05-03 22:45:37.672018 UTC
15
2019-04-25 22:02:35.649821 UTC
16
2019-04-16 22:28:23.332535 UTC
17
2019-04-08 21:31:29.647127 UTC
18
2019-04-08 21:28:08.048893 UTC
19
2019-04-02 10:48:26.426473 UTC
20
2019-04-02 03:14:26.484550 UTC
21
2019-04-02 03:04:35.950857 UTC
22
2019-04-02 02:58:24.057070 UTC
23
2019-03-26 18:47:56.158290 UTC
24
2019-03-20 17:42:17.897407 UTC
25
2019-03-19 19:07:31.649379 UTC
26
2019-03-19 19:00:57.671292 UTC
27
2019-03-14 05:35:05.859369 UTC
28
2019-03-14 05:24:51.512822 UTC
29
2019-03-14 05:22:11.353553 UTC

Even if we broke 32bit linux builds after this builders removal, we will accept patch to fix that (as we do for cl.exe build).

And if we (infra team) has less number of builders, our operational cost become lower.
That is my main objective of this proposal.

If you still concern this builder removal, how do you think removing bots for 32bit linux test only?

Reid Kleckner

unread,
Jun 11, 2019, 1:41:55 PM6/11/19
to Takuto Ikuta, Jakob Kummerow, Chromium-dev, Dirk Pranke, infra-dev, Nico Weber
On Tue, Jun 11, 2019 at 6:22 AM 'Takuto Ikuta' via Chromium-dev <chromi...@chromium.org> wrote:
If you still concern this builder removal, how do you think removing bots for 32bit linux test only?

I think doing compile-only testing of 32-bit Linux is a good idea, assuming that compilation is relatively cheap compared to testing. 

Takuto Ikuta

unread,
Jun 14, 2019, 1:44:03 AM6/14/19
to Reid Kleckner, Jakob Kummerow, Chromium-dev, Dirk Pranke, infra-dev, Nico Weber
OK.

Currently, my decision is keeping 32bit linux CI/CQ builder only do building.
Message has been deleted

Darwin Huang

unread,
Jun 18, 2019, 4:12:03 PM6/18/19
to Chromium-dev, r...@google.com, jkum...@chromium.org, dpr...@chromium.org, infr...@chromium.org, tha...@chromium.org
Is there a bug I could link to for this builder change?


On Thursday, June 13, 2019 at 10:44:03 PM UTC-7, Takuto Ikuta wrote:
OK.

Currently, my decision is keeping 32bit linux CI/CQ builder only do building.

On Wed, Jun 12, 2019 at 2:40 AM Reid Kleckner <> wrote:
On Tue, Jun 11, 2019 at 6:22 AM 'Takuto Ikuta' via Chromium-dev <> wrote:
If you still concern this builder removal, how do you think removing bots for 32bit linux test only?

I think doing compile-only testing of 32-bit Linux is a good idea, assuming that compilation is relatively cheap compared to testing. 

Takuto Ikuta

unread,
Jun 18, 2019, 7:43:13 PM6/18/19
to Darwin Huang, Chromium-dev, Reid Kleckner, Jakob Kummerow, Dirk Pranke, infra-dev, Nico Weber

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/a32a0197-80d6-4cb6-9e78-b42a0caef736%40chromium.org.

Nico Weber

unread,
Jul 16, 2019, 10:17:35 AM7/16/19
to Takuto Ikuta, Chromium-dev, Dirk Pranke, infra-dev, John Abd-El-Malek
Sorry for the slow reply here, I've been on vacation.

From what I understand, while we no longer ship 32-bit linux, we still want to support building and running tests on it, for several reasons:

- Clusterfuzz wants to use 32-bit linux because fuzzers find more issues when pointers are smaller.
- 32-bit ARM is still very common, and 32-bit linux x86 finds issues that 32-bit ARM finds, like Jakob says. You replied to this point with an analysis that failures on that bot aren't 32-bit specific, but that doesn't answer that it's still useful for development.

There's an old, internal thread ("Deprecating x86 (32-bit) Linux Support") on an internal list from 2015 where the conclusion was to keep bots for this config around. (+jam since he was active on that thread.)

I'm all in favor of having fewer configurations and if we could completely stop supporting 32-bit Linux builds, I think that'd be awesome. But as-is, we still support them, but have fewer public bots that check that things work, so we now learn about breakages from Clusterfuzz bugs, which seems worse than before.

So I think you should either try to convince the clusterfuzz folks to stop using 32-bit binaries and make sure v8 is happen without a 32-bit x86 build as well, or we should keep bots for this config around.

Nico

Takuto Ikuta

unread,
Jul 16, 2019, 10:57:39 AM7/16/19
to Nico Weber, Chromium-dev, Dirk Pranke, infra-dev, John Abd-El-Malek
On Tue, Jul 16, 2019 at 11:16 PM Nico Weber <tha...@chromium.org> wrote:
Sorry for the slow reply here, I've been on vacation.

From what I understand, while we no longer ship 32-bit linux, we still want to support building and running tests on it, for several reasons:

- Clusterfuzz wants to use 32-bit linux because fuzzers find more issues when pointers are smaller.
- 32-bit ARM is still very common, and 32-bit linux x86 finds issues that 32-bit ARM finds, like Jakob says. You replied to this point with an analysis that failures on that bot aren't 32-bit specific, but that doesn't answer that it's still useful for development.

There's an old, internal thread ("Deprecating x86 (32-bit) Linux Support") on an internal list from 2015 where the conclusion was to keep bots for this config around. (+jam since he was active on that thread.)

I'm all in favor of having fewer configurations and if we could completely stop supporting 32-bit Linux builds, I think that'd be awesome. But as-is, we still support them, but have fewer public bots that check that things work, so we now learn about breakages from Clusterfuzz bugs, which seems worse than before.

So I think you should either try to convince the clusterfuzz folks to stop using 32-bit binaries and make sure v8 is happen without a 32-bit x86 build as well, or we should keep bots for this config around.


Considering response from fuzzing/v8 folks, I decided to keep both 32bit x86 builder (not tester) and 32bit libfuzzer builder.
I only removed 32bit linux tester here.
Reply all
Reply to author
Forward
0 new messages