Automatic Commits to Chromium

[Joshua Pawlicki]

Hi Chromium-dev,

The tree contains many files, such as plugins_win.json (a list of plugins and their minimum versions) or transport_security_state_static.json (HSTS list), which could be kept up to date by automation.

Has anyone looked into automatically creating and CQ'ing CLs? If not, who should I talk to about such a project?

- Josh

[Dirk Pranke]

There are many various kinds of "auto-rollers" we use to update things, so such ideas are at least plausible.

I'd be a decent person to start with, and could pull in others if we thought that was useful.

-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Lucas Garron]

(Re-sending from my Chromium address, since Inbox didn't correctly reply from it.)

On Mon, Jan 30, 2017 at 1:24 PM Lucas Garron <lga...@google.com> wrote:

I'm hoping to automate the transport_security_state_static.json to the point where a bot files a weekly CL that I can just LGTM and land.

Full automation would certainly be nice.

If anyone wants to write a Chrome auto-roller for this, I can prioritize implementing an up-to-date "source of truth" for it: https://github.com/chromium/hstspreload.org/issues/76

»Lucas

[Anthony Berent]

I am not sure what these files are, but if they are automatically generated, or generated outside Chromium, and don't need human review, is it actually appropriate to keep them in the Chromium repositories at all? If it isn't possible (or unacceptably slow) to generate these at build time would it be better to treat them as data files, and to download them from storage at gclient sync time?

[Joshua Pawlicki]

I assume we don't want to generate them at build time since we'll need a network request to get them, and it seems like the build should operate without an Internet connection once you have the artifacts. I know no details about fetching them from storage during gclient sync, but in principle the idea seems sound - whether we use that or an auto-roller, we will have canonical copies available by some network API.

Which is the better way to go? Is it acceptable if the thing that generates the canonical copy is closed-source, and if so, does that lean more toward an auto-roller solution?

[Aaron Gable]

One of the goals of the project is reproducible builds. This means that, if I check out an old version of Chrome, I should be able to compile it and get the same result as someone compiling months ago when that version was tip-of-tree. We're not very good at that, but watch out for things which make it worse.

In particular, if you download artifacts during the checkout process via "gclient hooks", those artifacts need to be content-addressed. When the artifact updates, a file inside the source repo needs to also update to point to the new version. That way, when you try to build an old version of Chrome, you also get the old versions of the artifacts which aren't stored in the build tree.

As a result of this goal, we don't generally store small, human-readable artifacts outside the build tree. Something like plugins_win.json is small and easily maintainable. Replacing it with a one-line file containing a hash of a json file in google storage doesn't meaningfully reduce the size of the repo, doesn't reduce the number of commits to the repo, and unfortunately does reduce the readability of changes to the file. So I would not use an outside-the-tree solution for that file. Automating the updating of it, however, seems like a great idea, and could be done along the lines of any of the autorollers which manipulate the DEPS file.

The HSTS list, on the other hand, is huge. It also doesn't directly affect the build, as far as I know -- someone building an old version of Chrome with a new HSTS list isn't going to fail any tests or introduce any security problems. So that seems like a good candidate for being stored outside the repo and fetched via gclient hooks.

[Joshua Pawlicki]

Thank you all for your replies. IIUC, regardless of whether the file (or its hash) is stored within the repo, the auto-roller is the way to automatically update it. Is there any particular approval process I need to go through for adding writing new auto-rollers?