Moving chromium/src git dependencies to git submodules

[Josip Sokcevic]

Hi all,

We are moving away from git dependencies being managed by gclient DEPS to native Git submodules. You can review our plans and reasoning in this document.

We are doing this to simplify tooling used by Chromium and unlock our long term vision of using industry standard systems and getting the Chromium codebase with just git client (i.e. via git clone). We will execute this migration in three stages to reduce potential disruptions and to smooth the transition.

In the first stage, we will introduce git submodules (.gitmodules and gitlinks) alongside DEPS. As part of this effort, we will be restructuring our DEPS to be submodule compatible. The source files will remain at the same location so there will be no need to change build rules, nor will there be any local migration needed. The user interface will remain the same. ETA: 2023 Q2.

In the second stage, we will be using git submodules as source of truth and git dependencies will be dropped from DEPS files. We understand that some tools parse DEPS files directly so we will give sufficient time for this migration to happen (at least three months). The user interface will remain the same. ETA: 2023 Q4

In the last stage, we will start using the git submodule command to manage git submodules. We expect this stage to take the longest as we will need to address git submodule behavior. In this phase, you can expect some UI changes since git will start tracking submodules. ETA: unscheduled.

Again, please review our detailed timeline and plan here, and let us know if you have any concerns by replying to this email.

Best,

Josip

[Yngve N. Pettersen]

Hi,

This sounds interesting, and might help the maintenance of our own fork of Chromium.

We already maintain our repos using submodules, so this will probably not change anything for our daily usage, although it will change (and, I think, improve) the update procedure when moving to a new Chromium version. For reference, Chromium is a submodule under our own main repository.

I do have some concerns/questions:

We are using our own mirror of the Chromium repositories, in part because we have our own patches on the main Chromium repo, as well as some submodules. Will the submodule configuration be compatible with such a system? In particular regarding submodules that are hosted on a different host in the upstream configuration and require absolute URLs? My suggestion is to make sure you are only using a single server for the repos, so that relative URLs can be used exclusively.

Chromium DEPS have a lot of of submodules listed, and some submodules (angle, vulkan-deps, others) have DEPS files of their own. How is this going to be handled?

Additionally, several of the submodules used by Chromium have, at least in the versions used today (Chromium 112), their own .gitmodules  file and submodule configurations, one example is the swiftshader submodule, which references 9 repositories, mostly on Github. If configured, these submodules (which are not required by Chromium)  will trigger a lot of unnecessary network traffic and delays during updates. How are you planning to handle these third-party submodule specifications? Vivaldi's fork have branches in these modules that remove such submodules.

A second point about the submodules in DEPS is that a large number of them are only used for target platforms that we don't use, especially ChromiumOS (and there are a number of repos that are for ChromiumOS but are specified in DEPS for Linux checkouts). Then there are all the submodules required to build Android. At present we are not including the Android-only modules in our normal checkout, but uses a custom script using gclient to check them out for users that are going to work on Android. We would like to avoid checking out unnecessary submodules, but I am not sure that is practical in current Git. (IIRC Chromium have/had an extension script for Git that implemented platform specific submodules.) Have you considered this aspect of the system?

At the very least, I think this will make the submodule step of our upgrade process easier, since we won't have to reconfigure the submodule hierarchy each time we update the source code.

BTW, one Git problem you should think about how to solve: When changing submodule URLs, the developer will have to run git submodule sync to fetch from the right repository, *before* running git submodule update on the changed module. For a single layer submodules that is easily accomplished, but with a second level a simple recusive sync won't work, because the first layer's gitmodules file  hasn't been updated yet! AFAIK there is no update flag that will do a sync before processing the next level (something like git submodule update --recursive --sync does not exist). AFAIK the only way to currently implement this is to run: git submodule foreach --recursive "git submodule sync; git submodule update"

The best solution would be for the Git team to implement support for a --sync parameter for update.

Another useful Git feature in the gitmodules file that AFAIK does not exist yet and would require Git team action, is to block recursive submodule update for the submodule update step, e.g. to prevent the submodules in swiftshader from being configured and cloned, without needing to fork the submodule.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAJiyOijiivqjHLFZJbnHRZqiaGx4FB2AyxdoicTc3PFYg2iPuA%40mail.gmail.com.

-- 
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS

[Josip Sokcevic]

Thanks for your feedback, appreciate it!

On Thu, Mar 2, 2023 at 3:22 PM Yngve N. Pettersen <yn...@vivaldi.com> wrote:

Hi,

This sounds interesting, and might help the maintenance of our own fork of Chromium.

We already maintain our repos using submodules, so this will probably not change anything for our daily usage, although it will change (and, I think, improve) the update procedure when moving to a new Chromium version. For reference, Chromium is a submodule under our own main repository.

I do have some concerns/questions:

We are using our own mirror of the Chromium repositories, in part because we have our own patches on the main Chromium repo, as well as some submodules. Will the submodule configuration be compatible with such a system? In particular regarding submodules that are hosted on a different host in the upstream configuration and require absolute URLs? My suggestion is to make sure you are only using a single server for the repos, so that relative URLs can be used exclusively.

We already rely on about a dozen other hosts (https://chromium.googlesource.com/chromium/src/+/abfd8051d5c288ee87706d11002967ba9e9fdfaa/DEPS#511) so some absolute URLs will be needed. We need to determine if we will use relative or absolute paths for the repos on the same host - there are some trade-offs to consider before making a decision.

 

Chromium DEPS have a lot of of submodules listed, and some submodules (angle, vulkan-deps, others) have DEPS files of their own. How is this going to be handled?

The plan is to add git submodules to all repositories where we parse their DEPS file (ie all recursedeps entries)

 

Additionally, several of the submodules used by Chromium have, at least in the versions used today (Chromium 112), their own .gitmodules  file and submodule configurations, one example is the swiftshader submodule, which references 9 repositories, mostly on Github. If configured, these submodules (which are not required by Chromium)  will trigger a lot of unnecessary network traffic and delays during updates. How are you planning to handle these third-party submodule specifications? Vivaldi's fork have branches in these modules that remove such submodules.

We are aware of this problem, and there are many third_party repositories that have git submodules. That's one of the reasons why we are doing this in stages and we still plan on using depot_tools gclient to manage git dependencies (instead of relying on git submodule).

Ideally, IMO, git users should be able to limit submodule only to specific repositories by setting some property in .gitmodules, e.g.

[submodule "third_party/dawn"]
path = third_party/dawn
url = https://dawn.googlesource.com/dawn.git
[submodule "third_party/snappy/src"]
path = third_party/snappy/src
url = https://chromium.googlesource.com/external/github.com/google/snappy.git

recurse = false

 

This is something we need to discuss with Git developers.

A second point about the submodules in DEPS is that a large number of them are only used for target platforms that we don't use, especially ChromiumOS (and there are a number of repos that are for ChromiumOS but are specified in DEPS for Linux checkouts). Then there are all the submodules required to build Android. At present we are not including the Android-only modules in our normal checkout, but uses a custom script using gclient to check them out for users that are going to work on Android. We would like to avoid checking out unnecessary submodules, but I am not sure that is practical in current Git. (IIRC Chromium have/had an extension script for Git that implemented platform specific submodules.) Have you considered this aspect of the system?

We experimented with git attributes to be able to filter out repositories that we don't want, but the UX wasn't great. Similarly to my previous answer, we are partially delaying this problem by using gclient to manage git dependencies. However, this will need to be solved before we start the last stage of the migration.

--

Josip Sokcevic

[Yuly Novikov]

Hi Josip,

I have some concerns about making this change in ANGLE repo (third_part/angle in chromium/src).

1. ANGLE is used in many different projects: Chrome, Skia, Dawn, as well as a standalone repo. DEPS file allows for project-specific customizations depending on where ANGLE is synced via recursedeps. Checking out everything everywhere doesn't make sense, e.g. you wouldn't want to get a 2nd copy of all third_party/android* (~10GB) related repoes when you sync ANGLE in Chromium - these are only needed when you do standalone ANGLE. Also, ANGLE's GN files are auto-configured differently based on what is checked out. We also sync different dependencies based on target platform and internal/public checkouts. Do git submodules allow such customizations?
2. What about CIPD dependencies, will we continue using DEPS files for these? Are we going to have both DEPS and .gitmodules?
3. ANGLE has a custom builder recipe https://source.chromium.org/chromium/chromium/tools/build/+/main:recipes/recipes/angle_chromium.py which allows us to use an LKGR Chromium to run WebGL tests with an ANGLE CL. We store LKGR Chromium revision in ANGLE's DEPS file, the builder uses that revision to check out an older version of Chromium and then it updates third_party/angle inside Chromium checkout to the ToT ANGLE and applies the ANGLE CL to be tested. LKGR chromium is updated by https://autoroll.skia.org/r/chromium-angle-autoroll. Will we be able to keep this mechanism as is in submodules world?

4. The above autoroller uses https://source.chromium.org/chromium/chromium/src/+/main:third_party/angle/scripts/roll_chromium_deps.py to update ANGLE dependencies shared with Chromium, like Clang. It basically updates ANGLE's DEPS file from Chromium's DEPS file and then autogenerates some ANGLE files based on the update. How do we implement this with submodules?

5. Occasionally we need to test ANGLE CLs on Chromium bots and vice-versa. E.g., if an ANGLE CL lands in ANGLE repo, then we roll ANGLE into Chromium and the roll fails on some Chromium CQ bot which doesn't have a corresponding ANGLE CQ bot. We then revert the ANGLE CL and when we reland, we want to run the Chromium bot on the ANGLE CL to ensure roll doesn't fail again. We do this by creating a dummy roll CL, which updates Chromium's DEPS to a ref of the ANGLE CL, e.g. crrev.com/c/4199015. Will we be able to do this with submodules?

6. ANGLE is used in AOSP, which doesn't allow submodules. This means we'll need to modify https://source.chromium.org/chromium/chromium/src/+/main:third_party/angle/scripts/roll_aosp.sh to work with submodules, e.g. delete all .gitmodules files. This is extra work that we'd like to avoid if possible.

Yuly.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAJiyOiiDhg3yTLj7sxj7d4-X-d3T7P77q_9fUnnRqGhh7SFjfw%40mail.gmail.com.

[Josip Sokcevic]

On Wed, Mar 8, 2023 at 9:51 AM Yuly Novikov <ynov...@chromium.org> wrote:

Hi Josip,

I have some concerns about making this change in ANGLE repo (third_part/angle in chromium/src).

1. ANGLE is used in many different projects: Chrome, Skia, Dawn, as well as a standalone repo. DEPS file allows for project-specific customizations depending on where ANGLE is synced via recursedeps. Checking out everything everywhere doesn't make sense, e.g. you wouldn't want to get a 2nd copy of all third_party/android* (~10GB) related repoes when you sync ANGLE in Chromium - these are only needed when you do standalone ANGLE. Also, ANGLE's GN files are auto-configured differently based on what is checked out. We also sync different dependencies based on target platform and internal/public checkouts. Do git submodules allow such customizations?

It's a bit complicated. Technically, Git allows for such customization using filters and git attributes but not user friendly as gclient. That's why we will still rely on depot_tools/gclient to manage git submodules for the first two stages. We will move onto the third stage (Git client manages git submodules) only when we are comfortable with the Git CLI; but we don't have a timeline for that.

2. What about CIPD dependencies, will we continue using DEPS files for these? Are we going to have both DEPS and .gitmodules?

Yes,  we have no plans for CIPD dependencies so they will remain in DEPS. 

3. ANGLE has a custom builder recipe https://source.chromium.org/chromium/chromium/tools/build/+/main:recipes/recipes/angle_chromium.py which allows us to use an LKGR Chromium to run WebGL tests with an ANGLE CL. We store LKGR Chromium revision in ANGLE's DEPS file, the builder uses that revision to check out an older version of Chromium and then it updates third_party/angle inside Chromium checkout to the ToT ANGLE and applies the ANGLE CL to be tested. LKGR chromium is updated by https://autoroll.skia.org/r/chromium-angle-autoroll. Will we be able to keep this mechanism as is in submodules world?

Similarly to the first answer, there will be no changes in behavior for the first two phases. We need to scope how this will work with the git client directly, but we know we have to support this case. As I wrote in the document I shared, we need to do more research.

 

4. The above autoroller uses https://source.chromium.org/chromium/chromium/src/+/main:third_party/angle/scripts/roll_chromium_deps.py to update ANGLE dependencies shared with Chromium, like Clang. It basically updates ANGLE's DEPS file from Chromium's DEPS file and then autogenerates some ANGLE files based on the update. How do we implement this with submodules?

Instead of modifying DEPS files directly, you should utilize `gclient setdep` (looks like we will need to support updating conditions). Alternatively, you can update git submodules using `git update-index` once git submodules are present. We plan to do an audit of tools that parse DEPS file directly and file bugs for each one that needs a change (https://crbug.com/1409743).

I filed https://crbug.com/1422693 to track this - feel free to assign an appropriate component.

 

5. Occasionally we need to test ANGLE CLs on Chromium bots and vice-versa. E.g., if an ANGLE CL lands in ANGLE repo, then we roll ANGLE into Chromium and the roll fails on some Chromium CQ bot which doesn't have a corresponding ANGLE CQ bot. We then revert the ANGLE CL and when we reland, we want to run the Chromium bot on the ANGLE CL to ensure roll doesn't fail again. We do this by creating a dummy roll CL, which updates Chromium's DEPS to a ref of the ANGLE CL, e.g. crrev.com/c/4199015. Will we be able to do this with submodules?

Yes, via roll-dep which already supports git submodules. You can also use gitlink directly using git update-index, but it's not as easy as updating DEPS file.

 

6. ANGLE is used in AOSP, which doesn't allow submodules. This means we'll need to modify https://source.chromium.org/chromium/chromium/src/+/main:third_party/angle/scripts/roll_aosp.sh to work with submodules, e.g. delete all .gitmodules files. This is extra work that we'd like to avoid if possible.

Just to clarify, there will be just one .gitmodules file; but there is a gitlink for each git submodule entry. If think some changes will be needed to the script, especially since you are "flattening" all dependencies into AOSP angle (https://source.chromium.org/chromium/chromium/src/+/main:third_party/angle/scripts/roll_aosp.sh;l=167;drc=024114b2131d588ff30d6185c30894fc6640e635).

I suggest you create a bug for this issue, and we can continue this conversation there.

--

Josip Sokcevic

[Ken Russell]

On Wed, Mar 8, 2023 at 1:14 PM 'Josip Sokcevic' via Chromium-dev <chromi...@chromium.org> wrote:

On Wed, Mar 8, 2023 at 9:51 AM Yuly Novikov <ynov...@chromium.org> wrote:

Hi Josip,

I have some concerns about making this change in ANGLE repo (third_part/angle in chromium/src).

1. ANGLE is used in many different projects: Chrome, Skia, Dawn, as well as a standalone repo. DEPS file allows for project-specific customizations depending on where ANGLE is synced via recursedeps. Checking out everything everywhere doesn't make sense, e.g. you wouldn't want to get a 2nd copy of all third_party/android* (~10GB) related repoes when you sync ANGLE in Chromium - these are only needed when you do standalone ANGLE. Also, ANGLE's GN files are auto-configured differently based on what is checked out. We also sync different dependencies based on target platform and internal/public checkouts. Do git submodules allow such customizations?

It's a bit complicated. Technically, Git allows for such customization using filters and git attributes but not user friendly as gclient. That's why we will still rely on depot_tools/gclient to manage git submodules for the first two stages. We will move onto the third stage (Git client manages git submodules) only when we are comfortable with the Git CLI; but we don't have a timeline for that.

How will conditional submodule references work? The design doc doesn't provide details and neither do the top-level bugs crbug.com/1409737 or crbug.com/1409734.

It seems to me that the .gitmodules syntax will have to be extended in incompatible ways.

Would appreciate more details being provided in an easy-to-consume form - the design doc would probably be best.

Thanks,

-Ken

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAJiyOiirLGLQbesJa_97evf-_23AijPMvb8EXkCf%3DVwq%2BEv2tQ%40mail.gmail.com.

[Josip Sokcevic]

On Wed, Mar 8, 2023 at 3:31 PM Ken Russell <k...@chromium.org> wrote:

On Wed, Mar 8, 2023 at 1:14 PM 'Josip Sokcevic' via Chromium-dev <chromi...@chromium.org> wrote:

On Wed, Mar 8, 2023 at 9:51 AM Yuly Novikov <ynov...@chromium.org> wrote:

Hi Josip,

I have some concerns about making this change in ANGLE repo (third_part/angle in chromium/src).

1. ANGLE is used in many different projects: Chrome, Skia, Dawn, as well as a standalone repo. DEPS file allows for project-specific customizations depending on where ANGLE is synced via recursedeps. Checking out everything everywhere doesn't make sense, e.g. you wouldn't want to get a 2nd copy of all third_party/android* (~10GB) related repoes when you sync ANGLE in Chromium - these are only needed when you do standalone ANGLE. Also, ANGLE's GN files are auto-configured differently based on what is checked out. We also sync different dependencies based on target platform and internal/public checkouts. Do git submodules allow such customizations?

It's a bit complicated. Technically, Git allows for such customization using filters and git attributes but not user friendly as gclient. That's why we will still rely on depot_tools/gclient to manage git submodules for the first two stages. We will move onto the third stage (Git client manages git submodules) only when we are comfortable with the Git CLI; but we don't have a timeline for that.

How will conditional submodule references work? The design doc doesn't provide details and neither do the top-level bugs crbug.com/1409737 or crbug.com/1409734.

It seems to me that the .gitmodules syntax will have to be extended in incompatible ways.

Would appreciate more details being provided in an easy-to-consume form - the design doc would probably be best.

We still need to come up with a design for this - ideally, it will be something that the git client can understand, and I plan to work with git contributors on this. For the first stage, we plan to keep conditions in DEPS. I realized the doc I shared is rather thin on implementation details, while some internal docs go into depth (e.g. go/chops-chrome-on-submodules). With that being said, I will create a public design doc that covers the first phase and follow up work. Note that each phase will have its own DD.

--

Josip Sokcevic