Deprecating repo

[Kenny Ho]

I understand that there is a talk at the Gerrit User Summit 2016 about deprecating repo ("bye bye repo"):
https://gerrit.googlesource.com/summit/2016/+/master/index.md

Can someone provide a summary of what was discussed please?  (For the benefits of those who couldn't attend the summit.)  I am also interested in the plans of major repo users on this change (Android, Chromium OS, CoreOS, etc.)

[Stefan Beller]

On Thu, Nov 17, 2016 at 7:43 AM, Kenny Ho <y2k...@gmail.com> wrote:

>
> Can someone provide a summary of what was discussed please? (For the
> benefits of those who couldn't attend the summit.) I am also interested in
> the plans of major repo users on this change (Android, Chromium OS, CoreOS,
> etc.)

* repo is open source and we do not plan to shut it down *forcefully*
-> feel free to keep using it.
* we consider repo to be suboptimal in terms of workflow
(we meaning the Gerrit/Git team at Google)
* replacing repo by native git submodules is being worked on,
but we're not quite there yet.
* No plans on how to migrate Android etc. were announced publicly.

[Pierre Tardy]

Hi Stefan,

To my understanding, this is a quite old discussion. I remember seing email from  Shawn saying more or less the same arguments when I started working on Android 6 years ago (I can't find that mail anymore).

I think those claims from Gerrit team at Google have prevented the repo tool to evolve as it could have.

I have spoke with several other members of the Android community complaining about repo, and waiting that Google come with a miracle workflow with git submodules.

I even saw people rewrite repo in Java (initially) to get windows support as it was so hopeless to try and improve repo.

Still in 2016 its not quite there yet. Android Industry have built lots of tooling around repo, and it will be very difficult imho to move to other thing.

I was planning to spend some time on the repo tool in order to give it the love that it needs, and make it more maintainable:

- add a modern setup.py, and make it a real python module available on pypi that you can pip install alongside the weird bootstrap that we currently have.

- add unit tests and integration tests so that we can refactor it without fearing to break everything.

- provide a proper python API to the manifest parsing, so that it is easier to create internal python tooling with manifest awareness.

- add a multithreaded repo-forall python API to implement diff tools like the one that was discussed yesterday on this list

I would really appreciate to see a little bit more information on what is the view of the git experts here on how git submodule can replace the manifest workflow.

Especially: 

- Will it scale with 700+ repositories that some android team have?

- What is the pre-commit/pre-submit check support from CI tools?

   I am not sure how well Jenkins supports it.

   I know that buildbot does not support it, and that it will be quite difficult to support. Only post commit submodules checks are possible with buildbot.

   mapping where a change should be cherry-picked, and what super-project would be interrested by one change is quite interresting chalenge.

   It is not easywith repo but relatively easier, as there is all the metadata needed in the manifest, including branches that are tracked in submodules, etc.

- How can we implement manifest filtering with git submodules?

  For example Android SoC vendors manage internal manifest with lots of socs in it, then their customer teams will take the internal release manifest, filter out the repositories according to the soc the customer needs, and convert to binaries the repositories that the customer does not have signed NDA for.

Regards,

Pierre

The opinions expressed in this email are my own and do not reflect the view of my employer.

[Stefan Beller]

On Thu, Nov 17, 2016 at 1:08 PM, Pierre Tardy <tar...@gmail.com> wrote:
>
> Le jeu. 17 nov. 2016 à 20:54, 'Stefan Beller' via Repo and Gerrit Discussion
> <repo-d...@googlegroups.com> a écrit :
>>
>> On Thu, Nov 17, 2016 at 7:43 AM, Kenny Ho <y2k...@gmail.com> wrote:
>>
>> >
>> > Can someone provide a summary of what was discussed please? (For the
>> > benefits of those who couldn't attend the summit.) I am also interested
>> > in
>> > the plans of major repo users on this change (Android, Chromium OS,
>> > CoreOS,
>> > etc.)
>>
>> * repo is open source and we do not plan to shut it down *forcefully*
>> -> feel free to keep using it.
>> * we consider repo to be suboptimal in terms of workflow
>> (we meaning the Gerrit/Git team at Google)
>> * replacing repo by native git submodules is being worked on,
>> but we're not quite there yet.
>> * No plans on how to migrate Android etc. were announced publicly.
>
>
> Hi Stefan,
>
> To my understanding, this is a quite old discussion. I remember seing email
> from Shawn saying more or less the same arguments when I started working on
> Android 6 years ago (I can't find that mail anymore).

Yes this discussion is on going for a few years now. However by now we
actually have
people working on submodules and not hoping to see a miracle from the
community. ;)

>
> I think those claims from Gerrit team at Google have prevented the repo tool
> to evolve as it could have.

That's certainly a possible. I don't know if this is the actual reason that
prevented repo from taking off.

> I was planning to spend some time on the repo tool in order to give it the
> love that it needs, and make it more maintainable:

I think stepping up to maintain repo is a honorable thing to do, however
as you point out, having no tests/documentation makes maintenance
not easy in the first place. Sure tests can be added, but that will consume some
time I would presume. (When coming up with tests after the fact, how do
you know you test for the actual feature and corner cases? I think it is hard
to write tests after the fact, so I would love to see how it is done properly.)

>
> I would really appreciate to see a little bit more information on what is
> the view of the git experts here on how git submodule can replace the
> manifest workflow.

The main issue that motivates the move to Submodules is this:
Android wants to have atomicity across their whole code base in
the development process, e.g. you have a breaking change to
some library to make the API nicer. In a single Git repository you
could make the change for the library as well as all users of the libraries
within one (atomic) commit, such that the project works before as well
as after the change.

When having such a API change across multiple repositories, this
is impossible to sync atomically except via indirection over another
single repository.

There are different approaches to that, another approach done by Juniper
was presented at the summit by Basavaraj Karadakal:

https://gerrit.googlesource.com/summit/2016/+/master/sessions/Atomicity-with-change-sets.md

When looking through the slides one of the first things to notice is that
they also use another Git repository to coordinate between the actual
repositories.

Now we could come up with such a sync repo under the hood within
git-repo as well. However instead of piling more on top of Git,
why not use Git directly? (Submodules solve this exact problem,
they are just hard to use, so we need to make it as nice as the repo tool
such that people will switch voluntarily).

>
> Especially:
> - Will it scale with 700+ repositories that some android team have?

I did some work on that, e.g. parallel fetching of submodules
https://kernel.googlesource.com/pub/scm/git/git/+/master/Documentation/RelNotes/2.8.0.txt

* Add a framework to spawn a group of processes in parallel, and use
it to run "git fetch --recurse-submodules" in parallel.

https://kernel.googlesource.com/pub/scm/git/git/+/master/Documentation/RelNotes/2.9.0.txt

* "git clone" learned the "--shallow-submodules" option.

Not sure if that is sufficient though. I guess some issues will still
come up later.

> - What is the pre-commit/pre-submit check support from CI tools?

How do current pre submit tests work? hint: they are incredible hard to
get right across multiple repositories; In the future it is as simple
as using one call
to get all changes across repositories (and possibly including the project
that manages the dependencies, the superproject) in one go
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#submit-preview

> It is not easywith repo but relatively easier, as there is all the
> metadata needed in the manifest, including branches that are tracked in
> submodules, etc.

All information that is in git-repo ought to be available via the
gitmodules file as well.

>
> - How can we implement manifest filtering with git submodules?
> For example Android SoC vendors manage internal manifest with lots of socs
> in it, then their customer teams will take the internal release manifest,
> filter out the repositories according to the soc the customer needs, and
> convert to binaries the repositories that the customer does not have signed
> NDA for.

You'd need some sort of grouping feature for Submodules, such as Git attributes
e.g. https://public-inbox.org/git/20161110203428....@google.com/

Thanks for these follow up question,
Stefan

[Pierre Tardy]

Hi Stefan,

Thanks for the awesomely detailed response.
It might take a while for me to digest, but I still have some quick precision I'd like to have, and experience I can share.

 

I think stepping up to maintain repo is a honorable thing to do, however
as you point out, having no tests/documentation makes maintenance
not easy in the first place. Sure tests can be added, but that will consume some
time  I would presume. (When coming up with tests after the fact, how do
you know you test for the actual feature and corner cases? I think it is hard
to write tests after the fact, so I would love to see how it is done properly.)

 

Having 100% coverage from the start will indeed not be possible, but 2% is better than 0%, and the 4% will be even better, until we can require 90% coverage of the patched lines for every contribution.

This is what we did for the venerable buildbot which I am now maintainer of.

That said indeed if submodule is say 6 month away from achieving what we do with repo, maybe its not worth to make such big investment.

>
> I would really appreciate to see a little bit more information on what is
> the view of the git experts here on how git submodule can replace the
> manifest workflow.

The main issue that motivates the move to Submodules is this:
Android wants to have atomicity across their whole code base in
the development process

When having such a API change across multiple repositories, this
is impossible to sync atomically except via indirection over another
single repository.

There are different approaches to that, another approach done by Juniper
was presented at the summit by Basavaraj Karadakal:

  https://gerrit.googlesource.com/summit/2016/+/master/sessions/Atomicity-with-change-sets.md

When looking through the slides one of the first things to notice is that
they also use another Git repository to coordinate between the actual
repositories.

So this is the concept of cross-project dependencies. I think the open-stack guys did contribute a depends-on plugin, which was renamed zuul (not sure exactly why).

There is also the topic metadata which some are using for cross-project dependencies.

I have resolved internally the same problem with a very similar solution, but implemented on buildbot side, and not on Gerrit side.

Our solution can use either topic, depends-on metadata or tracked-on metadata. It works well, except people sometime complains it takes too much dependencies.

Our CI is doing the atomic merges on its own environment then forcepush the affected repositories, then wait for mirrors to mirror the new ref, and finally push a new manifest to the manifest server.

The manifest is then acting as a top project.

The problem that I did not automate is atomic cross-project revert. What we did is revert the problematic patches manually, and merge them together atomically.

I wonder if this is a problem that you have a solution for. Will git revert in a top dir will revert the patches in the subrepositories instead of just pointing the top dir to the ancient commit (and break tracability in the subrepositories)

 

Now we could come up with such a sync repo under the hood within
git-repo as well. However instead of piling more on top of Git,
why not use Git directly? (Submodules solve this exact problem,
they are just hard to use, so we need to make it as nice as the repo tool
such that people will switch voluntarily).

For me the main reason is to have a scripting language to handle those high level problems instead of having to write it in C (and then rewrite it in java in JGIT)

 

>
> Especially:
> - Will it scale with 700+ repositories that some android team have?

I did some work on that, e.g. parallel fetching of submodules
https://kernel.googlesource.com/pub/scm/git/git/+/master/Documentation/RelNotes/2.8.0.txt

 * Add a framework to spawn a group of processes in parallel, and use
   it to run "git fetch --recurse-submodules" in parallel.

https://kernel.googlesource.com/pub/scm/git/git/+/master/Documentation/RelNotes/2.9.0.txt

 * "git clone" learned the "--shallow-submodules" option.

Not sure if that is sufficient though. I guess some issues will still
come up later.

One important problem I wanted to upstream a solution in repo is to avoid pinging the state of each and every repository for a repo sync.

When doing git fetch of a branch the git protocol will send all the refs available for the repo (including the refs/changes!). This was a 22MB of sha1+branchref lines with a noop repo sync.

We have a repo optimization which will use the gerrit ls-project api to query the state of only the branches that we are following and decide if there is a need to use git fetch.

I wonder if you have a solution in mind for this problem.

> - What is the pre-commit/pre-submit check support from CI tools?

How do current pre submit tests work? hint: they are incredible hard to
get right across multiple repositories; In the future it is as simple
as using one call
to get all changes across repositories (and possibly including the project
that manages the dependencies, the superproject) in one go
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#submit-preview

That is a quite interesting API indeed!

The first question that come in mind is how fast is it? how much load will gerrit be able to achieve?

Gerrit is already quite loaded, I am not sure if this is a good idea to put it on the master. I guess this API will work on a read-only mirror, right?

 

The second remark is that there are two needs for such API.

The first need is for the scheduling.

CI needs to figure out which job needs to be triggered by a change. Usually there is one job per (topproject, branch), then those job trigger parallel subjobs.

The complexities of the programs makes it not always feasible to use same branch for subprojects and topprojects.

So in order to fully use of this great stuff, there is a need for an API which tell the list of (topprojects, branch) that are affected more or less recursively.

The second is the cherry-picking, and this idea of packing the bundles looks pretty cool. BTW, we use cherry-pick policy for most of our projects and not merge, in order to keep the git history clean.

so the apply of the bundle will be a git rebase for us, which looks fine enough.

Another remark is a problem that we had, and could not resolve yet is how to atomically test and update very top project that are affected by a change.

This was a problem that our kernel team asked. They wanted to have the kernel branch to be the same in the current dessert, and the next dessert manifests.

Again, how do you see this problem to be resolved?

The main problem that we had to resolve here is how to track the user spaces dependencies. sometime, e.g. graphics kernel patches have to be merged with associated graphics userland, and then you have to describe somehow the dependency list for each dessert, and this was too much complexity.

>    It is not easywith repo but relatively easier, as there is all the
> metadata needed in the manifest, including branches that are tracked in
> submodules, etc.

All information that is in git-repo ought to be available via the
gitmodules file as well.

 

I can't find from the git documentation if it is now possible to not fix the commit of a submodule, i.e just always sync the latest commit of the tracked branch.

>
> - How can we implement manifest filtering with git submodules?
>   For example Android SoC vendors manage internal manifest with lots of socs
> in it, then their customer teams will take the internal release manifest,
> filter out the repositories according to the soc the customer needs, and
> convert to binaries the repositories that the customer does not have signed
> NDA for.

You'd need some sort of grouping feature for Submodules, such as Git attributes
e.g. https://public-inbox.org/git/20161110203428....@google.com/

Indeed that looks like pretty bleeding edge stuff.

 

[Stefan Beller]

On Fri, Nov 18, 2016 at 5:31 AM, Pierre Tardy <tar...@gmail.com> wrote:

>
> Having 100% coverage from the start will indeed not be possible, but 2% is
> better than 0%, and the 4% will be even better, until we can require 90%
> coverage of the patched lines for every contribution.
> This is what we did for the venerable buildbot which I am now maintainer of.
>
> That said indeed if submodule is say 6 month away from achieving what we do
> with repo, maybe its not worth to make such big investment.

While I do not promise a 6 month here, I actually think it is a
reasonable time frame.

Then the manifest is tracking the projects by sha1 and not by target branch?

>
> The problem that I did not automate is atomic cross-project revert. What we
> did is revert the problematic patches manually, and merge them together
> atomically.
> I wonder if this is a problem that you have a solution for. Will git revert
> in a top dir will revert the patches in the subrepositories instead of just
> pointing the top dir to the ancient commit (and break tracability in the
> subrepositories)

For now I'd expect revert to be broken in the sense it would point at
the ancient commits. (Well actually revert would have a merge conflict
with follow ups that also touched the submodule pointer, I would assume)

>> Now we could come up with such a sync repo under the hood within
>> git-repo as well. However instead of piling more on top of Git,
>> why not use Git directly? (Submodules solve this exact problem,
>> they are just hard to use, so we need to make it as nice as the repo tool
>> such that people will switch voluntarily).
>
> For me the main reason is to have a scripting language to handle those high
> level problems instead of having to write it in C (and then rewrite it in
> java in JGIT)

The serving side (speaking for JGit/Gerrit) seems surpringly advanced w.r.t.
submodules already IMHO, so I don't think a lot of work needs to be done there.

>
>>
>> >
>> > Especially:
>> > - Will it scale with 700+ repositories that some android team have?
>>
>> I did some work on that, e.g. parallel fetching of submodules
>>
>> https://kernel.googlesource.com/pub/scm/git/git/+/master/Documentation/RelNotes/2.8.0.txt
>>
>> * Add a framework to spawn a group of processes in parallel, and use
>> it to run "git fetch --recurse-submodules" in parallel.
>>
>>
>> https://kernel.googlesource.com/pub/scm/git/git/+/master/Documentation/RelNotes/2.9.0.txt
>>
>> * "git clone" learned the "--shallow-submodules" option.
>>
>> Not sure if that is sufficient though. I guess some issues will still
>> come up later.
>
>
> One important problem I wanted to upstream a solution in repo is to avoid
> pinging the state of each and every repository for a repo sync.

Heh, yeah we'd expect server load to go down by a lot once submodules
are in use. This is because you would fetch the superproject, which records
all submodules that were changed, and then you only have to fetch these
~5 / 800 submodules that actually changed.

But server load is not the main selling point.

>
> When doing git fetch of a branch the git protocol will send all the refs
> available for the repo (including the refs/changes!). This was a 22MB of
> sha1+branchref lines with a noop repo sync.

I agree that the protocol needs some overhaul to accommodate Gerrits
excessive use of refs.

>
> We have a repo optimization which will use the gerrit ls-project api to
> query the state of only the branches that we are following and decide if
> there is a need to use git fetch.
>
> I wonder if you have a solution in mind for this problem.

"Shortterm" we'll migrate to submodules, which requires less repositories
to be fetched (superproject + all changed submodules; lots of submodules
are expected to be unchanged) and longer term we'd want to fix the protocol,
e.g. protocol version 2 ("client speaks first", first round is
capability negotiation
only, instead of having the server blast 22MB of refs at the client
without a way to stop it)

>
>>
>> > - What is the pre-commit/pre-submit check support from CI tools?
>>
>> How do current pre submit tests work? hint: they are incredible hard to
>> get right across multiple repositories; In the future it is as simple
>> as using one call
>> to get all changes across repositories (and possibly including the project
>> that manages the dependencies, the superproject) in one go
>>
>> https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#submit-preview
>
> That is a quite interesting API indeed!
> The first question that come in mind is how fast is it? how much load will
> gerrit be able to achieve?

We haven't measured yet.

> Gerrit is already quite loaded, I am not sure if this is a good idea to put
> it on the master. I guess this API will work on a read-only mirror, right?

I don't see any reason why it would not work there.

>
> The second remark is that there are two needs for such API.
>
> The first need is for the scheduling.
> CI needs to figure out which job needs to be triggered by a change. Usually
> there is one job per (topproject, branch), then those job trigger parallel
> subjobs.
> The complexities of the programs makes it not always feasible to use same
> branch for subprojects and topprojects.
> So in order to fully use of this great stuff, there is a need for an API
> which tell the list of (topprojects, branch) that are affected more or less
> recursively.

What are you recursing into here? (I assume you do not mean submodules,
but superprojects?) The submit_preview would include any affected
superproject recursively though.

>
> The second is the cherry-picking, and this idea of packing the bundles looks
> pretty cool. BTW, we use cherry-pick policy for most of our projects and not
> merge, in order to keep the git history clean.
> so the apply of the bundle will be a git rebase for us, which looks fine
> enough.

Well the bundle contains a dry run of "what would happen if I were to
submit right now", i.e. the bundles would contain the cherrypicks already,
no need to rebase. (You could have a check in there to see if Gerrit
put the correct footers while cherry picking for example).

> Another remark is a problem that we had, and could not resolve yet is how to
> atomically test and update very top project that are affected by a change.
> This was a problem that our kernel team asked. They wanted to have the
> kernel branch to be the same in the current dessert, and the next dessert
> manifests.
> Again, how do you see this problem to be resolved?
>
> The main problem that we had to resolve here is how to track the user spaces
> dependencies. sometime, e.g. graphics kernel patches have to be merged with
> associated graphics userland, and then you have to describe somehow the
> dependency list for each dessert, and this was too much complexity.

So ideally we'd have Depends-On footers in Gerrit as well as the topics,
such that the user land patches could depend on the kernel patches?

The submit_preview API would include the changes to the superproject
but also having the state of the rest of the superproject, e.g. you'd
notice that
the kernel submodule advances by $sha1..$sha1 but the userspace
submodule `foo` is pinned at $sha1, and then the build may fail because
the dependency was not there?

>
>> > It is not easywith repo but relatively easier, as there is all the
>> > metadata needed in the manifest, including branches that are tracked in
>> > submodules, etc.
>>
>> All information that is in git-repo ought to be available via the
>> gitmodules file as well.
>
>
> I can't find from the git documentation if it is now possible to not fix the
> commit of a submodule, i.e just always sync the latest commit of the tracked
> branch.

By design submodules are defined via the sha1. However some repo
workflow is to use moving targets (refs) instead of fixed sha1s for
the subprojects,
so you may be interested in

`git submodule update --remote`

which would fetch all submodules and checkout the branch that is specified
via $(git config --file .gitmodules submodule.<name>.branch).

[Martin Fick]

On Friday, November 18, 2016 11:57:03 AM 'Stefan Beller' via

Repo and Gerrit Discussion wrote:
> On Fri, Nov 18, 2016 at 5:31 AM, Pierre Tardy
<tar...@gmail.com> wrote:
> > Having 100% coverage from the start will indeed not be
> > possible, but 2% is better than 0%, and the 4% will be
> > even better, until we can require 90% coverage of the
> > patched lines for every contribution. This is what we
> > did for the venerable buildbot which I am now
> > maintainer of.

I don't see any reason to not create tests for repo now. As
you mention, you start small, and you might soon be
surprised who helps contribute to them. I love the idea of
eventually getting rid of repo, however I know that my
company is heavily built around it, and I still would like
to fix some bugs in it. If I have tests that give me
confidence that repo will not break with my fixes, great! I
suspect that realistically, we will need to wrap submodules
with repo for a while if we ever want to have an actual
migration path forward. So I doubt that tests will be
wasted. Tests will also help solidify what features of repo
are important to users. This will help guide what is
actually needed for git submodules to replace it (and
perhaps even outline some similar tests that are needed for
submodules).


...

> >> The main issue that motivates the move to Submodules is
> >> this: Android wants to have atomicity across their
> >> whole code base in the development process
> >>
> >>
> >> When having such a API change across multiple
> >> repositories, this is impossible to sync atomically
> >> except via indirection over another single repository.

And, of course, this can also be done with manifests using
sha1s or tags. We do this in many places. This confirms
that the git submodule approach will be good.

The new 'manifest' plugin that I mentioned during my summit
talk will also soon (we have changes under review) be able
to help create these versioned manifests server side instead
of having to use repo and having to have a local copy of the
repos that you want to version.

Even though the submodules support in Gerrit should
eventually provide the same feature, I think there is still
value in migrating to a server side (plugin) versioning
scheme for manifests before that is ready, and some of our
internal projects will do so. This will help people get
used to the work-flow, and likely will be easy to migrate to
a similar git submodules approach after (or even in
parallel), all while keeping approaches for new and legacy
work-flows close to the same.


...

> > Our CI is doing the atomic merges on its own environment
> > then forcepush the affected repositories, then wait for
> > mirrors to mirror the new ref,

You may want to check out the new 'batch' plugin that I also
discussed at the summit. It does this exact thing for you
server side which is much more efficient (and uses less server
load) for large projects, and it allows you to do this
without a local copy of the repositories. We have been
doing this server side for years, and it works very well.

> > and finally push a new
> > manifest to the manifest server. The manifest is then
> > acting as a top project.

We are currently enhancing the manifest plugin to work with
the batch plugin to be able to do something like this also.

...

> > One important problem I wanted to upstream a solution in
> > repo is to avoid pinging the state of each and every
> > repository for a repo sync.

This already happens if you use versioned manifests. Such
an approach is basically the same as how git submodules
would do it.

...

> > We have a repo optimization which will use the gerrit
> > ls-project api to query the state of only the branches
> > that we are following and decide if there is a need to
> > use git fetch.

I think we have wrappers that do this, having it built into
repo would be cool. Perhaps you could contribute your repo
modification to upstream repo?

Great discussion, thanks for bringing up such good points!

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Pierre Tardy]

Hi Stefan

On Fri, Nov 18, 2016 at 8:57 PM 'Stefan Beller' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

On Fri, Nov 18, 2016 at 5:31 AM, Pierre Tardy <tar...@gmail.com> wrote:

>
> That said indeed if submodule is say 6 month away from achieving what we do
> with repo, maybe its not worth to make such big investment.

While I do not promise a 6 month here, I actually think it is a
reasonable time frame.

So we are now 6 month after this discussion.

Can you update us on the status of git submodules at Android scale?

Do you think it would be feasible for a new Android integration team to start with git submodules instead of repo?

Can you communicate on the probably PoC you have done at Google on the current pain points if any?

Regards,

Pierre

[Stefan Beller]

On Wed, Apr 26, 2017 at 6:26 AM, Pierre Tardy <tar...@gmail.com> wrote:
>
> So we are now 6 month after this discussion.

Time flies. :)

> Can you update us on the status of git submodules at Android scale?

On the Gerrit side:
6 month ago core Gerrit could update the superproject gitlinks
via submodule subscriptions already, but changes to the manifest
(new/deleted/renamed projects in the manifest) are not handled by
core Gerrit. There is a new plugin 'Supermanifest'[1], which fits this gap.
It monitors changes to an existing manifest repo and will update the
submodules and superproject subscriptions in the superproject for that.

[1] https://gerrit.googlesource.com/plugins/supermanifest

On the Git side:
* commands that learned the '--recurse-submodules' flag:
ls-files, grep, checkout, reset

* submodule activeness is can be configured differently now.
"git config submodule.active <pathspec>" allows to select broad
groups of submodules. (the pathspec mechanis itself learned
new magic, such that you have a pathspec respecting gitattributes
which can be used to group submodules in another fashion, not just
via directories and path names)

By using such a grouping mechanis (for example via "x86" attribute
you can select submodules that may be added later in the future,
such that a "git submodule update --init" will catch these newly added
submodules.

* push options and refspecs being propagated to submodules.
One can push submodules, without pushing the superproject
This covers the Gerrit review workflow.

> Do you think it would be feasible for a new Android integration team to
> start with git submodules instead of repo?

We have not started internal dogfooding at scale, yet.
For that we're still planing on improving some working tree manipulators
(e.g. "git checkout --recurse-submodules -b" may want to create branches
in all submodules as well).

> Can you communicate on the probably PoC you have done at Google on the
> current pain points if any?

Gerrit itself is a good example for submodule use, though we haven't made
progress there[2].
[2] https://gerrit-review.googlesource.com/#/c/78805/

Thanks,
Stefan

[toko...@gmail.com]

Time flies fast.. Any updates on this repo replacement project status (for AOSP)?

[Wink Saville]

More time has flown by, any further updates?

[唐木村]

Dear  Stefan Beller

Now 6 years after this discussion.

Any update use repo or submodules for large projects like Android

Thanks