Git Superproject Support: Conflicts in the module cache

[Michael Devine]

Does anyone have an example of steps that work for using the git superproject to clone AOSP without using repo? From what I can tell, there are problems in the way AOSP's .gitmodules files are defined that prevent them from being used in a git project.

$ git clone https://android.googlesource.com/platform/superproject aosp/

$ pushd aosp/

$ git checkout android-15.0.0_r1

$ git submodule init

$ git submodule sync

$ git submodule update

... hours later ...

fatal: could not get a repository handle for submodule 'build/make'

Whenever I run the instructions above, I get an error from git related to the project in build/make. It seems to be related to the fact that git expects .git/modules/platform/build to be formatted like a proper .git directory, but it can't be because it needs to put data in .git/modules/platform/build/bazel and .git/modules/platform/build/soong and those would appear in the same directory.

Is this a known problem? Is there a way around it?

[Michael Devine]

It appears that tools/tradefederation have the exact same problem. The following commands should fix up the superproject if run after "git checkout" and before "git submodule init" in the instructions above.

$ sed --in-place -E 's/submodule "platform\/build"/submodule "platform\/build\/make"/g' .gitmodules

$ sed --in-place -E 's/submodule "platform\/tools\/tradefederation"/submodule "platform\/tools\/tradefederation\/core"/g' .gitmodules

$ git commit -a -m “Fixed up build/make and tools/tradefederation modules”

[Michael Devine]

One more question: The .gitmodules for android-15.0.0_r1 has a bunch of mentions of a "ref" field for each submodule. From what I can tell, that field is not documented anywhere in the Git documentation for gitmodules. Is there a place to find out more about this and how it relates to the "branch" field that is in the documentation?

[Ivan Frade]

Hi,

The ref field is something specific of repo and gerrit. We write
"ref" to hint the branch of the SHA1 when we cannot use the "branch"
field.

Gerrit uses the branch field to autoupdate the submodules in a
repository [1]. No "branch" means "disable autoupdate". At the same
time, knowing the branch where the sha1 came from is a big performance
boost for other tools browsing the code (finding the branch of a sha1
is expensive). So when we need to remove the "branch" field (to stop
autoupdates), we write the branch hint in the "ref" field.

Hope this helps,

Ivan

[1] https://gerrit-review.googlesource.com/Documentation/user-submodules.html

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/e99b9139-932b-48b5-b18c-f7242c6aaef3n%40googlegroups.com.

[Michael Devine]

Thanks Ivan! 

Does that mean that we should not expect a git clone/checkout of the superproject to work at all? Is the ref field actually used in determining the correct commit to use in each submodule?

Mike

[Ivan Frade]

Hi,

> Does that mean that we should not expect a git clone/checkout of the superproject to work at all? Is the ref field actually used in determining the correct commit to use in each submodule?

In principle, the clone/checkout of the superproject should work. The
superproject is a regular git repository with submodules, the only
special thing is that they get autoupdated on the server side. IIUC
the branch field is only relevant to update submodules to more recent
commits and if not set defaults to HEAD

That being said, many android superprojects are generated from
manifest files. The problem in your first message could be a some typo
in the manifest (e.g. build/make using the wrong origin), or (more
unlikely;f a bug in the translation from manifest to repo.

Ivan

On Thu, Mar 13, 2025 at 12:36 AM Michael Devine

> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/735f53ba-8d47-46a5-988f-e6b42761d9fan%40googlegroups.com.

[Michael Devine]

From what I can tell, the problem is that the translation of the manifest to the .gitmodules file just uses the name field of each project as the name of the submodule. The problem with this is that the name is based on the URI, and that the names of git submodules have to be defined so that they aren't ancestor paths of other submodules. e.g. if you have a .gitmodules that defines [submodule "a/b"] and [submodule "a/b/c"] it causes conflicts in the cache. 

This happens because it initializes .git/modules/a/b/c to be a bare git repository. Then it tries to do the same thing in .git/modules/a/b and says "Wait a minute, I can't make this a bare git repository because there are already subfolders here." It probably could get away with it in most cases, but if you had the case of "a/b" and "a/b/refs" the submodule with "refs" would collide with the "refs" directory in the a/b bare git repository and it would be a mess.

One way of fixing this would be to automatically suffix every submodule name when we generate the .gitmodules file. E.g. adding the "-module" suffix would mean that the above case would store cached data in ".git/modules/a/b-module" and ".git/modules/a/b/c-module" which are distinct paths.