Alternative to Google's repo script

[Mikael Persson]

Hello,

I am new to this group, and not sure if this is the right place, but at least it is related.

Like many, I have used Google's repo script for years, but never really used it more than

repo init and sync. After the projects were cloned, I solely relied on git for all other

operations, including uploading patches for Gerrit review.

I find that repo has its limitations in terms of manifest syntax (not flexible enough)

and would have preferred a tool which in general is closer to git - but running over several repos

like an "iterator". I even want to be able to use my own favorite git aliases, but for many repos

at the same time (in a "bulk" fashion).

Same feeling? As a good citizen, I started to make an alterative to repo, with the goal of fulfilling

my needs. Now reached a point where I am interested in some feedback before I move on...

Please check out https://github.com/rabarberpie/grit and read the documentation and/or try the script.

(Yeah, I probably should rename the tool/project to something better.)

Thanks!

Mikael

[Gareth Bowles]

As an alternative that is built into Git and available in newer Gerrit versions, have you looked at Git superprojects ?

[Nicholas Mucci]

Back in the day, since repo wouldn't run on Windows (symlinking didn't work) and a department we worked with didn't trust submodules, we reimplemented a lot of repo's feature set for a department at our company.  We used the repo manifest xml format, and the repo functionality was reimplemented with as close to regular porcelain git commands as possible.  This was like 8 years ago, and despite there being better approaches today, it is still running and basically does the same thing.  In retrospect, I wish I would've pushed submodules harder, because now they are locked into this tool that was developed just for them, and that has a nonzero support cost.  

Your script looks pretty handy, and if it works for you, that's awesome, and its great that you've shared it.  Its always great to have options.  I like your run-after-clone feature.  Your tool would be a really cool way for an open source project to grab dependencies and configure them.

In general, for big projects or collaborative efforts, especially in a corporate environment, I second Gareth's suggestion of looking at superprojects/submodules.  

-Nick

[Mikael Persson]

Thanks Gareth and Nicholas for feedback!

I was slightly aware of the git submodules feature but I have never used it in practise. I have now played around with it a bit
and indeed it is quite similar to a manifest (repo or in my tool grit) - in "simple" scenarios at least.

I see benefits with an explicit manifest though:

* Possibility to add additional metadata to each repo (both standard git options missing in submodule, but also things like groups).

* With profiles (in my tool grit) you can change remote server in a snap; or any other common git option/parameter, like clone depth or default branch.

* With the possibility to overlay manifests, you can more easily handle more complicated setups (see CONFIGS.md).

To summarize, I find git submodules as a good addition, but it is not sufficient to meet my needs/requirements. 

- Mikael

[Martin Fick]

On Wednesday, April 11, 2018 01:36:41 PM Mikael Persson
wrote:

>
> To summarize, I find git submodules as a good addition,
> but it is not sufficient to meet my needs/requirements.

I suspect that git community would be receptive to
enhancements in this area. I know that the google git team
wants to bring submodules on par with repo to eventually be
able to replace the use of repo in android. Maybe pick your
most needed repo feature and work on getting it into git?

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Stefan Beller]

Hi Mikael,

thanks for trying out submodules!

On Wed, Apr 11, 2018 at 1:36 PM, Mikael Persson <rabar...@gmail.com> wrote:
> Thanks Gareth and Nicholas for feedback!
> I was slightly aware of the git submodules feature but I have never used it
> in practise. I have now played around with it a bit
> and indeed it is quite similar to a manifest (repo or in my tool grit) - in
> "simple" scenarios at least.
>
> I see benefits with an explicit manifest though:
> * Possibility to add additional metadata to each repo (both standard git
> options missing in submodule, but also things like groups).

You can add additional metadata to it, see
https://git-scm.com/docs/gitmodules for some features.

For now all these metadata are versioned in the superproject,
but there are plans to have other places, where submodule information
will be stored, i.e. on another branch in the repository,
similar to refs/meta/config in Gerrit, Git may adopt a
refs/meta/submodules ref, which is overlayed on the .gitmodules
file in the repo itself. (That allows for a decoupled history of submodule
changes)

https://git-scm.com/docs/gitsubmodules

> * With profiles (in my tool grit) you can change remote server in a snap; or
> any other common git option/parameter, like clone depth or default branch.

That is an interesting thing to want, so essentially you would want a command
like git config --change-all submodule.*.url s/my-server.com/other-server.com/
or such.

> * With the possibility to overlay manifests, you can more easily handle more
> complicated setups (see CONFIGS.md).

Can you talk more about that?

> To summarize, I find git submodules as a good addition, but it is not
> sufficient to meet my needs/requirements.

Thanks for the evaluation!

Thanks,
Stefan

[Mikael Persson]

Den torsdag 12 april 2018 kl. 02:04:45 UTC+2 skrev Stefan Beller:

Hi Mikael,

thanks for trying out submodules!

On Wed, Apr 11, 2018 at 1:36 PM, Mikael Persson <rabar...@gmail.com> wrote:
> Thanks Gareth and Nicholas for feedback!
> I was slightly aware of the git submodules feature but I have never used it
> in practise. I have now played around with it a bit
> and indeed it is quite similar to a manifest (repo or in my tool grit) - in
> "simple" scenarios at least.
>
> I see benefits with an explicit manifest though:
> * Possibility to add additional metadata to each repo (both standard git
> options missing in submodule, but also things like groups).

You can add additional metadata to it, see
https://git-scm.com/docs/gitmodules for some features.

Can you add your own metadata as well (similar to "x-" in grit)? In my experience, the most common usage

is license info, to keep track of all code licenses in a project.

Looking at the link I mainly miss the ability to set group memberships (in the repo meaning) and single-branch.

(I guess this should be fairly easy to fix though...)

Groups are quite handy when you work within a domain in a larger system, e.g. I would like to do things like

"grit -g camera checkout -b mybranch" to checkout mybranch on all camera gits.

For now all these metadata are versioned in the superproject,
but there are plans to have other places, where submodule information
will be stored, i.e. on another branch in the repository,
similar to refs/meta/config in Gerrit, Git may adopt a
refs/meta/submodules ref, which is overlayed on the .gitmodules
file in the repo itself. (That allows for a decoupled history of submodule
changes)

https://git-scm.com/docs/gitsubmodules

> * With profiles (in my tool grit) you can change remote server in a snap; or
> any other common git option/parameter, like clone depth or default branch.

That is an interesting thing to want, so essentially you would want a command
like git config --change-all submodule.*.url s/my-server.com/other-server.com/
or such.

> * With the possibility to overlay manifests, you can more easily handle more
> complicated setups (see CONFIGS.md).

Can you talk more about that?

I spend some more time adding examples at:

https://github.com/rabarberpie/grit/blob/master/CONFIGS.md

The fundamental approach in grit is that shared git settings/parameters (among repositories), like remote name and URL, are placed in profiles. And profiles can inherit from other profiles to further encapsulate common settings, like branch name. In this way, there is essentially no duplication of information in the manifest and with properly defined profiles, you can easily change settings for all, or a subset, of the repositories.

But even above is not always sufficient. The possibility to overlay manifest enables you to share common manifest files into different combinations (see the link above for example use-cases). Overlaying is more powerful than include (like repo has), since it can not only add things, but also modify or remove things.

  

> To summarize, I find git submodules as a good addition, but it is not
> sufficient to meet my needs/requirements.

Thanks for the evaluation!

Thanks for the feedback! :-)

 

Thanks,
Stefan

[Mikael Persson]

Den onsdag 11 april 2018 kl. 23:35:32 UTC+2 skrev MartinFick:

On Wednesday, April 11, 2018 01:36:41 PM Mikael Persson
wrote:
>
> To summarize, I find git submodules as a good addition,
> but it is not sufficient to meet my needs/requirements.

I suspect that git community would be receptive to
enhancements in this area.  I know that the google git team
wants to bring submodules on par with repo to eventually be
able to replace the use of repo in android.  Maybe pick your
most needed repo feature and work on getting it into git?

I wasn't aware of that ambition/goal! It could be a good idea,

but I guess I need to understand submodules a bit better first;

currently there are some differences and gaps and I am not sure
whether they should converge, or one should build on the other.

[Stefan Beller]

On Tue, Apr 17, 2018 at 1:35 PM, Mikael Persson <rabar...@gmail.com> wrote:
>
>
> Den torsdag 12 april 2018 kl. 02:04:45 UTC+2 skrev Stefan Beller:
>>
>> Hi Mikael,
>>
>> thanks for trying out submodules!
>>
>> On Wed, Apr 11, 2018 at 1:36 PM, Mikael Persson <rabar...@gmail.com>
>> wrote:
>> > Thanks Gareth and Nicholas for feedback!
>> > I was slightly aware of the git submodules feature but I have never used
>> > it
>> > in practise. I have now played around with it a bit
>> > and indeed it is quite similar to a manifest (repo or in my tool grit) -
>> > in
>> > "simple" scenarios at least.
>> >
>> > I see benefits with an explicit manifest though:
>> > * Possibility to add additional metadata to each repo (both standard git
>> > options missing in submodule, but also things like groups).
>>
>> You can add additional metadata to it, see
>> https://git-scm.com/docs/gitmodules for some features.
>
> Can you add your own metadata as well (similar to "x-" in grit)? In my
> experience, the most common usage
> is license info, to keep track of all code licenses in a project.

You can add any metadata to it using the git config file format.

> Looking at the link I mainly miss the ability to set group memberships (in
> the repo meaning) and single-branch.

group memberships can be done via git attributes, and then

git clone --recurse-submodules=:(attr:default,my-group)

The single-branch is just a missing piece we'll have to look into

> Groups are quite handy when you work within a domain in a larger system,
> e.g. I would like to do things like
> "grit -g camera checkout -b mybranch" to checkout mybranch on all camera
> gits.

In a submodule world you can configure the active submodules via
submodule.active, which again takes a pathspec, including grouping
by git attributes.

>> > * With the possibility to overlay manifests, you can more easily handle
>> > more
>> > complicated setups (see CONFIGS.md).
>>
>> Can you talk more about that?
>
> I spend some more time adding examples at:
> https://github.com/rabarberpie/grit/blob/master/CONFIGS.md
> The fundamental approach in grit is that shared git settings/parameters
> (among repositories), like remote name and URL, are placed in profiles. And
> profiles can inherit from other profiles to further encapsulate common
> settings, like branch name. In this way, there is essentially no duplication
> of information in the manifest and with properly defined profiles, you can
> easily change settings for all, or a subset, of the repositories.
>
> But even above is not always sufficient. The possibility to overlay manifest
> enables you to share common manifest files into different combinations (see
> the link above for example use-cases). Overlaying is more powerful than
> include (like repo has), since it can not only add things, but also modify
> or remove things.

Thanks!
Stefan