Maintaining a repo mirror

[Tom Parkin]

Hi folks,

I am starting work on a project which uses Android, and am looking into setting up a repo tree to use for development work, and I was hoping to get some pointers on how best to do this.

We will be taking our initial repo tree from the silicon vendor, who hosts their platform support in a repo tree we can init and sync from.

My current plan is broadly as follows:

1. We will create a mirror of the si vendor tree using "repo init --mirror", and host this for our developers in our internal network.

2. We will set up a gerrit instance, with the gerrit.config gerrit.basePath set up to include the repo mirror

3. To develop our product, we will create a branch of manifests.git, and add a new manifest file for our product.  This will:

  * include the si vendor manifest file as a starting point

  * add in new project elements for code components we add (our own product code as well as external dependencies, etc)

  * use extend-project to switch to our own dev branches for specific projects where we need tweaks for our product (e.g. kernel, u-boot)

I think this should allow us to carry out development on our own branches of each repo project, and our developers will be able to use repo upload to submit their work to the branches using gerrit.

I'm less clear about taking updates from the silicon vendor.  From the perspective of updating the repo mirror, I think it is just a case of running "repo sync" to get the changes, and then updating all our branched components (manifests.git, plus anything else from the si vendor that we branched).

But it's not clear to me how this interacts with gerrit.  Firstly, do people typically manage updates like this via. gerrit?  Or just use git commands from the shell?  If the latter, is there any concerns around concurrency if the gerrit server is running while the updates are taking place?

Any comments on any of the above are welcome, especially around taking updates from the silicon vendor.

Many thanks!

Tom

[Martin Fick]

On 2022-02-25 11:09, Tom Parkin wrote:
> I'm less clear about taking updates from the silicon vendor. From the
> perspective of updating the repo mirror, I think it is just a case of
> running "repo sync" to get the changes, and then updating all our
> branched components (manifests.git, plus anything else from the si
> vendor that we branched).

Do you plan to keep the vendor's branches named the same thing that the
vendor named them? If so, then if you ever want to merge any changes
of your own on those branches, you might have to manage the merge
conflicts during the sync.

We place any upstream branches in separate upstream branch hierarchies
named after the upstream, so something like:

refs/heads/<upstream>/<upstream_branch> and generally consider those
branches read-only.

> But it's not clear to me how this interacts with gerrit. Firstly, do
> people typically manage updates like this via. gerrit? Or just use
> git commands from the shell? If the latter, is there any concerns
> around concurrency if the gerrit server is running while the updates
> are taking place?

We manage them via git commands in a cron job. Generally git/Gerrit is
fine with concurrency, NFS sometimes doesn't like it. Be sure to use
the same OS user for syncing that Gerrit runs as!

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Tom Parkin]

Hi Martin,

Thanks for your response!

On Thursday, 3 March 2022 at 01:19:39 UTC MartinFick wrote:

Do you plan to keep the vendor's branches named the same thing that the
vendor named them? If so, then if you ever want to merge any changes
of your own on those branches, you might have to manage the merge
conflicts during the sync.

We place any upstream branches in separate upstream branch hierarchies
named after the upstream, so something like:

refs/heads/<upstream>/<upstream_branch> and generally consider those
branches read-only.

I think there are two options:

1. We keep most of the vendor's branches the same as the vendor named them, and only locally branch things we want to modify directly ourselves.  The upside of this I think is that we can easily see in our manifest file what components we have changed (I am envisaging adding a new toplevel manifest which includes the vendor manifest, and then adding extend-projects elements for things we modify).  The downside is a bit more of a dance to ensure we branch stuff before we work on it.

2. We create all our own branches on all components at the point of setting up our tree.  The upside of this is that we can work on components without worrying about branching.  The downside is that when it's time to pull an update from the vendor it's a little less clear what components have been modified in our tree and hence need porting to the vendor's updated branches.

My gut feel on this is that it's a trade-off and there's no "right" way to go -- would you agree?

 

> But it's not clear to me how this interacts with gerrit. Firstly, do
> people typically manage updates like this via. gerrit? Or just use
> git commands from the shell? If the latter, is there any concerns
> around concurrency if the gerrit server is running while the updates
> are taking place?

We manage them via git commands in a cron job. Generally git/Gerrit is
fine with concurrency, NFS sometimes doesn't like it. Be sure to use
the same OS user for syncing that Gerrit runs as!

Yep, that all sounds good - -we would use the same OS, and would not be using NFS I don't think.  Thanks :-)

Since my original posting I thought of another question around managing the manifest.

I get the impression that people generally define the manifest with branches of all components. Hence a given client sync gets the tip of each git branch at that time.  I can see that this means there's no need to fiddle with the manifest day-in-day-out (compared with a pinned manifest).

But I worry slightly about potential instabilities whereby a multi-component change gets partially merged with one component held up due to e.g. merge conflicts.  That could leave the tree in an inconsistent state.  Do people find this a practical problem, or is it sufficiently rare to not be worth worrying about?

Thanks again,

Tom