Problems re-using "repo manifest -r" manifests with repo

[Paul Sokolovsky]

Hello,

Here at Linaro, we pull components for Android builds from various
sources, like AOSP upstream, our forks of AOSP components (to
fix compatibility issues with gcc 4.5/4.6, etc.), bootloaders and
kernels from SoC support teams, etc. All in all, that means that we
don't have full control over those repositories for example to tag them.

So, to achieve build reproducibility, we decided to use a feature that
repo provides, exporting "release" manifest for a build using
"repo manifest -r", which dumps manifest with SHA1 revision ids of
components used for particular build.

However, we found that such release manifests are not robust over time.
If an upstream project uses rebase work model, then, even if a
particular revision was tagged which ensures that this revision won't be
garbage-collected if rebased later in main branch, the repo tool may not
be able to fetch it.

This happens because, unlike git clone, the repo tool limits what it
fetches to refs/heads/*. So, if some revision is not reachable from one
of branches (but still reachable from refs/tags/*), it will lead to
an error like:

error.GitError: boot/u-boot-linaro-stable update-ref: error: Trying to write ref refs/remotes/m/linaro_android_2.3.3 with nonexistant object 9736a9332fcfe5fef1361a6d91740e160ad04bd5
fatal: Cannot update the ref 'refs/remotes/m/linaro_android_2.3.3'.

So, we would like to know if this can be considered a bug (it looks
pretty much such for us, after all, we're just trying to feed up repo
with what it generated, while 100% having valid revisions in the
underlying repositories), and can be fixed. The patch would be pretty
simple:

diff --git a/project.py b/project.py
index 9d67dea..31ee90f 100644
--- a/project.py
+++ b/project.py
@@ -1369,7 +1374,7 @@ class Project(object):
else:
ref_dir = None

- cmd = ['fetch']
+ cmd = ['fetch', '--tags']

# The --depth option only affects the initial fetch; after that we'll do
# full fetches of changes.

--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

[Magnus Bäck]

On Tuesday, July 12, 2011 at 16:01 CEST,
Paul Sokolovsky <paul.so...@linaro.org> wrote:

> Here at Linaro, we pull components for Android builds from various
> sources, like AOSP upstream, our forks of AOSP components (to fix
> compatibility issues with gcc 4.5/4.6, etc.), bootloaders and kernels
> from SoC support teams, etc. All in all, that means that we don't have
> full control over those repositories for example to tag them.

Not if you always pull straight from them, but you can keep a local
mirror of them (on your server, along with your own gits). Then you can
do whatever you want with them, including tagging or branching them.

> So, to achieve build reproducibility, we decided to use a feature that
> repo provides, exporting "release" manifest for a build using
> "repo manifest -r", which dumps manifest with SHA1 revision ids of
> components used for particular build.
>
> However, we found that such release manifests are not robust over
> time. If an upstream project uses rebase work model, then, even if a
> particular revision was tagged which ensures that this revision won't
> be garbage-collected if rebased later in main branch, the repo tool
> may not be able to fetch it.

Gah, which project uses rebase for public branches that people are
expected to pull from?

[...]

> The patch would be pretty simple:

Looks reasonable to me.

[...]

--
Magnus Bäck Opinions are my own and do not necessarily
SW Configuration Manager represent the ones of my employer, etc.
Sony Ericsson

[Paul Sokolovsky]

Hello Magnus,

On Jul 12, 8:56 pm, Magnus Bäck <magnus.b...@sonyericsson.com> wrote:
> On Tuesday, July 12, 2011 at 16:01 CEST,

>      Paul Sokolovsky <paul.sokolov...@linaro.org> wrote:
>
> > Here at Linaro, we pull components for Android builds from various
> > sources, like AOSP upstream, our forks of AOSP components (to fix
> > compatibility issues with gcc 4.5/4.6, etc.), bootloaders and kernels
> > from SoC support teams, etc. All in all, that means that we don't have
> > full control over those repositories for example to tag them.
>
> Not if you always pull straight from them, but you can keep a local
> mirror of them (on your server, along with your own gits). Then you can
> do whatever you want with them, including tagging or branching them.

Yes, we considered complete mirroring as one solution, but we'd like
to
concentrate on doing our stuff like testing/fixing support for newer
GCC versions
and development hardware, rather than maintaining complete fork-
mirror, so
we'd like to consider other options too.

>
> > So, to achieve build reproducibility, we decided to use a feature that
> > repo provides, exporting "release" manifest for a build using
> > "repo manifest -r", which dumps manifest with SHA1 revision ids of
> > components used for particular build.
>
> > However, we found that such release manifests are not robust over
> > time. If an upstream project uses rebase work model, then, even if a
> > particular revision was tagged which ensures that this revision won't
> > be garbage-collected if rebased later in main branch, the repo tool
> > may not be able to fetch it.
>
> Gah, which project uses rebase for public branches that people are
> expected to pull from?

Well, it's not standalone project of course. Such cases usually start
with
someone hosting own patch queue branch for upstream submission, and
rightly
using rebases on it. But then patch queue gets long, other folks may
need that
stuff or at least want to test it, etc. So, it becomes kind of "proxy"
repository
which serves both upstream and downstream, and the latter can be
served reasonably
well with regular tagging. I guess this is not uncommon and reasonable
workflow,
at Linaro it's in particular used for U-boot patch queue, and we'd be
interested
to make repo work robustly even with such repositories.

>
> [...]
>
> > The patch would be pretty simple:
>
> Looks reasonable to me.

I assume this means "go on and submit that to Gerrit", and I'll do
that as
soon as I fix a contributor agreement.

>
> [...]
>
> --
> Magnus Bäck                   Opinions are my own and do not necessarily
> SW Configuration Manager      represent the ones of my employer, etc.
> Sony Ericsson

Thanks,

[Magnus Bäck]

On Tuesday, July 12, 2011 at 16:01 CEST,

Paul Sokolovsky <paul.so...@linaro.org> wrote:

[...]

> However, we found that such release manifests are not robust over
> time. If an upstream project uses rebase work model, then, even if a
> particular revision was tagged which ensures that this revision won't
> be garbage-collected if rebased later in main branch, the repo tool
> may not be able to fetch it.
>
> This happens because, unlike git clone, the repo tool limits what it
> fetches to refs/heads/*. So, if some revision is not reachable from
> one of branches (but still reachable from refs/tags/*), it will lead
> to an error like:
>
> error.GitError: boot/u-boot-linaro-stable update-ref: error: Trying to write ref refs/remotes/m/linaro_android_2.3.3 with nonexistant object 9736a9332fcfe5fef1361a6d91740e160ad04bd5
> fatal: Cannot update the ref 'refs/remotes/m/linaro_android_2.3.3'.

Ignoring the patch that you've uploaded (http://r.android.com/25843),
would it be an option to instead set the tagopt configuration variable
to --tags (see git-config(1))? After all, Repo is just following Git's
default behavior for fetches so it isn't totally obvious that this is
a bug in Repo. But yeah, I can agree that it violates the principle of
least surprise.