New repo tool to support submodule approach?
630 views
Skip to first unread message
Matt Fischer
Oct 1, 2010, 3:41:06 PM10/1/10
to Repo and Gerrit Discussion
There is an ongoing effort to move repo to a submodule-based tool,
instead of manual repository management. The manifest format is in
place, and the patch to add submodule smarts to Gerrit (https://
review.source.android.com/#change,14033) seems to be nearly done.
What I haven't seen yet are any references to a new incarnation of the
repo tool itself to actually make use of all of this. Has any work
gone on yet to develop it? My company is planning on making use of
the submodule framework, and I'm trying to determine whether I'll need
to write this tool myself, or if there is already work going on
somewhere that we can use/contribute to.
Thanks,
Matt
instead of manual repository management. The manifest format is in
place, and the patch to add submodule smarts to Gerrit (https://
review.source.android.com/#change,14033) seems to be nearly done.
What I haven't seen yet are any references to a new incarnation of the
repo tool itself to actually make use of all of this. Has any work
gone on yet to develop it? My company is planning on making use of
the submodule framework, and I'm trying to determine whether I'll need
to write this tool myself, or if there is already work going on
somewhere that we can use/contribute to.
Thanks,
Matt
Nasser Grainawi
Oct 1, 2010, 5:00:48 PM10/1/10
to Matt Fischer, Repo and Gerrit Discussion
I believe it's already there:
http://android.git.kernel.org/?p=tools/repo.git;a=commit;h=57272ba82e3e1baa2bd7743d799f7dbc2acd43f8
Nasser
--
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum
Matt Fischer
Oct 1, 2010, 5:40:07 PM10/1/10
to Repo and Gerrit Discussion
I'm not sure if that's what I was looking for. That change seems to
add an option to convert the manifest.xml format into a .gitmodules
file, but the process of checking out a repository (i.e. running 'repo
init' or 'repo sync') still runs through all the machinery of manually
checking out each repository into the proper place, etc. Under a
submodule system, repo shouldn't have to do any of that, it just needs
to call 'git submodule update' and it's done. Has anybody tackled
that piece of it yet?
--Matt
add an option to convert the manifest.xml format into a .gitmodules
file, but the process of checking out a repository (i.e. running 'repo
init' or 'repo sync') still runs through all the machinery of manually
checking out each repository into the proper place, etc. Under a
submodule system, repo shouldn't have to do any of that, it just needs
to call 'git submodule update' and it's done. Has anybody tackled
that piece of it yet?
--Matt
On Oct 1, 4:00 pm, Nasser Grainawi <nas...@codeaurora.org> wrote:
> On 10/01/2010 01:41 PM, Matt Fischer wrote:
>
> > There is an ongoing effort to move repo to a submodule-based tool,
> > instead of manual repository management. The manifest format is in
> > place, and the patch to add submodule smarts to Gerrit (https://
> > review.source.android.com/#change,14033) seems to be nearly done.
> > What I haven't seen yet are any references to a new incarnation of the
> > repo tool itself to actually make use of all of this. Has any work
> > gone on yet to develop it? My company is planning on making use of
> > the submodule framework, and I'm trying to determine whether I'll need
> > to write this tool myself, or if there is already work going on
> > somewhere that we can use/contribute to.
>
> > Thanks,
> > Matt
>
> I believe it's already there:http://android.git.kernel.org/?p=tools/repo.git;a=commit;h=57272ba82e...
> On 10/01/2010 01:41 PM, Matt Fischer wrote:
>
> > There is an ongoing effort to move repo to a submodule-based tool,
> > instead of manual repository management. The manifest format is in
> > place, and the patch to add submodule smarts to Gerrit (https://
> > review.source.android.com/#change,14033) seems to be nearly done.
> > What I haven't seen yet are any references to a new incarnation of the
> > repo tool itself to actually make use of all of this. Has any work
> > gone on yet to develop it? My company is planning on making use of
> > the submodule framework, and I'm trying to determine whether I'll need
> > to write this tool myself, or if there is already work going on
> > somewhere that we can use/contribute to.
>
> > Thanks,
> > Matt
>
Nasser Grainawi
Oct 1, 2010, 11:03:33 PM10/1/10
to Matt Fischer, Repo and Gerrit Discussion
No, AFAIK the plan didn't ever include switching fully over to using git submodule, only to take advantage of the git data model provided by submodule (gitlinks).
Nasser
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
Matt Fischer
Oct 2, 2010, 12:38:18 PM10/2/10
to Repo and Gerrit Discussion
Ok, but that still means that there's a plan to eventually set up repo
to make use of gitlinks to store the SHA-1 hash of each project,
correct? My understanding is that right now, it doesn't do that, but
simply uses the .gitmodules file to determine paths and revisions, and
then downloads everything manually just like it did before.
Is there work going on to make that change?
--Matt
> > More info athttp://groups.google.com/group/repo-discuss?hl=en
to make use of gitlinks to store the SHA-1 hash of each project,
correct? My understanding is that right now, it doesn't do that, but
simply uses the .gitmodules file to determine paths and revisions, and
then downloads everything manually just like it did before.
Is there work going on to make that change?
--Matt
Ulrik Sjölin
Oct 2, 2010, 1:44:59 PM10/2/10
to Repo and Gerrit Discussion
We have mainly concentrated on getting this functionality into Gerrit
itself and for the moment just left repo behind. You are of course
right though, we have to update repo, otherwise we will have a bunch
of confused users once we go live with the git-links ;-) As far as the
changes into Gerrit goes, we have divided this functionality into 2
parts. 1 part to handle the automatic updating of the "superproject"
on all relevant branches. The second part we are starting Monday next
week and it will handle the atomicity part. Basically the second part
should make it possible for a user to say "change x in git 1 is
dependant on change y in git 2" and Gerrit should be able to handle
this. I suppose a part 3 is to make sure that repo is taught to handle
submodules.
Ulrik
On Oct 2, 6:38 pm, Matt Fischer <mattfische...@gmail.com> wrote:is
itself and for the moment just left repo behind. You are of course
right though, we have to update repo, otherwise we will have a bunch
of confused users once we go live with the git-links ;-) As far as the
changes into Gerrit goes, we have divided this functionality into 2
parts. 1 part to handle the automatic updating of the "superproject"
on all relevant branches. The second part we are starting Monday next
week and it will handle the atomicity part. Basically the second part
should make it possible for a user to say "change x in git 1 is
dependant on change y in git 2" and Gerrit should be able to handle
this. I suppose a part 3 is to make sure that repo is taught to handle
submodules.
Ulrik
On Oct 2, 6:38 pm, Matt Fischer <mattfische...@gmail.com> wrote:is
Anthony
Oct 7, 2010, 9:24:52 AM10/7/10
to Repo and Gerrit Discussion
Ulrik,
Matt and I are more interested in the repo change coming first than
the dependencies for our needs.
Had you given much consideration to functionality of the tool? It
seems to me that the main reason for repo going forward is just to
make uploading a bit easier for people.
We believed that a new tool, completely separate from the current
repo, should be used. We were also interested in having Windows
support, so it could be easier if the tool was in a language like
Java.
It would be quite a bit more difficult to deal with creating a .repo/
projects and symlinking out git information when using submodules, but
I don't feel like that functionality currently is beneficial.
Matt and I are more interested in the repo change coming first than
the dependencies for our needs.
Had you given much consideration to functionality of the tool? It
seems to me that the main reason for repo going forward is just to
make uploading a bit easier for people.
We believed that a new tool, completely separate from the current
repo, should be used. We were also interested in having Windows
support, so it could be easier if the tool was in a language like
Java.
It would be quite a bit more difficult to deal with creating a .repo/
projects and symlinking out git information when using submodules, but
I don't feel like that functionality currently is beneficial.
jesse greenwald
Oct 14, 2010, 10:39:21 AM10/14/10
to Repo and Gerrit Discussion
Hi Ulrik,
Do you guys have a timeline for "part 2" - change dependencies?
I'd also be interested in learning more about your approach if you'd
like to share. For instance, I imagine it may not be too difficult to
add a new table to track dependencies between project. The GUI may
not be so bad either.
However, trying to atomically commit change might start to get tricky.
- Would you try to merge all of a change's dependencies before the
change itself?
- What happens if one of the dependencies won't cleanly merge?
- If you allow for circular dependencies then you might have to set
up some kind of transaction in which ref updates happen on the
projects only once all the merges are performed (and you may need to
lock down all repos to prevent ref updates from happening in the
middle).
Anyway, those are some thoughts I had rolling around. I look forward
to seeing what you guys come up with :).
Thanks,
Jesse
Do you guys have a timeline for "part 2" - change dependencies?
I'd also be interested in learning more about your approach if you'd
like to share. For instance, I imagine it may not be too difficult to
add a new table to track dependencies between project. The GUI may
not be so bad either.
However, trying to atomically commit change might start to get tricky.
- Would you try to merge all of a change's dependencies before the
change itself?
- What happens if one of the dependencies won't cleanly merge?
- If you allow for circular dependencies then you might have to set
up some kind of transaction in which ref updates happen on the
projects only once all the merges are performed (and you may need to
lock down all repos to prevent ref updates from happening in the
middle).
Anyway, those are some thoughts I had rolling around. I look forward
to seeing what you guys come up with :).
Thanks,
Jesse
Ulrik Sjölin
Oct 14, 2010, 3:26:39 PM10/14/10
to Repo and Gerrit Discussion
Hi there,
As a matter of fact we are working on part 2 right now and I think we
are making good progress, I think we are ready to push in 2-3 weeks.
So this is the basic use case:
1) user makes changes and creates commits in submodule a, b and c.
2) the user issues a "atomic" command in gerrit along with the sha1s
of commit a', b' and c'.
3) user pushes
4) when gerrit sees that all 3 commits has come in a commit on the
super project (s') is created on a review-branch (refs/changes/xyz)
where the git links points to a', b' and c'
5) upon review the b' change needs a new patchset and when the user
uploads b'', gerrit will create a new commit on the super-project s''.
6) This time all changes pass review and gets merged, however b'' does
not merge clean. Gerrit will have detected this beforehand and the
submit button is greyed out.
7) The user uploads a b''' that can be merged cleanly and passes
review ;-) This time gerrit will merge a', b''' and c' and create a
new commit on the destination branch of the super project where the
gitlinks points to the merged changes of the subprojects.
There is not much point in reviewing a super-project commit, OTOH
other users are probably interested in being able to download the full
change and perhaps its best to create a proper change for the super-
project as well.... what do you guys think? From a continous
integration perspective this is a non-issue since the stream event
will give the CI a way to get access to these super project commits
anyway.
Ahh well.... thoughts and/or ideas are very welcome.
Ulrik
As a matter of fact we are working on part 2 right now and I think we
are making good progress, I think we are ready to push in 2-3 weeks.
So this is the basic use case:
1) user makes changes and creates commits in submodule a, b and c.
2) the user issues a "atomic" command in gerrit along with the sha1s
of commit a', b' and c'.
3) user pushes
4) when gerrit sees that all 3 commits has come in a commit on the
super project (s') is created on a review-branch (refs/changes/xyz)
where the git links points to a', b' and c'
5) upon review the b' change needs a new patchset and when the user
uploads b'', gerrit will create a new commit on the super-project s''.
6) This time all changes pass review and gets merged, however b'' does
not merge clean. Gerrit will have detected this beforehand and the
submit button is greyed out.
7) The user uploads a b''' that can be merged cleanly and passes
review ;-) This time gerrit will merge a', b''' and c' and create a
new commit on the destination branch of the super project where the
gitlinks points to the merged changes of the subprojects.
There is not much point in reviewing a super-project commit, OTOH
other users are probably interested in being able to download the full
change and perhaps its best to create a proper change for the super-
project as well.... what do you guys think? From a continous
integration perspective this is a non-issue since the stream event
will give the CI a way to get access to these super project commits
anyway.
Ahh well.... thoughts and/or ideas are very welcome.
Ulrik
jesse greenwald
Oct 14, 2010, 5:26:11 PM10/14/10
to Repo and Gerrit Discussion
Hi Ulrik,
Thanks for the quick response and the detailed work flow. I have some
comments in-line.
> 1) user makes changes and creates commits insubmodulea, b and c.
another entry box on the change page which would allow a change to be
linked to another one. Gerrit could check to make sure the change
you're linking to exists first to validate the dependency.
> 4) when gerrit sees that all 3 commits has come in a commit on the
> super project (s') is created on a review-branch (refs/changes/xyz)
> where the git links points to a', b' and c'
>
project changes (especially compared to the possible complications).
Here are a few of issues I can think of off the top of my head:
- Several super projects tracking the submodules. This would result
in multiple super project changes for every creation of an atomic
group or every change to an atomic group.
- What happens if the super-project change is accidentally submitted
before all the submodule changes are in? The gitlink entries would
then point to non-existent commits (which is the typical problem
with git submodules)
- What happens if a super project starts tracking one of the
submodules that is currently part of an atomic group? Would Gerrit
need to create a change for the super project right away?
Likewise, what happens if a super-project stops tracking one of the
submodules involved in an atomic group.
- Once the submodules are merged in, the super-project change may
become out of date if something is merged in before it goes in.
Would the user be required to rebase the super project change? I
don't want to explain how to our users how to rebase a change with
gitlink updates :)
Instead, I would probably modify the submodule updating logic you're
working on in change 14033 to look for these atomic submissions. It
would need to be smart enough to ignore individual merges that are a
part of an atomic transaction. Once an atomic transaction is complete
it would go about automatically updating the super-project's gitlinks
to point to the current commit of all it's submodules that have a
"revision" specified.
If there is a desire to somehow download all the changes of an atomic
group to a users working copy, I would think about adding this into
repo some how. Perhaps a new SSH command can be added to get a list
of all dependent changes.
I'm certainly not an expert on the Gerrit code base, so these
suggestions may not be the realistic or the best thing to do. But
hopefully they still provide some useful feedback.
Thanks,
Jesse
> > > > > No, AFAIK the plan didn't ever include switching fully over to using gitsubmodule, only to take advantage of the git data model provided bysubmodule(gitlinks).
> > > > > > to call 'gitsubmoduleupdate' and it's done. Has anybody tackled
Thanks for the quick response and the detailed work flow. I have some
comments in-line.
> 1) user makes changes and creates commits insubmodulea, b and c.
> 2) the user issues a "atomic" command in gerrit along with the sha1s
> of commit a', b' and c'.
> 3) user pushes
Can this be done after changes are uploaded? I could see there being
> of commit a', b' and c'.
> 3) user pushes
another entry box on the change page which would allow a change to be
linked to another one. Gerrit could check to make sure the change
you're linking to exists first to validate the dependency.
> 4) when gerrit sees that all 3 commits has come in a commit on the
> super project (s') is created on a review-branch (refs/changes/xyz)
> where the git links points to a', b' and c'
>
> There is not much point in reviewing a super-project commit, OTOH
> other users are probably interested in being able to download the full
> change and perhaps its best to create a proper change for the super-
> project as well.... what do you guys think? From a continous
> integration perspective this is a non-issue since the stream event
> will give the CI a way to get access to these super project commits
> anyway.
Yeah, I'm not sure I see much value in automatically creating super-
> other users are probably interested in being able to download the full
> change and perhaps its best to create a proper change for the super-
> project as well.... what do you guys think? From a continous
> integration perspective this is a non-issue since the stream event
> will give the CI a way to get access to these super project commits
> anyway.
project changes (especially compared to the possible complications).
Here are a few of issues I can think of off the top of my head:
- Several super projects tracking the submodules. This would result
in multiple super project changes for every creation of an atomic
group or every change to an atomic group.
- What happens if the super-project change is accidentally submitted
before all the submodule changes are in? The gitlink entries would
then point to non-existent commits (which is the typical problem
with git submodules)
- What happens if a super project starts tracking one of the
submodules that is currently part of an atomic group? Would Gerrit
need to create a change for the super project right away?
Likewise, what happens if a super-project stops tracking one of the
submodules involved in an atomic group.
- Once the submodules are merged in, the super-project change may
become out of date if something is merged in before it goes in.
Would the user be required to rebase the super project change? I
don't want to explain how to our users how to rebase a change with
gitlink updates :)
Instead, I would probably modify the submodule updating logic you're
working on in change 14033 to look for these atomic submissions. It
would need to be smart enough to ignore individual merges that are a
part of an atomic transaction. Once an atomic transaction is complete
it would go about automatically updating the super-project's gitlinks
to point to the current commit of all it's submodules that have a
"revision" specified.
If there is a desire to somehow download all the changes of an atomic
group to a users working copy, I would think about adding this into
repo some how. Perhaps a new SSH command can be added to get a list
of all dependent changes.
I'm certainly not an expert on the Gerrit code base, so these
suggestions may not be the realistic or the best thing to do. But
hopefully they still provide some useful feedback.
Thanks,
Jesse
>
> > > > > Nasser
>
> > > > > On Oct 1, 2010, at 3:40 PM, Matt Fischer wrote:
>
> > > > > > I'm not sure if that's what I was looking for. That change seems to
> > > > > > add an option to convert the manifest.xml format into a .gitmodules
> > > > > > file, but the process of checking out a repository (i.e. running 'repo
> > > > > > init' or 'repo sync') still runs through all the machinery of manually
> > > > > > checking out each repository into the proper place, etc. Under a
> > > > > >submodulesystem, repo shouldn't have to do any of that, it just needs
> > > > > Nasser
>
> > > > > On Oct 1, 2010, at 3:40 PM, Matt Fischer wrote:
>
> > > > > > I'm not sure if that's what I was looking for. That change seems to
> > > > > > add an option to convert the manifest.xml format into a .gitmodules
> > > > > > file, but the process of checking out a repository (i.e. running 'repo
> > > > > > init' or 'repo sync') still runs through all the machinery of manually
> > > > > > checking out each repository into the proper place, etc. Under a
> > > > > > to call 'gitsubmoduleupdate' and it's done. Has anybody tackled
> > > > > > that piece of it yet?
>
> > > > > > --Matt
> > > > > > On Oct 1, 4:00 pm, Nasser Grainawi <nas...@codeaurora.org> wrote:
> > > > > >> On 10/01/2010 01:41 PM, Matt Fischer wrote:
>
> > > > > >>> There is an ongoing effort to move repo to asubmodule-based tool,
>
> > > > > > --Matt
> > > > > > On Oct 1, 4:00 pm, Nasser Grainawi <nas...@codeaurora.org> wrote:
> > > > > >> On 10/01/2010 01:41 PM, Matt Fischer wrote:
>
> > > > > >>> instead of manual repository management. The manifest format is in
> > > > > >>> place, and the patch to addsubmodulesmarts to Gerrit (https://
> > > > > >>> review.source.android.com/#change,14033) seems to be nearly done.
> > > > > >>> What I haven't seen yet are any references to a new incarnation of the
> > > > > >>> repo tool itself to actually make use of all of this. Has any work
> > > > > >>> gone on yet to develop it? My company is planning on making use of
> > > > > >>> thesubmoduleframework, and I'm trying to determine whether I'll need
> > > > > >>> What I haven't seen yet are any references to a new incarnation of the
> > > > > >>> repo tool itself to actually make use of all of this. Has any work
> > > > > >>> gone on yet to develop it? My company is planning on making use of
Ulrik Sjölin
Oct 15, 2010, 12:28:50 AM10/15/10
to Repo and Gerrit Discussion
> Thanks for the quick response and the detailed work flow. I have some
> comments in-line.
>
> > 1) user makes changes and creates commits insubmodulea, b and c.
> > 2) the user issues a "atomic" command in gerrit along with the sha1s
> > of commit a', b' and c'.
> > 3) user pushes
>
> Can this be done after changes are uploaded? I could see there being
> another entry box on the change page which would allow a change to be
> linked to another one. Gerrit could check to make sure the change
> you're linking to exists first to validate the dependency.
>
> > 4) when gerrit sees that all 3 commits has come in a commit on the
> > super project (s') is created on a review-branch (refs/changes/xyz)
> > where the git links points to a', b' and c'
>
> > There is not much point in reviewing a super-project commit, OTOH
> > other users are probably interested in being able to download the full
> > change and perhaps its best to create a proper change for the super-
> > project as well.... what do you guys think? From a continous
> > integration perspective this is a non-issue since the stream event
> > will give the CI a way to get access to these super project commits
> > anyway.
>
> Yeah, I'm not sure I see much value in automatically creating super-
> project changes (especially compared to the possible complications).
> Here are a few of issues I can think of off the top of my head:
Creating the super project commits is definetly something that is up
> comments in-line.
>
> > 1) user makes changes and creates commits insubmodulea, b and c.
> > 2) the user issues a "atomic" command in gerrit along with the sha1s
> > of commit a', b' and c'.
> > 3) user pushes
>
> Can this be done after changes are uploaded? I could see there being
> another entry box on the change page which would allow a change to be
> linked to another one. Gerrit could check to make sure the change
> you're linking to exists first to validate the dependency.
>
> > 4) when gerrit sees that all 3 commits has come in a commit on the
> > super project (s') is created on a review-branch (refs/changes/xyz)
> > where the git links points to a', b' and c'
>
> > There is not much point in reviewing a super-project commit, OTOH
> > other users are probably interested in being able to download the full
> > change and perhaps its best to create a proper change for the super-
> > project as well.... what do you guys think? From a continous
> > integration perspective this is a non-issue since the stream event
> > will give the CI a way to get access to these super project commits
> > anyway.
>
> Yeah, I'm not sure I see much value in automatically creating super-
> project changes (especially compared to the possible complications).
> Here are a few of issues I can think of off the top of my head:
for discussion, the way I see it, the change for the user is rather
big if we force them to update/commit/upload the superproject for each
commit. Repo can be changed to help out, but you will still have to
review and submit the submodule + superproject change, many users
would probably not like this new behavior.
>
> - Several super projects tracking the submodules. This would result
> in multiple super project changes for every creation of an atomic
> group or every change to an atomic group.
> - What happens if the super-project change is accidentally submitted
> before all the submodule changes are in? The gitlink entries would
> then point to non-existent commits (which is the typical problem
> with git submodules)
set up in Gerrit between the superproject and the submodules.
> - What happens if a super project starts tracking one of the
> submodules that is currently part of an atomic group? Would Gerrit
> need to create a change for the super project right away?
> Likewise, what happens if a super-project stops tracking one of the
> submodules involved in an atomic group.
need to upload and submit that change by hand. When Gerrit sees a
commit with a filename like this Gerrit will setup/remove subscription
for the submodule changes (change 14033 handles the subscription
part). Basically every user of a superproject has to do a "git pull"
and "git submodule update" to get changes locally.
> - Once the submodules are merged in, the super-project change may
> become out of date if something is merged in before it goes in.
> Would the user be required to rebase the super project change? I
> don't want to explain how to our users how to rebase a change with
> gitlink updates :)
atomc commit and this will ensure that all changes + the super project
changes are merged in all at the same time (the submodules first and
the superproject last).
>
> Instead, I would probably modify the submodule updating logic you're
> working on in change 14033 to look for these atomic submissions. It
is there a clever way to group some of these changes together? Without
more information I would say no, but if anyone has an idea here this
would be most welcome. The current solution is a new gerrit command to
let the user specify before the upload "hey, Gerrit, I will upload SHA
x,y and z and they belong together".
> would need to be smart enough to ignore individual merges that are a
> part of an atomic transaction. Once an atomic transaction is complete
> it would go about automatically updating the super-project's gitlinks
> to point to the current commit of all it's submodules that have a
> "revision" specified.
server to achieve that?
What you are say is pretty much what we intend to do ;-)
>
> If there is a desire to somehow download all the changes of an atomic
> group to a users working copy, I would think about adding this into
> repo some how. Perhaps a new SSH command can be added to get a list
> of all dependent changes.
glossy nice web UI and I would think that there would be a few visits
to my desk about this.
>
> I'm certainly not an expert on the Gerrit code base, so these
> suggestions may not be the realistic or the best thing to do. But
> hopefully they still provide some useful feedback.
>
it.
Ulrik
Magnus Bäck
Oct 15, 2010, 1:52:10 AM10/15/10
to Repo and Gerrit Discussion
On Friday, October 15, 2010 at 06:28 CEST,
Ulrik Sj�lin <ulrik....@gmail.com> wrote:
Ulrik Sj�lin <ulrik....@gmail.com> wrote:
> On 14 Okt, 23:26, jesse greenwald <jesse.greenw...@gmail.com> wrote:
[...]
> > Instead, I would probably modify the submodule updating logic you're
> > working on in change 14033 to look for these atomic submissions. �It
>
> So the question is: based on the stream of commit coming into Gerrit
> is there a clever way to group some of these changes together? Without
> more information I would say no, but if anyone has an idea here this
> would be most welcome. The current solution is a new gerrit command to
> let the user specify before the upload "hey, Gerrit, I will upload SHA
> x,y and z and they belong together".
Could the recently added topic tag for changes (Repo's -t option) be
useful? One drawback could be that the topic namespace is (I guess)
global. People have poor imagination, so we'd have a bunch of unrelated
changes by different people having the same tag ("fix" or some other
generic string), emailing the support staff about why they can't submit
their changes. Still, adding separate tagging for dependencies might be
too similar in spirit, or...?
Hmm, actually not. Dependencies have a direction and are certainly not
always bidirectional. Reusing the tag feature would mean that all
changes with dependencies have bidirectional dependencies. Bad.
[...]
> > If there is a desire to somehow download all the changes of an
> > atomic group to a users working copy, I would think about adding
> > this into repo some how. �Perhaps a new SSH command can be added
> > to get a list of all dependent changes.
>
> Yes good point.... its just that we have made our users very used
> to a glossy nice web UI and I would think that there would be a
> few visits to my desk about this.
Oh, if it was just *your* desk that would get frequent visits :-)
On the other hand, this should be an additional option to the repo
download command. I'm pretty sure the vast majority of users obtain
their download commands through the glossy web interface so we could
basically change those commands in any way we please.
--
Magnus B�ck Opinions are my own and do not necessarily
SW Configuration Manager represent the ones of my employer, etc.
Sony Ericsson
jesse greenwald
Oct 15, 2010, 12:03:28 PM10/15/10
to Repo and Gerrit Discussion
explicitly reviewed. I'm assuming that in order to verify a change
the user will need to download it into some super-project that
contains it and test it out there. If the change depends on other
changes (because it is in an atomic group), repo could be updated to
help you pull down any dependent changes.
With Gerrit/Repo, I don't think there has ever been a requirement to
review a change in the context of all projects that are tracking it
(historically using repo manifests). I'm not sure that the
introduction of "atomically grouped" changes would necessarily imply
the creation of this requirement. If this requirement is introduced,
why would it not be extended to non-grouped changes as well? For
example, if I upload a single change to submodule a', shouldn't a
corresponding change be created on super-project s' automatically that
is part of the automatic group (and t' as well if it also tracks a'
and so on)?
What I'm getting at is that it may be worth treating (requirement to
support atomically grouped changes) and (requirement to verify any
change in all relevant super-projects) as two separate requirements.
> > - Several super projects tracking the submodules. This would result
> > in multiple super project changes for every creation of an atomic
> > group or every change to an atomic group.
> > - What happens if the super-project change is accidentally submitted
> > before all the submodule changes are in? The gitlink entries would
> > then point to non-existent commits (which is the typical problem
> > with git submodules)
>
> It cant be submitted in wrong order since there will be dependencies
> set up in Gerrit between the superproject and the submodules.
the submodule changes. It seems like that would take of the problem.
> > - What happens if a super project starts tracking one of the
> > submodules that is currently part of an atomic group? Would Gerrit
> > need to create a change for the super project right away?
> > Likewise, what happens if a super-project stops tracking one of the
> > submodules involved in an atomic group.
>
> Yes when the submodule file is changed in the superproject you will
> need to upload and submit that change by hand. When Gerrit sees a
> commit with a filename like this Gerrit will setup/remove subscription
> for the submodule changes (change 14033 handles the subscription
> part). Basically every user of a superproject has to do a "git pull"
> and "git submodule update" to get changes locally.
would changed be created or abandoned for s' corresponding to all the
atomic group changes that a' is currently participating in?
> > - Once the submodules are merged in, the super-project change may
> > become out of date if something is merged in before it goes in.
> > Would the user be required to rebase the super project change? I
> > don't want to explain how to our users how to rebase a change with
> > gitlink updates :)
>
> I dont want to do that either ;-) The superproject is part of the
> atomc commit and this will ensure that all changes + the super project
> changes are merged in all at the same time (the submodules first and
> the superproject last).
applies because some unrelated change has been made to the super-
project in the mean time? It seems that either the change to the
super-module project would need to be updated automatically anytime a
change occurred on the super-module project or the user will have to
manually do this.
> > Instead, I would probably modify the submodule updating logic you're
> > working on in change 14033 to look for these atomic submissions. It
>
> So the question is: based on the stream of commit coming into Gerrit
> is there a clever way to group some of these changes together? Without
> more information I would say no, but if anyone has an idea here this
> would be most welcome. The current solution is a new gerrit command to
> let the user specify before the upload "hey, Gerrit, I will upload SHA
> x,y and z and they belong together".
same time the change is created in Gerrit? If this isn't a
requirement, it should be fairly easy to change the UI to allow users
to enter dependencies between changes. If this is a requirement, what
use case is it trying to solve?
> > would need to be smart enough to ignore individual merges that are a
> > part of an atomic transaction. Once an atomic transaction is complete
> > it would go about automatically updating the super-project's gitlinks
> > to point to the current commit of all it's submodules that have a
> > "revision" specified.
>
> How would you update them?
When a change comes in, determine if everything else the change
depends on (or is in the same atomic group) has also been committed.
If not all dependencies have been submitted, don't update the
gitlinks. Otherwise, go ahead and update gitlinks accordingly. When
determining the dependencies, you would need to take into account what
the current super-project is actually tracking and filter accordingly.
> Dont you need a super project commit on the server to achieve that?
Two ways this could be done are 1) creating the super-project
change(s) automatically in advance and then creating new revisions on
those changes as necessary automatically. Or 2) Don't make an
explicit Gerrit change on the server. Instead, let the merge op do
this for you automatically as described above.
> What you are say is pretty much what we intend to do ;-)
> > If there is a desire to somehow download all the changes of an atomic
> > group to a users working copy, I would think about adding this into
> > repo some how. Perhaps a new SSH command can be added to get a list
> > of all dependent changes.
>
> Yes good point.... its just that we have made our users very used to a
> glossy nice web UI and I would think that there would be a few visits
> to my desk about this.
could be wrapped by Repo:
1) User runs a command like
"repo <checkout/pull/cherry-pick> tools/gerrit 14033/6"
2) Repo in turn runs something like
"ssh gerrit.server.com gerrit get-dependencies 14033/6".
3) Gerrit returns a list of all the dependencies
4) Gerrit then checks out/pulls/cherry-picks all the appropriate
changes.
It can look at the current .gitmodules file to filter the list of
changes and determine where the submodules are on disk.
> > I'm certainly not an expert on the Gerrit code base, so these
> > suggestions may not be the realistic or the best thing to do. But
> > hopefully they still provide some useful feedback.
>
> This is a big change and think it is very important that we discuss
> it.
Ryan Alberts
Oct 15, 2010, 6:30:49 PM10/15/10
to Repo and Gerrit Discussion
I know this doesn't add much to the discussion but I am really excited
about this feature of adding in submodule support! Keep up the good
work on creating such a valuable tool!
On Oct 15, 11:03 am, jesse greenwald <jesse.greenw...@gmail.com>
wrote:
about this feature of adding in submodule support! Keep up the good
work on creating such a valuable tool!
On Oct 15, 11:03 am, jesse greenwald <jesse.greenw...@gmail.com>
wrote:
Ulrik Sjölin
Oct 18, 2010, 2:11:16 PM10/18/10
to Repo and Gerrit Discussion
On Oct 15, 6:03 pm, jesse greenwald <jesse.greenw...@gmail.com> wrote:
and convenient way to give the user access to the block of changes. I
like the notion of the user downloading one change and get all that is
connected with that change without and fuzz, but in any case this
change will never be merged, a new change will be created on the
destination branch once the submodule changes has been merged.
>
> With Gerrit/Repo, I don't think there has ever been a requirement to
> review a change in the context of all projects that are tracking it
> (historically using repo manifests). I'm not sure that the
> introduction of "atomically grouped" changes would necessarily imply
> the creation of this requirement. If this requirement is introduced,
> why would it not be extended to non-grouped changes as well? For
> example, if I upload a single change to submodule a', shouldn't a
> corresponding change be created on super-project s' automatically that
> is part of the automatic group (and t' as well if it also tracks a'
> and so on)?
placed on the super project are "working" (what ever that means ;-)).
Basically we link our CI up to Gerrit and what I want to do is to
make Hudson listen to all changes made on the super project, build,
test and pass a verdict on those changes. It is in this respect simply
not good enough to upload the changes that are part of the atomic
commit first and then later atomize them. The reason is of course that
its will not work for us to have those extra commits on the super
project that may not be buildable/testable.
>
> What I'm getting at is that it may be worth treating (requirement to
> support atomically grouped changes) and (requirement to verify any
> change in all relevant super-projects) as two separate requirements.
>
be able to pick up changes across gits made us implement this patch in
the first place.
file, Gerrit will track or untrack as needed and update the gitlinks
accordingly.
>
> > > - Once the submodules are merged in, the super-project change may
> > > become out of date if something is merged in before it goes in.
> > > Would the user be required to rebase the super project change? I
> > > don't want to explain how to our users how to rebase a change with
> > > gitlink updates :)
>
> > I dont want to do that either ;-) The superproject is part of the
> > atomc commit and this will ensure that all changes + the super project
> > changes are merged in all at the same time (the submodules first and
> > the superproject last).
>
> How would the super-project change be updated if it no longer cleanly
> applies because some unrelated change has been made to the super-
> project in the mean time? It seems that either the change to the
> super-module project would need to be updated automatically anytime a
> change occurred on the super-module project or the user will have to
> manually do this.
>
gitlinkes, no merges. The user handles the content of the .gitmodules
file and conflicts as well ;-)
> > > Instead, I would probably modify the submodule updating logic you're
> > > working on in change 14033 to look for these atomic submissions. It
>
> > So the question is: based on the stream of commit coming into Gerrit
> > is there a clever way to group some of these changes together? Without
> > more information I would say no, but if anyone has an idea here this
> > would be most welcome. The current solution is a new gerrit command to
> > let the user specify before the upload "hey, Gerrit, I will upload SHA
> > x,y and z and they belong together".
>
> Is there a requirement that the dependencies are created at the exact
> same time the change is created in Gerrit? If this isn't a
> requirement, it should be fairly easy to change the UI to allow users
> to enter dependencies between changes. If this is a requirement, what
> use case is it trying to solve?
>
solve the use case of CI as described above.
> > > would need to be smart enough to ignore individual merges that are a
> > > part of an atomic transaction. Once an atomic transaction is complete
> > > it would go about automatically updating the super-project's gitlinks
> > > to point to the current commit of all it's submodules that have a
> > > "revision" specified.
>
> > How would you update them?
>
> In the submodule op, the following logic could be used.
>
> When a change comes in, determine if everything else the change
> depends on (or is in the same atomic group) has also been committed.
> If not all dependencies have been submitted, don't update the
> gitlinks. Otherwise, go ahead and update gitlinks accordingly. When
> determining the dependencies, you would need to take into account what
> the current super-project is actually tracking and filter accordingly.
>
> > Dont you need a super project commit on the server to achieve that?
>
> Yes, there does need to be some change to the super-module project.
> Two ways this could be done are 1) creating the super-project
> change(s) automatically in advance and then creating new revisions on
> those changes as necessary automatically. Or 2) Don't make an
> explicit Gerrit change on the server. Instead, let the merge op do
> this for you automatically as described above.
>
> > What you are say is pretty much what we intend to do ;-)
> > > If there is a desire to somehow download all the changes of an atomic
> > > group to a users working copy, I would think about adding this into
> > > repo some how. Perhaps a new SSH command can be added to get a list
> > > of all dependent changes.
>
> > Yes good point.... its just that we have made our users very used to a
> > glossy nice web UI and I would think that there would be a few visits
> > to my desk about this.
>
> The user wouldn't necessarily call the ssh-command directly, all that
> could be wrapped by Repo:
>
> 1) User runs a command like
> "repo <checkout/pull/cherry-pick> tools/gerrit 14033/6"
> 2) Repo in turn runs something like
> "ssh gerrit.server.com gerrit get-dependencies 14033/6".
> 3) Gerrit returns a list of all the dependencies
> 4) Gerrit then checks out/pulls/cherry-picks all the appropriate
> changes.
> It can look at the current .gitmodules file to filter the list of
> changes and determine where the submodules are on disk.
>
commit on refs/changes/x that can be accessed through the ordinary
repo download command.
> > > I'm certainly not an expert on the Gerrit code base, so these
> > > suggestions may not be the realistic or the best thing to do. But
> > > hopefully they still provide some useful feedback.
>
> > This is a big change and think it is very important that we discuss
> > it.
>
>
> - Show quoted text -
jesse greenwald
Oct 19, 2010, 11:40:15 AM10/19/10
to Repo and Gerrit Discussion
One thing that isn't clear to me is what the model is for these atomic
"groups". For example, are individual dependencies specified between
changes? If so, can dependencies be circular? Or, is a group some
how specified and everything in the group must be committed
atomically.
Below you mentioned that dependencies would be set up between the
super-project change and the submodule changes. Is that just a one
way dependency then? I'd be curious to know more about how
dependencies and groups will be set up. I.e., do you set up a bunch
of singular dependencies (that may be circular), or would you just
create a "group" of changes that must be committed together.
One other issue I think that you'll need to handle with super-project
changes is making sure to somehow pull down refs/changes/* in each
submodule project. Otherwise, you may not be able to checkout super-
project changes because of missing SHA1s.
When 'git submodule update' runs, it will go through each submodule
and run "git fetch". This should fetch the refspec
'remote.origin.fetch' from remote 'origin' (see .git/config). By
default, 'remote.origin.fetch' is set to '+refs/heads/*:refs/remotes/
origin/*'. However, changes are currently stored in 'refs/changes/*'
which won't be pulled down by the default ref spec.
You might get lucky if the repository on the server has been packed
and you pull down pack files that happen to have everything you need.
I'm not sure how often Gerrit does that, or if it does it at all.
Otherwise, you would probably want some type of tool support on the
client to actually pull down the commits pointed to by the gitlink
entries in the super-project change before trying to pull down the
super-parent change. At which point, you might just want the tool to
handle figuring out what changes it needs to download instead of
relying on the super-project change.
> > With Gerrit/Repo, I don't think there has ever been a requirement to
> > review a change in the context of all projects that are tracking it
> > (historically using repo manifests). I'm not sure that the
> > introduction of "atomically grouped" changes would necessarily imply
> > the creation of this requirement. If this requirement is introduced,
> > why would it not be extended to non-grouped changes as well? For
> > example, if I upload a single change to submodule a', shouldn't a
> > corresponding change be created on super-project s' automatically that
> > is part of the automatic group (and t' as well if it also tracks a'
> > and so on)?
>
> For us at Sony Ericsson it is requirement that the atomic commits
> placed on the super project are "working" (what ever that means ;-)).
> Basically we link our CI up to Gerrit and what I want to do is to
> make Hudson listen to all changes made on the super project, build,
> test and pass a verdict on those changes. It is in this respect simply
> not good enough to upload the changes that are part of the atomic
> commit first and then later atomize them. The reason is of course that
> its will not work for us to have those extra commits on the super
> project that may not be buildable/testable.
I understand the desire to group together submodule updates on the
super-project. It's definitely a requirement for the submodule
updates to all be part of one change that is merged in to the super
project.
When you say commits, are you referring to gerrit changes (i.e. refs/
changes/*) or changes that have been submitted (i.e. refs/heads/
master).
> > What I'm getting at is that it may be worth treating (requirement to
> > support atomically grouped changes) and (requirement to verify any
> > change in all relevant super-projects) as two separate requirements.
>
> I not sure I understand what you mean, for us it is the requirement to
> be able to pick up changes across gits made us implement this patch in
> the first place.
I think there are different ways this can be done as I described
below.
Ok. The description above of the user issuing an "atomic" command
sounded a bit clunky, but perhaps this can be prettied up in the web
interface. It would also be nice to have the flexibility to still add
dependencies after a change has been uploaded. Perhaps this can be
done later.
One option for specifying the dependencies in advance could be via the
commit message:
..
Change-Id: If22a4404e807182d202318d1536831223ba2364d
Depends: If1a1fdfead16e2f94d3b42c6421487cc77a00ab0
See my comments above about trying to download commits that are
referenced only by heads in /refs/changes/*.
I think using a tool would give you more flexibility in the way the
changes are pulled in to the working copy. For instance, right now in
the user interface I can choose to do a "checkout out", "pull", or
"cherry-pick". These have different semantics, namely they allow a
user to either pull down the exact commit pointed to by the change or
try to merge the change into the current revision.
Usually when verifying a change I want to actually merge the change
into the latest version to see how it will actually work with the
latest version. Just checking out the exact commit isn't a true
verification if other changes have happened on the branch since was
uploaded. I don't see how this could be done by downloading a super
project change the way that is described.
Also, if you had a tool you could do things like cherry-pick an atomic
group of changes that have already been committed from one branch to
another. For instance, if a fix has been made on "refs/heads/master"
and it affects multiple projects and I'd like to pull it into "refs/
heads/release". The tool could determine everything that the change
needs and pull it over. This is something we often do by hand.
"groups". For example, are individual dependencies specified between
changes? If so, can dependencies be circular? Or, is a group some
how specified and everything in the group must be committed
atomically.
super-project change and the submodule changes. Is that just a one
way dependency then? I'd be curious to know more about how
dependencies and groups will be set up. I.e., do you set up a bunch
of singular dependencies (that may be circular), or would you just
create a "group" of changes that must be committed together.
One other issue I think that you'll need to handle with super-project
changes is making sure to somehow pull down refs/changes/* in each
submodule project. Otherwise, you may not be able to checkout super-
project changes because of missing SHA1s.
When 'git submodule update' runs, it will go through each submodule
and run "git fetch". This should fetch the refspec
'remote.origin.fetch' from remote 'origin' (see .git/config). By
default, 'remote.origin.fetch' is set to '+refs/heads/*:refs/remotes/
origin/*'. However, changes are currently stored in 'refs/changes/*'
which won't be pulled down by the default ref spec.
You might get lucky if the repository on the server has been packed
and you pull down pack files that happen to have everything you need.
I'm not sure how often Gerrit does that, or if it does it at all.
Otherwise, you would probably want some type of tool support on the
client to actually pull down the commits pointed to by the gitlink
entries in the super-project change before trying to pull down the
super-parent change. At which point, you might just want the tool to
handle figuring out what changes it needs to download instead of
relying on the super-project change.
> > With Gerrit/Repo, I don't think there has ever been a requirement to
> > review a change in the context of all projects that are tracking it
> > (historically using repo manifests). I'm not sure that the
> > introduction of "atomically grouped" changes would necessarily imply
> > the creation of this requirement. If this requirement is introduced,
> > why would it not be extended to non-grouped changes as well? For
> > example, if I upload a single change to submodule a', shouldn't a
> > corresponding change be created on super-project s' automatically that
> > is part of the automatic group (and t' as well if it also tracks a'
> > and so on)?
>
> For us at Sony Ericsson it is requirement that the atomic commits
> placed on the super project are "working" (what ever that means ;-)).
> Basically we link our CI up to Gerrit and what I want to do is to
> make Hudson listen to all changes made on the super project, build,
> test and pass a verdict on those changes. It is in this respect simply
> not good enough to upload the changes that are part of the atomic
> commit first and then later atomize them. The reason is of course that
> its will not work for us to have those extra commits on the super
> project that may not be buildable/testable.
super-project. It's definitely a requirement for the submodule
updates to all be part of one change that is merged in to the super
project.
When you say commits, are you referring to gerrit changes (i.e. refs/
changes/*) or changes that have been submitted (i.e. refs/heads/
master).
> > What I'm getting at is that it may be worth treating (requirement to
> > support atomically grouped changes) and (requirement to verify any
> > change in all relevant super-projects) as two separate requirements.
>
> I not sure I understand what you mean, for us it is the requirement to
> be able to pick up changes across gits made us implement this patch in
> the first place.
below.
sounded a bit clunky, but perhaps this can be prettied up in the web
interface. It would also be nice to have the flexibility to still add
dependencies after a change has been uploaded. Perhaps this can be
done later.
One option for specifying the dependencies in advance could be via the
commit message:
..
Change-Id: If22a4404e807182d202318d1536831223ba2364d
Depends: If1a1fdfead16e2f94d3b42c6421487cc77a00ab0
referenced only by heads in /refs/changes/*.
I think using a tool would give you more flexibility in the way the
changes are pulled in to the working copy. For instance, right now in
the user interface I can choose to do a "checkout out", "pull", or
"cherry-pick". These have different semantics, namely they allow a
user to either pull down the exact commit pointed to by the change or
try to merge the change into the current revision.
Usually when verifying a change I want to actually merge the change
into the latest version to see how it will actually work with the
latest version. Just checking out the exact commit isn't a true
verification if other changes have happened on the branch since was
uploaded. I don't see how this could be done by downloading a super
project change the way that is described.
Also, if you had a tool you could do things like cherry-pick an atomic
group of changes that have already been committed from one branch to
another. For instance, if a fix has been made on "refs/heads/master"
and it affects multiple projects and I'd like to pull it into "refs/
heads/release". The tool could determine everything that the change
needs and pull it over. This is something we often do by hand.
jesse greenwald
Oct 20, 2010, 8:52:22 AM10/20/10
to Repo and Gerrit Discussion
On Oct 19, 10:40 am, jesse greenwald <jesse.greenw...@gmail.com>
wrote:
> I think using a tool would give you more flexibility in the way the
> changes are pulled in to the working copy. For instance, right now in
> the user interface I can choose to do a "checkout out", "pull", or
> "cherry-pick". These have different semantics, namely they allow a
> user to either pull down the exact commit pointed to by the change or
> try to merge the change into the current revision.
>
> Usually when verifying a change I want to actually merge the change
> into the latest version to see how it will actually work with the
> latest version. Just checking out the exact commit isn't a true
> verification if other changes have happened on the branch since was
> uploaded. I don't see how this could be done by downloading a super
> project change the way that is described.
It looks like newer versions of git allow a --merge or --rebase option
> changes are pulled in to the working copy. For instance, right now in
> the user interface I can choose to do a "checkout out", "pull", or
> "cherry-pick". These have different semantics, namely they allow a
> user to either pull down the exact commit pointed to by the change or
> try to merge the change into the current revision.
>
> Usually when verifying a change I want to actually merge the change
> into the latest version to see how it will actually work with the
> latest version. Just checking out the exact commit isn't a true
> verification if other changes have happened on the branch since was
> uploaded. I don't see how this could be done by downloading a super
> project change the way that is described.
to be specified with 'git submodule update'. Cherry pick checkouts
probably still wouldn't be supported, but I don't think that's deal
breaker. Otherwise, it sounds like the git submodule command already
handles the use case I described above pretty well.
Jesse
Reply all
Reply to author
Forward
0 new messages