[RFC] Extension of tracking ids

[Peter Jönsson P]

Hi!

Sorry for the wall-of-text, but here we go:

At $DAY_JOB we are now getting ready to roll out git and Gerrit, which is great! But now we are trying to figure out how to best keep track of what changes are included in our commits. For example, which bugs are fixed with the commit or for which feature the commit was pushed (e.g. IPv6 or multicore-support). This makes it easier for the regression test team to pinpoint which commit causes the particular regression found. They don't normally dig into the actual source code to find that out.

For our current version control system we have our own developed tool chain for this. Included in this tool-chain is a web based tool where you can associate your commit/delivery with features and/or bug-reports. So people are quite used to having this functionality and we also use it for creating automatic delivery reports before releasing our product(s).

We have looked at the Gerrit tracking-id support, but to use it properly one needs to 'force' the developers to use a particular syntax in commit messages. Also since it will be part of the commit message you need to resubmit the patch set if you did a trivial spelling mistake. In our current tool you can always edit the metadata about your commit/delivery after it has been published.

So we are now discussing if we should implement something similar to our existing setup in Gerrit. It could be similar to "labels" that is available in Atlassian Jira [1] which one could add to the change one has pushed for review. Base functionality could be free form text but regexp-based enforcement should of course be available. Another extension would be to have search-as-you-type towards the ticketing system of your choice.

Does this make sense to implement inside Gerrit, or is there some better way to do this?

// Peter Jönsson

[1] Atlassian Jira Labels Plugin
https://plugins.atlassian.com/plugins/com.atlassian.jira.plugin.labels

[Magnus Bäck]

On Wednesday, April 25, 2012 at 11:39 EDT,
Peter Jönsson P <peter.p...@ericsson.com> wrote:

[...]

> We have looked at the Gerrit tracking-id support, but to use it
> properly one needs to 'force' the developers to use a particular
> syntax in commit messages. Also since it will be part of the commit
> message you need to resubmit the patch set if you did a trivial
> spelling mistake. In our current tool you can always edit the metadata
> about your commit/delivery after it has been published.
>
> So we are now discussing if we should implement something similar to
> our existing setup in Gerrit. It could be similar to "labels" that is
> available in Atlassian Jira [1] which one could add to the change one
> has pushed for review. Base functionality could be free form text but
> regexp-based enforcement should of course be available. Another
> extension would be to have search-as-you-type towards the ticketing
> system of your choice.

This sounds very similar to the planned and partially implemented label
feature, see below. I don't know if there's a design document or similar
that explains the big picture, but I believe the idea is to allow more
or less arbitrary labels to be connected to changes, and to allow
queries of labels.

https://gerrit-review.googlesource.com/#/q/project:gerrit+topic:edit-labels-in-change-screen,n,z

(Most changes in this screen are displayed three times iff the
status:open search predicate is included. Known bug?)

It seems the current implementation still uses the database to store
the labels, but using the refs/notes namespace (or some other place in
Gitland) would seem like a more appropriate place to put them and more
in line with Gerrit's future direction.

One nice thing about keeping this information in the commit messages is
that trivial "git log" operations can be used to make queries. But yes,
the immutability is a problem.

Another problem -- regardless of how you store the information -- is
how to deal with corrections that span over multiple commits. How do
you know when the complete solution to a bug (or a new feature) is in?
And what if those commits are in different gits on different branches,
where these branches are used differently in different projects or on
different system-level branches of the same project? But solving that
is probably not something Gerrit can do anyway, and you might not have
that problem (but I know others that do).

--
Magnus Bäck
ba...@google.com

[Mihai Rusu]

"git notes" can be used to add mutable metadata to commits. Our in house
command line tool used to work with the Gerrit Git repositories also provides
commands to setup the local Git repository so the notes are being saved to
Gerrit (it's just a branch with certain files) and tells local "git" to lookup
commit notes (when displaying commit metadata with "git log", etc) using it.

--
Mihai Rusu

[Peter Jönsson P]

Luca Milanesio wrote:
> Hi Peter,
> here at GerritForge.com are exactly implementing this feature, integrated
> with Rational Team Concert and Jira.

Sounds great! What kind of integration to RTC/Jira do you have there? Search-as-you-type-lookup or running of a hook to update RTC/Jira after a change is updated with a label? How pluggable is the model? We might need to add support for our internal bug tracking system.

// peter

[...]

> Luca Milanesio
> Lu...@Milanesio.org <mailto:Lu...@Milanesio.org>
> Mobile: +44-(0)7928-617383
> Skype: lucamilanesio

[Peter Jönsson P]

Magnus Bäck wrote:
> On Wednesday, April 25, 2012 at 11:39 EDT,
> Peter Jönsson P <peter.p...@ericsson.com> wrote:
>
[...]
>

> This sounds very similar to the planned and partially implemented label
> feature, see below. I don't know if there's a design document or similar that
> explains the big picture, but I believe the idea is to allow more or less
> arbitrary labels to be connected to changes, and to allow queries of labels.
>
> https://gerrit-review.googlesource.com/#/q/project:gerrit+topic:edit-labels-in-change-screen,n,z
>

This looks great!

[...]

>
> It seems the current implementation still uses the database to store the
> labels, but using the refs/notes namespace (or some other place in
> Gitland) would seem like a more appropriate place to put them and more in
> line with Gerrit's future direction.

Git notes sounds like a better approach for this, I guess the topic is stalled because it would need to be rewritten to use git notes instead of extending the db-model with another table? Maybe Gustaf could clarify the current state of this? :)

>
> One nice thing about keeping this information in the commit messages is that
> trivial "git log" operations can be used to make queries. But yes, the
> immutability is a problem.

Yes, our developers are still free to include the information inside the commit message.
But for the delivery information we think this is a better solution for our needs.

>
> Another problem -- regardless of how you store the information -- is how to
> deal with corrections that span over multiple commits. How do you know when
> the complete solution to a bug (or a new feature) is in?
> And what if those commits are in different gits on different branches, where
> these branches are used differently in different projects or on different
> system-level branches of the same project? But solving that is probably not
> something Gerrit can do anyway, and you might not have that problem (but I
> know others that do).

Yes, we have this in our current system as well. Normally the developer would also close the bug report so some cross-referencing between the bug tracker and the source-code-content-track would need to done.

For fixing the same bug on different releases we would clone the bug into a new number for the different branches/releases. Then let the bug-tracker maintain the information that connects them together.

// Peter Jönsson

>
> --
> Magnus Bäck
> ba...@google.com

[Luca Milanesio]

Hi Peter,

here at GerritForge.com are exactly implementing this feature, integrated with Rational Team Concert and Jira.

For "blocking the commit" though we would need to extend the concept of "hook" in Gerrit to allow a "pre-hook" (we have the code for it, implemented and rolled out on GerritForge for TeamForge ... but need a clean-up and review).

@Shawn: I have a lot of things planned for the May hacking week ... not sure this can fit as well ... but what do you think in principle ?

Luca.

---

Luca Milanesio

Lu...@Milanesio.org
Mobile: +44-(0)7928-617383
Skype: lucamilanesio

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

[Peter Jönsson P]

Mihai Rusu wrote:
[...]

> "git notes" can be used to add mutable metadata to commits. Our in house
> command line tool used to work with the Gerrit Git repositories also provides
> commands to setup the local Git repository so the notes are being saved to
> Gerrit (it's just a branch with certain files) and tells local "git" to
> lookup commit notes (when displaying commit metadata with "git log", etc)
> using it.

Thanks for pointing out git notes, I had forgotten about that. Definitely sounds like the approach to take!

But you let the individual developer add the notes locally before pushing to Gerrit our how do you do it? If you need to re-work the patch set or rebase it, you have to move the note to the correct commit right? But perhaps your internal tool handles that?

// peter

[Magnus Bäck]

On Thursday, April 26, 2012 at 10:25 EDT,

Peter Jönsson P <peter.p...@ericsson.com> wrote:

[...]

> But you let the individual developer add the notes locally before
> pushing to Gerrit our how do you do it? If you need to re-work the
> patch set or rebase it, you have to move the note to the correct
> commit right? But perhaps your internal tool handles that?

Reasonably recent versions of Git (1.7.1+) does that automatically for
amend and rebase operations, although this can be disabled. Quoting
git-notes(1):

notes.rewrite.<command>
When rewriting commits with <command> (currently amend or
rebase), if this variable is false, git will not copy notes
from the original to the rewritten commit. Defaults to true.
See also "notes.rewriteRef" below.

This setting can be overridden by the GIT_NOTES_REWRITE_REF
environment variable.

Now, the notes feature is rather new and may has a few rough edges.
I looked into using it for similar things (annotating existing commits
wiht metadata) probably a year ago and I think I found a few snags that
made adoption a bit tricky. I don't recall what, though. Maybe just the
inability of "git log --grep" to search notes?

--
Magnus Bäck
ba...@google.com

[Mihai Rusu]

On Thursday, April 26, 2012 07:25:37 am Peter Jönsson P wrote:
> Mihai Rusu wrote:
> [...]
>
> > "git notes" can be used to add mutable metadata to commits. Our in house
> > command line tool used to work with the Gerrit Git repositories also
> > provides commands to setup the local Git repository so the notes are
> > being saved to Gerrit (it's just a branch with certain files) and tells
> > local "git" to lookup commit notes (when displaying commit metadata with
> > "git log", etc) using it.
>
> Thanks for pointing out git notes, I had forgotten about that. Definitely
> sounds like the approach to take!
>
> But you let the individual developer add the notes locally before pushing
> to Gerrit our how do you do it?

Yes, like everything else with Git, you perform changes locally and maybe push
them somewhere remotely. We have a set of command wrappers that
add/edit/remove notes by performing a full cycle of "git fetch <notes-ref> &&
git note <command> && git push <notes-ref>" so developers don't need to care
about pushing the notes (when using those commands). But the important bit is
that notes are mutable, if someone forgets to add them or adds them
incorrectly, they can always add/edit notes associated to existing (already
pushed for review to Gerrit) commits.

> If you need to re-work the patch set or
> rebase it, you have to move the note to the correct commit right? But
> perhaps your internal tool handles that?

Hmm I don't think we handle that so I guess somehow "git" does it (since I
don't think we've hit a problem with this). Note that although we have support
for mutable arbitrary commit metadata using "git notes" we don't have many
tools relying on this mechanism. I think only a sub-team is making heavy use
of it, most people don't. I was only suggesting it as something to look into.

--
Mihai Rusu

[Luca Milanesio]

We are basically working on the bi-directional linking of:

- WorkItems to ChangeIds (and viceversa)

- WorkItems to Git Commit Ids (and viceversa)

- Status transitions on WorkItems based on Gerrit actions

- Comments and Related Links in WorkItems

Integration is based on standard Gerrit hooks and we use the "commentLink" configuration regex.

The all way is designed is for maximum plugg-ability 

(we have Jira as well in our pipeline but then anybody could extend to other systems)

Luca.

---

Luca Milanesio

Lu...@Milanesio.org
Mobile: +44-(0)7928-617383
Skype: lucamilanesio

[Luca Milanesio]

We are basically working on the bi-directional linking of:

- WorkItems to ChangeIds (and viceversa)

- WorkItems to Git Commit Ids (and viceversa)

- Status transitions on WorkItems based on Gerrit actions

- Comments and Related Links in WorkItems

Integration is based on standard Gerrit hooks and we use the "commentLink" configuration regex.

The all way is designed is for maximum plugg-ability 

(we have Jira as well in our pipeline but then anybody could extend to other systems)

Luca.

---

Luca Milanesio

Lu...@Milanesio.org
Mobile: +44-(0)7928-617383
Skype: lucamilanesio

On 26 Apr 2012, at 16:15, Peter Jönsson P wrote:

[Rich Schmitt]

I think we are interested in doing something very similar.

Is there a reason why gerrit puts the change-id in the commit-msg rather then in the notes.  For that matter, the minor bugzilla integration also expects the bug-id to be noted in the commit-msg.  We do not want to preserve this process specific information when the patch is ultimately pulled upstream.   In order to preserve commit SHAs when upstreaming, we'd rather gerrit leverage this information in a git note.

I believe it is straight forward enough to do and we are prepared to take it on, unless of course there has been some work on this already.

If anyone has any code to share, Please do.

Thanks
Rich

[Alex Blewitt]

On 13 Jan 2013, at 15:59, Rich Schmitt <blue6...@gmail.com> wrote:

> Is there a reason why gerrit puts the change-id in the commit-msg rather then in the notes.

Gerrit does not mutate the commit message in any way, not is it necessary to do this to use gerrit. You can push changes to gerrit without a change-id. The change-id, if any, is applied by the client.

What you can't do without change-id labels is have multiple commits be treated as patch sets to an existing review. Instead, every commit pushed becomes its own separate review and there's no way to tie comments made on one patch set to another.

Alex

[Rich Schmitt]

Thanks.  I know territory doesn't change the commit msg.   Its just that I find the change-id function useful and wish gerrit was set up to look for it in a note. 

Rich

[Rich Schmitt]

My Android phone insists on changing gerrit to territory.  Sorry for the confusion.