atomic commits

[Wink Saville]

Any head way on supporting the ability to submit changes across multiple projects atomically?

[Marcin Cieslak]

On Tue, 9 Jun 2015, Wink Saville wrote:

> Any head way on supporting the ability to submit changes across multiple
> projects atomically?

How should what work in pure git terms?

~Marcin

[Bassem Rabil]

There was a change [1] to support this, but there was no progress on this change since November 2014. However it looks like there is another implementation on master branch for "change.submitWholeTopic" [2] which supports different projects.

[1] https://gerrit-review.googlesource.com/#/c/55243/

[2] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#change

[Wink Saville]

Thanks, the most recent changes are here for "submit whole topic", does anyone know the current status?

Are these ready to be used?

[Alex Blewitt]

Even if they were, there's no such thing as an atomic commit across repositories. 

Sent from my iPhat 6

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Wink Saville]

What's your definition of "repository"? My definition of an "atomic commit across projects" is the following:

Repo manages multiple git projects and if I create several gerrit commits for some of the git projects then the commits may only be submitted to their respective projects as a transaction. Thus the commits are either all submitted or none are submitted.

So my hope is that "submit whole topic" is in fact "atomic commits across projects".

[Alex Blewitt]

Your hope is incorrect. Each project is its own repository and therefore such submits are not in fact atomic. 

Sent from my iPhat 6

[Stefan Beller]

Hi,

I have been working on the submitWholeTopic changes. Currently this is
just emulating clicking submit for all the changes in that topic in a
fast sequential manner. (No atomicity garantuees! But slightly better
than clicking submit yourself for all changes in a topic.)

I am working on getting this more into Gerrit currently. This starts
with removing the merge queue and then eventually I want to have all
the changes of one topic being merged within one database transaction.
This is the Gerrit side, where a user would observe "either all
changes of the topic are new" or "all changes of the topic are
merged", so it should look like an atomic update.

The git side is a bit more complicated as Alex said. Because each is
its own Git repository, it is easy to observe a non-atomic update of a
change. (Say there is a topic spanning 2 repos and you pull in one
repo but not the other)

> So my hope is that "submit whole topic" is in fact "atomic commits across projects".

Too much hope :( It was just a bug fix where Gerrit would consider
"foo" and "foo-bar" the same topic when merging topic "foo".

The long term plan for us at Google is to use git submodules instead
of the repo tool, where you have one Git history in the superproject
and all changes of one topic are updated within one commit of the
superproject. (updating more than one submodule at a time, but all the
submodules involved in the topic.)

Stefan

[Wink Saville]

If/when Gerrit gets the atomic commit then I would hope git could be taught about topic's and "guarantee" that a "git pull/repo sync" would only pull complete transactions.

I ask because one of the things that is most irritating about gerrit/repo is when you sync large projects, such as Android, at times its very hard to get a good build because the change rate is so high. In addition presubmit queues and continuous testing are nearly impossible without atomic commits, so I look forward to a "proper" solution :)

BTW, I recently learned about a project that support subrepo's that look interesting.

[Stefan Beller]

On Tue, Jun 9, 2015 at 4:14 PM, Wink Saville <wi...@saville.com> wrote:
> If/when Gerrit gets the atomic commit then I would hope git could be taught
> about topic's and "guarantee" that a "git pull/repo sync" would only pull
> complete transactions.

I don't think Git will be taught about Gerrits topics as they are too specific.

>
> I ask because one of the things that is most irritating about gerrit/repo is
> when you sync large projects, such as Android, at times its very hard to get
> a good build because the change rate is so high. In addition presubmit
> queues and continuous testing are nearly impossible without atomic commits,
> so I look forward to a "proper" solution :)

Yes Android. We're aware of the problem you're hinting at. And we want a
proper solution too. So we plan to migrate off of repo and have a single
git superproject where all of Android is in using git submodules[1].

Then there would be this superproject, which tracks all the other git repos
(that's what repo currently does) but in an atomic fashion on the commit
level.

>
> BTW, I recently learned about a project that support subrepo's that look
> interesting.
>

Looking at a random update via subrepo[2], this looks like a conceptual
combination of submodules and subtrees of Git to me. So I did not get the
advantage yet. (Just looking briefly). Maybe they selling point is the UI?

>

[1] https://git-scm.com/book/en/v2/Git-Tools-Submodules
[2] https://github.com/ingydotnet/git-subrepo/commit/f23f1d1a2352f401cb0bd7e23b10726d8a06c96b

[Wink Saville]

I'm hopeful and waiting with baited breath for the solution, txs :)

[Luca Milanesio]

Any rough plan of when we think to have a design and solution for atomic commits? It was first mentioned more than a year ago but so far we've made any progress :-(

Shall we tackle and implement it at the next Hackathon?

Do we think the goal is achievable?
Maybe even just with pseudo transactions and compensation logic to start?

Something is better than nothing ;-)

Luca

Sent from my iPhone

[Martin Fick]

Atomic commits aren't super useful without git tools which
which can tie together fetches across repos (because even if
you commit atomically, the repos can be fetched at differnt
times). But once you have tools that can tie together
fetches across repos (submodule commits, or versioned repo
manifests), then atomic commits become more of a nice to
have idea, and are way less important than originally
believed. You can switch your processes today to use
submodules or versioned repo manifests, without waiting for
atomic commits, and you will likely have all of what you
actually need. If that is not the case, I would be curious
why,

-Martin

On Tuesday, June 09, 2015 11:14:52 PM Wink Saville wrote:
> If/when Gerrit gets the atomic commit then I would hope
> git could be taught about topic's and "guarantee" that a
> "git pull/repo sync" would only pull complete
> transactions.
>
> I ask because one of the things that is most irritating
> about gerrit/repo is when you sync large projects, such
> as Android, at times its very hard to get a good build
> because the change rate is so high. In addition presubmit
> queues and continuous testing are nearly impossible
> without atomic commits, so I look forward to a "proper"
> solution :)
>
> BTW, I recently learned about a project that support

> subrepo <https://github.com/ingydotnet/git-subrepo/>'s

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Wink Saville]

I come from the "normal" Android repo/gerrit model and I'm not familiar with using submodules or versioned repo manifests to solve the problems I've experienced in Android. Can you give me some links to documents/web pages that might be educational?

[Stefan Beller]

On Wed, Jun 10, 2015 at 11:00 AM, Wink Saville <wi...@saville.com> wrote:
> I come from the "normal" Android repo/gerrit model and I'm not familiar with
> using submodules or versioned repo manifests to solve the problems I've
> experienced in Android. Can you give me some links to documents/web pages
> that might be educational?

I was laying out dreams and plans for a submodules based Android,
so I cannot point to anything yet.

[Magnus Bäck]

On Wednesday, June 10, 2015 at 20:00 CEST,

Wink Saville <wi...@saville.com> wrote:

> I come from the "normal" Android repo/gerrit model and I'm not
> familiar with using submodules or versioned repo manifests to
> solve the problems I've experienced in Android. Can you give me
> some links to documents/web pages that might be educational?

With submodules there’s a file that locks down all participating gits
to SHA-1s. A commit that updates that file can change the SHA-1 of more
than one git, guaranteeing consistency when fetching source code. This
is equivalent to a Repo manifest that lists SHA-1s instead of a branch
name (the kind of manifest you can get with "repo manifest").

Of course, the SHA-1 updates (whether submodules or Repo manifests)
still need to be made in a single commit if you have multiple updates,
and the user interface for doing this isn’t obvious. We actually had a
submodule-based atomic commit prototype several years ago but the user
experience was just too awful. Maybe today’s topic model (which I don’t
particularly like) can be used.

--
Magnus Bäck | Software Engineer, Development Tools
magnu...@sonymobile.com | Sony Mobile Communications

[Phil Hord]

It is not important that the commits are applied atomically. It's more important to me that the merges are prevented ("pending") until all of them can be applied. It's also a nice feature if the gitlinks in related superprojects are updated all in one commit.

[Luca Milanesio]

Shouldn’t be difficult to implement, including the associated compensation logic.

In the shopping list for the Hackathon then?

Luca

[Stefan Beller]

I am working towards it actually,

I had some changes floating around, which would make the submodule
submission atomic per topic across submodules, but it would have made
the merge queue even more complicated, hence I am currently trying to
kill off the merge queue first. That makes the cross repo atomic
submission
later easier.

[Wink Saville]

Another feature needed is a "pre-submit queue" so sanity checks can be made so people dont sync to bad commits.

[Igor]

submitWholeTopic is a great feature, which is very much anticipated in our development. 

I just installed v2.11.1 and after adding submitWholeTopic=true I see the submit button changed to "Submit whole topic", however 2.11.1 documentation doesn't mention this setting.

Can somebody please clarify what's the status of this feature?

Thanks.

[Stefan Beller]

It's a feature which was not stable at the time of the 2.11 release,
so it should have been reverted in full on the 2.11 branch (including
documentation of course). However maybe some spurious code was left
in because the revert did not pick up all the changes (other features in
the meantime depended on some refactoring the submitWholeTopic did
iirc).

So I am not sure about the state of the submitWholeTopic in 2.11.
On master it is considered stable and we're using it for some time now
for Android development.

[David Pursehouse]

On 06/16/2015 05:52 AM, Stefan Beller wrote:
> It's a feature which was not stable at the time of the 2.11 release,
> so it should have been reverted in full on the 2.11 branch (including
> documentation of course). However maybe some spurious code was left
> in because the revert did not pick up all the changes (other features in
> the meantime depended on some refactoring the submitWholeTopic did
> iirc).
>

In [1] I reverted only the changes that had caused regressions.

What was left of the feature worked (as far as it was developed at the
time) but since it was still in development on master, I thought it best
to consider "unsupported" and therefore removed the documentation.

> So I am not sure about the state of the submitWholeTopic in 2.11.

I don't think it's ready for use on 2.11.x; what is there is quite
behind what is now on master.

I've uploaded a change [2] to disable it completely and will include
that in 2.11.2 unless there are objections.

> On master it is considered stable and we're using it for some time now
> for Android development.
>

And on gerrit-review of course. I'm pleased to say it's working very
well so far :)


[1] https://gerrit-review.googlesource.com/65121
[2] https://gerrit-review.googlesource.com/69010