[RFC] Limiting and simplifying refs advertising for changes

[Luca Milanesio]

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

2. I want to allow the people to do a simgle "git pull" and get the latest patch-set for a change

Example Scenarios:

Scenario1 - CI: I want to build all the active changes:

- configure +refs/active-changes/*:refs/remotes/review/*

- just pull and build all the branches

- keep them up-date by simply fetching them again

Scenario2 - Local dev: I want to work on change 5617

- add Gerrit as 'review' remote with the following ref-spec: +refs/active-changes/*:refs/remotes/review/*

- git checkout -b 5617 review/5617 (it will fetch the latest patch-set of change 5617)

- make some local modifications

- git commit -a --amend -m "My contribution"

- git push review

How does it sound?

Comments, suggestions are more than welcome :-)

Luca.

[Edwin Kempin]

On Thu, Aug 17, 2017 at 10:14 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Is this a use-case that many people have?

Normally it's one person working on a change. In this case you always have the latest change state in your repo, since as change author you pushed it.

I agree that sometimes you want to work on somebody else's change, but this is rather seldom.

I guess trying out somebody else's change happens more often, but normally you do this when you review the change and then you are already in the UI where you can copy the download command to the clipboard.

 

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

We once tried to add a magic ref that always points to the latest patch set of a change (something like refs/changes/XX/XXXX/latest), so that you can easily fetch the latest state of a change. However this was given up since it was too expensive to determine the latest patch set that is visible to you, e.g. the latest patch set could be a draft patch set that is not visible to you. This problem will be solved once draft patch sets are gone. So I guess after draft patch sets are gone it's a good idea to resume this idea. Not sure, but I think Changcheng or Patrick were looking into this back then.

 

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

2. I want to allow the people to do a simgle "git pull" and get the latest patch-set for a change

Example Scenarios:

Scenario1 - CI: I want to build all the active changes:

- configure +refs/active-changes/*:refs/remotes/review/*

- just pull and build all the branches

- keep them up-date by simply fetching them again

For CI this would be nice.

 

Scenario2 - Local dev: I want to work on change 5617

- add Gerrit as 'review' remote with the following ref-spec: +refs/active-changes/*:refs/remotes/review/*

- git checkout -b 5617 review/5617 (it will fetch the latest patch-set of change 5617)

- make some local modifications

- git commit -a --amend -m "My contribution"

- git push review

As said above I don't think that many users will benefit from this as it's rather rare that you update changes of other users.

 

How does it sound?

I like the idea of reattempting to implement refs/changes/XX/XXXX/latest after draft patch sets are gone, but I'm not sure that many users would benefit from the 'active-changes' ref filter. It's yet another concept and each new concept adds a bit of complexity so i'm not sure it's worth doing.

 

Comments, suggestions are more than welcome :-)

Luca.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luca Milanesio]

Hi Edwin, thanks for your feedback.

See below my replies.

On 17 Aug 2017, at 10:18, Edwin Kempin <eke...@google.com> wrote:

On Thu, Aug 17, 2017 at 10:14 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Is this a use-case that many people have?

Normally it's one person working on a change. In this case you always have the latest change state in your repo, since as change author you pushed it.

I agree that sometimes you want to work on somebody else's change, but this is rather seldom.

Actually happens quite frequently that more than one people is working on the same change ... is this called pair-programming in XP ? ;-)

The "pair" can be remote and they may interleave themselves in the patch-sets.

I do a lot on the analytics plugin with Claudio.

I guess trying out somebody else's change happens more often, but normally you do this when you review the change and then you are already in the UI where you can copy the download command to the clipboard.

Yes, but people then end up in a "detached head" state and most novice users are quite "scared" about detached state in Git ;-)

Of course, experienced Git and Gerrit users have no issues with the current Download mechanism.

 

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

We once tried to add a magic ref that always points to the latest patch set of a change (something like refs/changes/XX/XXXX/latest), so that you can easily fetch the latest state of a change. However this was given up since it was too expensive to determine the latest patch set that is visible to you, e.g. the latest patch set could be a draft patch set that is not visible to you. This problem will be solved once draft patch sets are gone. So I guess after draft patch sets are gone it's a good idea to resume this idea. Not sure, but I think Changcheng or Patrick were looking into this back then.

If you recall .. draft patch-sets in a non-draft change had many many other issues :-(

I am so happy that drafts are going away :-)

And yes, will be worth revisiting it and making that happen :-)

 

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

2. I want to allow the people to do a simgle "git pull" and get the latest patch-set for a change

Example Scenarios:

Scenario1 - CI: I want to build all the active changes:

- configure +refs/active-changes/*:refs/remotes/review/*

- just pull and build all the branches

- keep them up-date by simply fetching them again

For CI this would be nice.

Currently the integration between Gerrit and the CI systems is *so complicated* because of the ref structure of the changes/CC/CCCCC/PP structure.

By including a simplified refspec, with the ability to have a "constant" ref name for a change, will make trivial the integration with a lot of CI systems.

 

Scenario2 - Local dev: I want to work on change 5617

- add Gerrit as 'review' remote with the following ref-spec: +refs/active-changes/*:refs/remotes/review/*

- git checkout -b 5617 review/5617 (it will fetch the latest patch-set of change 5617)

- make some local modifications

- git commit -a --amend -m "My contribution"

- git push review

As said above I don't think that many users will benefit from this as it's rather rare that you update changes of other users.

Experienced users? no, they won't use it.

Novice users? maybe yes?

Gerrit workflow is *way simpler* than GitHub, GitLab or Stash workflows ... BUT the download + amend + push required locally is scaring a lot of novice users that are just giving up.

I bet that:

- with a simplified local Git workflow (as above)

- with PolyGerrit UX

... the Gerrit adoption will definitely take off and a lot more people will start using it :-)

 

How does it sound?

I like the idea of reattempting to implement refs/changes/XX/XXXX/latest after draft patch sets are gone, but I'm not sure that many users would benefit from the 'active-changes' ref filter. It's yet another concept and each new concept adds a bit of complexity so i'm not sure it's worth doing.

Yes, we could start by breaking down the concept in separate changes:

C1: allow fetching from /XX/XXXX/latest magic ref

C2: refs/active-changes/* for CI systems to fetch only latest active changes

C3: enable push to XX/XXXX/latest magic ref 

C1 and C2 are independent

C3 depends on C1

[Edwin Kempin]

On Thu, Aug 17, 2017 at 12:02 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi Edwin, thanks for your feedback.

See below my replies.

On 17 Aug 2017, at 10:18, Edwin Kempin <eke...@google.com> wrote:

On Thu, Aug 17, 2017 at 10:14 AM, Luca Milanesio <luca.milanesio@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Is this a use-case that many people have?

Normally it's one person working on a change. In this case you always have the latest change state in your repo, since as change author you pushed it.

I agree that sometimes you want to work on somebody else's change, but this is rather seldom.

Actually happens quite frequently that more than one people is working on the same change ... is this called pair-programming in XP ? ;-)

The "pair" can be remote and they may interleave themselves in the patch-sets.

Yes, I'm aware of such a workflow, but I would claim that this is rather for experienced users, since you have to take good care to agree on who is pushing next so that you don't overwrite each others changes. 

 

I do a lot on the analytics plugin with Claudio.

I guess trying out somebody else's change happens more often, but normally you do this when you review the change and then you are already in the UI where you can copy the download command to the clipboard.

Yes, but people then end up in a "detached head" state and most novice users are quite "scared" about detached state in Git ;-)

Of course, experienced Git and Gerrit users have no issues with the current Download mechanism.

 

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

We once tried to add a magic ref that always points to the latest patch set of a change (something like refs/changes/XX/XXXX/latest), so that you can easily fetch the latest state of a change. However this was given up since it was too expensive to determine the latest patch set that is visible to you, e.g. the latest patch set could be a draft patch set that is not visible to you. This problem will be solved once draft patch sets are gone. So I guess after draft patch sets are gone it's a good idea to resume this idea. Not sure, but I think Changcheng or Patrick were looking into this back then.

If you recall .. draft patch-sets in a non-draft change had many many other issues :-(

I am so happy that drafts are going away :-)

And yes, will be worth revisiting it and making that happen :-)

 

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

2. I want to allow the people to do a simgle "git pull" and get the latest patch-set for a change

Example Scenarios:

Scenario1 - CI: I want to build all the active changes:

- configure +refs/active-changes/*:refs/remotes/review/*

- just pull and build all the branches

- keep them up-date by simply fetching them again

For CI this would be nice.

Currently the integration between Gerrit and the CI systems is *so complicated* because of the ref structure of the changes/CC/CCCCC/PP structure.

By including a simplified refspec, with the ability to have a "constant" ref name for a change, will make trivial the integration with a lot of CI systems.

Isn't it enough if we have support for /XX/XXXX/latest?

 

 

Scenario2 - Local dev: I want to work on change 5617

- add Gerrit as 'review' remote with the following ref-spec: +refs/active-changes/*:refs/remotes/review/*

- git checkout -b 5617 review/5617 (it will fetch the latest patch-set of change 5617)

- make some local modifications

- git commit -a --amend -m "My contribution"

- git push review

As said above I don't think that many users will benefit from this as it's rather rare that you update changes of other users.

Experienced users? no, they won't use it.

Novice users? maybe yes?

Gerrit workflow is *way simpler* than GitHub, GitLab or Stash workflows ... BUT the download + amend + push required locally is scaring a lot of novice users that are just giving up.

I bet that:

- with a simplified local Git workflow (as above)

- with PolyGerrit UX

... the Gerrit adoption will definitely take off and a lot more people will start using it :-)

 

How does it sound?

I like the idea of reattempting to implement refs/changes/XX/XXXX/latest after draft patch sets are gone, but I'm not sure that many users would benefit from the 'active-changes' ref filter. It's yet another concept and each new concept adds a bit of complexity so i'm not sure it's worth doing.

Yes, we could start by breaking down the concept in separate changes:

C1: allow fetching from /XX/XXXX/latest magic ref

C2: refs/active-changes/* for CI systems to fetch only latest active changes

C3: enable push to XX/XXXX/latest magic ref 

SGMT, but still not convinced that C2 is crucial.

 

C1 and C2 are independent

C3 depends on C1

Comments, suggestions are more than welcome :-)

Luca.

-- 
-- 
To unsubscribe, email repo-discuss+unsub...@googlegroups.com

[Luca Milanesio]

On 17 Aug 2017, at 12:09, Edwin Kempin <eke...@google.com> wrote:

On Thu, Aug 17, 2017 at 12:02 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi Edwin, thanks for your feedback.

See below my replies.

On 17 Aug 2017, at 10:18, Edwin Kempin <eke...@google.com> wrote:

On Thu, Aug 17, 2017 at 10:14 AM, Luca Milanesio <luca.milanesio@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Is this a use-case that many people have?

Normally it's one person working on a change. In this case you always have the latest change state in your repo, since as change author you pushed it.

I agree that sometimes you want to work on somebody else's change, but this is rather seldom.

Actually happens quite frequently that more than one people is working on the same change ... is this called pair-programming in XP ? ;-)

The "pair" can be remote and they may interleave themselves in the patch-sets.

Yes, I'm aware of such a workflow, but I would claim that this is rather for experienced users, since you have to take good care to agree on who is pushing next so that you don't overwrite each others changes. 

True, but people coming from GitFlow are used to it and having a "surrogate" of that workflow in Gerrit would ease the transition IMHO.

But that's just my opinion, I may be completely wrong :-(

 

I do a lot on the analytics plugin with Claudio.

I guess trying out somebody else's change happens more often, but normally you do this when you review the change and then you are already in the UI where you can copy the download command to the clipboard.

Yes, but people then end up in a "detached head" state and most novice users are quite "scared" about detached state in Git ;-)

Of course, experienced Git and Gerrit users have no issues with the current Download mechanism.

 

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

We once tried to add a magic ref that always points to the latest patch set of a change (something like refs/changes/XX/XXXX/latest), so that you can easily fetch the latest state of a change. However this was given up since it was too expensive to determine the latest patch set that is visible to you, e.g. the latest patch set could be a draft patch set that is not visible to you. This problem will be solved once draft patch sets are gone. So I guess after draft patch sets are gone it's a good idea to resume this idea. Not sure, but I think Changcheng or Patrick were looking into this back then.

If you recall .. draft patch-sets in a non-draft change had many many other issues :-(

I am so happy that drafts are going away :-)

And yes, will be worth revisiting it and making that happen :-)

 

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

2. I want to allow the people to do a simgle "git pull" and get the latest patch-set for a change

Example Scenarios:

Scenario1 - CI: I want to build all the active changes:

- configure +refs/active-changes/*:refs/remotes/review/*

- just pull and build all the branches

- keep them up-date by simply fetching them again

For CI this would be nice.

Currently the integration between Gerrit and the CI systems is *so complicated* because of the ref structure of the changes/CC/CCCCC/PP structure.

By including a simplified refspec, with the ability to have a "constant" ref name for a change, will make trivial the integration with a lot of CI systems.

Isn't it enough if we have support for /XX/XXXX/latest?

Yes, but we need to make sure as well that closed or obsolete changes aren't going to pollute the 'ls-remote' listing, otherwise the CI system would end up building what doesn't make sense anymore.

That's the reason why I proposed a separate ref-spec (refs/active-changes).

Alternatively it could be implemented as a plugin (thanks to Shawn's permission backend) that hides the non-active changes in the refs listing.

That would work as well because it is only for CI users that you want to define such behaviour.

 

 

Scenario2 - Local dev: I want to work on change 5617

- add Gerrit as 'review' remote with the following ref-spec: +refs/active-changes/*:refs/remotes/review/*

- git checkout -b 5617 review/5617 (it will fetch the latest patch-set of change 5617)

- make some local modifications

- git commit -a --amend -m "My contribution"

- git push review

As said above I don't think that many users will benefit from this as it's rather rare that you update changes of other users.

Experienced users? no, they won't use it.

Novice users? maybe yes?

Gerrit workflow is *way simpler* than GitHub, GitLab or Stash workflows ... BUT the download + amend + push required locally is scaring a lot of novice users that are just giving up.

I bet that:

- with a simplified local Git workflow (as above)

- with PolyGerrit UX

... the Gerrit adoption will definitely take off and a lot more people will start using it :-)

 

How does it sound?

I like the idea of reattempting to implement refs/changes/XX/XXXX/latest after draft patch sets are gone, but I'm not sure that many users would benefit from the 'active-changes' ref filter. It's yet another concept and each new concept adds a bit of complexity so i'm not sure it's worth doing.

Yes, we could start by breaking down the concept in separate changes:

C1: allow fetching from /XX/XXXX/latest magic ref

C2: refs/active-changes/* for CI systems to fetch only latest active changes

C3: enable push to XX/XXXX/latest magic ref 

SGMT, but still not convinced that C2 is crucial.

It could be implemented as a plugin as I mentioned before.

[Edwin Kempin]

Cool, I would prefer it to be a plugin :)

If we later see that everyone wants it we can still consider implementing it in core.

 

 

C1 and C2 are independent

C3 depends on C1

Comments, suggestions are more than welcome :-)

Luca.

-- 
-- 
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

[Luca Milanesio]

True, it is not worth including into core, with all the extra documentation effort, training, adoption, coding and testing associated.

Whatever can be done in a plugin, it should be moved outside of core :-)

[Dave Borowitz]

On Thu, Aug 17, 2017 at 4:14 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

FWIW, this is a huge issue for us; we have several repos getting towards 1M refs. So, we are planning on adding a single config option to hide *all* refs from advertisement so we can use it on gerrit-review. You can still fetch by SHA-1, and the download-commands plugin will reflect this.

This obviously won't improve the workflow of a single "git pull", but it will cut down on ref advertisements for sure.

 

2. I want to allow the people to do a simgle "git pull" and get the latest patch-set for a change

Example Scenarios:

Scenario1 - CI: I want to build all the active changes:

- configure +refs/active-changes/*:refs/remotes/review/*

- just pull and build all the branches

- keep them up-date by simply fetching them again

Scenario2 - Local dev: I want to work on change 5617

- add Gerrit as 'review' remote with the following ref-spec: +refs/active-changes/*:refs/remotes/review/*

- git checkout -b 5617 review/5617 (it will fetch the latest patch-set of change 5617)

- make some local modifications

- git commit -a --amend -m "My contribution"

- git push review

How does it sound?

Comments, suggestions are more than welcome :-)

Luca.

[Luca Milanesio]

On 17 Aug 2017, at 15:09, Dave Borowitz <dbor...@google.com> wrote:

On Thu, Aug 17, 2017 at 4:14 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

FWIW, this is a huge issue for us; we have several repos getting towards 1M refs. So, we are planning on adding a single config option to hide *all* refs from advertisement so we can use it on gerrit-review. You can still fetch by SHA-1, and the download-commands plugin will reflect this.

We have clients in a similar situation, where replication to slaves become a real issue.

90% of the times slaves are there for CI jobs, and you only want to build changes that are "recent" (let's say, the last month) and "live" (no closed or abandoned changes).

With those conditions, our clients could have a significant improvement in the reduction of the advertised refs to the slaves and consequent speedup of their CI builds as well.

This obviously won't improve the workflow of a single "git pull", but it will cut down on ref advertisements for sure.

Yep.

So possibly a "core-plugin" idea? Or as you said makes sense in the kernel as well?

[Dave Borowitz]

On Thu, Aug 17, 2017 at 10:21 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 17 Aug 2017, at 15:09, Dave Borowitz <dbor...@google.com> wrote:

On Thu, Aug 17, 2017 at 4:14 AM, Luca Milanesio <luca.milanesio@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

FWIW, this is a huge issue for us; we have several repos getting towards 1M refs. So, we are planning on adding a single config option to hide *all* refs from advertisement so we can use it on gerrit-review. You can still fetch by SHA-1, and the download-commands plugin will reflect this.

We have clients in a similar situation, where replication to slaves become a real issue.

90% of the times slaves are there for CI jobs, and you only want to build changes that are "recent" (let's say, the last month) and "live" (no closed or abandoned changes).

Could your CI slaves do this without modifying the wire protocol? Use the search API to search for changes matching whatever definition of "interesting" you want. Search results include the SHA-1 of the current revision. Stitch those together into a single, long "git fetch" command line.

ISTM improving the default behavior of "git pull" is something that applies to humans, not to bots.

 

With those conditions, our clients could have a significant improvement in the reduction of the advertised refs to the slaves and consequent speedup of their CI builds as well.

This obviously won't improve the workflow of a single "git pull", but it will cut down on ref advertisements for sure.

Yep.

So possibly a "core-plugin" idea? Or as you said makes sense in the kernel as well?

Our proposal, omitting refs/changes/ entirely, is likely to go in core.

I think adding additional things to the ref advertisement is actually already doable with ReceivePackInitializers. Other people may have different feelings, but my gut says it'd be hard to come up with a one-size-fits-all approach. Although maybe Edwin's idea of a /latest ref is broadly applicable enough *shrug*.

 

 

2. I want to allow the people to do a simgle "git pull" and get the latest patch-set for a change

Example Scenarios:

Scenario1 - CI: I want to build all the active changes:

- configure +refs/active-changes/*:refs/remotes/review/*

- just pull and build all the branches

- keep them up-date by simply fetching them again

Scenario2 - Local dev: I want to work on change 5617

- add Gerrit as 'review' remote with the following ref-spec: +refs/active-changes/*:refs/remotes/review/*

- git checkout -b 5617 review/5617 (it will fetch the latest patch-set of change 5617)

- make some local modifications

- git commit -a --amend -m "My contribution"

- git push review

How does it sound?

Comments, suggestions are more than welcome :-)

Luca.

-- 
-- 
To unsubscribe, email repo-discuss+unsub...@googlegroups.com

[Luca Milanesio]

On 17 Aug 2017, at 15:28, Dave Borowitz <dbor...@google.com> wrote:

On Thu, Aug 17, 2017 at 10:21 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 17 Aug 2017, at 15:09, Dave Borowitz <dbor...@google.com> wrote:

On Thu, Aug 17, 2017 at 4:14 AM, Luca Milanesio <luca.milanesio@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

FWIW, this is a huge issue for us; we have several repos getting towards 1M refs. So, we are planning on adding a single config option to hide *all* refs from advertisement so we can use it on gerrit-review. You can still fetch by SHA-1, and the download-commands plugin will reflect this.

We have clients in a similar situation, where replication to slaves become a real issue.

90% of the times slaves are there for CI jobs, and you only want to build changes that are "recent" (let's say, the last month) and "live" (no closed or abandoned changes).

Could your CI slaves do this without modifying the wire protocol?

Yes, but the problem is that CI are reading from Gerrit slaves that are typically in the same local network or region.

The problem is replicating *only recent stuff* to the Gerrit slaves and optimising the ref advertising phase.

The Gerrit slaves would then keep only the "freshest" set of refs, avoiding the overload of having tens of thousands of refs to check, most of them 100% identical all the time because they refer to outdated or closed changes.

Use the search API to search for changes matching whatever definition of "interesting" you want. Search results include the SHA-1 of the current revision. Stitch those together into a single, long "git fetch" command line.

Yep, they do ... but again only against the Gerrit Slaves :-(

The master to slave delay is still there, because of the huge amounts of advertised refs.

ISTM improving the default behavior of "git pull" is something that applies to humans, not to bots.

 

With those conditions, our clients could have a significant improvement in the reduction of the advertised refs to the slaves and consequent speedup of their CI builds as well.

This obviously won't improve the workflow of a single "git pull", but it will cut down on ref advertisements for sure.

Yep.

So possibly a "core-plugin" idea? Or as you said makes sense in the kernel as well?

Our proposal, omitting refs/changes/ entirely, is likely to go in core.

Cool, could we then define an extension point for plugins to alter that behaviour?

Alternatively I could just implement an authentication backend, which could be slow because it works at permission level :-(

I think adding additional things to the ref advertisement is actually already doable with ReceivePackInitializers.

Interesting, let me dig into it :-) Thanks for the info.

Other people may have different feelings, but my gut says it'd be hard to come up with a one-size-fits-all approach. Although maybe Edwin's idea of a /latest ref is broadly applicable enough *shrug*.

The changes/NN/NNNN/latest is going to be very popular I believe :-)

[Martin Fick]

On Thursday, August 17, 2017 11:18:35 AM 'Edwin Kempin' via

Repo and Gerrit Discussion wrote:
> On Thu, Aug 17, 2017 at 10:14 AM, Luca Milanesio
> <luca.mi...@gmail.com>
> wrote:

> > *What:*

> > Introduce a new 'active-changes' for filtering out
> > closed or obsolete changes (merged, abandoned, older
> > than N days) from the "ls-remote" command and, for the
> > active changes that always point to the latest
> > patch-set.
> We once tried to add a magic ref that always points to the
> latest patch set of a change (something like
> refs/changes/XX/XXXX/latest), so that you can easily
> fetch the latest state of a change. However this was
> given up since it was too expensive to determine the
> latest patch set that is visible to you, e.g. the latest
> patch set could be a draft patch set that is not visible
> to you. This problem will be solved once draft patch sets
> are gone. So I guess after draft patch sets are gone it's
> a good idea to resume this idea. Not sure, but I think
> Changcheng or Patrick were looking into this back then.

One issue with a magic ref pointing to the latest patchset
that will likely make it non usable for enterprise users is
replication. A changing ref will be outof date on slaves
for a while, so users will likely not benefit much from this
if they operate across the globe.

-Martin


--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Luca Milanesio]

How is that different from a ref-update on a regular branch?
When master replicates to his slaves, every new or updated refs gets replicated in the same push, isn't it?
But true, it will ad "one extra ref to update" for every "live change".

[Martin Fick]

On Thursday, August 17, 2017 04:04:33 PM Luca Milanesio
wrote:

It's different because users expect a branch to change. If
you are providing a magic ref that is supposed to be the
latest, but it isn't much of the time, then you aren't
really helping users, and they are likely to just prefer
sticking to the immutable explicit ref which includes the
patchset (I would),

[Luca Milanesio]

Yes, but we were talking about Gerrit slaves, not people.
You said "A changing ref will be out of date on slaves for a while" and this is true, due to replication delays.
But how is that different from a replication delay in pushing a new ref?

> If you are providing a magic ref that is supposed to be the
> latest, but it isn't much of the time, then you aren't
> really helping users, and they are likely to just prefer
> sticking to the immutable explicit ref which includes the
> patchset (I would),

As a Gerrit experienced user, yes. As a novice user, I don't know ... possibly having something "similar" to a feature branch could be useful.
But again, I have don't have a crystal ball :-)

[Martin Fick]

On Thursday, August 17, 2017 04:21:18 PM Luca Milanesio

Slaves across the globe are used by people to fetch their
code. Most of our off-site users have instead-of rules
installed into their git-configs to point them to local
replicas of the master for fetches. Thus they generally are
fetching from a slave. I suspect this is similar to how
most large corps do it internally. Again, users expect
heads to be delayed, they won't expect a ref named "latest"
to be delayed. And if they understand that "latest" may not
be the latest, then why would they use it as a ref instead
of the explicit ref they want?

> > If you are providing a magic ref that is supposed to be
> > the>
> > latest, but it isn't much of the time, then you aren't
> > really helping users, and they are likely to just prefer
> > sticking to the immutable explicit ref which includes
> > the
> > patchset (I would),
>
> As a Gerrit experienced user, yes. As a novice user, I
> don't know ...
> possibly having something "similar" to a
> feature branch could be useful. But again, I have don't
> have a crystal ball :-)

I think the imroved workflows would be valuable. My nit is
really just in the naming of such a ref. Maybe you can
integrate it and make it look more like a feature branch.
Perhaps then it shouldn't really use change numbers either?
Especially since a feature branch likely inlcudes more than
one change? Perhaps you are looking for something that
allows topics to be fetched and pushed? Maybe you can come
up with a magic ref for topics that has some well defined
rules that maps a topic to a change series?

[Luca Milanesio]

Gotcha. However, in case of a slave delayed (let's say by 30'), the latest won't be available anyway because the new patch-set ref isn't there.

Let's say the master is:

p1

p2

p3 <- latest

and the slave is:

p1

p2 <- latest

Then p3 isn't on the slave anyway, and people can't get it from the master either.

Having said that ... the probability that two people on the other side of the planet will be pushing the same patch-set to the same change at the same time ... would be a bit unlikely ;-)

But agreed, in large-scale installations for enterprise-users (like you guys), the "novice user use-case" isn't really applicable after all.

My proposal was more for small-scale early adopters that need to get up to speed from a typical gitflow scenario, where features are in feature branches rather than in immutable patch-set refs.

 And if they understand that "latest" may not
be the latest, then why would they use it as a ref instead 
of the explicit ref they want?

If you are providing a magic ref that is supposed to be
the> 
latest, but it isn't much of the time, then you aren't
really helping users, and they are likely to just prefer
sticking to the immutable explicit ref which includes
the
patchset (I would),

As a Gerrit experienced user, yes. As a novice user, I
don't know ... 
possibly having something "similar" to a
feature branch could be useful. But again, I have don't
have a crystal ball :-)

I think the imroved workflows would be valuable.  My nit is 
really just in the naming of such a ref.  Maybe you can 
integrate it and make it look more like a feature branch.  
Perhaps then it shouldn't really use change numbers either?  
Especially since a feature branch likely inlcudes more than 
one change?  Perhaps you are looking for something that 
allows topics to be fetched and pushed?  Maybe you can come 
up with a magic ref for topics that has some well defined 
rules that maps a topic to a change series?

That's another interesting idea and it is what "git review" client-based workflow does.

[Oswald Buddenhagen]

On Thu, Aug 17, 2017 at 10:09:50AM -0400, 'Dave Borowitz' via Repo and Gerrit Discussion wrote:
> FWIW, this is a huge issue for us; we have several repos getting towards
> 1M refs. So, we are planning on adding a single config option to hide
> *all* refs from advertisement so we can use it on gerrit-review.
>  

what has become of the idea to replace git's ref advertisment protocol?
i know you guys actually made a proposal on the git list, but it seems
to have just petered out?

[lucamilanesio]

I've posted a Gerrit module to enable the reduction of refs advertising at:

https://github.com/GerritForge/gerrit-refsfilter

I've implemented as Gerrit PermissionsBackend module at the moment: that's the only possible way at the moment to plug additional logic into the refs filtering (see [1]).

I've tested locally and works fine :-)

@Dave: are you going to do something similar? Should we generalize an extension point in Gerrit to perform *exactly* that functionality without the need to hijack the permisson backend?

If you guys like the idea, can we create a Gerrit Module on gerrit.googlesource.com?

Name proposed: modules/refsfilter

Description: Gerrit lib module to allow filtering out refs in the Git advertizing protocol phase

[1] https://github.com/GerritForge/gerrit-refsfilter

[Dave Borowitz]

It still might happen.

Whether it does or not, we still want to make this change to Gerrit in the near term, because it will still be several years before we can depend on all clients speaking any new protocol.

[lucamilanesio]

Ping ... is anyone willing to create the repo for my Gerrit module proposal?

Name proposed: modules/refsfilter

Description: Gerrit lib module to allow filtering out refs in the Git advertising protocol phase

See below the full story associated.

[lucamilanesio]

For those Gerrit maintainers that are back on Monday: what do you think about creating a new Gerrit module for my extension?

See below:

Name proposed: modules/refsfilter

Description: Gerrit module to allow filtering closed refs in the Git refs advertising phase

I tested on Gerrit master and it works fine, the CI users won't be polluted any more with tons of closed and useless refs for them :-)

Thanks in advance.

Luca.

[Saša Živkov]

On Thu, Aug 17, 2017 at 4:09 PM, 'Dave Borowitz' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

On Thu, Aug 17, 2017 at 4:14 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

FWIW, this is a huge issue for us; we have several repos getting towards 1M refs.

We have exactly the same issue with several repos with huge number of refs.

 

So, we are planning on adding a single config option to hide *all* refs from advertisement so we can use it on gerrit-review. You can still fetch by SHA-1, and the download-commands plugin will reflect this.

I experimented with the (receivepack|transfer|uploadpack).hiderefs=refs/changes/ on a large repo.

This cuts fetch time to a fraction of what it was without this setting.

A problem with that approach is that one cannot fetch a single change using its change refs refs/changes/NN/NNNNN/N.

The download-commands plugin can provide download commands to fetch by SHA1 and this will solve the issue for (human) users.

However, I suspect that most of CI systems do not rely on the download-commands plugin and/or have the

refs/changes/NN/NNNNN/N structure hardcoded and would therefore still try fetching by refs/changes/...

Therefore, not advertising the whole refs/changes/ namespace might be quite disruptive for CI systems?

Maybe limiting advertising of refs/changes/* to the refs of all open changes and recently closed changes would be

less disruptive but would bring similar benefits?

Accessing older closed changes would still require a different approach but this is less disruptive.

[Luca Milanesio]

Bingo, *that's exactly* what my new Gerrit Module does ... and works like a charm :-)

And can be allocated to *only* specific groups of users, so that can be limited to those CI jobs (or replication nodes) that really need that type of speedup.

Can we create the repo on gerrit-review?

(the module is currently hosted on GitHub).

Luca.

[Saša Živkov]

On Tue, Aug 22, 2017 at 10:08 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 22 Aug 2017, at 09:02, Saša Živkov <ziv...@gmail.com> wrote:

On Thu, Aug 17, 2017 at 4:09 PM, 'Dave Borowitz' via Repo and Gerrit Discussion <repo-discuss@googlegroups.com> wrote:

On Thu, Aug 17, 2017 at 4:14 AM, Luca Milanesio <luca.milanesio@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

FWIW, this is a huge issue for us; we have several repos getting towards 1M refs.

We have exactly the same issue with several repos with huge number of refs.

 

So, we are planning on adding a single config option to hide *all* refs from advertisement so we can use it on gerrit-review. You can still fetch by SHA-1, and the download-commands plugin will reflect this.

I experimented with the (receivepack|transfer|uploadpack).hiderefs=refs/changes/ on a large repo.

This cuts fetch time to a fraction of what it was without this setting.

A problem with that approach is that one cannot fetch a single change using its change refs refs/changes/NN/NNNNN/N.

The download-commands plugin can provide download commands to fetch by SHA1 and this will solve the issue for (human) users.

However, I suspect that most of CI systems do not rely on the download-commands plugin and/or have the

refs/changes/NN/NNNNN/N structure hardcoded and would therefore still try fetching by refs/changes/...

Therefore, not advertising the whole refs/changes/ namespace might be quite disruptive for CI systems?

Maybe limiting advertising of refs/changes/* to the refs of all open changes and recently closed changes would be

less disruptive but would bring similar benefits?

Bingo, *that's exactly* what my new Gerrit Module does ... and works like a charm :-)

One issue with your module is that it requires removal of the global READ on refs/* permission.

I hope we can come up with a solution which doesn't necessarily rely on permissions.

But we should start with what you have currently :-)

 

And can be allocated to *only* specific groups of users, so that can be limited to those CI jobs (or replication nodes) that really need that type of speedup.

Can we create the repo on gerrit-review?

 

This shouldn't be a problem. If no one objects I would create the repo today.

 

(the module is currently hosted on GitHub).

Luca.

Accessing older closed changes would still require a different approach but this is less disruptive.

This obviously won't improve the workflow of a single "git pull", but it will cut down on ref advertisements for sure.

2. I want to allow the people to do a simgle "git pull" and get the latest patch-set for a change

Example Scenarios:

Scenario1 - CI: I want to build all the active changes:

- configure +refs/active-changes/*:refs/remotes/review/*

- just pull and build all the branches

- keep them up-date by simply fetching them again

Scenario2 - Local dev: I want to work on change 5617

- add Gerrit as 'review' remote with the following ref-spec: +refs/active-changes/*:refs/remotes/review/*

- git checkout -b 5617 review/5617 (it will fetch the latest patch-set of change 5617)

- make some local modifications

- git commit -a --amend -m "My contribution"

- git push review

How does it sound?

Comments, suggestions are more than welcome :-)

Luca.

-- 
-- 
To unsubscribe, email repo-discuss+unsub...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- 
To unsubscribe, email repo-discuss+unsub...@googlegroups.com

[Luca Milanesio]

Yes, I used the permission backend because it was the *only* way to introduce further ref-filtering in Gerrit.

However, I do quite like the idea of enabling / disabling this filtering on a per-group basis.

I would then stick with the idea of using a group-based configuration system, possibly even at repo-level in the refs/meta/config.

*Only large repos* need this type of filtering, it would be nice to limit the scope then.

The *ideal* solution would be:

- introduce a new extension for refs-filtering 

- implement the extension in a plugin

But we should start with what you have currently :-)

 

And can be allocated to *only* specific groups of users, so that can be limited to those CI jobs (or replication nodes) that really need that type of speedup.

Can we create the repo on gerrit-review?

 

This shouldn't be a problem. If no one objects I would create the repo today.

I would hold until we get Dave's feedback on the extension mentioned above: filtering through permission backend isn't going to perform well in terms of CPU usage and, as you said already, requires to skip the "quick ACL validation" of refs/* which would increase CPU consumption even more.

If Dave agrees on the new extension, we could push a change in Gerrit for that and *then* create a standard plugin for the refsfilter.

[Martin Fick]

On Tuesday, August 22, 2017 09:36:54 AM Luca Milanesio

wrote:
> > On 22 Aug 2017, at 09:21, Saša Živkov <ziv...@gmail.com>
> > wrote:>
> > On Tue, Aug 22, 2017 at 10:08 AM, Luca Milanesio

<luca.mi...@gmail.com <mailto:luca.mi...@gmail.com>>

wrote:
> >> On 22 Aug 2017, at 09:02, Saša Živkov <ziv...@gmail.com

> >> <mailto:ziv...@gmail.com>> wrote:
> >>
> >>
> >>
> >> On Thu, Aug 17, 2017 at 4:09 PM, 'Dave Borowitz' via
> >> Repo and Gerrit Discussion

> >> <repo-d...@googlegroups.com

> >> <mailto:repo-d...@googlegroups.com>> wrote:
> >>
> >>
> >> On Thu, Aug 17, 2017 at 4:14 AM, Luca Milanesio

> >> <luca.mi...@gmail.com
> >> <mailto:luca.mi...@gmail.com>> wrote: Hi all,

On the surface it seems like you are proposing
reimplementing the ACL mechanism under a different name. Do
you have any thoughts on how such a plugin would not suffer
the same performance problems as using the current ACL
mechanisms which are already highly optimized for
performance?

[luca.mi...@gmail.com]

Sent from my iPhone

My current implementation can only respond to single change permissions request and throw exception for all the closed changes: throwing 1M exceptions doesn't seems something highly optimised.

Additionally it forces to give up the refs/* rule which is a Hughes minus.

Having an interface that says: do you have a filter to apply seems something more sensible to me ;-)

But again, before start to implement a new extension point, I need feedback and ideas.

Luca

[lucamilanesio]

No more feedback for around a week ... @Gerrit Maintainers - shall we then create the module on gerrit-review.googlesource.com?

Thanks.

Luca.

On Tuesday, August 22, 2017 at 2:58:51 PM UTC-7, lucamilanesio wrote:

Sent from my iPhone

> On 22 Aug 2017, at 19:14, Martin Fick <mf...@codeaurora.org> wrote:
>
> On Tuesday, August 22, 2017 09:36:54 AM Luca Milanesio
> wrote:
>>> On 22 Aug 2017, at 09:21, Saša Živkov <ziv...@gmail.com>
>>> wrote:>
>>> On Tue, Aug 22, 2017 at 10:08 AM, Luca Milanesio

> <luca.mi...@gmail.com <mailto:luca.milanesio@gmail.com>>

> wrote:
>>>> On 22 Aug 2017, at 09:02, Saša Živkov <ziv...@gmail.com
>>>> <mailto:ziv...@gmail.com>> wrote:
>>>>
>>>>
>>>>
>>>> On Thu, Aug 17, 2017 at 4:09 PM, 'Dave Borowitz' via
>>>> Repo and Gerrit Discussion
>>>> <repo-d...@googlegroups.com

>>>> <mailto:repo-discuss@googlegroups.com>> wrote:
>>>>
>>>>
>>>> On Thu, Aug 17, 2017 at 4:14 AM, Luca Milanesio
>>>> <luca.mi...@gmail.com

>>>> <mailto:luca.milanesio@gmail.com>> wrote: Hi all,

[Shawn Pearce]

On Sat, Aug 26, 2017 at 9:21 AM, lucamilanesio <luca.mi...@gmail.com> wrote:

No more feedback for around a week ... @Gerrit Maintainers - shall we then create the module on gerrit-review.googlesource.com?

I've been dragging my feet on this because its not efficient to run this through the permission system, and Dave thinks this should be a standard config option available in the server.

 

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

[Luca Milanesio]

Hi Shawn,

thanks for the feedback, will wait for Dave to formulate a proposal for the config option.

I agree, its not efficient to "hijack" the permissions system for this.

Luca.

[Saša Živkov]

On Thu, Aug 17, 2017 at 4:21 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 17 Aug 2017, at 15:09, Dave Borowitz <dbor...@google.com> wrote:

On Thu, Aug 17, 2017 at 4:14 AM, Luca Milanesio <luca.milanesio@gmail.com> wrote:

Hi all,

I need the community's opinion on a change I am about to propose on Gerrit master branch.

See below what I want to do and why :-)

Goal:

Making people's lives easier when fetching active changes locally and keeping the up-to-date.

Giving a more "feature-branch-like" workflow that would accelerate the learning curve for Gerrit novice users

What:

Introduce a new 'active-changes' for filtering out closed or obsolete changes (merged, abandoned, older than N days) from the "ls-remote" command and, for the active changes that always point to the latest patch-set.

Why:

1. For busy repos, the number of changes refs could be *huge*, up to tens of thousands or even more: I want to make that list shorter and more tied to "live" changes.

FWIW, this is a huge issue for us; we have several repos getting towards 1M refs. So, we are planning on adding a single config option to hide *all* refs from advertisement so we can use it on gerrit-review. You can still fetch by SHA-1, and the download-commands plugin will reflect this.

We have clients in a similar situation, where replication to slaves become a real issue.

Today we succeeded making replication of a repo with 400K+ refs at least 10 times faster.

The main thing was to remove advertising of the refs/changes for receive-pack.

On the receiving side we configured:

  [receive]

    hideRefs = refs/changes/

    hideRefs = refs/cache-automerge/

Since, with this setting, replication of refs/changes is not possible we defined replication of changes as:

  push = refs/changes/*:refs/tmp/changes/*

and on the receiving side a post-receive hook [1] which moves all refs/tmp/changes/* to refs/changes.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/126031

 

90% of the times slaves are there for CI jobs, and you only want to build changes that are "recent" (let's say, the last month) and "live" (no closed or abandoned changes).

With those conditions, our clients could have a significant improvement in the reduction of the advertised refs to the slaves and consequent speedup of their CI builds as well.

This obviously won't improve the workflow of a single "git pull", but it will cut down on ref advertisements for sure.

Yep.

So possibly a "core-plugin" idea? Or as you said makes sense in the kernel as well?

2. I want to allow the people to do a simgle "git pull" and get the latest patch-set for a change

Example Scenarios:

Scenario1 - CI: I want to build all the active changes:

- configure +refs/active-changes/*:refs/remotes/review/*

- just pull and build all the branches

- keep them up-date by simply fetching them again

Scenario2 - Local dev: I want to work on change 5617

- add Gerrit as 'review' remote with the following ref-spec: +refs/active-changes/*:refs/remotes/review/*

- git checkout -b 5617 review/5617 (it will fetch the latest patch-set of change 5617)

- make some local modifications

- git commit -a --amend -m "My contribution"

- git push review

How does it sound?

Comments, suggestions are more than welcome :-)

Luca.

-- 
-- 
To unsubscribe, email repo-discuss+unsub...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luca Milanesio]

That is *really* cool, outstanding results !

Does this mean that the concept works ... and we should now do something about as "out-of-the-box" feature?

Luca.

[Matthias Sohn]

On Fri, Sep 8, 2017 at 5:56 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

That is *really* cool, outstanding results !

Does this mean that the concept works ... and we should now do something about as "out-of-the-box" feature?

yes, this improved replication performance a lot. We observe that time to replicate a single

from Germany to Vancouver dropped by roughly a factor of 10 down from 10min to 1-1.5min.

It seems more improvements are possible since we see that the pack arrives pretty quickly

already after 10-15sec (in order to simplify observing these timings we set receive.unpacklimit=1

so that we only need to watch the pack directory). We observed the process tree on the slave when

a change arrives and found that now we spend most time (40-50sec out of 60-80sec) running

git rev-list --objects --stdin --not --all

it looks like this is the connectivity check in check_connected() in connected.c  which is called from

receive-pack [2]. On the slave side we run git http-backend on apache. And this is a pretty large

repository.

Is this slowness due to running http-backend cgi program not caching between subsequent requests ?

We'll try to use gerrit on the slave side in order to test if jgit object cache does improve performance.

[2] https://github.com/git/git/blob/master/connected.c#L56

https://github.com/git/git/blob/master/builtin/receive-pack.c#L1487

https://github.com/git/git/blob/master/builtin/receive-pack.c#L1992

-Matthias