Scaling: Many Questions For Those with Large Installations

[Martin Fick]

I have some questions about practices that I was hoping some
of the folks with larger installations might be willing to
share with the community to help bring the Scaling

https://code.google.com/p/gerrit/wiki/Scaling?ts=1414115745&updated=Scaling

page up-to-date.


For example:

* Who is using DB replication? Which DB (which replication
type)?

* Which java gc collector settings have you found to be most
useful?

* Are there known scalability issues not mentioned on this
page?

* What is your biggest current scalability issue?


But also, I have some sizing questions:

* Does anyone have over 20 slaves for the same master? Over
30? More?

* Anyone have a lot more than 20K projects? How many?

* Anyone have a lot more than 500GB in repo data? How much?

* Anyone have a lot more than 1M changes? How many?

* Anyone have more than 500K refs in a single repo? How many
more?

* How many receive-packs in a day does your master get?

* How many hooks does your server run per day?

* What is your largest repository size on disk?

I may be asking more questions for this page soon, but I hope
this seems like a lot already to get started.

Thanks,

-Martin


--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Lundh, Gustaf]

Sony Mobile's main Gerrit instance:

* Who is using DB replication? Which DB (which replication type)?

We have one master and ~25 slaves (on 3 continents). All slaves have a local postgres database replicated from master.

* Which java gc collector settings have you found to be most useful?

We have tried many different settings, and even custom benchmarks apps to evaluate GC Collector performance. But the default (at least in Oracles Java 7) seems to work best in our environment.

* Are there known scalability issues not mentioned on this page?

I see some old issues that have been fixed, but most seem covered.

Perhaps a separate chapter on Storage Solutions and File Systems makes sense to have.

* What is your biggest current scalability issue?

Availability. We need to minimize downtime during upgrades and fixes. Replicating data to 25 slaves is also a bit heavy.

We are building like crazy, getting out reference mirrors to the 1000-1500 or so CI build machines can also be a challenge.

* Does anyone have over 20 slaves for the same master? Over 30? More?

25

* Anyone have a lot more than 20K projects? How many?

6000

* Anyone have a lot more than 500GB in repo data? How much?

We are around 500GB today

* Anyone have a lot more than 1M changes? How many?

Close to 1M

* Anyone have more than 500K refs in a single repo? How many more?

Not close afaik. We try to avoid unnecessary tagging and branching.

* How many receive-packs in a day does your master get?

We will get back to you on this one.

* How many hooks does your server run per day?

We don't use Hooks.

* What is your largest repository size on disk?

We will get back to you on this one.

Best regards

Gustaf

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bassem Rabil]

Here below scaling information for Ericsson instance. 

On Thursday, October 23, 2014 10:27:46 PM UTC-4, MartinFick wrote:

I have some questions about practices that I was hoping some 

of the folks with larger installations might be willing to 

share with the community to help bring the Scaling 

https://code.google.com/p/gerrit/wiki/Scaling?ts=1414115745&updated=Scaling 

page up-to-date. 

For example:

Ericsson main instance.

* Who is using DB replication?  Which DB (which replication 

type)?

 

We use PostGreSQL replication to ~7 DB slaves in 3 continents which are read-only DBs.

* Which java gc collector settings have you found to be most useful? 

We use the default java garbage collector. 

* Are there known scalability issues not mentioned on this page?

Recently we faced some issues with changes with large number of lines diff, i.e. in range of millions of diff lines. Of course Gerrit is not supposed to handle such changes for review, but in some cases such changes consume lots of resources to get and display such huge diff. We think that the server should treat such changes differently to avoid abusing the server resources.

 

* What is your biggest current scalability issue? 

- Repositories with large number of references, which slows down any operation towards such repositories

- Repositories with large size, i.e. 30-40 GB.

But also, I have some sizing questions: 

* Does anyone have over 20 slaves for the same master?  Over 30? More?

We have 7 slave instances with total up to 15 slave backends  

* Anyone have a lot more than 20K projects?  How many? 

Currently we have around 6k projects  

* Anyone have a lot more than 500GB in repo data?  How much?

We are approaching 1 TB of repo data.

* Anyone have a lot more than 1M changes? How many? 

Currently we have around 350k changes. 

* Anyone have more than 500K refs in a single repo?  How many more? 

The maximum number of refs we have is around 120k refs

 

* How many receive-packs in a day does your master get?

Around 20k per day 

 

* How many hooks does your server run per day? 

No hooks are run on server side, hooks are run on client side. 

* What is your largest repository size on disk? 

30-40 GB  

[Nasser Grainawi]

On Oct 23, 2014, at 8:27 PM, Martin Fick <mf...@codeaurora.org> wrote:

> I have some questions about practices that I was hoping some
> of the folks with larger installations might be willing to
> share with the community to help bring the Scaling
>
> https://code.google.com/p/gerrit/wiki/Scaling?ts=1414115745&updated=Scaling
>
> page up-to-date.
>
>
> For example:
>
> * Who is using DB replication? Which DB (which replication
> type)?

PostgreSQL. No replication.

>
> * Which java gc collector settings have you found to be most
> useful?

JVM 7 with ParallelOldGC collector.

>
> * Are there known scalability issues not mentioned on this
> page?
>
> * What is your biggest current scalability issue?

Availability.

>
>
> But also, I have some sizing questions:
>
> * Does anyone have over 20 slaves for the same master? Over
> 30? More?

24 slaves via 6 replication endpoints.

>
> * Anyone have a lot more than 20K projects? How many?

2506

>
> * Anyone have a lot more than 500GB in repo data? How much?

~600GB

>
> * Anyone have a lot more than 1M changes? How many?

995K+. We'll hit 1M in a couple weeks.

>
> * Anyone have more than 500K refs in a single repo? How many
> more?

~400K

>
> * How many receive-packs in a day does your master get?

~50K

>
> * How many hooks does your server run per day?

~30K

>
> * What is your largest repository size on disk?

~26GB

>
> I may be asking more questions for this page soon, but I hope
> this seems like a lot already to get started.
>
> Thanks,
>
> -Martin
>
>
> --
> The Qualcomm Innovation Center, Inc. is a member of Code
> Aurora Forum, hosted by The Linux Foundation
>

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

[Martin Fick]

Awesome info, thanks for the quick reply!

---"Lundh, Gustaf" <Gustaf...@sonymobile.com>--- wrote:
> Perhaps a separate chapter on Storage Solutions and File Systems makes
sense to have.

Yes, I think that is a good suggestion. Care to give us some info on what
you have seen that works well, and what problems you have seen? It might
be a good starting place? (Note I added a section about slaves sharing a
local backend)

> * What is your biggest current scalability issue?
> Availability. We need to minimize downtime during upgrades and fixes.

What sorts of things do you think would help in this area?

Has anyone addressed this for their site?

> Replicating data to 25 slaves is also a bit heavy.

Are these all in your replication.config or do they use local shared NFS
servers?

We have been playing with some multi tier replication scripts internally
to further reduce our master load to our sites where latency is not as
important. Anyone else doing anything like this?

> We are building like crazy, getting out reference mirrors to the
1000-1500
> or so CI build machines can also be a challenge.

Good point, that is a challenge for us also. That seems like a subject
that might perhaps be worth a page of its own eventually? It would be
great to hear how others are addressing this.

-Martin

[Martin Fick]

Thanks for the great info!

----"Bassem Rabil" <bassem.ra...@ericsson.com> wrote:
>
> Recently we faced some issues with changes with large number of lines
> diff,
> i.e. in range of millions of diff lines. Of course Gerrit is not supposed
> to handle such changes for review, but in some cases such changes consume
> lots of resources to get and display such huge diff. We think that the
> server should treat such changes differently to avoid abusing the server
> resources.

So just to clarify, these problems are mainly in the UI, or are there
other areas that this affects? I will try to add something about this to
the page.

>> * What is your biggest current scalability issue?
>>
> - Repositories with large number of references, which slows down any
> operation towards such repositories

I am a bit surprised that 120K refs is causing issues. We have one repo
with ~400K refs, and it isn't a speed daemon, but it works reasonably
well. How slow are we talking? I wonder if you could benefit from this
jgit patch: https://git.eclipse.org/r/#/c/24295/ ? If it helps, it will
mention it on the scaling page.

> - Repositories with large size, i.e. 30-40 GB.

Any thoughts on this? What specific problems are you running into. Just
slow downloads (due to the amount of data being transferred), or are there
other issues?

>> But also, I have some sizing questions:
>>
>> * Does anyone have over 20 slaves for the same master? Over 30? More?
>>
> We have 7 slave instances with total up to 15 slave backends

This sounds like you are sharing the storage for some of your slaves. How
are you managing gc/repacking on these shared backends? I would like to
add some suggestions to the Scaling page on what to do for this. Are you
using Gerrit gc, or git gc on them? Do you just have one of the slaves on
the shared back end configured to run gc, or do you try to load balance
this somehow? We have a script with per repo server locks which helps
load balance this across all the slaves on the back end. This also helps
ensure that it will get done eve n if slaves are added/removed.

Thanks again for the info!

-Martin

[Bassem Rabil]

Hi Martin

Here below my answers inline, thanks for this initiative to collect and update scalability issues page.

Regards

Bassem

On Friday, October 24, 2014 5:21:32 PM UTC-4, MartinFick wrote:

Thanks for the great info!

----"Bassem Rabil" <bassem.ra...@ericsson.com> wrote:
>
> Recently we faced some issues with changes with large number of lines
> diff,
> i.e. in range of millions of diff lines. Of course Gerrit is not supposed
> to handle such changes for review, but in some cases such changes consume
> lots of resources to get and display such huge diff. We think that the
> server should treat such changes differently to avoid abusing the server
> resources.

So just to clarify, these problems are mainly in the UI, or are there
other areas that this affects?  I will try to add something about this to
the page.

There are issues with reindexing such changes which is triggered upon merging any change to a branch, where all open changes to this changes are reindexed and the mergeabiliity check is recalculated. This process can hijack processing threads to get the diiff for such changes and thus affects the server resources. For the web UI, this is guarded by a mechanism to abort the trial to get diff if it takes longer time. For more details check this discussion [1].

[1] https://groups.google.com/forum/#!searchin/repo-discuss/jgit/repo-discuss/ZtiCilM3wFA/LijfZ4YkLHsJ

>> * What is your biggest current scalability issue?
>>
> - Repositories with large number of references, which slows down any
> operation towards such repositories

I am a bit surprised that 120K refs is causing issues.  We have one repo
with ~400K refs, and it isn't a speed daemon, but it works reasonably
well.  How slow are we talking?  I wonder if you could benefit from this
jgit patch: https://git.eclipse.org/r/#/c/24295/ ?  If it helps, it will
mention it on the scaling page.

For our case this 120k refs repository is one of the most busy repositories with enormous changes/pushes and also huge number of pulls for different test flows. The replication performance is affected by this large number of refs.

We will look into this patch for jgit, hopefully it will improve our experience with such repositories.

 

> - Repositories with large size, i.e. 30-40 GB.

Any thoughts on this?  What specific problems are you running into.  Just
slow downloads (due to the amount of data being transferred), or are there
other issues?

Yeah, slow downloads/replications and in many cases timeouts. Very slow and resource exhausting during git garbage collection.

>> But also, I have some sizing questions:
>>
>> * Does anyone have over 20 slaves for the same master?  Over 30? More?
>>
> We have 7 slave instances with total up to 15 slave backends

This sounds like you are sharing the storage for some of your slaves.  How
are you managing gc/repacking on these shared backends?  I would like to
add some suggestions to the Scaling page on what to do for this.   Are you
using Gerrit gc, or git gc on them?  Do you just have one of the slaves on
the shared back end configured to run gc, or do you try to load balance
this somehow?  We have a script with per repo server locks which helps
load balance this across all the slaves on the back end.  This also helps
ensure that it will get done eve n if slaves are added/removed.

You are right, we are sharing NFS storage for slave backend hosts. We use load balancing mechanism based on ha-proxy among these backend machines.

Git garbage collection is run on the backend machines so far, however we might consider soon allocating separate hosts for running garbage collection specially for increasing total repository size. 

[Oswald Buddenhagen]

On Fri, Oct 24, 2014 at 03:05:42PM -0600, Martin Fick wrote:
> > We are building like crazy, getting out reference mirrors to the
> > 1000-1500 or so CI build machines can also be a challenge.
>
> Good point, that is a challenge for us also.
>

that's a more or less universal problem for CI systems.

> It would be great to hear how others are addressing this.
>

one of our colleagues once tried to create a transparent caching proxy:
https://github.com/rohanpm/ngitcached - it's pre-alpha according to him.

it would be also possible to have the CI explicitly request replication
of specific refs to a local mirror before commencing work. that requires
a custom command interface on the mirror.

in principle, one can even make a hierarchy of mirrors to optimize
throughput.

all of these approaches are pull-based. i don't think gerrit's push
replication is of much use in the CI scenario (synchronization is much
harder with push).

[Bassem Rabil]

The large diff is the most recent challenge we encountered, however to summarize different challenges we have with 2 Gerrit instances sharing the same repos and DB (in hot standby model) is the availability challenge which was brought up in the discussion here, i.e.

- Sharing web sessions between 2 instances and this applies to other caches as well

- Sharing secondary index (I know there is currently an effort to introduce Elastic search which brings a shared secondary index)

- Conveying stream events to another instance.

... and so on.

This should bring us more into running Gerrit in a cluster of hosts sharing everything to reach the maximum availability.

Regards

Bassem 

[Vlad Canţîru]

Hi,

For sharing the web session, Martin you mentioned earlier this year that you were cooking something there. Did you reach any conclusion yet?

Luca might come up soon with a solution for stream events using Apache Flume, hope he haven't dropped this idea. Oswald mentions cache proxy and I don't have an example on hand but I think sharing index was already discussed somewhere here. Looks like all ingredients are getting together to finally have a cluster of small masters and avoid using 48 core machines.

I also think that going away from ssh would help to scale and have more options for improving availability. For small instances ssh is convenient but becomes too rigid once the system usage gets heavier. 

Best Regards

[Martin Fick]

> Hi,
>
>
> For sharing the web session, Martin you mentioned earlier this year that
> you were cooking something there. Did you reach any conclusion yet?

The websession code is indeed merged in the multi-master plugin, give it a
spin.

> Luca might come up soon with a solution for stream events using Apache
> Flume, hope he haven't dropped this idea.

I actually just was thinking about coding up a zookeper based idea for
this. I think it would be trivial. I have been looking into putting the
replciation queue in zookeper also, but it is a bit harder since unlike
stream events the objects won't serialize as easily. If zookeeper becomes
expected anyway in MM setups, I wonder if it would also be worth doing
cache evictions that way?

> Oswald mentions cache proxy and
> I don't have an example on hand but I think sharing index was already
> discussed somewhere here. Looks like all ingredients are getting together
> to finally have a cluster of small masters and avoid using 48 core
> machines.

Yes, although we need it even with our 48 core machines. :)

> I also think that going away from ssh would help to scale and have more
> options for improving availability. For small instances ssh is convenient
> but becomes too rigid once the system usage gets heavier.

For git operations at least, you can use https instead already. The REST
api makes many other operations available over http,

[David Pursehouse]

On 10/28/2014 07:03 AM, Martin Fick wrote:
>> Hi,
>>
>>
>> For sharing the web session, Martin you mentioned earlier this year that
>> you were cooking something there. Did you reach any conclusion yet?
>
> The websession code is indeed merged in the multi-master plugin, give it a
> spin.
>

Did you mean the websession-flatfile plugin?

https://gerrit-review.googlesource.com/#/q/status:merged+project:plugins/websession-flatfile+branch:master


Changes merged on multi-master seem to be only documentation.

https://gerrit-review.googlesource.com/#/q/status:merged+project:plugins/multi-master+branch:master

[Martin Fick]

On Monday, October 27, 2014 07:13:13 pm David Pursehouse
wrote:

> On 10/28/2014 07:03 AM, Martin Fick wrote:
>
> >> For sharing the web session, Martin you mentioned earlier
this year that
> >> you were cooking something there. Did you reach any
conclusion yet?
> >
> > The websession code is indeed merged in the multi-master
plugin, give it
> > a spin.
>
> Did you mean the websession-flatfile plugin?
>
> https://gerrit-
review.googlesource.com/#/q/status:merged+project:plugins/we
> bsession-flatfile+branch:master

Doh! Yes, that's what I get for answering on the way out the
door. Thanks!

> Changes merged on multi-master seem to be only
documentation.
>
> https://gerrit-
review.googlesource.com/#/q/status:merged+project:plugins/mu
> lti-master+branch:master

Right you are. Useful for using the websessions plugin. :)

[Vlad Canţîru]

>For git operations at least, you can use https instead already.  The REST
>api makes many other operations available over http,

 

Git operations over http is not a challenge even for slaves. As of today even if all git operations go over http, gerrit stream events part is still tied to ssh. First, only one master can generate events and second no matter how good your network is a persistent ssh connection cannot be considered reliable for any CI machinery. 

If I recall well you don't have this issue as you use polling approach only but in our case Jenkins trigger plugin is heavily used.

BTW: how is your effort on getting back on master going? 

-Regards 

  

[Martin Fick]

> As of today even if all git operations go over http, gerrit stream
events part is
> still tied to ssh. First, only one master can generate events and second no
> matter how good your network is a persistent ssh connection cannot be
> considered reliable for any CI machinery.

Agreed, but no connection mechanism can. There have been alternate
suggestions, but I suspect that for us we will want to support legacy ssh
connections. I don't think this is a very difficult problem to solve.

I started putting events into zookeeper last night, it was trivial as a
hack. A proper solution would take a little longer (to potentially use an
executor to queue them while getting them into zookeeper). Getting them
out is a bit harder, I will hack that up soon to see how to do it. So far
this seems a bit harder than putting them in, but not much, zookeeper does
have watches. To put them in, I gave the events an integer id, and just
incremented it. I then stored the events in a node named after their id:
"/events/<id>". SInce the event is piped as json, that is what I put in
zookeeper. That is very convenient as long as the event doesn't grow over
1MB (zookeper limit). Once in zookeeper, a mechanism will be needed for
flushing old events, I imagine time being an easy way to prune things.

With persistent events and ids it should become trivial to specify a
starting id on reconnect. That should make it possible to request events
while the connection was down. I suspect that this will be desired no
matter what sort of connection is used.

Do you think the above features should be enough for a decent MM solution?

I am thinking that if we get the above, it would be neat to enable
stream-event only hosts. All they do is listen to zookeeper and pipe the
data out over ssh. Of course, other tools could potentially just listen
to zookeeper directly. However, the idea of single purpose servers feels
kinda neat. A "read-only" event server like this could stay running even
when the "write-servers" go down. Such a server would only have
dependencies on the DB for user auth. It would have no dependencies on
the repos, so it doesn't even require NFS... This might make it possible
to keep this server running even during schema upgrades? It could be
useful even for single master solutions.

> If I recall well you don't have this issue as you use polling approach
> only but in our case Jenkins trigger plugin is heavily used.

For CI, yes, correct. We do use events for other data though now...

> BTW: how is your effort on getting back on master going?

Uhm, I think it will still be a while, :(

[Shawn Pearce]

Interesting solution.

Certainly would make this simple. Different masters can easily write to the same Zookeeper cluster to have their events mixed together.

That might be overkill as some message queue systems may be cheaper to run, but may also give you less reliable delivery semantics.

After EclipseCon Europe I am thinking about an MQTT system to push events, as JSON topics are fairly common and the brokers are able to push the stuff around easily.

But we may need to run a lock server like Zookeeper anyway so one less dependency is very appealing.

[Martin Fick]

On Tuesday, October 28, 2014 01:37:45 pm Martin Fick wrote:
> Getting them out is a bit harder, I will hack that up soon
> to see how to do it. So far this seems a bit harder than
> putting them in, but not much, zookeeper does have watches.

A few snags I ran into:

1) Auth. Events are filtered by user visibility. We would need
to put all events in zookeeper, and then filter them on retrieval
by user. This likely means decoding the json, and querying the
DB on the change id to get a ChangeControl to then perform the
isVisible() test on the event based on the current user. This
means that if events are in the queue for a while before the user
retrieves them, their visibility could change! Will this be a
problem in practice?

2) The only way I see to get a watch on children of a node is to
request all the children of the node. Watches seem to need to be
reset after every event they trigger. Retrieving the full list
of events in the queue every time that one is added (every time
the watch fires) will be very expensive as the list grows. This
likely will not scale. Some way of managing this will be
required. Maybe events get moved to another queue as soon as
they are added to reduce the size of the list being retrieved
each time? I suspect that this is solvable, but it might require
some creative thinking. :)

[Kenny Ho]

A side question for you folks running multiple slaves... how do you assign users to the slaves?  Is it a automatic process?  Or do you do it manually by function and/or geography?

Regards,

Kenny

On Saturday, October 25, 2014 8:07:13 PM UTC-4, Bassem Rabil wrote:

[Bassem Rabil]

The users are distributed among slaves mainly depending on the geographical location to achieve the least network delays working on the same LAN.

[Shawn Pearce]

On Wed, Oct 29, 2014 at 10:20 PM, Martin Fick <mf...@codeaurora.org> wrote:
>
> On Tuesday, October 28, 2014 01:37:45 pm Martin Fick wrote:
> > Getting them out is a bit harder, I will hack that up soon
> > to see how to do it. So far this seems a bit harder than
> > putting them in, but not much, zookeeper does have watches.
>
> A few snags I ran into:
>
> 1) Auth. Events are filtered by user visibility. We would need
> to put all events in zookeeper, and then filter them on retrieval
> by user. This likely means decoding the json, and querying the
> DB on the change id to get a ChangeControl to then perform the
> isVisible() test on the event based on the current user.

Yes. This is a necessary evil if we want to maintain the permission
system controlling event visibility.

> This
> means that if events are in the queue for a while before the user
> retrieves them, their visibility could change! Will this be a
> problem in practice?

No. This is reality. If the visibility changes the user can or cannot
see the event now, and that should be reflected at polling/delivery
time.

> 2) The only way I see to get a watch on children of a node is to
> request all the children of the node. Watches seem to need to be
> reset after every event they trigger. Retrieving the full list
> of events in the queue every time that one is added (every time
> the watch fires) will be very expensive as the list grows. This
> likely will not scale. Some way of managing this will be
> required. Maybe events get moved to another queue as soon as
> they are added to reduce the size of the list being retrieved
> each time? I suspect that this is solvable, but it might require
> some creative thinking. :)

Ah, right. I forgot about this "feature" of Zookeeper watches. Its
pretty useless.

Maybe you can keep the queue in one directory, and an atomic counter
in another. Put the watch on the atomic counter to let you know what
the "high water mark" is for event assignment, but the queue itself
isn't watched. Instead the server pulls events between the client's
last/current position and the high water mark obtained from that
counter file.

[Bassem Rabil]

Hi Martin 

We experimented by cherry picking this jgit patch on top of jgit 3.4 and tried it with Gerrit 2.9.1 for example with gerrit gc for the 120k refs repository, it took 11:55 mins instead of 11:58-12:30 mins
We have the replication delay set to few seconds e.g. in range of 25-30 seconds, during the replication listing all refs is taking time especially our repositories are stored on NFS, did you consider before archiving older refs ?

For example archive closed changes refs which is older than 6 months, to avoid carrying a heavy heritage of such references which are no longer used. May be push them to an archive repository which is not used for production.

I appreciate if you elaborate more on how you handle the repository you have with 500k refs, how long it takes normally to replicate over WAN ? How much is the total size on disk for such repository ?

With the current adoption rate of code reviews we have in our instance I think we might be hitting higher number of refs per repository soon.

Thanks and Regards

Bassem Guendy

On Saturday, October 25, 2014 8:07:13 PM UTC-4, Bassem Rabil wrote:

[Bassem Rabil]

Hi Martin

I was wondering about the example repository you were describing in this post with 400k refs. Are you still using this repository with this massive number of refs ? How many refs are there now ?

Did you consider deleting older changes refs from both git repository and Gerrit DB to cleanup obsoleted changes and downsize the repository?  Any other ideas on optimizing such large number of refs ?

Thanks

Bassem

[Martin Fick]

On Thursday, July 09, 2015 07:15:13 AM Bassem Rabil wrote:
> Hi Martin
>

> I was wondering about the example repository you were
> describing in this post with 400k refs. Are you still
> using this repository with this massive number of refs ?
> How many refs are there now ?
> Did you consider deleting older changes refs from both git
> repository and Gerrit DB to cleanup obsoleted changes and
> downsize the repository?

Seeing as Gerrit is a version control system, I am generally
against such ideas.

> Any other ideas on optimizing such large number of refs ?

Lots that I like, lots that I don't like. :)

It really depends on the use case, there is no general
approach. We attack each use case that is a problem for us.
Which specific use case are you trying to improve? What are
the stats for that use case?

[Bassem Rabil]

On Thursday, July 9, 2015 at 1:23:22 PM UTC-4, MartinFick wrote:

It really depends on the use case, there is no general
approach.  We attack each use case that is a problem for us.  
Which specific use case are you trying to improve?  What are
the stats for that use case?

We have a fast growing number of reference for one of our repositories, i.e. almost 100k new refs every 6 months totaling up to more than 220k. Most of these references are refs/changes, and they are growing fast, i.e. almost 100k refs every 6 months with more adoptions to code review. At this point the git  bitmaps generated with garbage collection does not help to improve the performance of git operations on such repository. Any ideas about how to avoid reaching a point where such repository is hard to be usable due to very slow git operations ?

Thanks

Bassem

[Martin Fick]

Can you give some stats on what slow means?

Thanks,

[Bassem Rabil]

On Thursday, July 9, 2015 at 1:59:40 PM UTC-4, MartinFick wrote:

Can you give some stats on what slow means?

For example the replication time increased from less than a minute 6 months ago  with 120k refs to 4 minutes with 220k refs.

[Oswald Buddenhagen]

On Thu, Jul 09, 2015 at 10:54:48AM -0700, Bassem Rabil wrote:
> We have a fast growing number of reference for one of our repositories,
> i.e. almost 100k new refs every 6 months totaling up to more than 220k.
> Most of these references are refs/changes, and they are growing fast, i.e.
> almost 100k refs every 6 months with more adoptions to code review. At this
> point the git bitmaps generated with garbage collection does not help to
> improve the performance of git operations on such repository.
>

our repos aren't growing quite as fast, but after three years, working
with the qtbase repository is becoming really painful as well
(approaching 100k refs).

> Any ideas about how to avoid reaching a point where such repository is
> hard to be usable due to very slow git operations ?
>

i don't think it can be avoided from the user side.
pulls can be made fast by directing people to a "thin" mirror without
all the magic refs. that doesn't help pushes, though - luckily, they
are quite a bit more rare.

edwin's patch at
https://gerrit-review.googlesource.com/69282 tries to address this
problem.
however, i'm not entirely convinced by the chosen approach, as
indicated in the linked issue.

[Nasser Grainawi]

On Jul 9, 2015, at 11:23 AM, Martin Fick <mf...@codeaurora.org> wrote:
>
> On Thursday, July 09, 2015 07:15:13 AM Bassem Rabil wrote:
>> Hi Martin
>>
>> I was wondering about the example repository you were
>> describing in this post with 400k refs. Are you still
>> using this repository with this massive number of refs ?
>> How many refs are there now ?

And since you asked, I think before it was approaching 400k (probably was 380k), it's now at 410k as usage of that repo has (obviously) slowed.

>> Did you consider deleting older changes refs from both git
>> repository and Gerrit DB to cleanup obsoleted changes and
>> downsize the repository?
>
> Seeing as Gerrit is a version control system, I am generally
> against such ideas.
>
>> Any other ideas on optimizing such large number of refs ?
>
> Lots that I like, lots that I don't like. :)
>
> It really depends on the use case, there is no general
> approach. We attack each use case that is a problem for us.
> Which specific use case are you trying to improve? What are
> the stats for that use case?
>
> -Martin
>
>
> --
> The Qualcomm Innovation Center, Inc. is a member of Code
> Aurora Forum, hosted by The Linux Foundation
>

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Martin Fick]

That sounds slow, I don't believe our replication is that
bad, but projects can vary in many ways. So I think you
probably want to quantify some things and do some testing.
I don't remember your details, are you pushing via ssh or
git-daemon?

You probably want to quantify what is causing this slowness.
I would first try to identify which piece is slow, the
sending end (Gerrit pushing), the connection (your WAN), or
the receiving end (your mirror)? Some ways to determine
the piece that is problematic in your system is to mirror to
a local disk on the same machine, how slow is that? Also,
look at CPU on both ends, and look at your network usage.

Also, are all replications to that project slow, or could it
be the amount of data being transferred? In other words, if
the project is up-to-date, and you trigger replication on it
via ssh, does it still take 4mins? Is the slowness ref
count related, or repo size related? In other words, if you
clone your repo to a different project name, and clone it
without all the refs (only leave a few heads), it is still
slow replicating?

I hope that gives you some ideas of how to at least narrow
down your problem. Once you have a better idea, there might
be ways to improve things,

[Martin Fick]

On Thursday, July 09, 2015 01:02:00 PM Martin Fick wrote:
> On Thursday, July 09, 2015 11:10:41 AM Bassem Rabil wrote:
> > On Thursday, July 9, 2015 at 1:59:40 PM UTC-4,
> > MartinFick
>
> wrote:
> > > Can you give some stats on what slow means?
> >
> > For example the replication time increased from less
> > than
> > a minute 6 months ago with 120k refs to 4 minutes with
> > 220k refs.
>
> That sounds slow, I don't believe our replication is that
> bad, but projects can vary in many ways.

FYI, our kernel/msm repo (a linux kernel project) with ~400K
refs replicates to a LAN based host usually between 30-60s.
If you are at 4min, I would be very doubtful that you cannot
improve things with better hardware, better config, better
software versions...

[Martin Fick]

On Thursday, July 09, 2015 08:21:16 PM Oswald Buddenhagen
wrote:

> On Thu, Jul 09, 2015 at 10:54:48AM -0700, Bassem Rabil
wrote:
> > We have a fast growing number of reference for one of
> > our repositories, i.e. almost 100k new refs every 6
> > months totaling up to more than 220k. Most of these
> > references are refs/changes, and they are growing fast,
> > i.e. almost 100k refs every 6 months with more
> > adoptions to code review. At this point the git
> > bitmaps generated with garbage collection does not help
> > to improve the performance of git operations on such
> > repository.
> our repos aren't growing quite as fast, but after three
> years, working with the qtbase repository is becoming
> really painful as well (approaching 100k refs).

Can you put some numbers to this pain?

> > Any ideas about how to avoid reaching a point where such
> > repository is hard to be usable due to very slow git
> > operations ?
> i don't think it can be avoided from the user side.
> pulls can be made fast by directing people to a "thin"
> mirror without all the magic refs. that doesn't help
> pushes, though - luckily, they are quite a bit more rare.

As a reference point, I can push a new change to our ~400K
ref kernel/msm repo in about 7s on a test server. So there
is lots that can be done to make things scale. I have tried
to share my knowldege in this area, but your specific cases
may need to be better understood.

Have you tried reading the scaling page
https://code.google.com/p/gerrit/wiki/Scaling?ts=1414115745&updated=Scaling
what things have you done? What things have not helped?
Where do you think improvements are still needed?

[Oswald Buddenhagen]

On Thu, Jul 09, 2015 at 03:06:16PM -0600, Martin Fick wrote:
> On Thursday, July 09, 2015 08:21:16 PM Oswald Buddenhagen wrote:
> > our repos aren't growing quite as fast, but after three
> > years, working with the qtbase repository is becoming
> > really painful as well (approaching 100k refs).
>
> Can you put some numbers to this pain?
>

pushing to qtbase just sits there for what feels like tens of seconds.
pulling of course also does - unlike when you use the thin mirror, when
it takes a second or two to do a small/zero update.
in contrast, when the repo was fresh in gerrit (not much smaller, but
very few refs), gerrit was fast, too.
operations on small (or even big, but inactive) repos continue to be no
problem at all, too.

the server is located "in the cloud", and our office network conection
positively sucks, which is definitely a contributing factor. but then,
this puts us on more equal footing with the outside contributors (and
home workers), who have only dsl or so (and it definitely feels even
slower from home).

> As a reference point, I can push a new change to our ~400K
> ref kernel/msm repo in about 7s on a test server.
>

and you don't see the problem there? 7s seems like quite a lot on what
is presumably a rather fast network.

> Have you tried reading the scaling page
> https://code.google.com/p/gerrit/wiki/Scaling?ts=1414115745&updated=Scaling
> what things have you done? What things have not helped?
>

any server tuning (beyond what i've already done) is utterly irrelevant
- the machine is positively *bored*. the bottleneck is the network
connection of the clients - which is no wonder, when every operation is
preceded by a multi-megabyte download of ref announcements. the git
protocol simply wasn't created with such an insane amount of refs in
mind.

[Lundh, Gustaf]

> and you don't see the problem there? 7s seems like quite a lot on what is presumably a rather fast network.

Quite a lot compared to what? kernel/msm.git is a huge git, with millions of objects. And with 400k branches and tags, I cannot imagine that many DVCS's handle a push or check in much faster than that. I mean, some graph connectivity checks and "which objects should I send"-negotiations should still be happening if we look past the annoying "here are all my refs"-handshake.

Best regards
Gustaf

[Lundh, Gustaf]

> As a reference point, I can push a new change to our ~400K ref kernel/msm repo in about 7s on a test server. So there is lots that can be done to make things scale. I have tried to share my knowldege in this area, but your specific cases may need to be better understood.

Have you tried experimenting with SSH compression to speed things up? Not sure if Mina SSHD handles it, but I just compared doing a "git show-ref | wc" (120k refs) over SSH, and the time was almost cut in half when turning on SSH Compression. Cannot replicate it when issuing a git push to Gerrit, so I'm guessing Mina SSHD may not support it.

Best regards
Gustaf

[Lundh, Gustaf]

Yeah. I see that SSH ZLib Compression is turned off in SSHDaemon.java (which makes sense, since the rest of the data of a typical push/fetch is already nicely packed). Could be fun to enable it and make some performance tests for smaller pushes though.

Best regards
Gustaf

-----Original Message-----

[Oswald Buddenhagen]

On Fri, Jul 10, 2015 at 03:00:27PM +0200, Lundh, Gustaf wrote:
> > and you don't see the problem there? 7s seems like quite a lot on
> > what is presumably a rather fast network.
>
> Quite a lot compared to what?
>

to what would feel reasonable for a very small operation.

> kernel/msm.git is a huge git, with millions of objects. And with 400k
> branches and tags, I cannot imagine that many DVCS's handle a push or
> check in much faster than that.
>

argument from incredulity. informal logical fallacy.

> I mean, some graph connectivity checks and "which objects should I
> send"-negotiations should still be happening if we look past the
> annoying "here are all my refs"-handshake.
>

the ref delta calculation itself takes a few milliseconds, the pack
calculation, upload, and delta resolution times are more or less
proportional to the size of the push (however defined, but usually
rather small), and the bookkeeping git does (creating/updating a ref,
mostly) may be even reasonably timed in microsecs (for gerrit certainly
in millisecs, due to evaluating the commits and updating/creating
changes from them). the bottom line is that typical pushes to gerrit are
very fast operations beyond the fixed cost of the handshake, which makes
the latter the elephant in the room.

we need to get creative. let's continue this in issue 175.

[Gustaf Lundh]

> the bottom line is that typical pushes to gerrit are very fast operations beyond the fixed cost of the handshake, which makes the latter the elephant in the room. 

There is still stuff that can be optimized. We suffered from extremely long push time, both to Gerrit and from Gerrit (replication). Never was the handshake our biggest problem, but most often connectivity and ACL calculations. The handshake is an issue, but not necessarily the biggest issue for us.

By minimizing refs used in Advertised Refs calculations we cut down push times from 42sec to 6.3 seconds [1]. By allowing us to "receive.checkReferencedObjectsAreReachable"[2] we could speed up things further. For replications this fix in JGit [3] made our replications 2-3 times faster. The point is, we measure and sample often, and I know there are still room for improvement besides transferring refs during the handshake. Especially if you have large gits with enormous amount of objects. Stating that all other calculations during a push are insignificant compared to a handshake is just wrong, at least in many, many installations and quite unfair.

Figuring out how to walk the graphs and marking objects as uninteresting is one of those things that could help many installations with gits like kernel/msm.

TDLR: Connectivity calculations are heavy. And extra heavy for Gerrit/JGit which needs to take ACLs into account. There are still room for improvements.

[1] https://gerrit-review.googlesource.com/#/c/51910/

[2] https://gerrit-review.googlesource.com/#/c/51910/

[3] https://git.eclipse.org/r/#/c/10307/

Best regards

Gustaf Lundh

[Martin Fick]

On Saturday, July 11, 2015 06:26:44 AM Gustaf Lundh wrote:
> Figuring out how to walk the graphs and marking objects as
> uninteresting is one of those things that could help many
> installations with gits like kernel/msm.
>
> TDLR: Connectivity calculations are heavy. And extra heavy
> for Gerrit/JGit which needs to take ACLs into account.
> There are still room for improvements.

Gustaf,

Have you considered using bitmaps for connectivity
calculations in jgit?

[Gustaf Lundh]

> Have you considered using bitmaps for connectivity calculations in jgit?

No, I have not really experimenting much with the bitmap support outside of the support that is already in place. Me and Sven tried briefly experimenting with building an ObjectWalk cache in JGit to get around reading and parsing TreeObjects when walking the graphs. While we could achieve great walking speeds, we could not build it a memory efficient way. And after this commit revert[1] was merged, we did not have the same burning need anymore to find a performance fix.

https://git.eclipse.org/r/#/c/46447/

Is this something you guys have been experimenting with?

/Gustaf

[Martin Fick]

On Monday, July 13, 2015 11:07:24 PM Gustaf Lundh wrote:
> > Have you considered using bitmaps for connectivity
> > calculations in jgit?
> No, I have not really experimenting much with the bitmap
> support outside of the support that is already in place.
> Me and Sven tried briefly experimenting with building an
> ObjectWalk cache in JGit to get around reading and
> parsing TreeObjects when walking the graphs. While we
> could achieve great walking speeds, we could not build it
> a memory efficient way. And after this commit revert[1]
> was merged, we did not have the same burning need anymore
> to find a performance fix.
>
> https://git.eclipse.org/r/#/c/46447/
>
> Is this something you guys have been experimenting with?

No, we are still on older versions, but would like to
upgrade to get that fix. I liked your caching series, but I
thought that if it was still too slow,that bitmaps would be
the right solution. It seems like it would take some work,
but it could likely be done. Maybe we need a jgit
performance hackathon! :)

Why does C git not suffer from this abismall peformance?

[Oswald Buddenhagen]

On Mon, Jul 13, 2015 at 03:43:03PM -0600, Martin Fick wrote:
> Why does C git not suffer from this abismall peformance?
>

it doesn't do per-object permission checks, so it doesn't have the need
to solve that problem in the first place.

i personally gave up on the issue an do access control on a repository
level (which implies a separate project (which is mostly a mirror) for
each group of branches with particular access rights). that's more
compatible with gitweb anyway ...