Gerrit service hang problem
seonguk.baek
Martin Fick
-Martin
"seonguk.baek" <baeks...@gmail.com> wrote:
>--
>To unsubscribe, email repo-discuss...@googlegroups.com
>More info at http://groups.google.com/group/repo-discuss?hl=en
Employee of Qualcomm Innovation Center,Inc. which is a member of Code Aurora Forum
seonguk.baek
2012년 3월 28일 수요일 오후 11시 19분 0초 UTC+9, MartinFick 님의 말:
>To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
>More info at http://groups.google.com/group/repo-discuss?hl=en
seonguk.baek
2012년 3월 28일 수요일 오후 11시 19분 0초 UTC+9, MartinFick 님의 말:
This sounds like java garbage collection. If your server is heavily loaded, which it sounds like it may be (2000%) CPU, it may simply be pausing for gc every now and then. I saw this a lot when I did performance testing with many large long lasting clones. Can you check your actual java heap usage using show-caches? If this is the case, you could try the concurrent garbage collector, but my experience with that was limited. If load is indeed your issue, your best bet is likely to reduce your load by offloafing much of your git traffic to slaves.-Martin
"seonguk.baek" <baeks...@gmail.com> wrote:
>To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
>More info at http://groups.google.com/group/repo-discuss?hl=en
Matthias Sohn
--
To unsubscribe, email repo-discuss...@googlegroups.com
Anatol Pomazau
Dear Martin.i''ve another question.What is concurrent garbage collector. and how to run it?
--
To unsubscribe, email repo-discuss...@googlegroups.com
seonguk.baek
is that a reason why gc need many time because large size of java heap memory(36GB)?
2012년 3월 28일 수요일 오후 11시 19분 0초 UTC+9, MartinFick 님의 말:
This sounds like java garbage collection. If your server is heavily loaded, which it sounds like it may be (2000%) CPU, it may simply be pausing for gc every now and then. I saw this a lot when I did performance testing with many large long lasting clones. Can you check your actual java heap usage using show-caches? If this is the case, you could try the concurrent garbage collector, but my experience with that was limited. If load is indeed your issue, your best bet is likely to reduce your load by offloafing much of your git traffic to slaves.-Martin
"seonguk.baek" <baeks...@gmail.com> wrote:
>To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
>More info at http://groups.google.com/group/repo-discuss?hl=en
seonguk.baek
2012년 3월 29일 목요일 오전 9시 7분 5초 UTC+9, Anatol Pomazau 님의 말:
Martin Fick
>
> As you said, cpu is paused when full gc is running.
>
> 1. So I applied openjdk option "-Xincgc -server" is it
> correct?
We don't use it yet, so I am not sure what works. Since you
seem to have an incredibly loaded server, please report back
with what works for you.
> 2. I gave heapLimit 100G and pactedGitLimit 64GB. that
> mean java is using 36GB.
> is that a reason why gc need many time because large
> size of java heap memory(36GB)?
I really don't know, but I suspect it has to do with the
fact that you have so many threads do uploads and downloads
and that as soon as those complete on large repos, the gc
will attempt to reclaim much of that data.
-Martin
--
Employee of Qualcomm Innovation Center, Inc. which is a
seonguk.baek
- registered users : 3000
2012년 3월 29일 목요일 오전 11시 12분 3초 UTC+9, MartinFick 님의 말:
Martin Fick
"seonguk.baek" <baeks...@gmail.com> wrote:
>1. information
>- registered users : 3000
>- concurrent users : about 400
>- download(repo sync) : 90% / upload(repo upload, git push) : 10%
>- total number of git repositories : 998 gits
>- size of total git repositories : 47GB
That is a huge load, you really should consider using git mirrors for your download traffic, that is what other large sites do. Since this is 90% of your load, it will likely fix your problem.
-Martin
Employee of Qualcomm Innovation Center,Inc. which is a member of Code Aurora Forum
Luciano Carvalho
We have a traffic and load heavier than that, and use several mirrors for downloads.
Gerrit runs smoothly, no hangs. You'll still have some outgoing load for the replications but it doesn't compare to hundreds of users syncing simultaneously.
--
To unsubscribe, email repo-discuss...@googlegroups.com
seonguk.baek
2012년 3월 29일 목요일 오후 12시 59분 38초 UTC+9, MartinFick 님의 말:
Martin Fick
"seonguk.baek" <baeks...@gmail.com> wrote:
>Hello. Martin.
>
>We set 3 slave servers for git project as you mentioned.
>So, normally 150 clients connected to one server.
>
>But cpu paused(hang) problem is still occurred which is gerrit 2.2.2.1
>version is installed one.
>This is not occurred with gerrit 2.1.8 version.
>Did you see this kind of problem before?
I saw this problem with 2.1.8 when doing 100 simultaneous clones of msm.
Martin Fick
Martin Fick <mf...@codeaurora.org> wrote:
>"seonguk.baek" <baeks...@gmail.com> wrote:
>
>>Hello. Martin.
>>
>>We set 3 slave servers for git project as you mentioned.
>>So, normally 150 clients connected to one server.
>>
>>But cpu paused(hang) problem is still occurred which is gerrit 2.2.2.1
>
>>version is installed one.
>>This is not occurred with gerrit 2.1.8 version.
>>Did you see this kind of problem before?
>
>I saw this problem with 2.1.8 when doing 100 simultaneous clones of
>msm.
I should add, consistently, which made it a good test case to attempt to fix it, I could not (yet).
seonguk.baek
Our load balancing is set 3 slave servers by replication.
2012년 3월 30일 금요일 오후 12시 19분 46초 UTC+9, MartinFick 님의 말:
Martin Fick
"seonguk.baek" <baeks...@gmail.com> wrote:
>You mean that gerrit 2.1.8 and gerrit 2.2.2.1 version have a same
>problem?
Yes, I think it is due to large allocations being freed all at once, but I don't really know.
>And currently it's impossible to fix it now? right?
I don't know of a fix besides reducing your load. Another way I reduced our load was to limit the ssh threads, I believe we keep our thread count much lower than you do, but I would have to check. I suggest that you set up some load tests and determine what your servers can handle and then set your thread limits accordingly. Since I noticed clones being very hard on the gc, I have wondered if it would make sense to create a separate thread pool in Gerrit for them?
I am picturing a config option which allows a separate thread pool to be defined for any specific ssh/git commanf, perhaps even allowing specific additional parameters such as project, ref, user... to be used to define such thread pools. This way it would be possible for wise admins to limit things with more fine grain. If this problem is serious enough for you, perhaps this is something you would be willing to attack?
>Our load balancing is set 3 slave servers by replication.
>Is this correct load balancing which you mentioned before?
I guess that all depends on how satisified you are with your current results? We probably only have 1/3 of your devs (but a lot more projects, 1000+), and we now have around 7 slaves.
Please, may I ask that you trim your replies (leave only the content you are replying to) and that you stop top posting (inline your replies). Thanks, since this list has a wide audience, a bit of work on your side may save a lot of work for others (and it will probably also get more people reading your posts),
-Martin
Luciano Carvalho
On Mar 29, 2012 10:39 PM, "seonguk.baek" <baeks...@gmail.com> wrote:
>
> You mean that gerrit 2.1.8 and gerrit 2.2.2.1 version have a same problem?
> And currently it's impossible to fix it now? right?
> Then how can we resolve this problem by workaround or some tip?
>
> Our load balancing is set 3 slave servers by replication.
> Is this correct load balancing which you mentioned before?
I'd also make sure you don't have any more git-upload-pack in the queue. You have to assure your users will sync from the mirrors. The way to force it is to set it on gerrit.config ( don't remember the setting, you'll have to take a look at the documentation)
Luciano.
>
> Thanks
>
>
> 2012년 3월 30일 금요일 오후 12시 19분 46초 UTC+9, MartinFick 님의 말:
>>
>> Martin Fick <mf...@codeaurora.org> wrote:
>> >"seonguk.baek" <baeks...@gmail.com> wrote:
>> >
>> >>Hello. Martin.
>> >>
>> >>We set 3 slave servers for git project as you mentioned.
>> >>So, normally 150 clients connected to one server.
>> >>
>> >>But cpu paused(hang) problem is still occurred which is gerrit 2.2.2.1
>> >
>> >>version is installed one.
>> >>This is not occurred with gerrit 2.1.8 version.
>> >>Did you see this kind of problem before?
>> >
>> >I saw this problem with 2.1.8 when doing 100 simultaneous clones of
>> >msm.
>>
>> I should add, consistently, which made it a good test case to attempt to fix it, I could not (yet).
>>
>> Employee of Qualcomm Innovation Center,Inc. which is a member of Code Aurora Forum
>
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
Sent from my Motorola RAZR
seonguk.baek
Since I added three slave servers, I got a little improvement. So I am
thinking to add more servers for better performance.
Thanks.
P.S) I will trim my comments after this as you pointed out.
Martin Fick
> Thanks for your reply :-)
>
> Since I added three slave servers, I got a little
> improvement. So I am thinking to add more servers for
> better performance.
You also may consider applying this patch to jgit:
https://git.eclipse.org/r/#/c/5491/1/org.eclipse.jgit/src/org/eclipse/jgit/revwalk/DateRevQueue.java
We have a similar patch running internally and it
drastically reduces the load which our larger repositories
put on our gerrit server. If you have any repositories with
a large amount of changes and tags in them, I suspect that
anything over 50K would be a lot, we have 100K and it was
really bad until we applied this patch. With less load,
your user requests will end quicker and your total memory
load should be reduced.
If any of your repos are that large, I would also consider
applying these git patches by René Scharfe:
http://marc.info/?l=git&m=133323194014727&w=2
http://marc.info/?l=git&m=133323194314728&w=2
http://marc.info/?l=git&m=133323763515996&w=2
to the git running on your mirrors. Without them, you will
also be holding memory resources for much longer than you
need to on your main Gerrit server since the git receiving
end will take much longer to process any Gerrit pushes.
These might help reduce your hardware needs somewhat, they
have done miracles for us,
-Martin
--
Employee of Qualcomm Innovation Center, Inc. which is a
seonguk.baek
As you said, I have pushing hang problems.
Can I use JGIT latest version instead of applying above patch? it's
JGIT 1.3.0.201202151440-r.
Was that patch applied in JGIT 1.3?
But I saw a post that Shawn found Auto CRLF bug in 1.3.
http://www.google.co.kr/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=4&ved=0CEgQFjAD&url=http%3A%2F%2Fgroups.google.com%2Fgroup%2Frepo-discuss%2Fbrowse_thread%2Fthread%2F641fccc69ae17b16&ei=Jrh-T8qKDaiXiQfnv6zPBA&usg=AFQjCNFdwKdm4ixXs5o0xdLatpmxntn3Mw&sig2=ZBaZNxW1Qo1KWMLFYvoc4Q
was that patch applied in above 1.3 version also?
Thank you.
Martin Fick
"seonguk.baek" <baeks...@gmail.com> wrote:
>> You also may consider applying this patch to jgit:
>
>>
>https://git.eclipse.org/r/#/c/5491/1/org.eclipse.jgit/src/org/eclipse...
>
>As you said, I have pushing hang problems.
>
>Can I use JGIT latest version instead of applying above patch? it's
>JGIT 1.3.0.201202151440-r.
>Was that patch applied in JGIT 1.3?
I don't believe that patch was applied to any version of jgit, and probably will not be as is.
>But I saw a post that Shawn found Auto CRLF bug in 1.3.
>
>http://www.google.co.kr/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=4&ved=0CEgQFjAD&url=http%3A%2F%2Fgroups.google.com%2Fgroup%2Frepo-discuss%2Fbrowse_thread%2Fthread%2F641fccc69ae17b16&ei=Jrh-T8qKDaiXiQfnv6zPBA&usg=AFQjCNFdwKdm4ixXs5o0xdLatpmxntn3Mw&sig2=ZBaZNxW1Qo1KWMLFYvoc4Q
>
>was that patch applied in above 1.3 version also?
I don't know how to look that up/verify that, sorry,
-Martin
Employee of Qualcomm Innovation Center,Inc. which is a member of Code Aurora Forum