repo sync is slow, and gerrit is busy on doing com.google.gerrit.server.project.RefControl.shortestExample

174 views
Skip to first unread message

Makson Lee

unread,
Jun 30, 2016, 9:56:35 PM6/30/16
to Repo and Gerrit Discussion
here is some information from java thread dump, my gerrit version is 2.12.2, when server is busy, we find lots of this in thread dump, can you help to explain what are gerrit actually doing? is this something related to reference access controls?

"SSH git-upload-pack '/platform/packages/providers/CalendarProvider' (username)" #98 prio=1 os_prio=0 tid=0x00007f0bd002a000 nid=0x696 runnable [0x00007f0cd6bcb000]
   java.lang.Thread.State: RUNNABLE
        at java.util.AbstractCollection.containsAll(AbstractCollection.java:318)
        at java.util.AbstractSet.equals(AbstractSet.java:95)
        at java.util.HashMap.getNode(HashMap.java:571)
        at java.util.HashMap.containsKey(HashMap.java:595)
        at dk.brics.automaton.BasicOperations.determinize(Unknown Source)
        at dk.brics.automaton.BasicOperations.determinize(Unknown Source)
        at dk.brics.automaton.Automaton.determinize(Unknown Source)
        at dk.brics.automaton.MinimizationOperations.minimizeHopcroft(Unknown Source)
        at dk.brics.automaton.MinimizationOperations.minimize(Unknown Source)
        at dk.brics.automaton.Automaton.minimize(Unknown Source)
        at dk.brics.automaton.RegExp.toAutomaton(Unknown Source)
        at dk.brics.automaton.RegExp.findLeaves(Unknown Source)
        at dk.brics.automaton.RegExp.toAutomaton(Unknown Source)
        at dk.brics.automaton.RegExp.toAutomatonAllowMutate(Unknown Source)
        at dk.brics.automaton.RegExp.toAutomaton(Unknown Source)
        at com.google.gerrit.server.project.RefControl.shortestExample(RefControl.java:654)
        at com.google.gerrit.server.util.MostSpecificComparator.distance(MostSpecificComparator.java:86)
        at com.google.gerrit.server.util.MostSpecificComparator.compare(MostSpecificComparator.java:61)
        at com.google.gerrit.server.util.MostSpecificComparator.compare(MostSpecificComparator.java:57)
        at com.google.gerrit.server.util.MostSpecificComparator.compare(MostSpecificComparator.java:47)
        at java.util.TimSort.binarySort(TimSort.java:296)
        at java.util.TimSort.sort(TimSort.java:221)
        at java.util.Arrays.sort(Arrays.java:1512)
        at java.util.ArrayList.sort(ArrayList.java:1454)
        at java.util.Collections.sort(Collections.java:175)
        at com.google.gerrit.server.project.SectionSortCache.sort(SectionSortCache.java:87)
        at com.google.gerrit.server.project.PermissionCollection$Factory.filter(PermissionCollection.java:117)
        at com.google.gerrit.server.project.ProjectControl.controlForRef(ProjectControl.java:227)
        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:127)
        at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:169)
        at com.google.gerrit.server.git.VisibleRefFilter.getAdvertisedRefs(VisibleRefFilter.java:158)
        at org.eclipse.jgit.transport.AbstractAdvertiseRefsHook.advertiseRefs(AbstractAdvertiseRefsHook.java:62)
        at org.eclipse.jgit.transport.UploadPack.sendAdvertisedRefs(UploadPack.java:867)
        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:715)
        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:666)
        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:80)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:437)
        - locked <0x00000007bc29ea68> (a com.google.gerrit.sshd.BaseCommand$TaskThunk)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:377)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Shawn Pearce

unread,
Jun 30, 2016, 11:04:19 PM6/30/16
to Makson Lee, Repo and Gerrit Discussion
Yes. This is part of the support for regex access controls. To determine how access controls match we construct shortest examples and rank them by length of the examples.

You can try increasing the permission_sort cache's memoryLimit:

I also had noticed this was a (small) bottleneck in our own servers, so master has a change that adds a cache for the shortestExample results. You could try cherry-picking this into 2.12.2 and see if it helps your use if increasing the permission_sort cache size doesn't help.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Makson Lee

unread,
Jun 30, 2016, 11:30:02 PM6/30/16
to Repo and Gerrit Discussion, cdle...@gmail.com
thank you for the quick reply, i will try both solutions to see if performance is improved.

Marco Miller

unread,
Jul 6, 2016, 3:13:19 PM7/6/16
to Repo and Gerrit Discussion, cdle...@gmail.com
We'd also be interested in knowing how likely performance can improve only by tuning permission_sort.
Did you get a chance to try increasing permission_sort's memoryLimit and see if it helps, Makson?

We are basically preparing to run 2.12 soon.
We currently set that cache's memoryLimit to 1572864.
The proposed cherry-pick involves quite a few commits made of refactorings.
I'm wondering if they could have some negative impact on 2.12's stability(?)
Also, the minimal list of commits to cherry-pick is not clear to me yet.
-Any further thoughts on this overall are welcome; thx.

Makson Lee

unread,
Jul 6, 2016, 9:31:21 PM7/6/16
to Repo and Gerrit Discussion, cdle...@gmail.com
On Thursday, July 7, 2016 at 3:13:19 AM UTC+8, Marco Miller wrote:
We'd also be interested in knowing how likely performance can improve only by tuning permission_sort.
Did you get a chance to try increasing permission_sort's memoryLimit and see if it helps, Makson?
actually, i didn't not tune permission_sort.

We are basically preparing to run 2.12 soon.
We currently set that cache's memoryLimit to 1572864.
The proposed cherry-pick involves quite a few commits made of refactorings.
I'm wondering if they could have some negative impact on 2.12's stability(?)
Also, the minimal list of commits to cherry-pick is not clear to me yet.
-Any further thoughts on this overall are welcome; thx.
i just modified v2.12.2 codebase directly according to change 79077, it is easy, but do this by your own risk.
RefControl.java

Marco Miller

unread,
Jul 7, 2016, 1:06:24 PM7/7/16
to Repo and Gerrit Discussion, cdle...@gmail.com
Thx Makson for the feedback and code.
We may decide to stick to permission_sort until we get a bottleneck, if any.
Reply all
Reply to author
Forward
0 new messages