Slow performance after upgrading to 3.5.2

[Nguyen Tuan Khang Phan]

Changes takes significantly longer to open. ls-groups now take 10 times longer compared to 3.4. Offline group re-index went from 1 min on 3.4 to 2.5 hours on 3.5. Changing the cache config for persisted groups_byuuid_persisted, groups_external_persisted to 1 gb each fixed the problem of long ls-groups. Is there any other cache that we need to take into account that might improve performance?

Also, under /opt/gerrit/review/cache we found 2 big files:

git_modified_files of 300gb and gerrit_file_diff of around 300gb as well. Are they affecting the performance?

gerrit cache config:

[cache "conflicts"]
       memoryLimit = 1024m
       diskLimit = 3g

[cache "groups"]
       memoryLimit = 57344

[cache "groups_byinclude"]
       memoryLimit = 57344

[cache "groups_byname"]
       memoryLimit = 57344

[cache "groups_byuuid"]
       memoryLimit = 57344

[cache "groups_members"]
       memoryLimit = 57344

[cache "groups_external"]
       diskLimit = 1g

[cache "groups_external_persisted"]
       diskLimit = 1g

[cache "ldap_groups"]
       memoryLimit = 35000
       maxAge = 4h

[cache "ldap_groups_byinclude"]
       memoryLimit = 140000
       maxAge = 12h

[cache "ldap_usernames"]
       memoryLimit = 75000
       maxAge = 4h

[cache "mergeability"]
       memoryLimit = 1024m
       diskLimit = 3g

[cache "permission_sort"]
       memoryLimit = 1500000

[cache "plugin_resources"]
       memoryLimit = 132m

[cache "projects"]
       memoryLimit = 65536
       loadOnStartup = true
       checkFrequency = off

[cache "sshkeys"]
       memoryLimit = 75000

[cache "diff"]
       memoryLimit = 1024m
       diskLimit = 3g
       timeout = 3m

[Nasser Grainawi]

On Mon, Jul 11, 2022 at 10:26 AM Nguyen Tuan Khang Phan <phan....@gmail.com> wrote:

Changes takes significantly longer to open. ls-groups now take 10 times longer compared to 3.4. Offline group re-index went from 1 min on 3.4 to 2.5 hours on 3.5. Changing the cache config for persisted groups_byuuid_persisted, groups_external_persisted to 1 gb each fixed the problem of long ls-groups. Is there any other cache that we need to take into account that might improve performance?

The only thing we've adjusted with group caches (so far) was to set refreshAfterWrite to ~80% of the maxAge value we had for any cache (other than ones where we intentionally want to expire, like websessions). Otherwise we saw a big performance hit on the first request after cache expiration. It would be good if you could narrow down what's taking more time in 3.5.

 

Also, under /opt/gerrit/review/cache we found 2 big files:

git_modified_files of 300gb and gerrit_file_diff of around 300gb as well. Are they affecting the performance?

Hmm, I think there's a section missing from https://www.gerritcodereview.com/3.5.html#important-notes that should cover changes to the diff caches. Change 318335: Remove old diff cache | https://gerrit-review.googlesource.com/c/gerrit/+/318335 removed any use of the 'diff' cache config section and it was split+replaced with the *modified_files caches (see https://gerrit-documentation.storage.googleapis.com/Documentation/3.5.2/config-gerrit.html#cache_names).

We did see that Gerrit was taking a long time to build the BloomFilter for very large (100s of GB) caches, so we decided that we would only use those during offline reindex and will use empty/smaller caches post-upgrade on the running server.

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/62b9da35-516a-4ba4-bc15-c9e20efe0bc0n%40googlegroups.com.

[Nguyen Tuan Khang Phan]

The only thing we've adjusted with group caches (so far) was to set refreshAfterWrite to ~80% of the maxAge value we had for any cache (other than ones where we intentionally want to expire, like websessions). Otherwise we saw a big performance hit on the first request after cache expiration. It would be good if you could narrow down what's taking more time in 3.5.

We will test this first. However, the general slowness is still there. Changes take a lot more time to open. Using the load-tests suite we see a 50% degradation in performance compared to 2.14. Stable-3.4 was more performing for us.

Hmm, I think there's a section missing from https://www.gerritcodereview.com/3.5.html#important-notes that should cover changes to the diff caches. Change 318335: Remove old diff cache | https://gerrit-review.googlesource.com/c/gerrit/+/318335 removed any use of the 'diff' cache config section and it was split+replaced with the *modified_files caches (see https://gerrit-documentation.storage.googleapis.com/Documentation/3.5.2/config-gerrit.html#cache_names).

We did see that Gerrit was taking a long time to build the BloomFilter for very large (100s of GB) caches, so we decided that we would only use those during offline reindex and will use empty/smaller caches post-upgrade on the running server.

What is the BloomFilter? We also use mergeability, can that affect performance?