Browse > Projects very slow
232 views
Skip to first unread message
Marcelo Ávila de Oliveira
Jun 8, 2018, 5:20:54 PM6/8/18
to Repo and Gerrit Discussion
We just upgraded to Gerrit 2.15.2 and the "Browse > Projects" page is very slow to load showing "Loading..." for minutes. The server has ~700 projects. Is it expected? Is there something we can check or do to improve the performance? Is Lucene used in this search?
PS: I know we can speed up using the "Filter" field but a lot of people still use the old UI where the "Filter" is only showed at the end.
Thanks a lot.
--
Marcelo Ávila de Oliveira
Luca Milanesio
Jun 8, 2018, 5:29:49 PM6/8/18
to Marcelo Ávila de Oliveira, Luca Milanesio, Repo and Gerrit Discussion
On 8 Jun 2018, at 22:20, Marcelo Ávila de Oliveira <mav...@cpqd.com.br> wrote:We just upgraded to Gerrit 2.15.2 and the "Browse > Projects" page is very slow to load showing "Loading..." for minutes. The server has ~700 projects. Is it expected? Is there something we can check or do to improve the performance? Is Lucene used in this search?
The project list is not indexed in Lucene. However, it is cached and should be a lot faster.
What is the output of the 'gerrit show-caches' when the listing is slow?
Do you see any differences in performance when you are logged in or not?
PS: I know we can speed up using the "Filter" field but a lot of people still use the old UI where the "Filter" is only showed at the end.Thanks a lot.--Marcelo Ávila de Oliveira
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Gert van Dijk
Jun 9, 2018, 4:49:02 AM6/9/18
to Repo and Gerrit Discussion
On Friday, 8 June 2018 23:20:54 UTC+2, Marcelo Ávila de Oliveira wrote:
We just upgraded to Gerrit 2.15.2 and the "Browse > Projects" page is very slow to load showing "Loading..." for minutes. The server has ~700 projects. Is it expected? Is there something we can check or do to improve the performance? Is Lucene used in this search?
I guess in the situation with Gerrit 2.14, your (ReviewDb) database was quick to respond without tweaking a cache. Now that this data comes from disk/filesystem directly with NoteDb, it could be required to tweak some caches as Luca indicated. I think you might want to take a look at the caches configuration and tweak the 'project_list' and 'projects' caches, then watch the cache states with the show-caches command as Luca also pointed out already.
On a side note, I noticed 'project_list' is not mentioned in the documentation, but it is with 'show-caches'. Additionally, the documentation still assumes the use of ReviewDb. I/we might want to update it.
Makson Lee
Jun 9, 2018, 4:55:37 AM6/9/18
to Repo and Gerrit Discussion
we have same situation after upgrade from 2.14.4 to 2.15.2.
David Pursehouse
Jun 9, 2018, 6:26:39 AM6/9/18
to Luca Milanesio, Marcelo Ávila de Oliveira, Repo and Gerrit Discussion
On Sat, Jun 9, 2018 at 6:29 AM Luca Milanesio <luca.mi...@gmail.com> wrote:
On 8 Jun 2018, at 22:20, Marcelo Ávila de Oliveira <mav...@cpqd.com.br> wrote:We just upgraded to Gerrit 2.15.2 and the "Browse > Projects" page is very slow to load showing "Loading..." for minutes. The server has ~700 projects. Is it expected? Is there something we can check or do to improve the performance? Is Lucene used in this search?The project list is not indexed in Lucene.
...in 2.15. A project index was later added and will be in 2.16/3.0
Marcelo Ávila de Oliveira
Jun 11, 2018, 9:11:24 AM6/11/18
to Repo and Gerrit Discussion
Thanks Luca, Gert and David.
My "gerrit show-caches" output is:
Name |Entries | AvgGet |Hit Ratio|
| Mem Disk Space| |Mem Disk|
--------------------------------+---------------------+---------+---------+
...
project_list | 1 | 38,8ms | 99% |
projects | 709 | 1,3ms | 99% |
...
SSH: 14 users, oldest session started 0 ms ago
Tasks: 2 total = 1 running + 0 ready + 1 sleeping
Mem: 2,13g total = 1,12g used + 1,00g free + 3,23m buffers
3,56g max
128 open files
I think the "projects" cache is ok (we have 709 projects), correct? What about the "project_list" one?
The output doesn't change if I'm trying to access "Browse > Projects" or not.
I added "cache.projects.maxAge = 0" to try force "no expire" to the cache (this should be the default, right?) and then I got the following:
project_list | 1 | 28,7ms | 95% |
projects | | 1,5ms | 0% |
It seems that the cache was desactivated...
Then I set "cache.projects.maxAge = 1w" and I got:
project_list | 1 | 52,4ms | 87% |
projects | 709 | 1,2ms | 87% |
thomasmu...@yahoo.com
Jun 11, 2018, 9:23:00 AM6/11/18
to Repo and Gerrit Discussion
This is fast for us @ the wmf https://gerrit.wikimedia.org/r/admin/projects
Hugo Arès
Jun 11, 2018, 9:28:15 AM6/11/18
to Repo and Gerrit Discussion
First, you should not set cache.projects.maxAge. Gerrit will invalidate projects when needed
and maintain project_list up to date with project creation/deletion, no need to have expiration.
You can take a few thread dumps (jstack -l <pid>, or use javamelody threads page) while your
list project request is happening, this should give you an idea what is happening.
Luca Milanesio
Jun 11, 2018, 9:32:10 AM6/11/18
to Repo and Gerrit Discussion, Luca Milanesio, Paladox
On 11 Jun 2018, at 14:23, thomasmulhall410 via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:This is fast for us @ the wmf https://gerrit.wikimedia.org/r/admin/projects
https://review.gerrithub.io/admin/projects is "relatively fast" as well for us:
- Page render time: 4s
- Projects list JSON call: 700 msec
I have to say that when the cache is suffering or not fully loaded, then it may take over 30s or even timeout :-O
Our cache size is https://github.com/wikimedia/puppet/blob/production/modules/gerrit/templates/gerrit.config.erb#L14
On Friday, June 8, 2018 at 10:20:54 PM UTC+1, Marcelo Ávila de Oliveira wrote:We just upgraded to Gerrit 2.15.2 and the "Browse > Projects" page is very slow to load showing "Loading..." for minutes.
Can you run the page refresh with the Chrome network inspector and send the results?
Have you tried to call the Project List API to see what's the net time?
The server has ~700 projects. Is it expected? Is there something we can check or do to improve the performance? Is Lucene used in this search?PS: I know we can speed up using the "Filter" field but a lot of people still use the old UI where the "Filter" is only showed at the end.Thanks a lot.--Marcelo Ávila de Oliveira
Marcelo Ávila de Oliveira
Jun 11, 2018, 9:32:41 AM6/11/18
to thomasmu...@yahoo.com, Repo and Gerrit Discussion
Em seg, 11 de jun de 2018 às 10:23, thomasmulhall410 via Repo and Gerrit Discussion <repo-d...@googlegroups.com> escreveu:
This is fast for us @ the wmf https://gerrit.wikimedia.org/r/admin/projects
No... your project list loads in a second!!! My project list takes 2 or 3 minutes to load...
Marcelo Ávila de Oliveira
Jun 11, 2018, 10:01:07 AM6/11/18
to Luca Milanesio, Repo and Gerrit Discussion, thomasmu...@yahoo.com
Em seg, 11 de jun de 2018 às 10:32, Luca Milanesio <luca.mi...@gmail.com> escreveu:
Can you run the page refresh with the Chrome network inspector and send the results?Have you tried to call the Project List API to see what's the net time?
$ time curl -s --request GET --netrc $REST_URL/projects/?d
...
real 3m15,917s
user 0m0,031s
sys 0m0,030s
The Chrome network inspector showed that the page spent 3.8 min waiting (TTFB) to https://gerrit.cpqd.com.br/projects/?d&n=26&S=0 GET request.
Luca Milanesio
Jun 11, 2018, 10:04:50 AM6/11/18
to Marcelo Ávila de Oliveira, Luca Milanesio, Repo and Gerrit Discussion, thomasmu...@yahoo.com
can you repeat by including the 'n and S' parameters? (remember to include the URL within single quotes)
The GUI should only load the first N of the first page.
Luca.
Luca Milanesio
Jun 11, 2018, 10:08:32 AM6/11/18
to Marcelo Ávila de Oliveira, Luca Milanesio, Repo and Gerrit Discussion, thomasmu...@yahoo.com
Ah, OK. Then it is really something to investigate on the API execution time on the backend.
Luca.
Luca Milanesio
Jun 11, 2018, 10:14:32 AM6/11/18
to Marcelo Ávila de Oliveira, Luca Milanesio, Repo and Gerrit Discussion, thomasmu...@yahoo.com
The only thing that it could then slowdown the listing is the permissions evaluations.
Which authentication mechanism you use? Where are the groups resolved?
Luca.
Marcelo Ávila de Oliveira
Jun 11, 2018, 10:20:40 AM6/11/18
to Luca Milanesio, Repo and Gerrit Discussion, thomasmu...@yahoo.com
Em seg, 11 de jun de 2018 às 11:14, Luca Milanesio <luca.mi...@gmail.com> escreveu:
The only thing that it could then slowdown the listing is the permissions evaluations.Which authentication mechanism you use? Where are the groups resolved?
Just to be clear:
$ time curl -s --request GET --netrc "$REST_URL/projects/?d&n=26&S=0"
...
real 3m41,191s
user 0m0,031s
sys 0m0,007s
We use LDAP. Sorry, I'm not sure if I understood the "where are the groups resolved"? question... we're using noteDB for accounts.
Luca Milanesio
Jun 11, 2018, 10:23:01 AM6/11/18
to Marcelo Ávila de Oliveira, Luca Milanesio, Repo and Gerrit Discussion, thomasmu...@yahoo.com
Groups are still resolved against LDAP anyway, even with NoteDb.
Do you typically use ldap groups for authentication?
It could be an issue with LDAP groups and the new authorization backend.Luca.
Luca Milanesio
Jun 11, 2018, 10:41:14 AM6/11/18
to Marcelo Ávila de Oliveira, Luca Milanesio, Repo and Gerrit Discussion, thomasmu...@yahoo.com
Typo: I meant for authorization in the Projects' ACLs
Marcelo Ávila de Oliveira
Jun 11, 2018, 12:55:22 PM6/11/18
to Repo and Gerrit Discussion
Em seg, 11 de jun de 2018 às 11:41, Luca Milanesio <luca.mi...@gmail.com> escreveu:
Do you typically use ldap groups for authentication?Typo: I meant for authorization in the Projects' ACLsIt could be an issue with LDAP groups and the new authorization backend.
Projects ACLs don't use LDAP groups directly but some intern groups (used in the project ACLs) include LDAP groups. In my case, I'm member of an intern group (admin) which has access to all projects.
BTW the same issue happens with Browse > Groups:
$ time curl -s --request GET --netrc "$REST_URL/groups/?n=26&S=0"
...
real 3m53,324s
user 0m0,048s
sys 0m0,048s
thomasmu...@yahoo.com
Jun 11, 2018, 1:09:13 PM6/11/18
to Repo and Gerrit Discussion
Groups are slow for us too (really slow). though the cache is unlimited it is very slow.
On Friday, June 8, 2018 at 10:20:54 PM UTC+1, Marcelo Ávila de Oliveira wrote:
Luca Milanesio
Jun 11, 2018, 5:21:37 PM6/11/18
to Marcelo Ávila de Oliveira, Luca Milanesio, Repo and Gerrit Discussion
The numbers are comparable => I am inclined to think that the problem is more related to the permissions resolution through ACLs.
Luca.
Marcelo Ávila de Oliveira
Jun 12, 2018, 9:06:17 AM6/12/18
to Repo and Gerrit Discussion
I can't see changes and groups cache info:
Name |Entries | AvgGet |Hit Ratio|
| Mem Disk Space| |Mem Disk|
--------------------------------+---------------------+---------+---------+
accounts | 101 | 6,6ms | 99% |
adv_bases | | | |
change_notes | 201 | 1,4ms | 98% |
changeid_project | 22 | | 82% |
changes | | | |
groups | | | |
groups_bymember | 20 | 1,9s | 96% |
groups_byname | | | |
groups_bysubgroup | 1024 | 34,4ms | 92% |
groups_byuuid | 1024 | 105,1ms | 84% |
groups_external | 1 | 124,3ms | 99% |
groups_subgroups | | | |
ldap_group_existence | | | |
ldap_groups | 38 | 9,5ms | 98% |
ldap_groups_byinclude | 195 | | 95% |
ldap_usernames | 31 | 482,5us | 73% |
permission_sort | 176 | | 99% |
plugin_resources | 17 | | 93% |
project_list | 1 | 136,4ms | 99% |
projects | 709 | 864,0us | 98% |
...
Is this expected?
--
Luca Milanesio
Jun 12, 2018, 9:51:08 AM6/12/18
to Marcelo Ávila de Oliveira, Luca Milanesio, Repo and Gerrit Discussion
Yeah, you've problems with your cache settings.
See my feedback below on what's wrong.
On 12 Jun 2018, at 14:05, Marcelo Ávila de Oliveira <mav...@cpqd.com.br> wrote:I can't see changes and groups cache info:Name |Entries | AvgGet |Hit Ratio|| Mem Disk Space| |Mem Disk|--------------------------------+---------------------+---------+---------+accounts | 101 | 6,6ms | 99% |adv_bases | | | |change_notes | 201 | 1,4ms | 98% |changeid_project | 22 | | 82% |changes | | | |
This is normal: changes cache is disabled by default.
groups | | | |
This means that this cache is never used.
groups_bymember | 20 | 1,9s | 96% |groups_byname | | | |groups_bysubgroup | 1024 | 34,4ms | 92% |
You have overflowed the resolution of groups by subgroup, which possibly involves LDAP queries for you.
groups_byuuid | 1024 | 105,1ms | 84% |
Same here.
groups_external | 1 | 124,3ms | 99% |groups_subgroups | | | |ldap_group_existence | | | |ldap_groups | 38 | 9,5ms | 98% |ldap_groups_byinclude | 195 | | 95% |ldap_usernames | 31 | 482,5us | 73% |permission_sort | 176 | | 99% |plugin_resources | 17 | | 93% |project_list | 1 | 136,4ms | 99% |projects | 709 | 864,0us | 98% |...Is this expected?
Add the following lines to your gerrit.config
[cache "groups_bysubgroup"]
memoryLimit = 2048
[cache "groups_byuuid"]
memoryLimit = 2048
P.S. Whilst the cache is "cold", it is absolutely normal that the listings are taking a very long time. Once the cache is loaded, the lists should be pretty quick. For 700+ repos should be less than 1s IMHO.
--Marcelo Ávila de Oliveira
thomasmu...@yahoo.com
Jun 12, 2018, 10:49:17 AM6/12/18
to Repo and Gerrit Discussion
Apparently in 2.15 it does not set defaults for "groups" https://gerrit.wikimedia.org/r/Documentation/config-gerrit.html#cache
compared to what the master docs say.
On Friday, June 8, 2018 at 10:20:54 PM UTC+1, Marcelo Ávila de Oliveira wrote:
Marcelo Ávila de Oliveira
Jun 12, 2018, 2:37:55 PM6/12/18
to Repo and Gerrit Discussion
Em ter, 12 de jun de 2018 às 10:51, Luca Milanesio <luca.mi...@gmail.com> escreveu:
Add the following lines to your gerrit.config[cache "groups_bysubgroup"]memoryLimit = 2048[cache "groups_byuuid"]memoryLimit = 2048
I did...
$ ... show-caches | grep group
groups | 2 | 1,7ms | 50% |
groups_bymember | 14 | 535,0ms | 94% |
groups_byname | | | |
groups_bysubgroup | 1142 | 12,8ms | 97% |
groups_byuuid | 2047 | 110,8ms | 73% |
groups_external | 1 | 63,4ms | 99% |
groups_subgroups | 3 | 72,6ms | 62% |
ldap_group_existence | | | |
ldap_groups | 46 | 11,6ms | 98% |
ldap_groups_byinclude | 226 | | 94% |
But the performance didn't change so much:
$ time curl ... GET "$REST_URL/projects/?d&n=1"
...
real 2m50,643s
It's interesting how the use of the "m=" parameter speed up the process:
$ time curl ... GET "$REST_URL/projects/?d&n=1&m=cod"
...
real 0m0,816s
Why is it faster to search for a specific project instead of just show the first one?
Gert van Dijk
Jun 12, 2018, 3:06:08 PM6/12/18
to Repo and Gerrit Discussion
On Tuesday, 12 June 2018 20:37:55 UTC+2, Marcelo Ávila de Oliveira wrote:
Em ter, 12 de jun de 2018 às 10:51, Luca Milanesio <luca.mi...@gmail.com> escreveu:
[cache "groups_bysubgroup"]memoryLimit = 2048[cache "groups_byuuid"]memoryLimit = 2048
[...]groups_byuuid | 2047 | 110,8ms | 73% |[...]
You're hitting the ceiling again (2048), I guess. Try raising this cache limit further until it doesn't reach its limit any longer (as Luca was trying to point out).
It's interesting how the use of the "m=" parameter speed up the process:$ time curl ... GET "$REST_URL/projects/?d&n=1&m=cod"...real 0m0,816sWhy is it faster to search for a specific project instead of just show the first one?
(Doing an educated *guess* here) I think limiting the list to a part of the name of the project will significantly reduce the amount of projects for which it has to look up the effective permissions for you (whether it might show it to you or not in the list). This could potentially be an expensive lookup with several levels of ACL, external groups, etc. If that has to be preformed in combination with a cache is overflowing (on groups in your case) then I am not surprised this can take minutes, really. And for Gerrit to know what the first project *for you* is, it performs all expensive lookups first and then order the results.
Marcelo Ávila de Oliveira
Jun 12, 2018, 3:51:54 PM6/12/18
to Repo and Gerrit Discussion
Em ter, 12 de jun de 2018 às 16:06, Gert van Dijk <gert...@gmail.com> escreveu:
You're hitting the ceiling again (2048), I guess. Try raising this cache limit further until it doesn't reach its limit any longer (as Luca was trying to point out).
I thought 2047 was not the ceiling yet... you were right:
$ ... show-caches | grep group
groups | | | |
groups_bymember | 5 | 184,1ms | 93% |
groups_byname | | | |
groups_bysubgroup | 288 | 49,1ms | 96% |
groups_byuuid | 2140 | 112,3ms | 93% |
groups_external | 1 | 107,7ms | 98% |
groups_subgroups | | | |
ldap_group_existence | | | |
ldap_groups | 26 | 15,0ms | 97% |
ldap_groups_byinclude | 215 | | 88% |
And the best of all... the issue was totally solved!!!
$ time curl ... GET "$REST_URL/projects/?d&n=1"
...
real 0m0,879s
$ time curl ... GET "$REST_URL/projects/?d"
...
real 0m1,476s
$ time curl ... GET "$REST_URL/groups/"
...
real 0m2,807s
This is really great!!!
Thanks a lot for the help (and patience) Gert, Luca, Thomas, David and Hugo.
Luca Milanesio
Jun 12, 2018, 4:13:29 PM6/12/18
to Marcelo Ávila de Oliveira, Luca Milanesio, Repo and Gerrit Discussion
I believe you guys have *a lot* of groups ... that slows down all the security filtering operations.
Until the cache is fully loaded, lists will take forever :-(
On GerritHub.io we consider a host "unhealthy" if the cache is not fully loaded, especially the projects and groups.
For us, the consequences of having a REST API taking minutes is an almost certain outage, due to the accumulation of incoming calls.
We prefer to take the node out of the load balancing and wait for it to be fully cache-loaded.
Luca.
Hugo Arès
Jun 12, 2018, 8:26:42 PM6/12/18
to Repo and Gerrit Discussion
On Tuesday, June 12, 2018 at 4:13:29 PM UTC-4, lucamilanesio wrote:
I believe you guys have *a lot* of groups ... that slows down all the security filtering operations.Until the cache is fully loaded, lists will take forever :-(On GerritHub.io we consider a host "unhealthy" if the cache is not fully loaded, especially the projects and groups.For us, the consequences of having a REST API taking minutes is an almost certain outage, due to the accumulation of incoming calls.
Same here, we never put a node in the load balancing before warming the projects and groups caches.
You should also consider periodically looking at your caches usage to make sure they are big enoungh to hold all the entries.
Luca.
To unsubscribe, email repo-disc...@googlegroups.com
Gert van Dijk
Jun 13, 2018, 2:43:09 AM6/13/18
to Repo and Gerrit Discussion
On Wednesday, 13 June 2018 02:26:42 UTC+2, Hugo Arès wrote:
You should also consider periodically looking at your caches usage to make sure they are big enoungh to hold all the entries.
It would help if Gerrit would show not only the absolute amount of entries that are being in use, but also the relative usage. E.g. "100%" somewhere in that table, in the case of Marcelo. That would be an easier to spot flag when running into these kind of issues. (Or maybe I'm missing something here.)
Luca Milanesio
Jun 13, 2018, 3:20:53 AM6/13/18
to Gert van Dijk, Luca Milanesio, Repo and Gerrit Discussion
That number is in there (on the right side) but represent the cache hit and not the used vs. total.
A cache hit of 100% is good, a cache use of 100% of its capacity is not so good.
What about putting a number vs. total?
Example:
Name |Entries | AvgGet |Hit Ratio|
| Mem Disk Space| |Mem Disk|
--------------------------------+--------------------------+---------+---------+
accounts | 263/1024 | 73.1ms | 64% |
vista...@gmail.com
Jun 15, 2018, 2:25:02 AM6/15/18
to Repo and Gerrit Discussion
Hi all:
I have same situation in gerrit 2.13.7.
My "show-caches" output is:
Name |Entries | AvgGet |Hit Ratio|
| Mem Disk Space| |Mem Disk|
--------------------------------+---------------------+---------+---------+
accounts | 3 | 281.8ms | 92% |
accounts_byemail | | | |
accounts_byname | 3 | | |
adv_bases | | | |
change_notes | | | |
changes | | | |
groups | 197 | 5.4ms | 12% |
groups_byinclude | | | |
groups_byname | | | |
groups_byuuid | | | |
groups_external | | | |
groups_members | 2068 | 5.9ms | 39% |
permission_sort | 4 | | 93% |
plugin_resources | | | |
project_list | 1 | 182.1ms | 80% |
projects | 10465 | 6.1ms | 89% |
sshkeys | 2 | 32.7ms | 50% |
please help!
Thanks!
Luca.
To unsubscribe, email repo-disc...@googlegroups.com
vista...@gmail.com
Jun 15, 2018, 3:06:16 AM6/15/18
to Repo and Gerrit Discussion
OK
My issue was solved!
My issue was solved!
lucamilanesio
Jan 24, 2019, 7:11:13 PM1/24/19
to Repo and Gerrit Discussion
On Saturday, June 9, 2018 at 9:55:37 AM UTC+1, Makson Lee wrote:
we have same situation after upgrade from 2.14.4 to 2.15.2.
I have finally found out that this was a *specific regression* in v2.15: instead of keeping the list of projects as a "virtual iterator" the list was completely materialized in memory, causing two problems:
a) The listing was super-slow, even when extracting a single project
b) The JVM Heap utilization was very high, because of the full scanning of all projects due to the early materialization of the iterator
This has now been fixed in both v2.15.9 and v2.16.4 ... apologies for being stubborn on this and have not looked at the code more carefully :-(
Upgrade now and you won't need anymore to have the *full list of projects* in cache, it will be fast also when the cache is almost empty.
HTH
Luca.
On Saturday, June 9, 2018 at 5:20:54 AM UTC+8, Marcelo Ávila de Oliveira wrote:
We just upgraded to Gerrit 2.15.2 and the "Browse > Projects" page is very slow to load showing "Loading..." for minutes. The server has ~700 projects. Is it expected? Is there something we can check or do to improve the performance? Is Lucene used in this search?PS: I know we can speed up using the "Filter" field but a lot of people still use the old UI where the "Filter" is only showed at the end.Thanks a lot.
--
Reply all
Reply to author
Forward
0 new messages