Gerrit 2.9 - slow web interface

1,990 views
Skip to first unread message

Leoš Junek

unread,
Oct 7, 2014, 8:52:57 AM10/7/14
to repo-d...@googlegroups.com
Hello to all,

I work as Gerrit administrator in SW company with only about 60 repositories (bare repos are in 5,4 GB in size) and about 70 users. Usually only 5 users at the maximum access Gerrit within a minute. Gerrit runs on dedicated virtual server with Oracle Linux 6.2, 5 GB physical RAM, Intel Xeon CPU E5504 @ 2.00GHz, uses Oracle Java 1.7.0_67. My configuration file gerrit.conf is listed below. Apache serves as a HTTP proxy.

Users complain about slow loading of some pages, inluding
- main Gerrit page
- list of project
- some open changes for code review

Slow page loading does not occurs on regular basis, but from time to time. Sometimes it takes serveral seconds to load one of mentioned pages, Gerrit seems to be frozen and writes "Working". It does not happen at each request, but serveral times a day to users who work with Gerrit regularly.  

When I checked memory usage, there was no visible increase.  

I used Firebug to measure loading time, it took almost 3 seconds to load. The most time consuming task (2,59s) was to perform XMLHTTP Request http://gerrittest.oksystem.local/r/changes/?n=25&O=1 Screenshot from Firebug is attached.  

Sometimes it takes even longer, as well as getting list of projects.

On testing Gerrit engine I added some more parameters to gerrit.config (in comparison to production Gerrit listed below)¨to improve Gerrit performance, but it did not help.
[database]
  poolMaxIdle
= 16
  poolLimit
= 64
[core]
  deltaBaseCacheLimit
= 50m
  packedGitOpenFiles
= 1024
[cache "accounts"]
  maxAge
= 1d
[cache "changes"]
  memoryLimit
= 200
  maxAge
= 1d
[cache "diff"]
  maxAge
= 1d
  memoryLimit
= 20m
[cache "diff_intraline"]
  maxAge
= 1d
  memoryLimit
= 20m
[cache "git_tags"]
  maxAge
= 1d
[cache "plugin_resources"]
  maxAge
= 1d
[cache "projects"]
  maxAge
= 1d




What could be reason of "Working" Gerrit web interface?
How can I debug it? I have no idea what to check (CPU usage, memory/swap, I/O operations, network connections) or what to ask users to measure.   

Do anybody reconfigure Gerrit to improve performance of Gerrit to reduce web pages load times?


Regards

Leos


=== gerrit.config (production) ===
[gerrit]
        basePath
= /opt/gitrepo
        canonicalWebUrl
= http://gerrit.oksystem.local/r/
[database]
        type
= h2
        database
= db/ReviewDB
[auth]
        type
= LDAP
[ldap]
        server
= ldaps://adc01.oksystem.local
        accountBase
= dc=oksystem,dc=local
        accountPattern
= (&(objectClass=user)(sAMAccountName=${username}))
        accountFullName
= displayName
        accountEmailAddress
= mail
        username
= CN=specialGerrit,OU=Foo,OU=Bar,DC=oksystem,DC=local
        password
= ****
        groupBase
= dc=oksystem,dc=local
        referral
= follow
        sslVerify
= false
        localUsernameToLowerCase
= true
[sendemail]
        smtpServer
= localhost
        smtpUser
= gerrit
[container]
        user
= gerrit2
        javaHome
= /usr/lib/jvm/jdk1.7.0_67
        heapLimit
= 3096m
[core]
        packedGitLimit
= 60m
        packedGitWindowSize
= 16k
[sshd]
        listenAddress
= *:29418
[httpd]
        listenUrl
= proxy-http://127.0.0.1:8080/r/
[cache]
        directory
= cache
[cache "web_sessions"]
        maxAge
= 1 y
[mimetype "image/*"]
        safe
= true
[index]
        type
= LUCENE





gerrit_firebug.PNG

Matthias Sohn

unread,
Oct 7, 2014, 11:05:49 AM10/7/14
to Leoš Junek, Repo and Gerrit Discussion
which Gerrit version are you using ?

Your jgit buffer cache (core.packedGitLimit) is pretty small, JGit loves memory and you only have
60MB cache for caching access to pack files. We dedicate half of the heap for JGit buffer cache.

--
Matthias

Leoš Junek

unread,
Oct 8, 2014, 3:29:28 AM10/8/14
to repo-d...@googlegroups.com, leos...@gmail.com
Hello Matthias,

Thanks for your reply. I'm running Gerrit 2.9. 
I am not sure, what is cached on server and what is cached in user's browser. It's obvious that loading time is longer after restart of Gerrit. 

Here is the list of HTTP requests on production server, where no "core.packedGitLimit" was set yet. Though Gerrit was not restarted for a month, it took more than 8 s to display details of one change.

Started Time Sent Received Method Result Type URL
00:00:01.164 7.166 326 1485 GET 200 application/json http://gerrit.oksystem.local/r/changes/5322/detail?O=404
00:00:08.355 0.033 373 276 GET 200 application/json http://gerrit.oksystem.local/r/changes/5322/revisions/cada5d4ac3d1838ea7df3163e3fb9331c8f141bd/comments
00:00:08.360 0.047 370 274 GET 200 application/json http://gerrit.oksystem.local/r/changes/5322/revisions/cada5d4ac3d1838ea7df3163e3fb9331c8f141bd/files
00:00:08.364 0.047 371 620 GET 200 application/json http://gerrit.oksystem.local/r/changes/5322/revisions/cada5d4ac3d1838ea7df3163e3fb9331c8f141bd/commit
00:00:08.369 0.041 334 585 GET 200 application/json http://gerrit.oksystem.local/r/projects/mobile.sms.shared/config
00:00:08.459 0.192 355 276 GET 200 application/json http://gerrit.oksystem.local/r/changes/?q=status:open+is:mergeable+conflicts:5322&O=a
00:00:08.463 0.191 372 276 GET 200 application/json http://gerrit.oksystem.local/r/changes/5322/revisions/cada5d4ac3d1838ea7df3163e3fb9331c8f141bd/related
00:00:08.468 0.185 417 276 GET 200 application/json http://gerrit.oksystem.local/r/changes/?q=project:mobile.sms.shared+change:I611fe777b013e7a2cad4d61045bb8ce9d9b71ca0+-change:5322+-is:abandoned&O=a
00:00:08.473 0.182 400 276 GET 200 application/json http://gerrit.oksystem.local/r/changes/?q=status:open+project:mobile.sms.shared+branch:master+topic:documentation+-change:5322&O=a
00:00:08.492 0.164 361 242 GET 404 text/plain (NS_ERROR_FAILURE) http://gerrit.oksystem.local/r/accounts/rauscher%40oksystem.cz/avatar?s=26
00:00:08.506 0.151 359 242 GET 404 text/plain (NS_ERROR_FAILURE) http://gerrit.oksystem.local/r/accounts/tethal%40oksystem.cz/avatar?s=26
00:00:08.513 0.145 358 242 GET 404 text/plain (NS_ERROR_FAILURE) http://gerrit.oksystem.local/r/accounts/kabat%40oksystem.cz/avatar?s=26
00:00:08.520 0.138 360 242 GET 404 text/plain (NS_ERROR_FAILURE) http://gerrit.oksystem.local/r/accounts/vykypel%40oksystem.cz/avatar?s=26
00:00:08.528 0.132 376 276 GET 200 application/json http://gerrit.oksystem.local/r/changes/5322/revisions/cada5d4ac3d1838ea7df3163e3fb9331c8f141bd/submit_type
00:00:08.627 0.125 376 -5380 GET (Cache) application/x-shockwave-flash http://gerrit.oksystem.local/r/gerrit_ui/98CCFC92412383C3C016A68FD84F0F1C.cache.swf
00:00:08.649 * 376/376 * GET (Cache) application/x-shockwave-flash http://gerrit.oksystem.local/r/gerrit_ui/98CCFC92412383C3C016A68FD84F0F1C.cache.swf

What are common response time in small companies like ours? Is it usual to wait for serveral seconds to load page? 

As soon as I change configuration on production server, I will let you know.

Leos


Dne úterý, 7. října 2014 17:05:49 UTC+2 Matthias Sohn napsal(a):

Alex Blewitt

unread,
Oct 8, 2014, 4:25:07 AM10/8/14
to Leoš Junek, repo-d...@googlegroups.com
What improvement did you see when you increased the config as suggested?

Bear in mind that this limit should be around 50% of the web servers heap and should also be proportional to the size of your active repositories. 

If you only have a couple of hello world repositories then 50m should be fine. If you are using it for many large repositories then it will not be. 

Sent from my iPhat 6
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Leoš Junek

unread,
Oct 9, 2014, 4:33:34 AM10/9/14
to repo-d...@googlegroups.com, leos...@gmail.com
Hello AlBlue, and others,

after configuration change I asked user to use Gerrit as usual and report me if it got frozen. Unfortunately it did not help, there are still moments when users must wait from 1 to 6 seconds to load page.
Here are some of URLs of pages, that caused frozen Gerrit:


What about size of our repositories? Total number of repositories is 60, where 90% of repositories consist of less than 300 changes/commits and 4 repositories consist of up to 1000 commits and only 2 repos more than 2000 commits. Total commits count is about 13700. In my opinion sum of our active repositories is pretty small, number of concurrent accesses as well, so I do not understand what is wrong.

To show "frozen moments" of our Gerrit I do attach some HAR files with recorded responses on HTTP request as well as screenshots of these files viewed in online viewer.

I will be grateful for any hint what should be examined to find root of that freezing.

Leoš


Dne středa, 8. října 2014 10:25:07 UTC+2 AlBlue napsal(a):
02-gerrit.har
03-gerrit.har
04-gerrit.har
05-gerrit.har
02-gerrit.HAR.png
03-gerrit.HAR.png
04-gerrit.HAR.png
05-gerrit.HAR.png

Alex Blewitt

unread,
Oct 9, 2014, 5:08:37 AM10/9/14
to Leoš Junek, repo-d...@googlegroups.com
In order to load a page, two things must happen. There must be a database connection (to query for the states) and the repository content must be loaded (to extract the data).

Databases are configured in the [databases] section, documented here: https://gerrit-documentation.storage.googleapis.com/Documentation/2.9.1/config-gerrit.html#database

If you are using connection pooling, you need to verify that the maximum number of concurrent user requests is less than the number of database pool connections, as otherwise users will have to wait. You can also check the performance of your database in case that is the bottleneck. If you are not using connection pooling, you should be using connection pooling.

To load a repository the pack files must be loaded from JGit. Various configuration settings are documented  in the [core] section https://gerrit-documentation.storage.googleapis.com/Documentation/2.9.1/config-gerrit.html#core

In particular, the size of the cached pack files can dramatically speed up operations (otherwise the repository will be reloaded from disk a lot). The general rule of thumb is that the cached pack size should be about 1/2 of the JVM size, and that the cached pack size should be proportionate to the number of repositories and their sizes. A single 1G repository will be around the same as 100 x 10M repositories (assuming they are all active). The default is 10Mb which is almost certainly too small for a reasonable installation, but you have to vary the JVM memory too.

If your repositories are not garbage collected (i.e. you have lots of pack or object files in each repo) then this will dramatically slow down performance. If you are using native git to do the gc, ensure you are using the one that supports bitmap creation (git 2.0 and above, I believe). You can also use jgit gc to do the same thing.

Finally, timing individual URLs and saying which ones are slow is of little relevance for others to be able to assist you. Not only are they URLs that won’t resolve externally, but the fact of the matter is that your configuration choices (which you have not yet shown) are the single point that explains your performance issues, and that the time it takes to load a specific URL is just a symptom for how you have your repository configured. I suggest that you post the relevant sections of the above configuration files, the number of individual pack/object files in the repositories (and their sizes) and your JVM/OS memory/limitations if you are still struggling with this.

Alex

<02-gerrit.har><03-gerrit.har><04-gerrit.har><05-gerrit.har><02-gerrit.HAR.png><03-gerrit.HAR.png><04-gerrit.HAR.png><05-gerrit.HAR.png>

Vlad Canţîru

unread,
Oct 9, 2014, 5:26:11 PM10/9/14
to Alex Blewitt, Leoš Junek, repo-d...@googlegroups.com

Hi,


Considering your set-up, usage and current settings I don't think you have any issue with DB, neither with Java. Keeping an eye on these though is still a good idea. Full java garbage collection can take long enough time to give you same symptom you described.  Since you have performance issues with WebUI I don't think this is related to core settings mentioned by Alex, could be but I don't think this is your case. 


I think your issue comes from the fact that you set memoryLimit on changes to 200.

 

[cache "changes"]
  memoryLimit = 200


I suggest you comment this part and leave it default which is higher anyway and see if this brings any improvement.


Hope this helps,

Vladimir Cantiru





Leoš Junek

unread,
Oct 20, 2014, 12:22:43 PM10/20/14
to repo-d...@googlegroups.com, alex.b...@gmail.com
Hi Vladimir,

from thanks for your hint. I have just removed memoryLimit line from my config and add Gerrit garbage collection. If it does not help, I am ready to monitor Java Garbage collection via jstat. Your answer lead me to focus on GC - it may be the light at the end of the tunnel cause I have no any idea about monitoring or reconfiguration. 

Now I have to wait for user experience, because I do not work with Gerrit regularly to discover slowdown events. Then I am going to leave a post if it helps or not.

Current config of my Gerrit is here:

[gerrit]
        basePath
= /opt/gitrepo
        canonicalWebUrl
= http://gerrit.oksystem.local/r/
[database]
        type
= h2
        database
= db/ReviewDB

        poolMaxIdle
= 16
        poolLimit
= 64
[auth]
        type
= LDAP
[ldap]

        server
= ldaps://pdc.oksystem.local
        accountBase
= dc=FFFF,dc=HHHH

        accountPattern
= (&(objectClass=user)(sAMAccountName=${username}))
        accountFullName
= displayName
        accountEmailAddress
=
mail
        username
= CN=XXXX,OU=DDDD,OU=EEEE,DC=FFFF,DC=HHHH
        password
= XXXX



        groupBase
= dc=oksystem,dc=local
        referral
= follow
        sslVerify
= false
        localUsernameToLowerCase
= true
[sendemail]
        smtpServer
= localhost
        smtpUser
= gerrit
[container]
        user
= gerrit2
        javaHome
= /usr/lib/jvm/jdk1.7.0
_67
        heapLimit
= 3g
[core]
        packedGitLimit
= 1536m
        packedGitWindowSize
= 16k
        deltaBaseCacheLimit
= 50m

[sshd]
        listenAddress
= *:29418

        threads
=24
        batchThreads
= 2

[httpd]
        listenUrl
= proxy-http://127.0.0.1:8080/r/
[cache]
        directory
= cache
[cache "accounts"]

        maxAge
= 1d
[cache "changes"]

        maxAge
= 1d
[cache "diff"]
        maxAge
= 1d
        memoryLimit
= 20m
[cache "diff_intraline"]
        maxAge
= 1d
        memoryLimit
= 20m
[cache "git_tags"]
        maxAge
= 1d
[cache "permission_sort"]

        maxAge
= 1d
[cache "plugin_resources"]
        maxAge
= 1d
[cache "projects"]
        maxAge
= 1d
[cache "sshkeys"]
        maxAge
= 1d

[cache "web_sessions"]
        maxAge
= 1 y
[mimetype "image/*"]
        safe
= true
[index]
        type
= LUCENE
[gc]
        startTime
= 0:00
        interval
= 6 hours


Regards

Leos




Dne čtvrtek, 9. října 2014 23:26:11 UTC+2 evlacan napsal(a):

Saša Živkov

unread,
Oct 27, 2014, 5:36:29 AM10/27/14
to Leoš Junek, repo-d...@googlegroups.com, Alex Blewitt
I guess your Gerrit is affected by a bug in gwtorm which I discovered last week.
Read the commit message of [1].

To check if you have wrong order of columns in (compound) primary keys check,
for example, the primary key of the PATCH_COMMENTS table.
If the columns in the key are ordered like: (FILE_NAME, CHANGE_ID, PATCH_SET_ID, UUID)
then you *are* affected by this bug.
The correct order, for this case is: (CHANGE_ID, PATCH_SET_ID, FILE_NAME, UUID)

While [1] will fix the bug for new Gerrit sites (created after [1] is used by Gerrit)
the existing sites will probably have to get fixed manually. I am going to write
an announcement and instructions/script on how to fix this issue in an existing site.

Saša

--

Leoš Junek

unread,
Jan 9, 2015, 7:39:59 AM1/9/15
to repo-d...@googlegroups.com, leos...@gmail.com, alex.b...@gmail.com
Hello Saša,

thanks for your hint. Our Gerrit is not affected by bug you mentioned.

Unfortunatelly our Gerrit is still slow. There was no strong need to solve it, so I have postponed solution till these days. Now I would like to focus on Alex reply, hope to move the problem forward. 

Regards

Leoš

Dne pondělí 27. října 2014 10:36:29 UTC+1 zivkov napsal(a):
Message has been deleted

Leoš Junek

unread,
Jan 13, 2015, 11:15:11 AM1/13/15
to repo-d...@googlegroups.com
Hello to all,

My problem seems to be solved, I do hope. Developers who use Gerrit does not complain any more. 

Probably too many HTTP connections were established from developers to Gerrit server. For several hours I monitored output of


 netstat | grep http |  wc -l

and it ranges from 0 to 226. Maximum number of http threads is set by httpd.maxThreads config option. Default value was 25. I have set it to 100 and frozening almost dissapeared. In a short time I am going to put there 300 to be sure number HTTP connections will not be limited.


 At the same time I have added daily garbage collection to crontab
0 3 * * * /usr/bin/ssh -p 29418 username@gerrit.mycompany.local gerrit gc --all


Thanks to all for your advice and suggestions!

Leoš

Matthias Sohn

unread,
Jan 13, 2015, 11:17:06 AM1/13/15
to Leoš Junek, Repo and Gerrit Discussion, Alex Blewitt
ensure that you also increase the size of database connection pool when you increase httpd thread pool size.
Otherwise HTTP threads may be unusable if they can't get a database connection since the database
connection pool is exhausted.

-Matthias 

Leoš Junek

unread,
Jan 13, 2015, 11:44:52 AM1/13/15
to repo-d...@googlegroups.com
Thanks, Matthias. 

Documentation says database.poolLimit should be a few units less that sum of httpd and sshd threads. I have completely forgotten about it. Now I set it to 310.

Leoš

Dne úterý 13. ledna 2015 17:17:06 UTC+1 Matthias Sohn napsal(a):

Leoš Junek

unread,
Jan 13, 2015, 12:52:49 PM1/13/15
to repo-d...@googlegroups.com

Mistake in my English, I'm sorry.
Should be "... a few units more than ..."

Dne 13.1.2015 17:44 "Leoš Junek" <leos...@gmail.com> napsal(a):
--
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/KCcpWroImEY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.

Leoš Junek

unread,
Jan 15, 2015, 7:52:15 AM1/15/15
to repo-d...@googlegroups.com, leos...@gmail.com
Hello Alex and others,

it seems setting httpd.maxThreads = 300 and database.poolLimit = 310 did not help to solve the problem. Gerrit still gets frozen from time to time. 

My database config section is
[database]
        type
= h2
        database
= db/ReviewDB

        poolMaxIdle
= 16
        poolLimit
= 310

database.connectionPool is true by default with H2 database - documentation says it

Here is example of content of single bare repository
20K     anaut.git/objects/pack/pack-34fce7e4af614b96b7b8a23279918dc5aeaba792.bitmap
180K    anaut.git/objects/pack/pack-34fce7e4af614b96b7b8a23279918dc5aeaba792.idx
11M     anaut.git/objects/pack/pack-34fce7e4af614b96b7b8a23279918dc5aeaba792.pack
20K     anaut.git/objects/pack/pack-fbed29b34dbd1caf39bcba1cd5a388ac08c679bc.idx
104K    anaut.git/objects/pack/pack-fbed29b34dbd1caf39bcba1cd5a388ac08c679bc.pack

There are 111 files with name "pack-*.pack", total size is 1082 MB. Where can I get object files? 


My container and core sections are
[container]
        user
= gerrit2
        javaHome
= /usr/lib/jvm/jdk1.7.0
_67
        heapLimit
= 3g

[core]
        packedGitLimit
= 1536m
        packedGitWindowSize
= 16k
        deltaBaseCacheLimit
= 50m

At night I perform garbage collection with crontab, running gerrit gc --all via ssh on daily basis.

There are 55 bare repositories on my Gerrit server, where size of
31 repos is in interval 1 MB - 9 MB
16 repos is in interval 10 MB - 30 MB
6 repos is in interval 40 - 100 MB
1 repo is 404 MB
Total size is 1157 MB

Number of connected developers is quite low, total number of all developers is about 70. No more than 5 developers access Gerrit at the same time.

OS is RedHat based linux distribution equipped with physical 5 GB RAM, though only 4968 MB are available to system (shown by free, htop). JVM heap is set to 3 GB.

I have no idea about way to set JGit parameters (those mentioned in [core] section). Do anybody have any idea how can I find proper values of these parameters?

Could anybody tell me what to explore to find source of freezing?



Regards

Leoš


Dne čtvrtek 9. října 2014 11:08:37 UTC+2 AlBlue napsal(a):

Matthias Sohn

unread,
Jan 15, 2015, 8:58:07 AM1/15/15
to Leoš Junek, Repo and Gerrit Discussion
I would create a couple of thread dumps on the server and check if there are any threads stuck on some resource e.g. database, LDAP.
You can check the ssh job queue for long runners using the ps command [1]. I would also check status of caches [2], maybe you need
to tweak some cache settings [3].


-Matthias

Leoš Junek

unread,
Jan 16, 2015, 5:07:54 AM1/16/15
to repo-d...@googlegroups.com, leos...@gmail.com
Thanks, Mattias.


Here is output of gerrit show-queue command
Task     State        StartTime         Command

------------------------------------------------------------------------------
0c38d57d 10:31:24.848 Jan-13 21:46      Reload Submit Queue
ac2c2940
22:46:07.428 Jan-13 21:46      Log File Compressor
------------------------------------------------------------------------------


And here cache statistics
Gerrit Code Review        2.9                       now    10:40:12   CET
                                                 uptime    
2 days 12 hrs


 
Name                          |Entries              |  AvgGet |Hit Ratio|
                               
|   Mem   Disk   Space|         |Mem  Disk|
--------------------------------+---------------------+---------+---------+
  accounts                      
|    37               |   1.3ms | 99%     |
  accounts_byemail              
|    54               |   1.3ms | 98%     |
  accounts_byname              
|    72               | 520.2us | 99%     |
  adv_bases                    
|                     |         |         |
  change_kind                  
|                     |  15.1ms |  0%     |
  changes                      
|                     |         |         |
  groups                        
|    15               |   1.2ms | 95%     |
  groups_byinclude              
|     3               | 345.7us | 66%     |
  groups_byname                
|                     |         |         |
  groups_byuuid                
|    52               | 288.7us | 94%     |
  groups_external              
|     1               | 560.3us | 80%     |
  groups_members                
|    13               |   6.6ms | 99%     |
  ldap_group_existence          
|                     |         |         |
  ldap_groups                  
|    15               |   3.2s  | 99%     |
  ldap_groups_byinclude        
|                     |         |         |
  ldap_usernames                
|    12               | 329.9us | 52%     |
  permission_sort              
|  1024               |         | 95%     |
  plugin_resources              
|     1               |         | 92%     |
  project_list                  
|     1               |  27.9ms | 99%     |
  projects                      
|    56               | 307.5us | 96%     |
  sshkeys                      
|    14               |   2.1ms | 99%     |
D conflicts                    
|     9      9   8.47k|         | 78%     |
D diff                          
|   241    431 804.27k|  22.0ms | 95% 100%|
D diff_intraline                
|    91    194 160.29k|  14.8ms | 44% 100%|
D git_tags                      
|                0.00k|         |  0%     |
D web_sessions                  
|    21   2548 904.34k|         | 99% 100%|


SSH
:      4  users, oldest session started    2 days 12 hrs ago
Tasks:    3  total =    1 running +      0 ready +    2 sleeping
Mem: 804.25m total = 594.94m used + 179.12m free +  30.19m buffers
       
2.90g max
         
28 open files,        1 cpus available,       90 threads


The longest time when accessing cache is at getting info from LDAP groups. Does it mean Gerrit stucks on connecting to LDAP? 

I would tweak cache settings, but unfortunately I have not idea about what is wrong. Could you see it from cache statistics? Or should I just try to modify parameters one-by-one and watch what changes?


What I found at my monitoring is list of current H2 database sessions remains same, there are still 11 sessions and it did not change for two days

 ID | USER_NAME | SESSION_START           | STATEMENT                                  | STATEMENT_START         | CONTAINS_UNCOMMITTED
 ---+-----------+-------------------------+--------------------------------------------+-------------------------+---------------------
 10 |           | 2015-01-14 09:12:34.8   | NULL                                       | 2015-01-14 09:34:51.374 | false
 11 |           | 2015-01-14 09:12:40.352 | NULL                                       | 2015-01-14 10:45:47.474 | false
 8  |           | 2015-01-14 08:05:55.839 | select * from information_schema.sessions; | 2015-01-14 08:33:10.629 | false
 9  |           | 2015-01-14 08:35:49.17  | NULL                                       | 2015-01-14 09:16:38.752 | false
 12 |           | 2015-01-14 10:45:23.585 | NULL                                       | 2015-01-16 11:02:20.869 | false
 13 |           | 2015-01-14 10:45:26.127 | NULL                                       | 2015-01-16 11:02:20.869 | false
 3  |           | 2015-01-13 21:45:54.52  | NULL                                       | 2015-01-13 22:01:47.579 | false
 4  |           | 2015-01-13 21:45:54.521 | NULL                                       | 2015-01-13 22:01:47.606 | false
 5  |           | 2015-01-13 21:45:54.521 | NULL                                       | 2015-01-13 22:01:46.632 | false
 6  |           | 2015-01-13 21:45:54.521 | NULL                                       | 2015-01-13 22:01:47.591 | false
 7  |           | 2015-01-13 21:50:22.495 | NULL                                       | 2015-01-13 22:01:47.617 | false
(11 rows; 4 ms)
 ID | USER_NAME | SESSION_START           | STATEMENT                                  | STATEMENT_START         | CONTAINS_UNCOMMITTED
 ---+-----------+-------------------------+--------------------------------------------+-------------------------+---------------------
 10 |           | 2015-01-14 09:12:34.8   | NULL                                       | 2015-01-14 09:34:51.374 | false
 11 |           | 2015-01-14 09:12:40.352 | NULL                                       | 2015-01-14 10:45:47.474 | false
 8  |           | 2015-01-14 08:05:55.839 | select * from information_schema.sessions; | 2015-01-14 08:33:10.629 | false
 9  |           | 2015-01-14 08:35:49.17  | NULL                                       | 2015-01-14 09:16:38.752 | false
 12 |           | 2015-01-14 10:45:23.585 | NULL                                       | 2015-01-16 11:02:20.869 | false
 13 |           | 2015-01-14 10:45:26.127 | NULL                                       | 2015-01-16 11:02:20.869 | false
 3  |           | 2015-01-13 21:45:54.52  | NULL                                       | 2015-01-13 22:01:47.579 | false
 4  |           | 2015-01-13 21:45:54.521 | NULL                                       | 2015-01-13 22:01:47.606 | false
 5  |           | 2015-01-13 21:45:54.521 | NULL                                       | 2015-01-13 22:01:46.632 | false
 6  |           | 2015-01-13 21:45:54.521 | NULL                                       | 2015-01-13 22:01:47.591 | false
 7  |           | 2015-01-13 21:50:22.495 | NULL                                       | 2015-01-13 22:01:47.617 | false
(11 rows; 4 ms)

Users request something from Gerrit and then leave Gerrit during the day, but the table remains same. Is that OK?


Leoš



Dne čtvrtek 15. ledna 2015 14:58:07 UTC+1 Matthias Sohn napsal(a):

Basavaraj Karadakal

unread,
Oct 11, 2015, 12:42:07 AM10/11/15
to Repo and Gerrit Discussion, leos...@gmail.com, alex.b...@gmail.com
Hi Sasa,
    I am debugging performance issue of gerrit 2.9.1 web interface. I came across this post. I see that we are affected by this bug. Are there instructions to fix it in current installation? I did a test upgrade to 2.11.3, the schema upgrade has fixed it. It will take some time for us to upgrade our production installation to 2.11.3, just curious to see if I can fix this before upgrade.

Thanks
Basavaraj

David Ostrovsky

unread,
Oct 11, 2015, 3:10:51 PM10/11/15
to Repo and Gerrit Discussion

Am Sonntag, 11. Oktober 2015 06:42:07 UTC+2 schrieb Basavaraj Karadakal:
Hi Sasa,
    I am debugging performance issue of gerrit 2.9.1 web interface. I came across this post. I see that we are affected by this bug. Are there instructions to fix it in current installation?

I don't remember that the instructions were ever posted. So here we go.

The source code in question is here: [1]. So after instrumentation of it: [2]
and upgrading affected Gerrit version 2.9.1: [3] with instrumented 2.10
branch, the SQL statements (H2 dialect) are: [4].

Note that for PostgreSQL "drop constraint foo" form must be used instead.


Saša Živkov

unread,
Oct 12, 2015, 7:34:50 AM10/12/15
to Basavaraj Karadakal, Repo and Gerrit Discussion, Leoš Junek, Alex Blewitt
On Sun, Oct 11, 2015 at 6:42 AM, Basavaraj Karadakal <bkr...@gmail.com> wrote:
Hi Sasa,
    I am debugging performance issue of gerrit 2.9.1 web interface. I came across this post. I see that we are affected by this bug. Are there instructions to fix it in current installation?

For 2.9.x (x >= 2) and 2.10.* releases, run the init program in order to fix this issue.
The init will detect all primary keys with the wrong column order and will recreate the
primary keys.

Basavaraj Karadakal

unread,
Oct 12, 2015, 5:39:09 PM10/12/15
to Repo and Gerrit Discussion, bkr...@gmail.com, leos...@gmail.com, alex.b...@gmail.com
Thanks for the information.
Basavaraj

Gardenia X

unread,
Dec 29, 2016, 9:59:43 AM12/29/16
to Repo and Gerrit Discussion, leos...@gmail.com, alex.b...@gmail.com
hi Alex

What about the mysql database, the default connectionPool is false, when i set it to be true,  get this error "Cannot get a connection, pool error Timeout waiting for idle object" , how to configure the connection pooling of mysql database? May you help me?

在 2014年10月9日星期四 UTC+8下午5:08:37,Alex Blewitt写道:
Reply all
Reply to author
Forward
0 new messages