Gerrit 2.14 starts very slowly.

[wei ren]

Have upgraded gerrit from 2.12 to 2.14 for a couple of months. It used to take less than one minute for starting gerrit. Since last week, it has suddenly slowed down to almost 7 mins. Below is the error_log during start.

[2017-10-10 08:36:40,015] [main] WARN  com.google.gerrit.sshd.SshDaemon : Disabling cipher aes192-ctr: Unable to initialize cipher BaseCipher[AES,16,24,AES/CTR/NoPadding]
[2017-10-10 08:36:40,020] [main] WARN  com.google.gerrit.sshd.SshDaemon : Disabling cipher aes256-ctr: Unable to initialize cipher BaseCipher[AES,16,32,AES/CTR/NoPadding]
[2017-10-10 08:36:40,023] [main] WARN  com.google.gerrit.sshd.SshDaemon : Disabling cipher arcfour256: Illegal key size or default parameters; try installing unlimited cryptography extension
[2017-10-10 08:36:40,040] [main] WARN  com.google.gerrit.sshd.SshDaemon : Disabling cipher aes192-cbc: Unable to initialize cipher BaseCipher[AES,16,24,AES/CBC/NoPadding]
[2017-10-10 08:36:40,040] [main] WARN  com.google.gerrit.sshd.SshDaemon : Disabling cipher aes256-cbc: Unable to initialize cipher BaseCipher[AES,16,32,AES/CBC/NoPadding]
[2017-10-10 08:36:41,945] [main] INFO  org.eclipse.jetty.util.log : Logging initialized @341569ms
[2017-10-10 08:36:42,108] [main] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 248m
[2017-10-10 08:36:42,584] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /home/gerrit2/gerrit_site/plugins
[2017-10-10 08:36:42,852] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin commit-message-length-validator, version v2.14
[2017-10-10 08:36:43,124] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin deleteproject, version v2.13-10-gee5d0a7
[2017-10-10 08:36:43,154] [DeleteTrashFolders] WARN  com.googlesource.gerrit.plugins.deleteproject.fs.DeleteTrashFolders : Exception while trying to delete trash folders
java.nio.file.AccessDeniedException: home/xxx/.ssh
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
    at java.nio.file.Files.newDirectoryStream(Files.java:457)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
    at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
    at java.nio.file.Files.walkFileTree(Files.java:2706)
    at java.nio.file.Files.walkFileTree(Files.java:2742)
    at com.googlesource.gerrit.plugins.deleteproject.fs.DeleteTrashFolders$1.run(DeleteTrashFolders.java:131)
    at java.lang.Thread.run(Thread.java:748)
[2017-10-10 08:36:43,328] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin download-commands, version v2.14
[2017-10-10 08:36:48,698] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin events-log, version v2.13-17-g2d5d48a
[2017-10-10 08:36:48,828] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin hooks, version v2.14
[2017-10-10 08:36:49,021] [main] WARN  com.googlesource.gerrit.plugins.replication.ReplicationFileBasedConfig : Config file /home/gerrit2/gerrit_site/etc/replication.config does not exist; not replicating
[2017-10-10 08:36:49,028] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin replication, version v2.14
[2017-10-10 08:36:49,110] [main] INFO  com.google.gerrit.server.config.PluginConfigFactory : No /home/gerrit2/gerrit_site/etc/reviewers.config; assuming defaults
[2017-10-10 08:36:49,163] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin reviewers, version 693779c
[2017-10-10 08:36:49,272] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin reviewnotes, version v2.14
[2017-10-10 08:36:49,325] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin singleusergroup, version v2.14
[2017-10-10 08:36:49,428] [main] INFO  com.google.gerrit.sshd.SshDaemon : Started Gerrit SSHD-CORE-1.4.0 on git.nane.cn:29418
[2017-10-10 08:36:49,430] [main] INFO  org.eclipse.jetty.server.Server : jetty-9.3.17.v20170317
[2017-10-10 08:36:50,120] [main] INFO  org.eclipse.jetty.server.handler.ContextHandler : Started o.e.j.s.ServletContextHandler@20a434a8{/,null,AVAILABLE}
[2017-10-10 08:36:50,130] [main] INFO  org.eclipse.jetty.server.AbstractConnector : Started ServerConnector@31d09031{HTTP/1.1,[http/1.1]}{0.0.0.0:8082}
[2017-10-10 08:36:50,132] [main] INFO  org.eclipse.jetty.server.Server : Started @349764ms
[2017-10-10 08:36:50,133] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.14 ready

From the log, we see that "Logging initialized @341569ms" and after about 8 secs, gerrit finishes loading plugins and has started. We do see some other warning messages but none of them seem to be related to the slow start. Specifically,

1. "WARN  com.google.gerrit.sshd.SshDaemon : Disabling cipher aes192-ctr: Unable to initialize cipher BaseCipher[AES,16,24,AES/CTR/NoPadding]  ==> We guess that those ciphers are not supported by gerrit?
   
2. "[DeleteTrashFolders] WARN  com.googlesource.gerrit.plugins.deleteproject.fs.DeleteTrashFolders : Exception while trying to delete trash folder
   java.nio.file.AccessDeniedException: home/xxx/.ssh"  ==> Have no idea why the delete-project plugin is trying to delete a user's (not the gerrit user) .ssh directory. 
   
3. "WARN  com.googlesource.gerrit.plugins.replication.ReplicationFileBasedConfig : Config file /home/gerrit2/gerrit_site/etc/replication.config does not exist; not replicating"
   

Below is gerit.config.

[addreviewer]
        maxWithoutConfirmation = 20
        maxAllowed = 50
[auth]
        type = LDAP
        gitBasicAuthPolicy = HTTP
[cache]
        directory = cache
[changeCleanup]
        abandonAfter = 3 day
        abandonMessage = "Auto-Abandoned due to inactivity, see ${URL}Documentation/user-change-cleanup.html#auto-abandon"\n\
\n\
 If this change is still wanted, it should be restored. If you have any question regarding auto-abandon, please email xxx@xxx.xx
        startTime = Mon 0:21
        interval = 1 day
[commentlink "issueid"]
        match = "\\s#(\\d+)[\\s,.]?"
        link = http://redmine.xxx.xx/redmine/issues/$1
[container]
        heapLimit = 1g
        user = gerrit2
        javaHome = /usr/lib/jvm/java-8-oracle/jre
[core]
        packedGitWindowSize = 16k
        packedGitLimit = 512m
        packedGitOpenFiles = 1024
[database]
        type = mysql
        url = jdbc:mysql://localhost:3306/reviewdb?user=xxx&password=xxx&useUnicode=true&characterEncoding=utf8
        hostname = localhost
        database = xxx
        username = xxx
[gc]
        aggressive = true
        startTime = Sun 1:21
        interval = 1 week
[gerrit]
        basePath = /home/gerrit2/repos
        canonicalWebUrl = http://git.xxx.xxx:8082/
        serverId = 54fc2da3-efff-482c-bdb2-f6e4fcc72a72
[gitweb]
        type = gitweb
        cgi = /usr/lib/cgi-bin/gitweb.cgi
[hooks]
        patchsetCreatedHook = patchset-created
        commitReceived = commit-received
        changeMergedHook = change-merged
[httpd]
        listenUrl = http://*:8082/
[index]
        type = LUCENE
[ldap]
        server = ldap://ldap.xxx.xx
        accountBase = ou=users,dc=xxx,dc=xx
        groupBase = ou=groups,dc=xxx,dc=xx
        accountBase = ou=users,dc=xxx,dc=xx
        groupBase = ou=groups,dc=xxx,dc=xx
        groupsVisibleToAll = true
[receive]
        enableSignedPush = false
        threadPoolSize = 48
[repository "*"]
        defaultSubmitType = REBASE_IF_NECESSARY
[sendemail]
        smtpServer = smtp.xxx.com
        smtpServerPort = 465
        smtpEncryption = SSL
        smtpUser = gerrit2@xxx.xx
        from = gerrit2@xxx.xx
[sshd]
        enableCompression = true
        backend = MINA
        listenAddress = git.xxx.xx:29418
        threads = 48
        batchThreads = 2
        streamThreads = 50
[plugin "events-log"]
        storeUrl = jdbc:h2:/home/gerrit2/db/ChangeEvents

The command for starting the gerrit is:

java -Xmx1g -jar /home/gerrit2/gerrit_site/bin/gerrit.war daemon -d /home/gerrit2/gerrit_site

The CPU of the gerrit server is Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz and it has 2GB memory.

Any help or sugestion will be highly appreciated. If any more info is needed, please kindly let me know. Thanks.

[wei ren]

Forgot to mention that gerrit is configured as an upstart service. The weird thing is that gerrit only starts very slowly if it is started by upstart. If it is started by the following command, it actually starts much faster (< 1 min).

java -Xmx1g -jar /home/gerrit2/gerrit_site/bin/gerrit.war daemon -d /home/gerrit2/gerrit_site

[2017-10-11 01:24:52,825] [main] INFO  org.eclipse.jetty.util.log : Logging initialized @22253ms
[2017-10-11 01:24:57,110] [main] INFO  org.eclipse.jetty.server.Server : Started @26539ms

Below is the upstart service conf file gerrit.conf in /etc/init.

# This task runs the gerrit code review.
# This conf file should be saved as /etc/init

description     "Gerrit code review"
author          "Wei Ren <ren...@smartconn.cc>"

start on startup
stop on shutdown

# Set the running userid and groupid.
setuid gerrit2
setgid gerrit2

# respawn the job up to 10 times within a 5 second period.
# If the job exceeds these values, it will be stopped and
# marked as failed.
respawn
respawn limit 10 5

# The arguments are copied from those generated by /home/gerrit2/gerrit_site/bin/gerrit.sh
exec java -Xmx1g -jar /home/gerrit2/gerrit_site/bin/gerrit.war daemon -d /home/gerrit2/gerrit_site

Also tried set the above uid and gid to root but didn't see any improvement. By the way, the running OS is Ubuntu 14.04LTS.

[baoshi li]

As suggested in this post, Gerrit 2.13.6 takes ages to start, it is probably due to missing entropy for the RNG. You can have a try to install haveged (How to Setup Additional Entropy for Cloud Servers Using Haveged).

wei ren於 2017年10月10日星期二 UTC+8上午11時42分43秒寫道：