Help Configuring Gerrit 2.11 -- Out of Heap

514 views

Skip to first unread message

Joel DuBien

unread,

May 1, 2015, 1:24:13 PM5/1/15

to repo-d...@googlegroups.com

Hello,

My gerrit instance has been very stable over the past 3 years. I've recently upgraded to gerrit 2.11 and am not experiencing "Out of heap" errors. I'm also getting many errors abotu BloomFilters and ClassNotFoundExceptions.

I have now increased the heapLimit from 8000m to 10000m to try to fix this problem. I am wondering if there are other options I should pursue as well. Please advise.

My server has the following:

32 GB Ram
256 GB SSD
24 CPU Core Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz
Size of all the git repos, 5.2G total

Current gerrit.config (only relevant sections shown):

[database]

poolLimit = 48

[auth]

type = LDAP

[container]

user = gerrit

javaHome = /usr/java/jdk1.7.0_17/jre

heapLimit = 8000m

[core]

packedGitLimit = 3000m

packedGitOpenFiles = 256

[cache]

directory = cache

Error log contents:

[2015-05-01 10:52:11,595] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot build BloomFilter for jdbc:h2:file:/usr/local/gerrit/

cache/conflicts

org.h2.jdbc.JdbcSQLException: Deserialization failed, cause: "java.lang.ClassNotFoundException: com.google.gerrit.extensions.common.SubmitTy

pe" [90027-174]

at org.h2.message.DbException.getJdbcSQLException(DbException.java:332)

at org.h2.message.DbException.get(DbException.java:161)

at org.h2.util.Utils.deserialize(Utils.java:413)

at org.h2.jdbc.JdbcConnection.convertToDefaultObject(JdbcConnection.java:1846)

at org.h2.jdbc.JdbcResultSet.getObject(JdbcResultSet.java:489)

at com.google.gerrit.server.cache.h2.H2CacheImpl$KeyType.get(H2CacheImpl.java:248)

at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.buildBloomFilter(H2CacheImpl.java:376)

at com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore.mightContain(H2CacheImpl.java:354)

at com.google.gerrit.server.cache.h2.H2CacheImpl.getIfPresent(H2CacheImpl.java:100)

at com.google.gerrit.server.query.change.ConflictsCacheImpl.getIfPresent(ConflictsCacheImpl.java:54)

at com.google.gerrit.server.query.change.ConflictsPredicate$1.match(ConflictsPredicate.java:106)

at com.google.gerrit.server.query.change.ConflictsPredicate$1.match(ConflictsPredicate.java:86)

at com.google.gerrit.server.query.AndPredicate.match(AndPredicate.java:76)

at com.google.gerrit.server.query.change.AndSource.readImpl(AndSource.java:116)

at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:99)

at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:153)

at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:102)

at com.google.gerrit.server.query.change.QueryChanges.query0(QueryChanges.java:143)

at com.google.gerrit.server.query.change.QueryChanges.query(QueryChanges.java:132)

at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:99)

at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:40)

at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:324)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)

at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:497)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:722)

Caused by: java.lang.ClassNotFoundException: com.google.gerrit.extensions.common.SubmitType

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:423)

at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:266)

at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:622)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1593)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)

at java.io.ObjectInputStream.readEnum(ObjectInputStream.java:1704)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)

at org.h2.util.Utils.deserialize(Utils.java:411)

... 61 more

[2015-05-01 10:59:42,443] ERROR com.google.gerrit.pgm.Daemon : Thread PluginScanner threw exception

java.lang.OutOfMemoryError: Java heap space

at java.util.jar.Manifest$FastInputStream.<init>(Manifest.java:332)

at java.util.jar.Manifest$FastInputStream.<init>(Manifest.java:327)

at java.util.jar.Manifest.read(Manifest.java:195)

at java.util.jar.Manifest.<init>(Manifest.java:69)

at java.util.jar.JarFile.getManifestFromReference(JarFile.java:180)

at java.util.jar.JarFile.getManifest(JarFile.java:166)

at com.google.gerrit.server.plugins.JarPluginProvider.getJarPluginName(JarPluginProvider.java:73)

at com.google.gerrit.server.plugins.JarPluginProvider.getPluginName(JarPluginProvider.java:63)

at com.google.gerrit.server.plugins.UniversalServerPluginProvider.getPluginName(UniversalServerPluginProvider.java:48)

at com.google.gerrit.server.plugins.PluginLoader.getGerritPluginName(PluginLoader.java:697)

at com.google.gerrit.server.plugins.PluginLoader.getPluginName(PluginLoader.java:75)

at com.google.gerrit.server.plugins.PluginLoader.asMultimap(PluginLoader.java:705)

at com.google.gerrit.server.plugins.PluginLoader.prunePlugins(PluginLoader.java:627)

at com.google.gerrit.server.plugins.PluginLoader.rescan(PluginLoader.java:357)

at com.google.gerrit.server.plugins.PluginScannerThread.run(PluginScannerThread.java:41)

[2015-05-01 10:59:43,274] WARN com.google.gerrit.sshd.GerritServerSession : Exception caught

java.lang.OutOfMemoryError: Java heap space

at java.util.Arrays.copyOf(Arrays.java:2271)

at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)

at org.bouncycastle.asn1.ASN1Object.getEncoded(Unknown Source)

at org.bouncycastle.openssl.jcajce.JcaPEMKeyConverter.getKeyPair(Unknown Source)

at org.apache.sshd.common.keyprovider.FileKeyPairProvider.doLoadKey(FileKeyPairProvider.java:135)

at org.apache.sshd.common.keyprovider.FileKeyPairProvider$1$1.setNextObject(FileKeyPairProvider.java:106)

at org.apache.sshd.common.keyprovider.FileKeyPairProvider$1$1.hasNext(FileKeyPairProvider.java:89)

at org.apache.sshd.common.keyprovider.AbstractKeyPairProvider.getKeyTypes(AbstractKeyPairProvider.java:54)

at org.apache.sshd.server.session.ServerSession.sendKexInit(ServerSession.java:106)

at org.apache.sshd.server.session.ServerSession.readIdentification(ServerSession.java:168)

at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:302)

at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)

at org.apache.sshd.common.io.mina.MinaService.messageReceived(MinaService.java:105)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:854)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)

at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)

at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:714)

at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)

at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)

at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)

at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:722)

[2015-05-01 11:02:44,794] WARN org.eclipse.jetty.util.thread.QueuedThreadPool :

java.lang.OutOfMemoryError: Java heap space

[2015-05-01 11:01:54,488] ERROR com.google.gerrit.pgm.Daemon : Thread pool-26-thread-926 threw exception

java.lang.OutOfMemoryError: Java heap space

Thanks for your help!

Joel DuBien

unread,

May 1, 2015, 1:41:33 PM5/1/15

to repo-d...@googlegroups.com

As a follow up here, I just ran gerrit gc across all of our git repos. That dropped the size of all git repos from 5.2G to 3.4G. Perhaps the size of the git repos was the cause of the heap OOMs?

Augusto Roman

unread,

May 26, 2015, 12:13:01 PM5/26/15

to repo-d...@googlegroups.com

Did you resolve this? We updated to 2.11 recently and only get the BloomFilter and ClassNotFoundExceptions (no OOMs)

Error log:

[2015-05-26 15:10:40,668] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot build BloomFilter for jdbc:h2:file:/home/gerrit2/review_site/cache/conflicts

org.h2.jdbc.JdbcSQLException: Deserialization failed, cause: "java.lang.ClassNotFoundException: com.google.gerrit.extensions.common.SubmitType" [90027-174]