Internal server error

[Faizan Ali]

We are getting internal server errors on a random basis and not very sure what could be causing this. Please suggest:

git push --progress --porcelain origin refs/heads/master:refs/for/master --set-upstream 
Counting objects: 23, done. 
Delta compression using up to 8 threads. 
Compressing objects: 100% (8/8), done. 
Writing objects: 100% (10/10), 1.10 KiB | 0 bytes/s, done. 
Total 10 (delta 4), reused 0 (delta 0) 
remote: Resolving deltas: 100% (4/4)[K 
remote: Processing changes: refs: 2, done [K 
error: failed to push some refs to 'ssh://user@git:29418/enhancedfile' 
To ssh://user@git:29418/enhancedfile 
! refs/heads/master:refs/for/master [remote rejected] (internal server error) 
  

Error in logs:

[2015-02-13 15:49:51,505] ERROR com.google.gerrit.server.git.ReceiveCommits : Can't insert change/patchset for PROJECT
com.google.gerrit.server.git.InsertException: Error inserting change/patchset
        at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:258)
        at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:249)
        at com.google.common.util.concurrent.Futures$MappingCheckedFuture.mapException(Futures.java:1672)
        at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
        at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:756)
        at com.google.gerrit.server.git.ReceiveCommits.processCommands(ReceiveCommits.java:564)
        at com.google.gerrit.server.git.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:90)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222)
        at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201)
        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator$1.call(ThreadLocalRequestScopePropagator.java:55)
        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:98)
        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:131)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:364)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.ExecutionException: com.google.gwtorm.server.OrmException: Cannot set auto commit mode
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
        at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:78)
        ... 17 more

[Matthew Webber]

What Gerrit version are you using?

[Faizan Ali]

Version 2.9.1 

[muhammad]

Faizan! verify that you have push permissions enabled for refs/for/ .  

[Faizan Ali]

Yes, we have the permissions because of the fact we can push for reviews most of the time and the error is intermittent. As per our research this could be related to connection pooling for Oracle db ( we use Oracle ). 

[database]

        type = ORACLE
        poolmaxidle = 16
        database = db/ReviewDB
        port = 1521
        instance = ########
        hostname = HOSTNAME
        connectionpool = true
        username = USERNAME
        poollimit = 32
        dataSourceInterceptorClass = com.googlesource.gerrit.plugins.javamelody.MonitoringDataSourceInterceptor

On a similar note we found this thread : https://groups.google.com/forum/#!topic/repo-discuss/ranb6VwsMOs with exactly the same problem but with a difference that it relates to mysql and we have oracle. Since, there is no official documentation available relating this issue to Oracle, I am not sure if we should go ahead and disable the connection pooling. Any suggestion please!

[David Ostrovsky]

Am Dienstag, 17. Februar 2015 07:11:56 UTC+1 schrieb Faizan Ali:

Yes, we have the permissions because of the fact we can push for reviews most of the time and the error is intermittent. As per our research this could be related to connection pooling for Oracle db ( we use Oracle ). 

[database]

        type = ORACLE
        poolmaxidle = 16
        database = db/ReviewDB
        port = 1521
        instance = ########
        hostname = HOSTNAME
        connectionpool = true
        username = USERNAME
        poollimit = 32
        dataSourceInterceptorClass = com.googlesource.gerrit.plugins.javamelody.MonitoringDataSourceInterceptor

Have you tried to disable javamelody datasource interceptor?

Can you reliably reproduce the problem? What are the steps?

There is dedicated universal connectin pool driver, so may be

trying to replace Apache commons DBCP with UCP driver

would help?