Gerrit reviews are stuck and complains on "502 Proxy Error"
475 views
Skip to first unread message
reddy
Jan 18, 2018, 1:52:54 AM1/18/18
to Repo and Gerrit Discussion
Hello,
We have a gerrit instance running on 7.2.0 version and started seeing issues with Gerrit reviews are stuck and the Gerrit Web UI Review page complains on "502 Proxy error"
The ssl_error_log has below errors
[Wed Jan 17 21:23:09 2018] [error] [client 10.255.85.217] proxy: Error reading from remote server returned by /gerrit/changes/2164/detail, referer: https://git-amr-7.devtools.intel.com/gerrit/
[Wed Jan 17 21:23:09 2018] [error] [client 10.255.85.217] no acceptable variant: /usr/share/apache2/error/HTTP_BAD_GATEWAY.html.var, referer: https://git-amr-7.devtools.intel.com/gerrit/
[Wed Jan 17 21:23:46 2018] [error] [client 10.252.88.36] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost
[Wed Jan 17 21:23:46 2018] [error] [client 10.252.88.36] proxy: Error reading from remote server returned by /gerrit/a/changes/2164/index
[Wed Jan 17 21:24:05 2018] [error] [client 10.254.68.174] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost, referer: https://git-amr-7.devtools.intel.com/gerrit/
[Wed Jan 17 21:24:05 2018] [error] [client 10.254.68.174] proxy: Error reading from remote server returned by /gerrit/changes/2164/detail, referer: https://git-amr-7.devtools.intel.com/gerrit/
[Wed Jan 17 21:24:05 2018] [error] [client 10.254.68.174] no acceptable variant: /usr/share/apache2/error/HTTP_BAD_GATEWAY.html.var, referer: https://git-amr-7.devtools.intel.com/gerrit/
[Wed Jan 17 21:24:06 2018] [error] [client 10.254.68.174] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost, referer: https://git-amr-7.devtools.intel.com/gerrit/
[Wed Jan 17 21:24:06 2018] [error] [client 10.254.68.174] proxy: Error reading from remote server returned by /gerrit/changes/2164/detail, referer: https://git-amr-7.devtools.intel.com/gerrit/
[Wed Jan 17 21:24:06 2018] [error] [client 10.254.68.174] no acceptable variant: /usr/share/apache2/error/HTTP_BAD_GATEWAY.html.var, referer: https://git-amr-7.devtools.intel.com/gerrit/
We initially started the gerrit reindex, and after sometime the job was killed.
Any suggestions or ideas on how to fix it ?
Gert van Dijk
Jan 18, 2018, 2:22:23 AM1/18/18
to Repo and Gerrit Discussion
What you're showing are the web server (reverse proxy) logs. However, it only shows the symptom that it cannot reach Gerrit and not much more. Please share what errors you have in the Gerrit server log for clues why it's not available to your reverse proxy. If possible, please also share relevant parts of your both configurations.
And how exactly did you perform the reindex?
And how exactly did you perform the reindex?
reddy
Jan 18, 2018, 7:15:28 PM1/18/18
to Repo and Gerrit Discussion
Below are the errors
2018-01-18 00:19:42,738|ERROR|ReceiveCommits-6|ReceiveCommits | Can't insert change/patchset for <myRepo>
com.google.gerrit.server.git.InsertException: Error inserting change/patchset
at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:272)
at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:263)
at com.google.common.util.concurrent.Futures$MappingCheckedFuture.mapException(Futures.java:1809)
at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:809)
at com.google.gerrit.server.git.ReceiveCommits.processCommands(ReceiveCommits.java:594)
at com.google.gerrit.server.git.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:89)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222)
at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201)
at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator$1.call(ThreadLocalRequestScopePropagator.java:55)
at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:98)
at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:131)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:399)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: java.lang.InterruptedException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:78)
... 17 more
Caused by: java.io.IOException: java.lang.InterruptedException
at com.google.gerrit.server.index.ChangeIndexer$1.apply(ChangeIndexer.java:76)
at com.google.gerrit.server.index.ChangeIndexer$1.apply(ChangeIndexer.java:67)
at com.google.common.util.concurrent.Futures$MappingCheckedFuture.mapException(Futures.java:1809)
at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:81)
at com.google.gerrit.server.change.ChangeInserter.insert(ChangeInserter.java:270)
at com.google.gerrit.server.git.ReceiveCommits$CreateRequest.insertChange(ReceiveCommits.java:1743)
at com.google.gerrit.server.git.ReceiveCommits$CreateRequest.access$900(ReceiveCommits.java:1673)
at com.google.gerrit.server.git.ReceiveCommits$CreateRequest$1.call(ReceiveCommits.java:1704)
at com.google.gerrit.server.git.ReceiveCommits$CreateRequest$1.call(ReceiveCommits.java:1700)
at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:96)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
at com.google.gerrit.server.git.ReceiveCommits$CreateRequest.insertChange(ReceiveCommits.java:1699)
at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:805)
... 16 more
Caused by: java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:78)
... 28 more
2018-01-18 00:19:42,739|WARN |SSH git-receive-pack '/jsvasish_test-repo-amr2' (someUser)|AsyncReceiveCommits| Error in ReceiveCommits while processing changes for project <MyRepo>
java.util.concurrent.ExecutionException: (timeout 157ms, cancelled)
at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:255)
at com.google.gerrit.server.git.AsyncReceiveCommits.onPreReceive(AsyncReceiveCommits.java:162)
at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:208)
at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:161)
at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:115)
at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:445)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:399)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.CancellationException
at java.util.concurrent.FutureTask.report(FutureTask.java:121)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at com.google.gerrit.server.git.WorkQueue$Task.get(WorkQueue.java:372)
at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:251)
... 16 more
Gerrit.config file:
[gerrit]
basePath = /XXX
secureStoreClass = com.collabnet.gerrit.EncryptedSecureStore
canonicalWebUrl = https://XXXXXXX/gerrit
secureStore = teamforge-securestore
[database]
type = POSTGRESQL
hostname = 127.0.0.1
database = reviewdb
username = gerrit
poolLimit = 150
poolMaxIdle = 16
[auth]
type = TEAMFORGE
gitBasicAuth = true
httpPasswordSettingsEnabled = true
cookiepath = /gerrit
[teamforge]
cache-path = /opt/collabnet/gerrit/cache
cache-ttl = 300
apiPort = 9081
refreshTimeout = 3600
jumboPushThreshold = 30
maxDiffSize = 25000
url = https://XXXXXXX
username = scmadmin
replicationMode = master
repositoryRoot = XXXX
maxFilesListedInTFCommitObject = 250
allowPushIfTeamForgeConnectionIsDown = false
parallelRemoteCallLimit = 9
maxRemoteCallRetry = 3
credentialsCache = true
credentialsCacheTimeOut = 3600
reconnectInterval = 30
notificationGitMultiMailPyPath = /opt/collabnet/gerrit/lib/
notificationMaxSize = 25000
notificationMaxPythonExecutors = 2
supportOrchestrateNotifications = true
commitProcessingTimeOut = 15min
createTFProjectLinkedApps = true
teamCachesTimout = 60
teamForgeMenuHeader = TeamForge
ensureStreamEventsForRegisteredUsers = true
ensureAdminRightsForSiteAdmins = true
notificationMaxOrchestrateExecutors = 2
syncTeamForgeProjectHierarchy = true
supportSiteWideRoles = true
supportDefaultAccessPermissions = true
externalSystemId = XXXX
[container]
user = gerrit
javaHome = /usr/java/jdk1.8.0_45
heapLimit = 40g
[gitweb]
cgi = /var/www/gitweb-caching/gitweb.cgi
forceTeamForgeCodeBrowser = false
[sshd]
listenAddress = *:29418
threads = 48
streamThreads = 5
maxConnectionsPerUser = 12
idleTimeout = 5min
backend = NIO2
[httpd]
listenUrl = proxy-https://localhost:9080/gerrit
registerMBeans = true
requestLog = false
maxQueued = 300
acceptorThreads = 3
minThreads = 10
maxThreads = 100
[cache]
directory = cache
[cache "diff"]
timeout = 120s
[cache "web_sessions"]
maxAge = 2d
[cache "projects"]
memoryLimit = 10m
maxAge = 2h
[cache "accounts"]
memoryLimit = 10m
maxAge = 2h
[cache "accounts_byname"]
memoryLimit = 10m
maxAge = 2h
[cache "groups"]
memoryLimit = 10m
maxAge = 2h
[cache "permission_sort"]
memoryLimit = 10m
maxAge = 2h
[sendemail]
smtpServer = localhost
[theme]
backgroundColor = FFFFFF
topMenuColor = B1C8E8
textColor = 264D69
trimColor = E5E5E5
selectionColor = D8EEFF
[commentlink "TeamForge"]
match = ((artf|doc|wiki|forum|topc|page|cmmt|rel|pkg|tracker|plan|taskgrp|task|team)\\d+)
html = <a target=_top href=https://XXXXX/sf/go/$1>$1</a>
[user]
anonymousCoward = User Data Not Available
[receive]
timeout = 8min
maxObjectSizeLimit = 150m
checkMagicRefs = true
checkReferencedObjectsAreReachable = false
changeUpdateThreads = 8
[plugins]
allowRemoteAdmin = true
jsLoadTimeout = 30s
[core]
packedGitWindowSize = 16k
packedGitOpenFiles = 4096
packedGitLimit = 2g
streamFileThreshold = 15m
[plugin "teamforge-replication"]
replicationDelay = 15s
replicationRetry = 5m
sshConnectionTimeout = 15s
sshCommandTimeout = 30s
pushTimeout = 30s
threads = 4
[index]
type = LUCENE
threads = 8
Index Command:
java -jar bin/gerrit.war reindex --index
Reply all
Reply to author
Forward
0 new messages