Hitting 504 gateway timeout intermittently while accessing gerrit web UI

332 views
Skip to first unread message

Raghavendra Pathi

unread,
Jun 26, 2018, 2:57:04 AM6/26/18
to Repo and Gerrit Discussion
We are using v2.13.9 and are experiencing 504 gateway timeout's while browsing the gerrit UI, What are the initial steps to identify the problem.

Bassem Rabil

unread,
Jun 27, 2018, 6:57:45 AM6/27/18
to Repo and Gerrit Discussion
You can consider installing JavaMelody plugin into your instance to have more insights on what is running  and CPU/Memory usage [1]. This could be a good starting point to troubleshoot your instance.

Raghavendra Pathi

unread,
Jul 3, 2018, 4:47:54 AM7/3/18
to Repo and Gerrit Discussion
Is it safe to install this plugin on a production environment. Once enabled will it further deteriorate end user experience.

Bassem Rabil

unread,
Jul 3, 2018, 7:01:34 AM7/3/18
to Repo and Gerrit Discussion
Yes, it is safe and won't affect user experience, it is used in most of big Gerrit instances for monitoring and troubleshooting.

Raghavendra Pathi

unread,
Jul 3, 2018, 8:19:24 AM7/3/18
to Repo and Gerrit Discussion
I am a Python dev. Could you please help with the instructions on how to build this Jar file?

I get this error every-time i try to build it,

/private/var/tmp/_bazel_root/495e9cfbcb2de4cfa1e16b4edb650ffe/external/local_config_cc/BUILD:50:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-darwin_x86_64: Xcode version must be specified to use an Apple CROSSTOOL. If your Xcode version has changed recently, try: "bazel clean --expunge" to re-run Xcode configuration
ERROR: Analysis of target '//:javamelody' failed; build aborted: Analysis of target '@local_config_cc//:cc-compiler-darwin_x86_64' failed; build aborted

David Pursehouse

unread,
Jul 3, 2018, 8:22:23 AM7/3/18
to Raghavendra Pathi, Repo and Gerrit Discussion
Did you try "bazel clean --expunge" as suggested in the error message?   This usually works for me.
 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

David Pursehouse

unread,
Jul 3, 2018, 8:23:17 AM7/3/18
to Raghavendra Pathi, Repo and Gerrit Discussion
On Tue, Jul 3, 2018 at 9:22 PM David Pursehouse <david.pu...@gmail.com> wrote:
On Tue, Jul 3, 2018 at 9:19 PM Raghavendra Pathi <raghave...@gmail.com> wrote:
I am a Python dev. Could you please help with the instructions on how to build this Jar file?


Also note that you can download the prebuilt jar rather than trying to build it yourself:

Raghavendra Pathi

unread,
Jul 3, 2018, 8:29:06 AM7/3/18
to Repo and Gerrit Discussion
Yes I did that and then it fails with same error.

Raghavendra Pathi

unread,
Jul 3, 2018, 8:34:08 AM7/3/18
to Repo and Gerrit Discussion
Oh ok. Thanks for pointing it out.

I see dependency jar files as well, The main jar javamelody.jar will go into gerrit plugins, what about the dependencies?

Also, In order to enable database interceptor do i have to do anything?

David Pursehouse

unread,
Jul 3, 2018, 8:50:41 AM7/3/18
to Raghavendra Pathi, Repo and Gerrit Discussion
It's documented here:


Put the javamelody-nodep.jar file in the site's plugins folder, and the javamelody-deps.jar and javamelody-data-interceptor.jar files in the site's lib folder.

Raghavendra Pathi

unread,
Jul 3, 2018, 10:04:53 AM7/3/18
to Repo and Gerrit Discussion
I followed the steps from the document and the monitoring tab now comes up which shows number of threads running, http requests etc. But the database interceptor is not getting activated as the sql statistics is not yet seen.

I also see some errors like below,

[2018-07-03 13:44:01,032] [HTTP-119] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /monitoring?width=200&height=50&graph=Free_disk_space
java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11FontManager
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)

Message has been deleted

Raghavendra Pathi

unread,
Jul 5, 2018, 5:27:45 AM7/5/18
to Repo and Gerrit Discussion
I see some URLs taking very long to respond ~900,000 ms. There are no slow queries. We have 8G, 4vcpu running in a container. Configured with 25 http threads. Any leads to diagnose this problem would be really helpful.

/changes/488/revisions/3ef73fd06e25webc8732daa50cb37b58be0cb2f8/files GET 33 7 545,201 957,008 509,959 0 2 0.00 0
/projects/xxxmng/config GET 16 13 146,386 954,152 357,310 0 1 0.00 0
/changes/488/revisions/3ef73fd06e25webc8732daa50cb37b58be0cb2f8/drafts GET 8 4 239,476 957,861 478,923 0 0 0.00 0
/changes/489/revisions/8cd3e3da50511d6fc649c3e29bb75dd1dce1bb3d/drafts GET 8 4 239,212 956,820 478,405 0 1 0.00 0
/changes/435/revisions/ad5891f0f04dead94e633fb9322fe7b744150765/drafts GET 8 1 954,171 954,171 0 0 5 0.00 0
/changes/435/revisions/ad5891f0f04dead94e633fb9322fe7b744150765/files GET 8 2 475,054 950,076 671,781 0 3 0.00 0
/changes/489/revisions/c3e03c228eba432a7a6e56de5b0ed279cef2ce81/actions GET 8 9 105,512 949,235 316,395 0 5 0.00 0
/changes/487/revisions/edc319d03b9c499c64b7744e9c225f38b7b2ebb6/files GET 8 2 474,492 948,913 670,932 0 1 0.00 0

Saša Živkov

unread,
Jul 5, 2018, 6:08:38 AM7/5/18
to raghave...@gmail.com, Repo and Gerrit Discussion
On Thu, Jul 5, 2018 at 11:27 AM Raghavendra Pathi <raghave...@gmail.com> wrote:
I see some URLs taking very long to respond ~900,000 ms. There are no slow queries. We have 8G, 4vcpu running in a container. Configured with 25 http threads. Any leads to diagnose this problem would be really helpful.

1. Do you regularly "git gc" all repositories?

2. 900s is quite long period which gives you enough time to create several java thread dumps. Look inside each of them and try to find which
thread is processing that long running request, then compare the same thread in subsequent thread dumps. This should give you a hint.


/changes/488/revisions/3ef73fd06e25webc8732daa50cb37b58be0cb2f8/files GET       33      7       545,201         957,008         509,959         0       2       0.00    0
/projects/xxxmng/config GET     16      13      146,386         954,152         357,310         0       1       0.00    0
/changes/488/revisions/3ef73fd06e25webc8732daa50cb37b58be0cb2f8/drafts GET      8       4       239,476         957,861         478,923         0       0       0.00    0
/changes/489/revisions/8cd3e3da50511d6fc649c3e29bb75dd1dce1bb3d/drafts GET      8       4       239,212         956,820         478,405         0       1       0.00    0
/changes/435/revisions/ad5891f0f04dead94e633fb9322fe7b744150765/drafts GET      8       1       954,171         954,171         0       0       5       0.00    0
/changes/435/revisions/ad5891f0f04dead94e633fb9322fe7b744150765/files GET       8       2       475,054         950,076         671,781         0       3       0.00    0
/changes/489/revisions/c3e03c228eba432a7a6e56de5b0ed279cef2ce81/actions GET     8       9       105,512         949,235         316,395         0       5       0.00    0
/changes/487/revisions/edc319d03b9c499c64b7744e9c225f38b7b2ebb6/files GET       8       2       474,492         948,913         670,932         0       1       0.00    0

Raghavendra Pathi

unread,
Jul 5, 2018, 7:35:23 AM7/5/18
to Repo and Gerrit Discussion
MY GC runs every day midnight.

And these long response times are not too often, But next time it happens I will get the dump and compare as you suggested.

Raghavendra Pathi

unread,
Jul 5, 2018, 10:38:01 PM7/5/18
to Repo and Gerrit Discussion
I've noticed these two behaviors, Some tasks are hung[0] for a long time and also I see this stacktrace [1] once in a while, could this stack-trace mean something wrong with the indexing?

[0]
Task State StartTime Command
------------------------------------------------------------------------------
933f22ae 23:00:00.003 13:08:12.764 Log File Compressor
d33590d1 Jul-08 07:00 13:08:17.607 GC runner
------------------------------------------------------------------------------

[1]
[2018-07-05 21:14:17,461] [ReceiveCommits-1] ERROR com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener : interrupted
com.google.gwtorm.server.OrmException: interrupted
at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource.read(LuceneChangeIndex.java:317)
at com.google.gerrit.server.index.change.IndexedChangeQuery.read(IndexedChangeQuery.java:90)
at com.google.gerrit.server.query.QueryProcessor.query(QueryProcessor.java:191)
at com.google.gerrit.server.query.QueryProcessor.query(QueryProcessor.java:139)
at com.google.gerrit.server.query.QueryProcessor.query(QueryProcessor.java:122)
at com.google.gerrit.server.query.InternalQuery.query(InternalQuery.java:71)
at com.google.gerrit.server.query.change.InternalChangeQuery.byBranchCommit(InternalChangeQuery.java:267)
at com.googlesource.gerrit.plugins.reviewnotes.CreateReviewNotes.loadPatchSet(CreateReviewNotes.java:256)
at com.googlesource.gerrit.plugins.reviewnotes.CreateReviewNotes.createNotes(CreateReviewNotes.java:159)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener.createReviewNotes(RefUpdateListener.java:102)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener.access$000(RefUpdateListener.java:37)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener$1.run(RefUpdateListener.java:65)
at com.googlesource.gerrit.plugins.reviewnotes.RefUpdateListener.onGitReferenceUpdated(RefUpdateListener.java:91)
at com.google.gerrit.server.extensions.events.GitReferenceUpdated.fire(GitReferenceUpdated.java:119)
at com.google.gerrit.server.extensions.events.GitReferenceUpdated.fire(GitReferenceUpdated.java:88)
at com.google.gerrit.server.git.ReceiveCommits.processCommands(ReceiveCommits.java:656)
at com.google.gerrit.server.git.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:89)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:221)
at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:200)
at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator$1.call(ThreadLocalRequestScopePropagator.java:55)
at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:99)
at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:131)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:417)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Raghavendra Pathi

unread,
Jul 10, 2018, 1:31:30 AM7/10/18
to Repo and Gerrit Discussion
Some more context on how we are running the application.

Its running on a docker based container with a NFS based storage attached as a data directory. I am suspecting lucene to be the bottle neck here. Is there a way to fine tune the Lucene configurations? Or Any leads to debug on that perspective.

Reply all
Reply to author
Forward
0 new messages