How to prevent thread dead lock in Gerrit version(2.16.22)

[Swetha Laxman]

We have recently faced a issue in one of our Gerrit instance regarding the dead locks.

In the Javamelody (version - JavaMelody 1.83.0)version which we are using under threads section at few times we were able to find category called Dead lock threads, Due to which load is being high in the server which causes performance issues.

Also killing of the blocked threads was not a recommended way according to the statistics.  Could anyone help on how to identify or prevent the dead thread lock in the server and cause for the lock.

Thanks.

Suvedha

[Nasser Grainawi]

A few things:
1) It's hard for anyone to help without more info on the threads that are reported as deadlocks. Can you provide that?

2) 2.16.28 is the latest 2.16.x release, so you should at least upgrade to that as it contains bug fixes that might be relevant

3) 2.16 is EOL, so you should plan an upgrade to a stable supported version

Thanks,

Nasser

 

Thanks.

Suvedha

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/21ad4d0d-0553-4813-aafa-046c09b9efd4n%40googlegroups.com.

[Antoine Musso]

Hello,

I rarely had thread deadlock on our Gerrit (and we have been running various versions over the last twelve years). There is one thought that is been causing me to wake up at night thinking about it and I never found the reason. A summary:

We ran Gerrit 2.15.14

symptom: gerrit show-queue -w -q shows SendEmail tasks stuck which on our setup are processed by a single thread. The threads pilled up until Gerrit became unresponsive and we had to restart it entirely.

The stacktraces were always pointing at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad which is was most probably some kind of deadlock happening in Guava com.google.common.cache. Upstream recommended to move to another caching system (Caffeine).

The fix: we had none!

Eventually we changed the hardware (more better CPUs and more memory) and we upgraded the operating system from Debian Jessie to Debian Buster.  After that the issue no more occurred.  Maybe it was a problem in one of the library java links to or a Linux Kernel issue (but afaik we ran the same kernel on all machines).

I highly recommend you upgrade Gerrit, your users will get the newer UI and will definitely enjoy it. Our upgrade from 2.15 to 3.2 was conducted by Christian Aistleitner who wrote a summary on this list detailing the plan he used and the many tweaks he had to do:

https://groups.google.com/g/repo-discuss/c/G5wucKJg9Ag/m/pLin-i3mBgAJ

Hope that helps!

References

Our task with lot of various uninformed attempts to debug it: https://phabricator.wikimedia.org/T224448

The Gerrit issue we filed https://bugs.chromium.org/p/gerrit/issues/detail?id=7645

Guava issue with a very nice response from a maintainer: https://github.com/google/guava/issues/3602#issuecomment-530457714

-- 
Antoine "hashar" Musso
Release Engineering