High memory and CPU spikes during High load and dev's clones fail with 502 error

[deekshith s n]

Hi Team, 

Recently we tried upgrading from 3.12.1 to 3.13.3. Post that during high load we see high memory and cpu spike which is causing downtime of our gerrit. Any suggestions on this below are our setup details 

We use active - active gerrit setup with HA cluster plugin, we host gerrit as docker container in aws ec2 instance 

Instance size: m7i.2xlarge ( 3 instances ) 
Shared EBS ( across 3 instance ): Type io2    ( throughput 12000) this is where we store git folder and lfs 
Clustering: PCS + DLM 

Gerrit config 

[gerrit]
    basePath = git
    serverId = "1233-5667-789"
    canonicalWebUrl = "https://git.uici.com/"
    instanceId = "git"
[container]
    user = gerrit
    javaHome = /opt/java/openjdk
    heapLimit = 24g
    javaOptions = "-Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance"
    javaOptions = "-Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance"
    javaOptions = -Djava.security.egd=file:/dev/./urandom
    javaOptions = --add-opens java.base/java.net=ALL-UNNAMED
    javaOptions = --add-opens java.base/java.lang.invoke=ALL-UNNAMED
[auth]
    type = OAUTH
    gitBasicAuthPolicy = HTTP
    logoutUrl = https://sso.com/idp/startSLO.ping?LogoutType=AsyncOnly
    cookiedomain = ".uic.com"
[receive]
    enableSignedPush = false
    timeout = 6min
[sendemail]
    from = "\"Gerrit\" <noreply@.com>"
    smtpServer = server
    smtpServerPort = 587
    smtpEncryption = tls
    smtpUser = "username"
    html = false
[sshd]
    threads = 32
    listenAddress = *:29418
    maxConnectionsPerUser = 0
    backend = NIO2
[httpd]
    listenUrl = proxy-https://*:8080
    maxQueued = 1000
    minThreads = 10
    maxThreads = 50
[cache]
    directory = cache
[gitweb]
    cgi = /usr/share/gitweb/gitweb.cgi
    type = gitweb
[change]
    submitWholeTopic = true
    mergeabilityComputationBehavior = API_REF_UPDATED_AND_CHANGE_REINDEX
[core]
    packedGitOpenFiles = 4096
[gc]
    startTime = Sat 23:59
    interval = 1 week
[lfs]
    plugin = lfs
    backend = fs
[index]
    type = lucene
[accountPatchReviewDb]
    url = "jdbc:postgresql://db:5432/gerritdatabase?user=user&password=gerrit1234"
[receive]
    autogc = false
[plugins]
    allowRemoteAdmin = true
[plugin "emoticons"]
    showEmoticons = true
[plugin "code-owners"]
    disabled = true
    requiredApproval = Code-Review+1
    overrideApproval = Owners-Override+1
    allowedEmailDomain = ui.com
    allowedEmailDomain = test.com

We tried reducing heap memory, set restriction threads didnt help. Also we tried increasing instance size m7.4xlarge couldnt really help. 

Did anyone face the same issue? if yes please suggest the changes.