Gerrit commands/UI hangs

Anu

unread,

Jul 6, 2015, 6:18:35 AM7/6/15

to repo-d...@googlegroups.com

Hello,

Recently I have noticed that there are intermittent access issues for any kind of Gerrit commands.

And after sometime, it will work normally and again it repeats after a while.

Could you please suggest if there is any fine tuning needed in our server setup?

1) Server is with 31 GB RAM with 16 cores.

2) In addition to that, I have conifgured Java heapLimit as "15GB".

Please find more details below:

Thanks, Anu

______________________________________________

[root@bhimscm51 etc]# free -g

total used free shared buffers cached

Mem: 31 29 1 0 0 13

-/+ buffers/cache: 16 15

Swap: 3 0 3

[root@bhimscm51 etc]#

$ ssh <server> -p 29418 gerrit show-caches

Gerrit Code Review 2.8.5 now 15:40:40 IST

uptime 7 days 2 hrs

Name |Entries | AvgGet |Hit Ratio|

| Mem Disk Space| |Mem Disk|

--------------------------------+---------------------+---------+---------+

accounts | 365 | 4.8ms | 99% |

accounts_byemail | 299 | 4.0ms | 99% |

accounts_byname | 367 | 4.1ms | 99% |

adv_bases | | | |

changes | | 8.7ms | 0% |

groups | 22 | 5.5ms | 91% |

groups_byinclude | 8 | 5.2ms | 66% |

groups_byname | | | |

groups_byuuid | 16 | 3.8ms | 95% |

groups_external | 1 | 4.3ms | 80% |

groups_members | 26 | 5.3ms | 99% |

ldap_group_existence | 4 | 9.1ms | 57% |

ldap_groups | 122 | 67.0ms | 99% |

ldap_groups_byinclude | | | |

ldap_usernames | 20 | 3.5ms | 51% |

permission_sort | 1024 | | 96% |

plugin_resources | 1 | | 99% |

project_list | 1 | 174.3ms | 99% |

projects | 105 | 2.0s | 99% |

sshkeys | 181 | 9.4ms | 99% |

D diff | 1263 3849 10.72m| 151.3ms | 78% 99%|

D diff_intraline | 798 798 474.09k| 15.6ms | 72% 100%|

D git_tags | 3 3 42.43k| | 74% 100%|

D web_sessions | 125 10904 4.31m| | 98% 8%|

SSH: 13 users, oldest session started 6 days 18 hrs ago

Tasks: 22 total = 14 running + 4 ready + 4 sleeping

Mem: 14.49g total = 9.58g used + 4.86g free + 49.27m buffers

14.49g max

128 open files, 16 cpus available, 162 threads

$

Saša Živkov

unread,

Jul 6, 2015, 8:09:59 AM7/6/15

to Anu, repo-d...@googlegroups.com

On Mon, Jul 6, 2015 at 12:18 PM, Anu <anugr...@gmail.com> wrote:

Hello,

Recently I have noticed that there are intermittent access issues for any kind of Gerrit commands.

Create a couple of thread dumps during the time a command hangs.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Anu

unread,

Jul 8, 2015, 10:12:29 AM7/8/15

to repo-d...@googlegroups.com, anugr...@gmail.com

Below is the error I see in "error_log" file....

[2015-07-08 19:24:55,854] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/7907/detail?O=404

org.eclipse.jetty.io.EofException

at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)

at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)

at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)

at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1078)

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:161)

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)

at org.eclipse.jgit.util.TemporaryBuffer.writeTo(TemporaryBuffer.java:248)

at com.google.gerrit.httpd.restapi.RestApiServlet$6.writeTo(RestApiServlet.java:981)

at com.google.gerrit.httpd.restapi.RestApiServlet.replyBinaryResult(RestApiServlet.java:722)

at com.google.gerrit.httpd.restapi.RestApiServlet.replyJson(RestApiServlet.java:626)

at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:330)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at com.google.gerrit.pgm.http.jetty.GetUserFilter.doFilter(GetUserFilter.java:76)