5s Delay During Push/Upload (It adds up!)

128 views
Skip to first unread message

andy

unread,
Aug 30, 2010, 9:29:51 PM8/30/10
to Repo and Gerrit Discussion
New gerrit setup on a server. Trying to push new branch to ~160
projects but it takes 5 seconds per project!

I think it has to do with getHostName in the code.... Check out this
example:

time ssh -p 29418 xx.xx.xx.xx gerrit show-connections
real 0m5.532s

time ssh -p 29418 xx.xx.xx.xx gerrit show-connections -n
real 0m0.494s

Anyone have any suggestions for a quick fix? Maybe an option in the
configuration?

PS. Huge thanks for the community's for all your hard work and support
on this amazing project!



Shawn Pearce

unread,
Aug 30, 2010, 9:43:41 PM8/30/10
to andy, Repo and Gerrit Discussion
On Mon, Aug 30, 2010 at 18:29, andy <andre...@gmail.com> wrote:
> New gerrit setup on a server. Trying to push new branch to ~160
> projects but it takes 5 seconds per project!

Ouch!

> I think it has to do with getHostName in the code.... Check out this
> example:
>
> time ssh -p 29418 xx.xx.xx.xx gerrit show-connections
> real    0m5.532s
>
> time ssh -p 29418 xx.xx.xx.xx gerrit show-connections -n
> real    0m0.494s

Hmm, yes, that does seem to be suggesting that looking up hostnames
through your DNS server is taking 5s. Or at least that's true coming
from the JVM you are running. :-)

Perhaps your primary DNS server in /etc/resolv.conf isn't responding?
So the JVM is looking up on that server first, waiting 5s for it to
timeout with no response, then going to the next fallback server and
finally getting an answer. But maybe other programs on your system
are searching them concurrently, or are starting a query on the
secondary server after a much lower timeout waiting for the primary,
so you aren't really noticing this issue with other applications.
Just a shot in the dark, but I'm reasonably certain the JVM does its
own DNS client these days.

> Anyone have any suggestions for a quick fix? Maybe an option in the
> configuration?

If re-ordering your resolv.conf doesn't work, you can try increasing
the number of threads in the replication.config remote block for the
relevant remote. Permitting more concurrent connections will help you
to do those DNS lookups in parallel, rather than serial, so its 5 *
100 / threads rather than 5 * 100 seconds. :-\

Shawn Pearce

unread,
Aug 30, 2010, 10:03:34 PM8/30/10
to andy, Repo and Gerrit Discussion
On Mon, Aug 30, 2010 at 18:43, Shawn Pearce <s...@google.com> wrote:
> On Mon, Aug 30, 2010 at 18:29, andy <andre...@gmail.com> wrote:
>> New gerrit setup on a server. Trying to push new branch to ~160
>> projects but it takes 5 seconds per project!
>
> Perhaps your primary DNS server in /etc/resolv.conf isn't responding?

Actually this old Sun bug:

http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=5092063

suggests that the 5s latency can occur when there is no reverse DNS
entry configured for a given IP address. Its possible that the JSch
client code is looking up the hostname, discarding it and getting the
IP address only, then creating an InetAddress from that, which is
triggering reverse DNS lookup again to find out how to contact the
hostname (e.g. should we use a socks proxy or not).

andy

unread,
Aug 31, 2010, 3:06:45 PM8/31/10
to Repo and Gerrit Discussion
Thank you, its FIXED! For those having the same issue, here is the
fix: Modify IdentifiedUser.java, replace getCanonicalHostName with
getHostAddress instead.

If you don't want to recompile its worth looking at your resolv.conf
to make sure its properly configured like Shawn said, and look at the
workaround suggestions in the sun bug link (None of those were
practical in my case).

On Aug 30, 10:03 pm, Shawn Pearce <s...@google.com> wrote:
> On Mon, Aug 30, 2010 at 18:43, Shawn Pearce <s...@google.com> wrote:

Joyer Huang

unread,
Nov 9, 2015, 3:19:20 AM11/9/15
to Repo and Gerrit Discussion, andre...@gmail.com
We have another new gerrit setup on a server.
Every run of `ssh xxx gerrit plugin ls' take about 5 secs to run.
Could it be the same kind of problem?
Reply all
Reply to author
Forward
0 new messages