Issue of "java.net.SocketException: Connection timed out"

[Chunlin Zhang]

I got this issue from time to time these weeks, related log below.
That is a socket postgresql using, that will be likely to be resolved if I do the database replication, but I still want to know what the timeout is exactly? Can I change the timeout value?

Thanks

-----
[2015-11-02 00:17:14,459] WARN  com.google.gerrit.server.account.GroupIncludeCacheImpl : Cannot load members of group
java.util.concurrent.ExecutionException: com.google.gwtorm.server.OrmException: fetch failure on account_groups
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
        at com.google.gerrit.server.account.GroupIncludeCacheImpl.subgroupsOf(GroupIncludeCacheImpl.java:91)
        at com.google.gerrit.server.account.IncludingGroupMembership.containsAnyOf(IncludingGroupMembership.java:93)
        at com.google.gerrit.server.account.IncludingGroupMembership.contains(IncludingGroupMembership.java:69)
        at com.google.gerrit.server.account.UniversalGroupBackend$UniversalGroupMembership.contains(UniversalGroupBackend.java:129)
        at com.google.gerrit.server.project.ProjectControl.match(ProjectControl.java:515)
        at com.google.gerrit.server.project.ProjectControl.match(ProjectControl.java:502)
        at com.google.gerrit.server.project.RefControl.access(RefControl.java:641)
        at com.google.gerrit.server.project.RefControl.access(RefControl.java:613)
        at com.google.gerrit.server.project.RefControl.doCanPerform(RefControl.java:549)
        at com.google.gerrit.server.project.RefControl.canPerform(RefControl.java:540)
        at com.google.gerrit.server.project.ProjectControl.canPerformOnAnyRef(ProjectControl.java:439)
        at com.google.gerrit.server.project.ProjectControl.isVisible(ProjectControl.java:261)
        at com.google.gerrit.server.project.ProjectControl$GenericFactory.validateFor(ProjectControl.java:95)
        at com.google.gerrit.server.args4j.ProjectControlHandler.parseArguments(ProjectControlHandler.java:77)
        at org.kohsuke.args4j.CmdLineParser.parseArgument(CmdLineParser.java:508)
        at com.google.gerrit.util.cli.CmdLineParser.parseArgument(CmdLineParser.java:221)
        at com.google.gerrit.sshd.BaseCommand.parseCommandLine(BaseCommand.java:205)
        at com.google.gerrit.sshd.BaseCommand.parseCommandLine(BaseCommand.java:188)
        at com.google.gerrit.sshd.AbstractGitCommand$1.executeParseCommand(AbstractGitCommand.java:65)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:440)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.gwtorm.server.OrmException: fetch failure on account_groups
        at com.google.gwtorm.schema.sql.SqlDialect.convertError(SqlDialect.java:158)
        at com.google.gwtorm.schema.sql.DialectPostgreSQL.convertError(DialectPostgreSQL.java:65)
        at com.google.gwtorm.jdbc.JdbcAccess.convertError(JdbcAccess.java:459)
        at com.google.gwtorm.jdbc.JdbcAccess.queryList(JdbcAccess.java:156)
        at com.google.gerrit.reviewdb.client.AccountGroup_Access_accountGroups_GwtOrm$$7.byUUID(Unknown Source)
        at com.google.gerrit.server.account.GroupIncludeCacheImpl$SubgroupsLoader.load(GroupIncludeCacheImpl.java:149)
        at com.google.gerrit.server.account.GroupIncludeCacheImpl$SubgroupsLoader.load(GroupIncludeCacheImpl.java:136)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
        ... 33 more
Caused by: org.postgresql.util.PSQLException: An I/O error occured while sending to the backend.
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:283)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
        at com.google.gwtorm.jdbc.JdbcAccess.queryList(JdbcAccess.java:144)
        ... 38 more
Caused by: java.net.SocketException: Connection timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
        at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
        at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
        at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1661)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
        ... 44 more
                                                                                                  

[Chunlin Zhang]

I often get this kind of issue these years for those gerrit mirror connect to remote db server, the network between them sometime not so stable.

Today after search lots of document, I see this:

- http://stackoverflow.com/questions/2166872/how-to-use-tcp-keepalives-settings-in-postgresql

- https://www.gnugk.org/keepalive.html

My assumption is that:

- Those "java.net.SocketException: Connection timed out" error indicate that the tcp connection is corrupted, but the db client in gerrit cannot detect and drop the db connection, so if I get this kind of timeout issue, the only thing I can do is restart gerrit

- If I change the keepalive timeout value from 7200(ubuntu default) to 60, the tcp connection will auto-detect the corruption and disconnect itself quickly

I have change the value, but need wait some time to see whether it works.

root@gerritserver:~# sysctl -A | grep net.ipv4|grep keep

net.ipv4.tcp_keepalive_intvl = 75

net.ipv4.tcp_keepalive_probes = 9

net.ipv4.tcp_keepalive_time = 7200

root@gerritserver:~# sysctl -w net.ipv4.tcp_keepalive_time=40 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=8

net.ipv4.tcp_keepalive_time = 40

net.ipv4.tcp_keepalive_probes = 3

net.ipv4.tcp_keepalive_intvl = 8

在 2015年11月2日星期一 UTC+8上午10:14:29，Chunlin Zhang写道：