[JDBC] Random UnknownHostExceptions

[George Woodring]

We are seeing random UnknownHostExceptions and are looking for suggestions on trying to find out the root cause.  We are seeing one error per 100K connections as my best guesstimation. 

We are running on CentOS 5.2, tomcat 6.0.29, jdbc 9.0-801.jdbc4 and using tomcat-dbcp connection pooling.

An example of the failure senario is as follows.

At 2012-06-28 08:46:43,886 we got the exception at the bottom.  According to the tcpdump, there was not a DNS lookup for the server at this time and we have active DB connections to the server.

According to the db log, we had successful connections before and after the failure:

Jun 28 08:46:02 iprobe113 postgres[12386]: [2-1] LOG:  connection received: host=192.168.175.21 port=53592

Jun 28 08:46:02 iprobe113 postgres[12387]: [2-1] LOG:  connection received: host=192.168.175.21 port=53593

Jun 28 08:47:58 iprobe113 postgres[12461]: [2-1] LOG:  connection received: host=192.168.175.21 port=53614

Jun 28 08:47:58 iprobe113 postgres[12462]: [2-1] LOG:  connection received: host=192.168.175.21 port=53615

There are no DNS attempts for any of these connections.  According to the tcpdump, the next DNS attempt is not until 8:50.

Any suggestions would be appreciated.

George Woodring

The following is the exception we are seeing

org.postgresql.util.PSQLException: The connection attempt failed.
       at org.postgresql.Driver$ConnectThread.getResult(Driver.java:348)
       at org.postgresql.Driver.connect(Driver.java:271)
       at org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
       at org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
       at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
       at org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:79)
       at org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
       at org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)

       at org.apache.commons.dbutils.QueryRunner.prepareConnection(QueryRunner.java:195)
       at net.iglass.db.QueryRunner.prepareConnection(QueryRunner.java:211)
       at net.iglass.db.QueryRunner.query(QueryRunner.java:287)
       at net.iglass.jglass.core.dao.StaticGraphDAO.fetchCpeHistHealthDataForDataSources(StaticGraphDAO.java:1172)
       at net.iglass.jglass.core.dao.StaticGraphDAO.fetchCpeHistHealthDataSources(StaticGraphDAO.java:591)
       at net.iglass.jglass.jrobin.builder.CpeHistHealthBuilder.addLines(CpeHistHealthBuilder.java:131)
       at net.iglass.jglass.jrobin.builder.AbstractBuilder.buildMiniGraph(AbstractBuilder.java:266)
       at net.iglass.jglass.jrobin.builder.AbstractBuilder.buildGraph(AbstractBuilder.java:179)
       at net.iglass.jglass.jrobin.action.GraphAction.getGraph(GraphAction.java:145)
       at net.iglass.jglass.jrobin.action.GraphAction.doExecute(GraphAction.java:75)

       at com.cc.framework.adapter.struts.ActionUtil.execute(ActionUtil.java:751)
       at com.cc.framework.adapter.struts.FWAction.execute(FWAction.java:279)
       at net.iglass.jglass.BaseAction.execute(BaseAction.java:93)
       at com.cc.framework.adapter.struts.FWAction.execute(FWAction.java:314)
       at com.cc.framework.adapter.struts.RequestProcessorUtil.processAction(RequestProcessorUtil.java:76)
       at com.cc.framework.adapter.struts.FWRequestProcessor.processActionPerform(FWRequestProcessor.java:90)
       at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
       at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
       at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at net.iglass.jglass.servlets.SessionListener.doFilter(SessionListener.java:529)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at net.iglass.jglass.servlets.AccessControlFilter.doFilter(AccessControlFilter.java:303)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
       at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:427)
       at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:384)
       at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1584)
       at java.lang.Thread.run(Thread.java:662)

Caused by: java.net.UnknownHostException: iprobe113

       at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:177)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
       at java.net.Socket.connect(Socket.java:529)
       at java.net.Socket.connect(Socket.java:478)
       at java.net.Socket.<init>(Socket.java:375)
       at java.net.Socket.<init>(Socket.java:189)
       at org.postgresql.core.PGStream.<init>(PGStream.java:62)
       at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76)
       at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
       at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
       at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
       at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
       at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
       at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
       at org.postgresql.Driver.makeConnection(Driver.java:393)
       at org.postgresql.Driver.access$100(Driver.java:47)
       at org.postgresql.Driver$ConnectThread.run(Driver.java:308)
       ... 1 more

--
iGLASS Networks
www.iglass.net

[Dave Cramer]

Not exactly sure how to help you here as clearly this is not a postgresql issue.

Dave Cramer

dave.cramer(at)credativ(dot)ca
http://www.credativ.ca

--
Sent via pgsql-jdbc mailing list (pgsql...@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

[Guillaume Cottenceau]

George Woodring <george.woodring 'at' iglass.net> writes:

> Caused by: java.net.UnknownHostException: iprobe113
>
>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:177)
>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>        at java.net.Socket.connect(Socket.java:529)
>        at java.net.Socket.connect(Socket.java:478)
>        at java.net.Socket.<init>(Socket.java:375)
>        at java.net.Socket.<init>(Socket.java:189)
>        at org.postgresql.core.PGStream.<init>(PGStream.java:62)

The PG driver seems to be using the standard Socket java
interface, so there's little chance it has anything to do with
it. I would try to run a local Java program using a Socket object
to connect to a fake port on the same machine, for example you
could base on network connection code from:

http://code.google.com/p/androidnetworktester/source/browse/trunk/src/org/gc/networktester/tester/TcpConnectionTester.java

Btw, one per 100K connections means you keep on creating new
connections. This usually has very poor performance. You should
use a connection pool.

--
Guillaume Cottenceau