Random Database Connection Error

[Nick Hatter]

Hello Lifters,

This morning our service was taken out of action all of a sudden by these database connection errors:

org.postgresql.util.PSQLException: Something unusual has occured to cause the driver to fail. Please report this exception.
        at org.postgresql.Driver.connect(Driver.java:287)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:215)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:45)

10:01:02.242 [http-bio-8080-exec-1723] ERROR net.liftweb.http.LiftRules - Exception being returned to browser when processing /playerkey
java.lang.NullPointerException: Looking for Connection Identifier ConnectionIdentifier(lift) but failed to find either a JNDI data source with the name lift or a lift connection manager with the correct name

 

Line 45 in our Boot.scala is this:

Full(DriverManager.getConnection(
"jdbc:postgresql://%s:%s/%s" format (host,port,db),
user, pass))

One of the instances was connecting just fine to our database, but the other instance was having a hissy fit. The database and instance metrics all look OK.

We've temporarily resolved it by firing up a new EC2 instance... but I'm scared of it happening again so randomly. This is the first time we've seen this occur.

If anyone has any ideas, please do let me know.

We're using Scala 2.10.4 with Lift version 2.6, AWS Elastic Beanstalk with Tomcat 7 Java 7, and have t2.large instances with 4GB heap + perm size. Our database is running with Postgres 9.3 on a db.m4.large instance with multi-AZ failover. Hope this helps.

Thanks in advance,

Nick

[Antonio Salazar Cardozo]

Hmmm… It's hard to tell for sure but this line:

  org.postgresql.util.PSQLException: Something unusual has occured to cause the driver to fail. Please report this exception.

Seems like it's probably the root cause, and it seems like reporting it/talking to

the Postgres folks might be the best first step to debugging. It seems like this

error in turn causes the NPE when looking up the connection identifier (because

the connection failed to be built).

Thanks,

Antonio