[play-2.2] Database exceptions since upgrade

[Niklas Nylund]

Since upgrading we have started to see some exceptions from time to time that didn't happen previously. We are using Play with Ebean and MySQL, has anyone seen similar behavior to this? The exceptions are thrown in controller actions not doing anything special (no async stuff, akka etc), it's just a web page and we have very light load. This is kind of odd since there hasn't been an update to Ebean in a year (i think). 

Caused by: javax.persistence.PersistenceException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
 at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:339)
 at com.avaje.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2018)
 at com.avaje.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:178)
 at com.avaje.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1459)
 at com.avaje.ebeaninternal.server.core.DefaultBeanLoader.loadMany(DefaultBeanLoader.java:140)
 at com.avaje.ebeaninternal.server.core.DefaultServer.loadMany(DefaultServer.java:499)
 at com.avaje.ebeaninternal.server.loadcontext.DLoadManyContext.loadMany(DLoadManyContext.java:139)
 at com.avaje.ebean.common.AbstractBeanCollection.lazyLoadCollection(AbstractBeanCollection.java:127)
 at com.avaje.ebean.common.BeanSet.init(BeanSet.java:91)
 at com.avaje.ebean.common.BeanSet.size(BeanSet.java:279)
......

Caused by: javax.persistence.PersistenceException: java.sql.SQLException: Timed out waiting for a free available connection.
       at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:339)

       at com.avaje.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2018)    
              at com.avaje.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:178)

       at com.avaje.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1459)

       at com.avaje.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:885)

      ... 48 more

Caused by: java.sql.SQLException: Timed out waiting for a free available connection.

       at com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal(DefaultConnectionStrategy.java:87)

       at com.jolbox.bonecp.AbstractConnectionStrategy.getConnection(AbstractConnectionStrategy.java:90)

       at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:540)

       at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:131)

       at play.db.ebean.EbeanPlugin$WrappingDatasource.getConnection(EbeanPlugin.java:147)

       at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:313)
.....

[virtualeyes]

What does
play> dependencies

show for boneCP version?

[Niklas Nylund]

Right, I found quite a few posts about problems with BoneCP while googling about this issue today.

dependencies says we are running with com.jolbox:bonecp:0.8.0-rc1. Should it be a newer one?


Niklas

> --
> You received this message because you are subscribed to a topic in the Google Groups "play-framework" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/play-framework/7PBnjiXkNuU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to play-framewor...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Niklas Nylund]

There seems to be a slightly newer version, BoneCP 0.8.0 Release. I'll try that and see if it helps. 

We are now seeing a couple of these exceptions every night so it may be caused by the BoneCP bug that has been discussed earlier, we have close to no traffic during the night and maybe the GC isn't run then and connections aren't freed.

Niklas

> To unsubscribe from this group and all its topics, send an email to play-framework+unsubscribe@googlegroups.com.

[Niklas Nylund]

Hmm, how do I get Play/SBT to pick up the new version? I added   "com.jolbox" % "bonecp" % "0.8.0-RELEASE"  to plugins.sbt but dependencies still show that I am using rc1.

Niklas

[Emre Çelikten]

Hello Niklas,

See: https://groups.google.com/forum/#!topic/play-framework/VPPYN_Vdrgg

Hope this helps,

Emre

> To unsubscribe from this group and all its topics, send an email to play-framewor...@googlegroups.com.

[Niklas Nylund]

Thanks for the link but unfortunately I am not able to get Play to pull in a new version of BoneCP.

I still get this when running dependencies,

| com.jolbox:bonecp:0.8.0-rc1                                       | com.typesafe.play:play-jdbc_2.10:2.2.0                 | As bonecp-0.8.0-rc1.jar                  |

|                                                                   | default:cmsapi_2.10:1.0-SNAPSHOT                       |                                          |

+-------------------------------------------------------------------+--------------------------------------------------------+------------------------------------------+

My build.sbt file looks like this,

name := "cmsapi"

version := "1.0-SNAPSHOT"

play.Project.playJavaSettings

libraryDependencies ++= Seq(

  javaJdbc,

  javaEbean,

  cache,

  "org.avaje.ebeanorm" % "avaje-ebeanorm-api" % "3.1.1",

  "mysql" % "mysql-connector-java" % "5.1.18",

  "com.typesafe" %% "play-plugins-mailer" % "2.1-RC2",

  "commons-collections" % "commons-collections" % "2.1.1",

  "org.jdom" % "jdom" % "2.0.2",

  "commons-io" % "commons-io" % "2.4",

  "commons-lang" % "commons-lang" % "2.6",

  "commons-codec" % "commons-codec" % "1.8",

  "com.typesafe.play" %% "play-jdbc" % "2.2.0" exclude("com.jolbox", "bonecp"),

  "com.jolbox" % "bonecp" % "0.8.0.RELEASE"

)   

Niklas

[Emre Çelikten]

Hello again,

Just asking to be sure: Have you tried reload and clean commands after changing your build.sbt?

Emre

[virtualeyes]

RC1 is asking for a hosing, everyone on 2.2 really needs 0.8 stable (likely one of the motivations for the quick 2.2.1 release).

Do something like this in your Build.scala:

object ApplicationBuild extends Build {
  import Deps._

  val rootDeps = bonecp ++ Seq(postgres, cache) ++ pdf
  lazy val root = play.Project("flubagge", "0.1.0", rootDeps, ...)

  object Deps {
    ...
    val bonecp = Seq(

      "com.typesafe.play" %% "play-jdbc" % "2.2.0" exclude("com.jolbox", "bonecp"),
      "com.jolbox" % "bonecp" % "0.8.0.RELEASE"
    )
  }
}

Play 2.2's built-in "jdbc" dep pulls in the to-be-avoided 0.8 RC1, which takes precedence over whatever you specify as the bonecp dep, so we manually exclude the f-er, thereby experience stability ;-)

Once in a great while I get a transient connection exception, but that may be network related, or perhaps caused by Linux and its uber long default keepalive

Good luck...

[Niklas Nylund]

Thanks for the tip.

I somehow completely missed that 2.2.1 is out now and it should have a new bonecp. I'll try upgrading to that first.

Niklas

[Niklas Nylund]

FYI since upgrading one project to 2.2.1 we only saw one of these exceptions during last night instead of 5-10. While this is better it's still a bit worrying.

Niklas

[Michael Seid]

Let me know if there is any update on this. I am also experiencing this, and not sure how to fix it.

Thanks for reporting this and helping.  I look forward to any news.

[Niklas Nylund]

2.2.1 and the new version of BoneCP doesn't fix this. I haven't found any work around for this.

Niklas

[Jxtps]

We are also seeing this issue on 2.2.1.

[Niklas Nylund]

Has anyone successfully downgraded BoneCP to the older version that was used in 2.1.x?

Niklas

[virtualeyes]

Likewise, same here, still occurring.

Is BoneCP 0.8 stable or is this Play 2.2 specific?

[virtualeyes]

Trying to sort out where these errors are coming from; the variables at play that I can see are:

1) Underlying DB will have a default idle connection timeout (mine is 8H)
2) Play's default idleConnectionMaxAge for BoneCP is 1H.
3) OS has its own default KeepAlive (2H on RedHat based distros).

In this scenario the OS never keeps the inactive connection alive since 2H > 1H, the BoneCP idle timeout. It is unlikely that the DB is closing connections given the 8H timeout (no DB log entries to that effect in my case), so it would appear that BoneCP is for some reason not properly closing idle connections, or is attempting to recycle already closed connections rather than drawing an active connection from the pool.

Side note: while benchmarking my application I increased the number of available threads (i.e. overriding Play default akka config) well beyond actual day-to-day server load. I left the benchmark config in place and have seen a 10X increase in jdbc.SQLError.createSQLException errors.

Probably best to tune the akka thread pool as close as possible to real life load in order to minimize the damage of whatever is happening with BoneCP and idle connections.

[Niklas Nylund]

I haven't had time to debug this now but in our case it happens mostly with a database that has light load and most likely is idle for long stretches. Most exceptions seems to be thrown when googlebot finds the pages in the middle of the night and causes a lot of requests. This is all circumstantial but it points towards the same things you mention about idle connections.

Niklas

[virtualeyes]

After trimming down pool size appropriate to actual server load, now I see the pattern:

At least 1 hour between errors, which is default max idle in play config I believe.

This can occur at anytime of day, so the oh-it's-just-a-bot notion doesn't apply.

Solution must be to just hammer one's site with traffic 24/7, if one could be so lucky ;-)

[Niklas Nylund]

Can someone from the core team comment on this? Do you have anything in production with the default Java stack and also getting these errors?

How do I go about downgrading BoneCP?


Niklas

[Volker Hatz]

I am also surprised that this is not treated more offensively. This is a pretty annoying bug and its hard to recommend to go productive with that error.

I have combined both proposed configuration changes and this seems to work. But time will tell.

application.conf: added db.default.connectionTestStatement="SELECT 1"

ApplciationBuild: remove "jdbc" and add:

    "com.typesafe.play" %% "play-jdbc" % "2.2.0" exclude("com.jolbox", "bonecp"),

    "com.jolbox" % "bonecp" % "0.8.0.RELEASE" 

Anyhow this cannot be a final solution.

[Niklas Nylund]

On Tuesday, November 19, 2013 9:11:01 PM UTC+2, Volker Hatz wrote:

I am also surprised that this is not treated more offensively. This is a pretty annoying bug and its hard to recommend to go productive with that error.

I have combined both proposed configuration changes and this seems to work. But time will tell.

application.conf: added db.default.connectionTestStatement="SELECT 1"

This seems to help somewhat but it doesn't fix the problem completely for us. The amount of errors in our logs have dropped over the last 2 days with roughly 75% after I added this.

Niklas

[virtualeyes]

Interesting, I thought that "Select 1" was the BoneCP default, which would mean this really has no affect:

db.default.connectionTestStatement="SELECT 1"

As setting it or not will result in the same thing.

What stopped the problem entirely was setting:

maxConnectionAge=0

Not a single connection error in 72 hours, vs. several per day.

Note: am also on 0.8.1-snapshot (to get fix for prepared statement cache bug found in 0.8-release), so not sure if that has also improved stability.

[Volker Hatz]

When I added the "Select 1" I got rid of the issues. Also I have the logging of database statements "on". It only shows the "Select 1" when you add it as a db.default.connectionTestStatement property.

[virtualeyes]

Thanks, you're right, I just checked BoneCP default config, they fallback to a slower db agnostic metadata test statement if not set.

Will add that in now ;-)

[Michael Seid]

Do you mind sharing some Build.scala and your application.conf snippets? I still am running into these errors.  All of these configs have helped, but it still comes up about 3 to 4 times a day.  

Thanks for your help.

[virtualeyes]

Are you on Play 2.2.1 or manually pulling in BoneCP 0.8.0 via an earlier version of Play?

I ask because it sounds like some users have found stability with 0.8.0 and there may be no need for 0.8.1-snapshot to address this particular issue. Play 2.2.1 comes with 0.8.0 out of the box so nothing to do short of setting:
maxConnectionAge=0

and perhaps setting connectionTestStatement specific to your DBMS (for MySQL, "SELECT 1").

[Michael Seid]

Thanks for the help! I am using 2.2.1 standard. But it still seems be borked for me. Also, I am connecting to a MySQL instance on Amazon RDS. Do you think there is any configuration I need to do their? Nothing changed over there, but I'm not sure if I should update any connection settings.

Here is my configuration. Any ideas?

db.default.partitionCount=2

db.default.maxConnectionsPerPartition=10

db.default.minConnectionsPerPartition=10

db.default.connectionTimeout=10 seconds

db.default.idleConnectionTestPeriod=1 minute

db.default.idleMaxAge=10 minutes

db.default.maxConnectionAge=0

db.default.initSQL="SELECT 1"

db.default.connectionTestStatement="SELECT 1"

Thanks for the help. This problem has really been a tough one to figure out.

[virtualeyes]

BoneCP author said partitionCount should = number of cpu core(s), so set accordingly for your ADS instance.

Unless your application is getting hammered, you may not need 20 connections in the pool created on startup, maybe try with a lower minConnections.

Check out Play's BoneCP defaults as well for reference.

[Niklas Nylund]

Adding,

db.default.maxConnectionAge=0

db.default.connectionTestStatement="SELECT 1"

with MySQL and 2.2.1 seems to do the trick, we haven't gotten any errors now for 48 hours.

Niklas

[Bruno Leite]

Not sure if this is going to help anyone but I had this "java.sql.SQLException: Timed out waiting for a free available connection." before, when I forgot to close a QueryIterator. 

The correct usage should be (for example) something like this:

QueryIterator<Some> iterator = Ebean.find(Some.class).findIterate();

try {     

     Ebean.delete(iterator);  

} finally {
     iterator.close();

}

If the iterator.close()is not invoked after using the QueryIterator, the DB connection will be kept open, meaning that after some time all DB connections are consumed but not released.

[Michael Seid]

Thanks for all the help guys. This seems to be working for me too. So far, so good. Haven't had any errors yet today.  Makes me feel much better.

[ronik]

I just hit the same problem and am hoping that the following lines will fix this problem:

db.default.maxConnectionAge=0

db.default.connectionTestStatement="SELECT 1"

Thinking of going into production with play soon so such bugs are always scary but the community-effect helps a bit. Play seems to have the potential but a large community will make it better.

[Joe Zulli]

Hi everyone,

I am getting the same error (No operations allowed after connection closed). I was getting it about 50 times a day. I then added the following lines mentioned above:

db.default.maxConnectionAge=0

db.default.connectionTestStatement="SELECT 1"

...and now I only get about 10 errors a day. Well, that's progress, but I'd love to get zero and stop getting paged. Has anyone had the above setting work for them, and are now getting zero errors? If so, would you mind sharing ALL of your db.default.* settings? Perhaps, there is something else I am doing in a different setting that is tripping it up.

Thanks in advance!

Joe

[Jxtps]

This seems to work for me:

db.default.connectionTimeout=10000
db.default.maxConnectionAge=0
db.default.connectionTestStatement="SELECT 1"
db.default.driver=com.mysql.jdbc.Driver
# db.default.logStatements=true

[Joe Zulli]

Thanks Jxtps! It's been about 24 hours and I haven't gotten a single error. Interestingly, my settings were already very close to yours, but I also changed a couple of other defaults. Possibly, most importantly, I changes the app to have two partitions of 15 connections each, instead of the default of 1 partition of 30 connections. For those who are interested, here are the properties that I commented out to get it to work.

# db.default.partitionCount=2

# db.default.maxConnectionsPerPartition=15

# db.default.acquireRetryDelay=5 seconds

Hope that helps,

Joe

[virtualeyes]

That doesn't make any sense, the default partition count is 1, so when you comment out:
# db.default.partitionCount=2

You have 1 partition ;-)

The solution is:
db.default.maxConnectionAge=0

BoneCP author has stated that this setting forces a hard timeout on all connections, regardless of whether or not the connection is active.

Not sure why Play sets the default maxConnectionAge to 1 hour or whatever it is, but it does wreak havoc, AFAICT

[Joe Zulli]

Yeah, I don't get it either. I'd love to get to the bottom of this, but this is just.... weird.

Virtualeyes, you mentioned a few messages above that the BoneCP author recommends having exactly on partition per core. I was originally using 2 partitions, even though I'm on EC2 m1.medium instances, which I'm pretty sure are single core machines. I don't know why having more partitions than cores would hurt anything but do you think it could be related?

Joe

[thoefer]

How about setting "default.maxConnectionAge=0" in your application.conf?

Until now we've not experienced any issue of this sort. Using Play 2.2.1 with Scala and bonecp:0.8.0.RELEASE

Am Dienstag, 5. November 2013 09:13:15 UTC+1 schrieb Niklas Nylund:

Since upgrading we have started to see some exceptions from time to time that didn't happen previously. We are using Play with Ebean and MySQL, has anyone seen similar behavior to this? The exceptions are thrown in controller actions not doing anything special (no async stuff, akka etc), it's just a web page and we have very light load. This is kind of odd since there hasn't been an update to Ebean in a year (i think). 

Caused by: javax.persistence.PersistenceException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
 at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:339)
 at com.avaje.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2018)
 at com.avaje.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:178)
 at com.avaje.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1459)
 at com.avaje.ebeaninternal.server.core.DefaultBeanLoader.loadMany(DefaultBeanLoader.java:140)
 at com.avaje.ebeaninternal.server.core.DefaultServer.loadMany(DefaultServer.java:499)
 at com.avaje.ebeaninternal.server.loadcontext.DLoadManyContext.loadMany(DLoadManyContext.java:139)
 at com.avaje.ebean.common.AbstractBeanCollection.lazyLoadCollection(AbstractBeanCollection.java:127)
 at com.avaje.ebean.common.BeanSet.init(BeanSet.java:91)
 at com.avaje.ebean.common.BeanSet.size(BeanSet.java:279)
......

Caused by: javax.persistence.PersistenceException: java.sql.SQLException: Timed out waiting for a free available connection.
       at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:339)

       at com.avaje.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2018)    
              at com.avaje.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:178)

       at com.avaje.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1459)

       at com.avaje.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:885)

      ... 48 more

Caused by: java.sql.SQLException: Timed out waiting for a free available connection.

[thoefer]

ok, virtualeyes already suggested this setting... I can confirm that is works for us!

[Niklas Nylund]

Argh, we started getting these errors again a lot when we changed some of our analytics code. Basically we tried to optimize one action by moving some analytics database updates out of the request action. It's ok to update it later asynchronously after the request has been handled and a reply sent to the client.

I still haven't had time to try to make an isolated test case for this but it seems fairly reproducible and happens in production on a continuous basis. All we need to do is move a single database update into an Akka job and then we run into this:

Akka.system().scheduler().scheduleOnce(delay, runnableInstanceWithDbUpdates, Akka.system().dispatcher());

javax.persistence.PersistenceException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.

at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:339)

at com.avaje.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2018)

at com.avaje.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:178)

at com.avaje.ebeaninternal.server.core.DefaultServer.findId(DefaultServer.java:1191)

at com.avaje.ebeaninternal.server.core.DefaultServer.find(DefaultServer.java:1046)

at com.avaje.ebeaninternal.server.core.DefaultServer.find(DefaultServer.java:1033)

at play.db.ebean.Model$Finder.byId(Model.java:261)

at models.analytics.TrackViewJob.runJob(TrackViewJob.java:34)

at controllers.helpers.Job$1.run(Job.java:30)

at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)

at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:534)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

at com.mysql.jdbc.Util.getInstance(Util.java:386)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)

at com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1205)

at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1197)

at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4954)

at com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)

at play.api.db.BoneCPApi$$anon$1.onCheckOut(DB.scala:329)

at com.jolbox.bonecp.AbstractConnectionStrategy.postConnection(AbstractConnectionStrategy.java:75)

at com.jolbox.bonecp.AbstractConnectionStrategy.getConnection(AbstractConnectionStrategy.java:92)

at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:553)

at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:131)

at play.db.ebean.EbeanPlugin$WrappingDatasource.getConnection(EbeanPlugin.java:147)

at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:313)

How is Play handling this? Is the Akka job trying to use the same DB connection as the controller/request handler code, which in these cases is probably finished already and have given up its handle?

Niklas

[Christopher Hunt]

There may be a problem with Bonecp 0.8.0 regarding connection management. My fix for it is here:

https://github.com/playframework/playframework/pull/2009

Looks as though this didn’t make it into the 2.2.x branch though, which means that it didn’t make 2.2.2-RC1. I have now included it for 2.2.2-RC2:

https://github.com/playframework/playframework/pull/2009#issuecomment-32581152

We’ll shortly make a 2.2.2-RC2 followed by a 2.2.2 release soon thereafter.

[Niklas Nylund]

This looks interesting, thanks for the heads up.

I'll be sure to test this out, it's a bit tricky since we only see the errors in production under real load and I won't upgrade our stuff to 2.2.2 at release day right away but I will let you know if this fixes things.

Niklas

[Christopher Hunt]

Great, FYI - 2.2.2-RC2 is now available.

On 18/01/2014, at 5:13 PM, Niklas Nylund <nyl...@gmail.com> wrote:

This looks interesting, thanks for the heads up.

I'll be sure to test this out, it's a bit tricky since we only see the errors in production under real load and I won't upgrade our stuff to 2.2.2 at release day right away but I will let you know if this fixes things.

-- 
Christopher Hunt
Senior Engineer
Typesafe – Build Reactive Apps on the JVM!

Twitter: @huntchr

[Thijs]

I also have these problems, I'm running a Play 2.2.1 application in a production env and these random connection problems don't feel right. Seeing all these issues BoneCP doesn't seem so stable.

It would be great if I could easily choose a different, more proven, connection pool like c3p0. Is this possible?

[virtualeyes]

BoneCP author himself mentioned this seemingly too-good-to-be-true alternative:
https://github.com/brettwooldridge/HikariCP

Performance is through the roof compared to BCP and c3p0, no idea about stability...

[Michael Slinn]

Wow, those graphs are impressive. I wonder if a different connection pool for Play could be plugged in without adverse affect?

[Niklas Nylund]

2.2.2 seems to fix these issues for us, no errors yet and we have been in production for 48 hours+.

Niklas

[Sebas]

I have just encountered this problem with play 2.2.2.

Runtime was about a week in production.

Seems 2.2.2 has not yet fixed this issue.

[virtualeyes]

One time in a week, or are you getting several per day?

Also, is your database getting hammered 24/7? I've noticed the issue comes up pre 2.2.2 when the database is relatively idle.

Thinking about switching to HikaraiCP if Play supports alternative CPs.

[Sebas]

The database is relatively idle at the moment, it is far from being hammered 24/7. 

When the application will be used more this of course will change, and by searching google a bit I have found that the problem will probably occur a lot more often (unless it's fixed of course).

Annoying part of the problem is that you have to restart play to resolve it, so scheduling a daily restart by cron seems a workable, but really temporary solution.

[Christopher Hunt]

Can you please confirm that you're actually using 2.2.2 by checking the play plugin version. Thanks

[Sebas]

com.typesafe.play.play_2.10-2.2.2.jar

package created with dist command

[sachin walia]

I have developed my own Database Plugin using HirakiCP. There were multiple reasons for building my own plugin:

- I was getting too frequent "no available connection" error with Bonecp and postgres (almost everyday on play 2.2.2). With HirakiCP I havent seen this error for some time.

- I had a requirement to encrypt complete DB details in application.conf. This was not possible with current play DB plugin.

- Once the new Postgres driver is stable (https://github.com/impossibl/pgjdbc-ng) then I have a plan to switch to this. As of now I am getting some exception using it. However Database Plugin implementation is flexible and all it needs is setting a flag as true to switch to this new driver. This driver requires Netty 4.0 though and play I think still uses 3.x. I havent seen any compatibility issue in my application though.

So far I am very content with the overall results.

thanks,

Sachin Walia

[Marcos Pereira]

Hi, Sachin, 

Do you have any plans to open source your plugin?

--
http://about.me/marcospereira

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[sachin walia]

Sure. I'll open source the plugin. I will just remove/alter my project specific code and make it open source. I'll try to complete it by this weekend.

Also, please note in order to use this plugin you'll have to remove the native DB plugin from play. I made this design choice to keep the development as simple as possible. However if there is lot of interest then people can enhance it as per their need.

thanks,

Sachin Walia

[Oleg Smirnov]

Hi,

See: https://bugs.launchpad.net/bonecp/+bug/1243551

I tried to roll back to the previous library BoneCP version 0.7.1.RELEASE, which had no problems.

val appDependencies = Seq(
   //jdbc,
   anorm,
   "mysql" % "mysql-connector-java" % "5.1.26",
   ...
   "com.typesafe.play" %% "play-jdbc" % "2.2.1" exclude("com.jolbox", "bonecp"),
   "com.jolbox" % "bonecp" % "0.7.1.RELEASE"
 )

[Yann Simon]

What is the current state of this problem?
Do you still experience problem with play 2.2 out of the box.
Does rollbacking the BoneCP version to 0.7.1.RELEASE fix this issue?

[Alan Lavintman]

Guys, We fall into this as well, any updates? We are currently upgrading to 2.2.2, any idea if the issue will continue to happen?

Bests, 

Alan.

[Yann Simon]

It would maybe help if somebody concerned by this problem opens an
issue on github.

[Niklas Nylund]

Since upgrading to 2.2.2 we haven't seen any of these errors, we used to get them multiple times a day previously.

Niklas 

You received this message because you are subscribed to a topic in the Google Groups "play-framework" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/play-framework/7PBnjiXkNuU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to play-framewor...@googlegroups.com.

[Yann Simon]

On Apr 22, 2014 7:44 PM, "Niklas Nylund" <nyl...@gmail.com> wrote:
>
> Since upgrading to 2.2.2 we haven't seen any of these errors, we used to get them multiple times a day previously.
>
> Niklas 

Thanks for the feedback!
Any other return of experience?

[Niklas Nylund]

On 23 apr 2014, at 23:26, Yann Simon <yann.s...@gmail.com> wrote:

On Apr 22, 2014 7:44 PM, "Niklas Nylund" <nyl...@gmail.com> wrote:
>
> Since upgrading to 2.2.2 we haven't seen any of these errors, we used to get them multiple times a day previously.
>
> Niklas 

Thanks for the feedback!
Any other return of experience?

No, not really. Everything is working as expected. The compilation speed up is also nice but I can't remember if that was in 2.2.2 or earlier.

Niklas

[Fransiskus Xaverius]

I am using Play 2.2.2 and still experiencing this problem. Any thoughts guys? I tried various solutions that I got online, but none of theme works.

-Frans

[Rich Dougherty]

On Fri, Jun 6, 2014 at 5:05 AM, Fransiskus Xaverius <frans...@gmail.com> wrote:

I am using Play 2.2.2 and still experiencing this problem. Any thoughts guys? I tried various solutions that I got online, but none of theme works.

I'm sorry I don't have anything to suggest, except to say that issues relating to closed connections and connection pools can be very tricky to debug. E.g. the last Play connection pool issue I debugged was caused by a memory leak elsewhere in the app. As the GC load got higher, database connections started failing.

To debug you either need to develop a reproducible test case or you need to have excellent production monitoring or logging so you can correlate events while you're searching for a diagnosis.

Cheers

Rich

[Nicolas Rémond]

Suffered from the same kind of issues.

Ditched BoneCP for HikariCP - things are much more stable now.

[Rupanjan Baidya]

I am using Play 2.2.3! .. getting the same problem, here is the stack-trace,


! @6k3clo9h0 - Internal server error, for (POST) [/items/add/2] ->

play.api.Application$anon$1: Execution exception[[PersistenceException: java.sql.SQLException: Connection is closed!]]
at play.api.Application$class.handleError(Application.scala:293) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.DefaultApplication.handleError(Application.scala:399) [play_2.10-2.2.3.jar:2.2.3]
at play.core.server.netty.PlayDefaultUpstreamHandler$anonfun$3$anonfun$applyOrElse$3.apply(PlayDefaultUpstreamHandler.scala:264) [play_2.10-2.2.3.jar:2.2.3]
at play.core.server.netty.PlayDefaultUpstreamHandler$anonfun$3$anonfun$applyOrElse$3.apply(PlayDefaultUpstreamHandler.scala:264) [play_2.10-2.2.3.jar:2.2.3]
at scala.Option.map(Option.scala:145) [scala-library.jar:na]
at play.core.server.netty.PlayDefaultUpstreamHandler$anonfun$3.applyOrElse(PlayDefaultUpstreamHandler.scala:264) [play_2.10-2.2.3.jar:2.2.3]

Caused by: javax.persistence.PersistenceException: java.sql.SQLException: Connection is closed!
at com.avaje.ebeaninternal.server.transaction.JdbcTransaction.rollback(JdbcTransaction.java:641) ~[avaje-ebeanorm.jar:na]
at com.avaje.ebeaninternal.server.transaction.JdbcTransaction.rollback(JdbcTransaction.java:622) ~[avaje-ebeanorm.jar:na]
at com.avaje.ebeaninternal.server.core.BeanRequest.rollbackTransIfRequired(BeanRequest.java:87) ~[avaje-ebeanorm.jar:na]
at com.avaje.ebeaninternal.server.persist.DefaultPersister.saveRecurse(DefaultPersister.java:284) ~[avaje-ebeanorm.jar:na]
at com.avaje.ebeaninternal.server.persist.DefaultPersister.save(DefaultPersister.java:248) ~[avaje-ebeanorm.jar:na]
at com.avaje.ebeaninternal.server.core.DefaultServer.save(DefaultServer.java:1568) ~[avaje-ebeanorm.jar:na]

Caused by: java.sql.SQLException: Connection is closed!
at com.jolbox.bonecp.ConnectionHandle.checkClosed(ConnectionHandle.java:459) ~[bonecp.jar:na]
at com.jolbox.bonecp.ConnectionHandle.rollback(ConnectionHandle.java:1270) ~[bonecp.jar:na]
at com.avaje.ebeaninternal.server.transaction.JdbcTransaction.rollback(JdbcTransaction.java:634) ~[avaje-ebeanorm.jar:na]
at com.avaje.ebeaninternal.server.transaction.JdbcTransaction.rollback(JdbcTransaction.java:622) ~[avaje-ebeanorm.jar:na]
at com.avaje.ebeaninternal.server.core.BeanRequest.rollbackTransIfRequired(BeanRequest.java:87) ~[avaje-ebeanorm.jar:na]
at com.avaje.ebeaninternal.server.persist.DefaultPersister.saveRecurse(DefaultPersister.java:284) ~[avaje-ebeanorm.jar:na]


Nicolas,

Could you please share your build.sbt for HikariCP plugin?

Thanks in advance.

-RB

[Laurent Fleuriot]

Hello,

I have the same error. Have you find the issue?

Thanks in advance.

Laurent