Integration tests and managing database pool

[Sam]

I am using dropwizard 0.8.2 and have written many Integration tests that run against a real postgres server.

I used

@ClassRule
public static final DropwizardAppRule<MRServiceConfiguration> RULE =
        new DropwizardAppRule<MRServiceConfiguration>(MRService.class, ResourceHelpers.resourceFilePath("testing.yml"));

classrule as described in the docs to start up the server before the test is run.

Everything runs great, if I run one test class at a time. If I run all the test classes I have, then the number of open connections to the database keeps increasing, till they are exhausted and tests start failing. Surefire starts and stops the dropwizard service with every new class, but the connections remain.

I believe that the database pool is not being 'managed' in this case and hence the connections stick around. The connections are all released when the surefire process finishes.

Need a clue on how to 'manage' the pool, so that the connections are released at the end of every class.

-Sam

[Ryan Kennedy]

I've been experiencing the same problem. When running locally I've been able to increase max_connections on PostgreSQL to kick the can down the road a bit, but that's more difficult to do in a managed environment like CircleCI.

Ryan

--
You received this message because you are subscribed to the Google Groups "dropwizard-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dropwizard-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ryan Kennedy]

Sam, are you using Hibernate with that DataSourceFactory by any chance?

Ryan

[Ryan Kennedy]

When using Hibernate in my own project and applying a liberal amount of DropwizardAppRule, I can see a constant increasing leak of connections in `lsof -iTCP:5432 | awk '{print $1 " " $10}' | sort | uniq -c`. I've noticed in the past that my app maintains a persistent count of at least 3 in-use database connections. Coincidentally, the count in the `lsof` output tends to leak in increments of 3. I suspect Hibernate isn't cleaning those connections as there's some indication in the source of org.apache.tomcat.jdbc.pool.ConnectionPool that connections won't be reclaimed until they themselves get closed ("Active connections will be closed upon the java.sql.Connection#close() method is called on the underlying connection instead of being returned to the pool").

If you have a lot of tests (I have 200+) and each test or test class is leaking a handful of connections at a time, it's easy to see how you could quickly run past PostgreSQL's default (in most cases) of max_connections=100.

I'm not sure of the best way to plug this leak. It's not going to show up in any production app since the app is going to exit shortly after the lifecycle stops, forcibly closing the connections in the process. It really seems to be just a test thing, which is probably going to be low priority for most folks (although maybe not for those of us running into this ourselves).

Ryan

[Yun Zhi Lin]

I'm also getting the same issue when running Integration tests locally using Hibernate. Fortunately this hasn't happened in our CI / CD environment. Max connection = 100 didn't really help either.

[Rupert Smith]

On Monday, September 7, 2015 at 2:14:47 PM UTC+1, Sam wrote:

@ClassRule
public static final DropwizardAppRule<MRServiceConfiguration> RULE =
        new DropwizardAppRule<MRServiceConfiguration>(MRService.class, ResourceHelpers.resourceFilePath("testing.yml"));

 

Everything runs great, if I run one test class at a time. If I run all the test classes I have, then the number of open connections to the database keeps increasing, till they are exhausted and tests start failing.

I am facing the same issue.

1. It sounds like it may be necessary to force Hibernate to close connections, in order for the pool to clean them up.

2. It may also be that each invocation of the DropwizardAppRule is causing a new pool to be created, and the old pools left running.

I will investigate whether I can find a way to clean up 1 or 2. Another possibility might be to force Maven surefire (or failsafe - same thing but for the ingrationtest phase), to run each test in a brand new JVM instance. I believe there is a flag in Maven to enable this, and it might be the easiest solution.

Rupert

[Rupert Smith]

On Friday, November 6, 2015 at 11:29:57 AM UTC, Rupert Smith wrote:

Another possibility might be to force Maven surefire (or failsafe - same thing but for the ingrationtest phase), to run each test in a brand new JVM instance. I believe there is a flag in Maven to enable this, and it might be the easiest solution.

So setting the forkmode didn't work for me for other reasons (could no longer find some stuff on the classpath - what?). But I recommend trying it, as it may work for you:

 

https://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html 

[Rupert Smith]

On Monday, September 7, 2015 at 2:14:47 PM UTC+1, Sam wrote:

I am using dropwizard 0.8.2 and have written many Integration tests that run against a real postgres server.

@ClassRule
public static final DropwizardAppRule<MRServiceConfiguration> RULE =
        new DropwizardAppRule<MRServiceConfiguration>(MRService.class, ResourceHelpers.resourceFilePath("testing.yml"));

The fundamental problem here is that each @ClassRule starts a new DropWizard application. That application has its own database connection pool. When the next test starts, it creates a new application with _another_ pool.

This really comes down to the drop wizard application lifecycle having no 'stop' method. You can start an application, but not stop one.

I tend to define an interface like this:

public interface StartStopLifecycle {

   void start();

   void shutdown();

   void shutdownNow();

}

Inspired the methods on java.util.ExecutorService that run thread pools.

I would then make all DW bundles implement this interface, as well as the application itself. There is then a natural way to 'stop' and application, and have that ripple down into all bundles, so all resources can also be stopped and cleaned up.

Is there a way to 'stop' a DW application? In particular, to tell it to release its connection pool?

Rupert

[Rupert Smith]

On Wednesday, November 18, 2015 at 11:16:16 AM UTC, Rupert Smith wrote:

Is there a way to 'stop' a DW application? In particular, to tell it to release its connection pool?

Or is there some quick-and-dirty way I can drill down into the DW objects and grab a reference to the connection pool, so I can call close() on it. Then I could just call that close() in the test @AfterClass method.

I tried figuring the above out by running in the debugger and putting in a break-point where I can get a reference to the application class, and then see if I can drill down from there to get to the pool. Haven't managed so far.

Rupert 

[Rupert Smith]

On Wednesday, November 18, 2015 at 11:16:16 AM UTC, Rupert Smith wrote:

I tend to define an interface like this:

public interface StartStopLifecycle {

   void start();

   void shutdown();

   void shutdownNow();

}

I see there is a io.dropwizard.lifecycle.Managed that has start() and stop() methods on it.

There is also io.dropwizard.db.ManagedPooledDataSource that implements it.

Any idea how I invoke its stop() method?

Rupert 

[Rupert Smith]

On Wednesday, November 18, 2015 at 11:39:54 AM UTC, Rupert Smith wrote:

Any idea how I invoke its stop() method?

Ok, so I think this is the answer. Put this code in your @AfterClass method on your tests:

    List<LifeCycle> managedObjects = environment.lifecycle().getManagedObjects();

    for (LifeCycle lifeCycle : managedObjects) {

        if (lifeCycle instanceof SessionFactoryManager) {

            lifeCycle.stop();

        }

    } 

This closes the Hibernate sessions, and release the connections back to the pool. I am not getting above the pool size with this in place. To watch the number of connections currently open on postgresql do:

watch "psql -c \"select count(*) from pg_stat_activity\""

===============

Perhaps I should close all managed lifecycles at the end of the tests...? Not sure if this would result in the correct order of closing though?

Rupert

[Rupert Smith]

On Wednesday, November 18, 2015 at 1:55:30 PM UTC, Rupert Smith wrote:

Ok, so I think this is the answer. Put this code in your @AfterClass method on your tests:

Actually, I spoke to soon. Its still going up, but less quickly. 

[Yun Zhi Lin]

Two of my colleagues discovered a permanent fix to this. I've written a gist: https://gist.github.com/yunspace/98602ced01dc7d774378

The key is in these 3 settings:

dataSource:
	abandonWhenPercentageFull: 9
	removeAbandonedTimeout: 2
	removeAbandoned: true

Because they are not exposed by the default DatasourceFactory, we wrote a wrapper. I will submit a PR later on.

[Rupert Smith]

On Thursday, November 19, 2015 at 1:29:28 PM UTC, Yun Zhi Lin wrote:

Two of my colleagues discovered a permanent fix to this. I've written a gist: https://gist.github.com/yunspace/98602ced01dc7d774378

The key is in these 3 settings:

dataSource:
	abandonWhenPercentageFull: 9
	removeAbandonedTimeout: 2
	removeAbandoned: true

Thanks, I'll try it out.