H2 running in Tomcat 7 produces memory leak reports in tomcat's detection mechanism

[Jim]

I am using Tomcat 7.0.25 in a Linux machine

I am using Tomcat's connection pool
(org.apache.tomcat.dbcp.dbcp.BasicDataSource). As database
I am using H2 as a file database.
All is ok, but I have the following problem.
On shutdown of Tomcat I see in catalina.out:


SEVERE: The web application [/GeneralApplication] appears to have
started a thread named [H2
Log Writer GENERICAPPLICATION] but has failed to stop it. This is very
likely to create a
memory leak.
Apr 4, 2012 2:32:54 PM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/GeneralApplication] appears to have
started a thread named [H2
File Lock Watchdog /opt/en/repos/tomcat/webapps/GeneralApplication/db/
internaldatabase.lock.db]
but has failed to stop it. This is very likely to create a memory
leak.


After contacting Tomcat dev (thread here ) the problem seems to be in
H2 (specifically the classloader that H2's JDBC is using is not the
proper one i.e. the same class loader that loaded the JDBC driver).
The root cause is pointed out to be:
"H2 creates those threads it should ensure that the
context class loader for the threads is set to the same class loader
that loaded the JDBC driver. We had similar issues with the evictor
thread in Apache Commons Pool (used by DBCP)."

Could you please look into this?
Is this a new issue.
I suspect that after the new memory leak detection mechanism
introduced in Tomcat 7, H2 will produce similar reports in all
installations

[Jim]

Link to discussion with tomcat dev:
http://mail-archives.apache.org/mod_mbox/tomcat-users/201204.mbox/%3C4F7C810...@apache.org%3E

[Thomas Mueller]

Hi,

I would like to understand what the problem is, but could you provide
the configuration you are using, and if possible a simple test case
please? That would help a lot.

Regards,
Thomas

[Jim]

Details also here:
http://tomcat.10.n6.nabble.com/Discrepancy-between-Tomcat-s-connection-pool-and-tomcat-s-report-on-memory-leaks-td4686308.html

I am using Tomcat 7.0.25 in a Linux machine

I am using Tomcat's connection pool
(org.apache.tomcat.dbcp.dbcp.BasicDataSource). As database I am using
H2 as a file database.

The following entry is in server.xml

<Resource name="jdbc_GENERIS_RS" auth="Container"
type="javax.sql.DataSource"
driverClassName="org.h2.Driver"
url="jdbc:h2:file:/opt/en/repos/tomcat/webapps/
GeneralApplication/db/internaldatabase;SCHEMA=genericschema"
username="root" password="sa"
maxActive="20" maxIdle="10" maxWait="-1"
/>


And the following entry is under $TOMCAT_HOME/conf/context.xml

<ResourceLink name="jdbc/GenericDataSource"
global="jdbc_GENERIS_RS"
type="javax.sql.DataSource"/>


If this info is not enough, I can send you a test sample.
How can I attach it?

On Apr 10, 10:01 pm, Thomas Mueller <thomas.tom.muel...@gmail.com>
wrote:

[Jim]

I have send a sample test case in your mail. Is this adequate for you
to investigate this?

On Apr 10, 10:01 pm, Thomas Mueller <thomas.tom.muel...@gmail.com>
wrote:

[Jim]

Hi,
Did you have a chance to look into this? Is this indeed an H2 problem
as stated by Tomcat dev?

On Apr 10, 10:01 pm, Thomas Mueller <thomas.tom.muel...@gmail.com>
wrote:

[Thomas Mueller]

Hi,

I didn't have time to look at it yet.

I believe the problem is that the database connections are not closed, probably because the connection pool is not disposed. So I believe it's an application problem, and not a problem within H2.

Regards,

Thomas

--
You received this message because you are subscribed to the Google Groups "H2 Database" group.
To post to this group, send email to h2-da...@googlegroups.com.
To unsubscribe from this group, send email to h2-database...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/h2-database?hl=en.

[Jim]

I am not sure what you mean. I always call connection.close() so that
the connections will go back to the pool. The pool (which is of
Tomcat) decides when to actually close the connections. So as far as
your comment that the connection pool is not disposed, there is an
analysis on this on the Tomcat dev group thread I posted here. Did you
get a chance to read that thread? Also I have send you a sample code.
Did you tried that?

On Apr 25, 9:52 am, Thomas Mueller <thomas.tom.muel...@gmail.com>

wrote:
> Hi,
>
> I didn't have time to look at it yet.
>
> I believe the problem is that the database connections are not closed,
> probably because the connection pool is not disposed. So I believe it's an
> application problem, and not a problem within H2.
>
> Regards,
> Thomas
>
>
>
>
>
>
>
> On Tuesday, April 24, 2012, Jim wrote:
> > Hi,
> > Did you have a chance to look into this? Is this indeed an H2 problem
> > as stated by Tomcat dev?
>

> > On Apr 10, 10:01 pm, Thomas Mueller <thomas.tom.muel...@gmail.com<javascript:;>

>
> > wrote:
> > > Hi,
>
> > > I would like to understand what the problem is, but could you provide
> > > the configuration you are using, and if possible a simple test case
> > > please? That would help a lot.
>
> > > Regards,
> > > Thomas
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "H2 Database" group.

> > To post to this group, send email to h2-da...@googlegroups.com<javascript:;>
> > .

> > To unsubscribe from this group, send email to

> > h2-database...@googlegroups.com <javascript:;>.

[Thomas Mueller]

Hi,

> So as far as
> your comment that the connection pool is not disposed, there is an
> analysis on this on the Tomcat dev group thread I posted here. Did you
> get a chance to read that thread?

No. I didn't have time to look at it yet.

> Also I have send you a sample code.
> Did you tried that?

No, not yet.

Regards,
Thomas

[Jim]

Ok. If you need anything else from me, please let me know. I assume
that what I have provided so far is sufficient for your investigation?

On Apr 26, 9:44 am, Thomas Mueller <thomas.tom.muel...@gmail.com>
wrote:

[Jim]

Hi,

Is there any progress on this? It's been a month and hadn't had any
kind of input.
This is starting to become important for me.
From my point of view I agree with Tomcat devs that this is H2 bug.
So any new?

Thanks!

[Thomas Mueller]

Hi,

Thanks for the test case! I have installed Tomcat 7 now. I couldn't reproduce the problem yet, but I think I will be able to do that later this week.I don't know yet how to fix it (if it is even possible to fix it). But I will continue working on it.

Regards,

Thomas

--
You received this message because you are subscribed to the Google Groups "H2 Database" group.

To post to this group, send email to h2-da...@googlegroups.com.
To unsubscribe from this group, send email to h2-database...@googlegroups.com.

[Thomas Mueller]

Hi,

I could reproduce the log message now and I think I found a way to avoid it.

Just to make sure: you didn't in fact see any out of memory problems,
or similar, right? It's just the message in the log file?

Regards,
Thomas

[Jim]

Hi Thomas,

1) So far I didn't get a memory leak problem e.g. reported by
profiler. I only get these log warnings which hint that there is such
a problem. At this point anyone reading these logs can only assume
that there is a memory leak.
2) I occusionally end up (when shutting down Tomcat) with a
"redirectServer.lock.db". I think this is related to the issue we are
discussing. This is also a problem for me since my app is part of an
rpm installation and on uninstall due to existing
redirectServer.lock.db the directories are not cleaned.

From what you are saying, your approach is to "mask" the warning? You
are saying that there is no actually memory issue here? Because in my
mind it seems to me that there is a subtle "race-condition" since H2
expects all connections to be closed in order to shutdown properly,
but it is Tomcat that decides when to close all the connections.


On May 24, 8:27 pm, Thomas Mueller <thomas.tom.muel...@gmail.com>
wrote:

[Thomas Mueller]

Hi,

> 1) So far I didn't get a memory leak problem e.g. reported by
> profiler.

OK that's good to know.

> I only get these log warnings which hint that there is such
> a problem.

I guess the classloader of the web application stays in memory
unnecessarily for the first time you restart the web application. But
only the first time, and not after subsequent restarts. As wether the
solution I found really solves this problem I don't know - it might or
it might not. But I believe even if it wouldn't solve that problem,
it's not such a big deal. If it would consume more *each time* you
restart the web application - yes that would be a big problem. But I
don't think that ever was the case (even with older versions of H2).

> At this point anyone reading these logs can only assume
> that there is a memory leak.

Yes. The problem seems to be that the threads (that belong to the
database) directly refer to the classloader of the web app. This I can
solve. Whether or not the classloader of the web app can be unloaded
now I can't tell for sure however.

> 2) I occusionally end up (when shutting down Tomcat) with a
> "redirectServer.lock.db".

This is a bit strange. Tomcat should dispose the connection pool,
which in turn should close all connections to the database. That would
delete the .lock.db file. If it's not deleted, then at least one
database connection remains open during the Tomcat shutdown.

Is there a way to reproduce it? How to you stop Tomcat? (it should
work both with ./shutdown.sh and with Ctrl+C).

> This is also a problem for me since my app is part of an
> rpm installation and on uninstall due to existing
> redirectServer.lock.db the directories are not cleaned.

If the database is still open, you will be able to delete the .lock.db
file, but the file is re-created automatically after one second or so.
If you see this behavior, then the process is still alive. That would
mean you didn't actually stop the process.

> From what you are saying, your approach is to "mask" the warning?

I think not - I did what I read in the discussion at the Tomcat
mailing list. But I'm not completely sure it will in all cases have
the desired effect, as I don't fully understand how classloading works
within Tomcat.

> You are saying that there is no actually memory issue here?

I believe if there is a memory issue, it's just a one time problem
(the first restart of the web app might not release some of the
memory). But not a real memory leak (where memory usage grows and
grows after each restart). You could actually test it, you would need
to restart the web app in a loop (about 100 times or so) and check if
it ever runs out of memory.

> Because in my
> mind it seems to me that there is a subtle "race-condition" since H2
> expects all connections to be closed in order to shutdown properly,
> but it is Tomcat that decides when to close all the connections.

I wouldn't call this race condition. Yes, H2 expects all connections
to be closed, but not necessarily when the web app is stopped. The
connections should be closed when the connection pool is disposed,
which should occur at the latest when Tomcat is stopped.

Regards,
Thomas

[Hermes Flying]

Hi Thomas,

I stop Tomcat using $CATALINA_HOME/bin/shutdown.sh and I occusionally expererience that the .lock.db file is not deleted.

You are saying:

>>This is a bit strange. Tomcat should dispose the connection pool,
>>which in turn should close all connections to the database. That would
>>delete the .lock.db file. If it's not deleted, then at least one
>>database connection remains open during the Tomcat shutdown.

But this is the issue of this thread. My application closes all the connections, which are returned to Tomcat's pool and on shutdown Tomcat does not dispose all of the connections before shutting down the web app and so on and as a result the .lock.db file remains.

Does your solution solve this issue as well?

Thank you

---

From: Thomas Mueller <thomas.to...@gmail.com>
To: h2-da...@googlegroups.com
Sent: Sunday, May 27, 2012 6:41 PM
Subject: Re: H2 running in Tomcat 7 produces memory leak reports in tomcat's detection mechanism

--
You received this message because you are subscribed to the Google Groups "H2 Database" group.
To post to this group, send email to h2-da...@googlegroups.com.

To unsubscribe from this group, send email to h2-database+unsub...@googlegroups.com.

[Thomas Mueller]

Hi,

> I stop Tomcat using $CATALINA_HOME/bin/shutdown.sh and I occusionally
> expererience that the .lock.db file is not deleted.

What happens if you delete this file? Does it stay deleted, or does it
re-appear after a few seconds? If it stays deleted, then the database
was not closed but the process was stopped. If it re-appears, then the
process is still running.

> You are saying:
>>>This is a bit strange. Tomcat should dispose the connection pool,
>>>which in turn should close all connections to the database. That would
>>>delete the .lock.db file. If it's not deleted, then at least one
>>>database connection remains open during the Tomcat shutdown.

> But this is the issue of this thread.

Well, it's a problem... It might not be your problem.

> My application closes all the
> connections, which are returned to Tomcat's pool and on shutdown Tomcat does
> not dispose all of the connections before shutting down the web app and so
> on and as a result the .lock.db file remains.

Well, at the beginning of the mail you wrote you shut down *Tomcat*,
not just the web app. If the web app is stopped, then the database
stays open, that's correct and OK.

> Does your solution solve this issue as well?

No.

Regards,
Thomas

[Jim]

Hi Thomas,
I think that the problem of the remaining lock.db file is caused by the problem discussed by the thread.
The problem is that when I shut down tomcat the web application is shutdown first but due to the way H2 driver is loaded (as discussed in the Tomcat thread) the pool is not being disposed of by Tomcat at that point and hence the problem appears (since H2 has open connections). This is an interaction of H2 driver with Tomcat
So I can't figure out what you mean that "the database stays open, that's correct and OK. "

[Thomas Mueller]

Hi,

In my previous mail I asked "What happens if you delete this file?" -
so what happens?

I understand the connection is not closed when the *web application* is stopped.

But the connection should be closed (by Tomcat) if *Tomcat* is stopped.

Regards,
Thomas

> --
> You received this message because you are subscribed to the Google Groups
> "H2 Database" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/h2-database/-/Xz5bGvt2F-kJ.

>
> To post to this group, send email to h2-da...@googlegroups.com.
> To unsubscribe from this group, send email to

> h2-database...@googlegroups.com.

[Jim]

Hi Thomas,
I took thread dumps of this and the "H2 File Lock Watchdog" remains hanging after Tomcat has been shutdown. As a result the JVM does not shutdown either. Does H2 start this thread as non-daemon perhaps?
Anyway the complaint of the memory leak is about the "H2 File Lock Watchdog" and I see that it is the Filewatch dog that remains hanging.
And yes if I delete the lock.db file it is recreated. I guess by H2?
How can this be resolved? I am stuck now. Could you please help me?

> h2-database+unsubscribe@googlegroups.com.

[Thomas Mueller]

Hi,

Well, that means some of the connections were not closed. H2 has no
way of knowing by itself that a connection is no longer needed.

> Does H2 start this thread as non-daemon perhaps?

No, H2 uses daemon threads. The database also uses a shutdown hook, to
close all forgotten open connection when calling System.exit.

> And yes if I delete the lock.db file it is recreated. I guess by H2?

Yes, if the connections are not closed.

> How can this be resolved? I am stuck now. Could you please help me?

If you use a connection pool, you need to ensure the connection pool
is disposed, so that all connections are closed. You need to check how
and when the connection pool disposes connections.

As an alternative, you could use another connection pool, or no
connection pool at all. If you have only one application that uses the
database, then you could instantiate the connection pool within the
web application (open the connection pool when the web application is
started, and dispose it when it is stopped).

Regards,
Thomas