Chasing down a memory leak, possibly stupid question about DBTCPConnector

[Alex]

MongoDB driver: 2.3
uname -a: Linux domU-12-31-39-08-13-C5 2.6.18-xenU-ec2-v1.0 #2 SMP Mon
Feb 18 14:28:43 UTC 2008 x86_64 x86_64 x86_64 GNU/Linux
(no logs)

I have an application where (sensibly or not), I create and close
large numbers of Mongo objects (say 1/s).

The application appears to be leaking memory (quite slowly), so I
dumped its heap. There appeared to be many live DBTCPConnector,
DBPortPool, DBPort; and HashMap entries held by the previously
mentioned classes. Everything traced back to the DBTCPConnector
objects, which were referenced by a ThreadLoad hashmap.

Looking at the code on git, I noticed in DBTCPConnector :

private final ThreadLocal<MyPort> _myPort = new
ThreadLocal<MyPort>(){
protected MyPort initialValue(){
return new MyPort();
}
};

shouldn't _myPort be static? As it stands it looks like it's
registering a new object as a thread local "static object" every time
a DBTCPConnector is created. Which would then leak in a container like
tomcat where many of the connection handling threads can persist for
the lifetime of the container.

I'll try editing it in a local copy and seeing what happens, but I
thought I'd post just in case someone looked at it and said "yes
that's a bug" while I go downstairs and make myself a coffee :P

Apologies if this is a stupid Java misunderstanding (I'm mostly a C++
programmer, so this is the first time I've encountered the ThreadLocal
construct).

[Eliot Horowitz]

You could have multiple DBTcpConnector instances, so that can't be
static at the moment.

I think there is a jira to try and clean that up more nicely on .close()

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Alex]

Thanks for the quick reply.

Yep, I think you might need to call _myport.remove() in the close()
method.

Here's a link with some handy analysis:

http://www.0xcafefeed.com/tag/threadlocal/

I'm trying it now, I'll post whether that works or not.

[Alex]

I actually didn't check whether the ThreadLocal::close worked,
although I see no reason why it shouldn't.

I just created a local version of the driver with the ThreadLocal code
stubbed out, since it so happens that my code structure guarantees a
single Mongo object accessible per thread anyway. I'm assuming the
purpose of the non-static thread local code is to allow many
DBTCPConnectors, each of which can serve multiple threads, but for
each (connector,thread) pair, a single TCP connection is used.

This fixes the memory leak.

FWIW I think you should get the fix in for 2.4 if that isn't already
the plan, because a reasonably standard usage can result in a pretty
slow leak, so people are going to deploy without noticing it.

I'd be interested in knowing the JIRA (or what keywords would narrow
down a search for it), so I can track for when I can return to just
using the off-the-shelf JAR.

[Eliot Horowitz]

Doesn't seem to be a case for that actually.
Can you open?
Will see if there is a way to squeeze into 2.4, but likely 2.5