Many Exceptions + OOM/Heap

[project2501]

Hi,
I'm using mongo 2.0.0 and java driver 2.6.5

I do periodic insert of small documents of about 20-50 every couple
minutes. Before long, mongo throws exceptions and the java-driver
loses its heap and gives OOM (even though app server was allocated 8GB
ram and had plenty).

Java driver complains many times:

Connection reset,

then

Exception determining maxBSON size using0

then

OutOfMemory: Java heap space

I boosted the heap 5 times and its now 8GB, yet this still happens.

mongod is still running, but can no longer connect to it from shell or
anything. I'm running on a NUMA and launch using numactl.

Any idea why mongo isn't working properly here?

thanks.

[Scott Hernandez]

It seems like this is completely a client-side (java) issue. Can you
please use jconsole/jvisualvm or a profiler to see where the memory is
going. The odds are very good this has nothing to do with mongo per se
and is just related to your java code.

http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
http://visualvm.java.net/

If you have a sample which causes the problem then post that could
would be a good idea.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>

[project2501]

I will post some code, but its only 1 jsp that is doing this. There is
no heavy logic or state being maintained. It seems the mongo driver
originates the exception well before the OOM. Also, as I mentioned, I
can no longer reach mongo when this happens, even from mongo shell.
Mongo dies.

Here is my JSP pseudo-code, simply.

Mongo mongo = new Mongo();
DB db = mongo.getDB('testdb');
DBCollection items = db.getCollection('items');

for i = 0; i< items.length; i++ {

BasicDBObject dbobject = new BasicDBObject();
dbobj.put("key","value");
....
....

items.insert(dbobj);

}

Ack, perhaps I need a close or commit or something?

On May 6, 11:31 am, Scott Hernandez <scotthernan...@gmail.com> wrote:
> It seems like this is completely a client-side (java) issue. Can you
> please use jconsole/jvisualvm or a profiler to see where the memory is
> going. The odds are very good this has nothing to do with mongo per se
> and is just related to your java code.
>

> http://java.sun.com/developer/technicalArticles/J2SE/jconsole.htmlhttp://visualvm.java.net/

[Scott Hernandez]

I'd guess it is because you should not create a new Mongo instance on
each page. You should treat the Mongo class as a Singleton and
basically only use one for your (whole) application.

Have you looked at the server logs, or run mongostat as this happens?

[project2501]

I will make those adjustments, but even still, re-creating a Mongo()
object should not have the ill side effect of crashing the server
after a few thousand times.
When those objects go out of scope, the driver should not leak
resources. It should act gracefully.

I will try the mongostat and report back. Thanks for the tip!
Hopefully, I find the solution because its a serious production system
experiencing this error.

> >>http://java.sun.com/developer/technicalArticles/J2SE/jconsole.htmlhtt...

[Scott Hernandez]

I don't think the server is crashing at all. I suspect you are simply
just using all the connections available.

You are probably seeing a combination of the objects staying in scope
too long (holding connections open) for the few number of connections
which the server support based on your system.

It is best not to guess and for you to just provide the logs and
stats. The answers will be there.

[project2501]

Gotcha. Will report back. Thanks!

[project2501]

When I try to run mongostat on the jammed server it says:

DBClientCursor::init call() failed
DBClientBase::findN: transport error: .....

mongod is running still.

[Scott Hernandez]

You need to start it before, if you have connection problems. Please post the logs.

[project2501]

Scott,
Thanks for the help. Appreciate it. The logs do indeed indicate too
many connections.

I changed the use of Mongo mongo = new Mongo(), to become a
<jsp:useBean ... with scope set to "application" for all my JSP's.

Watching mongostat it seems a new connection is still made every time
the application is visited or refreshed.

Is there some other code change I can do to fix this? Should there
only be one connection for all JSP's,sessions,etc? I'm new to mongo,
so not sure how the connection management is to be handled.

On May 7, 9:49 am, Scott Hernandez <scotthernan...@gmail.com> wrote:
> You need to start it before, if you have connection problems. Please post
> the logs.

[Scott Hernandez]

You can use the Mongo.Holder instance which is a "static" singleton.

Mongo yourMongo = Mongo.Holder.instance().connect(...);
http://api.mongodb.org/java/current/com/mongodb/Mongo.Holder.html#connect(com.mongodb.MongoURI)

[project2501]

Thanks Scott. Seems to be working now. I've seen the light!
Also, I need to RTFM more....

On May 7, 11:33 am, Scott Hernandez <scotthernan...@gmail.com> wrote:
> You can use the Mongo.Holder instance which is a "static" singleton.
>

> Mongo yourMongo = Mongo.Holder.instance().connect(...);http://api.mongodb.org/java/current/com/mongodb/Mongo.Holder.html#con...)