MongoDB Java driver & Connection reset problems..
Jeff
I'm trying to work out a problem where it appears all my available
connections on the server-side are being consumed by a single app
after it's been running awhile. i think this is likely because of how
the app is structured. Here's what's goign on;
1. App runns MongoDB Java driver 2.7.0. There's a Quartz job that
fires every 5 minutes that does the following:
a. query the MongoDB for a list of values (hosts in this case)
b. for every value, spawn a new thread via Executor/Callable that:
i. Performs a SNMP query on the host passed as an argument.
ii. Writes results into Mongo.
After this runs about 24 hours or so, any new connections from the app
begin to get "Connection Reset" errors. Interestingly, the same
application from a new host (My local IDE), ALSO gets connection
resets. The mongo shell does not however.
As soon as I restart the main tomcat container running my problematic
app, things appear to go back to normal. Both the app running in the
tomcat container, and the app running inside my IDE can connect to the
DB w/o issue.
So my question is ;
1. Do I somehow need to unload the mongo driver or de-register it to
close out connections?
2. What else am I missing?
This is my first java app, so be kind please if my terminology is off.
I hate being a newb. :)
Scott Hernandez
each time, or reusing one?
Can you provide the code for this?
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>
Jeff
on a singleton. So get instance, connect, select database/collection,
read/write.
Code wise.. sure. the code itself is rather long, and I'm using the
webUI at the moment, so I can't attach the code directly. I've edited/
stripped out most of the non-essential stuff (I know.. who's to tell
what's essential vs non-essential in this context?).
Really all that's missing is the Apache commons configurator stuff,
log4j statements, imports, SNMP helper and SNMP4J imports, etc.
Note that this also has about 60 SNMP queries per host that get run,
and there are 3 different instances of this job running, each polling
about 100 physical servers... so at any one point in time, I could be
looking at upwards of 60 or so connections to the DB... might be that
connect statement with 10 blocking threads.. but still, that should
only impact a single copy of the running application, not the MongoDB
server itself, would it?
Classes below are:
HostPoller -- This is the class that is called()/run by the
executor service. Polls the host via SNMP, writes the data into the
database.
quartzJob.class -- this is executed by the Quartz scheduler. --
looks up hosts from an active-host table, creates threads for polling
hosts individually.
MongoConnection.class -- Singleton helper routines for working
with Mongo. Based on Greylog's MongoDB code (some of it was lifted
directly).
-------------- Begin hostPoller.class
public classs hostPoller implements Callable<Boolean> {
MongoConnection mc;
String hostname;
public hostPoller(String hostname) {
this.mc = MongoConnection.getInstance();
this.hostname = hostname;
}
public Boolean call() throws Exception {
SnmpHelper helper = new SnmpHelper;
String targetAddress = "udp:" + this.hostname + "/161";
helper.setAddress(targetAddress);
// read this from a properties file later.
helper.setCommunity("public");
helper.start();
BasicDBObject record = new BasicDBObject();
record.append("host", this.hostname);
DBCollection metricsCollection =
mc.getDatabase().getCollection("raw_metrics_table");
Boolean polledOK = true;
try {
String oid = ".1.3.6.1.4.1.32229.1.2.1.1.0"
String result = helper.getAsString(new OID(oid));
record.append("num200s", result);
oid = ".1.3.6.1.4.1.32229.1.2.1.2.0"
result = helper.getAsString(new OID(oid)):
record.append("num400s", result);
} catch (Exception e) {
// log an error
polledOK = false;
} finally {
helper.stop();
} // end try-catch
metricsCollection.save(record);
return polledOK;
} // end call();
}
-------------- End hostPoller.class
-------------- Begin quartzjob.class
// QUARTZ Class.
// some stuff omitted here (imports, log4j, commons-Configuration
stuff to pickup properties files.
public class pollManager implements Job {
MongoConnection mc;
public pollManager() {
mc = MongoConnection.getInstance();
}
// Meat.
public void execute (JobExecutionContext context) throws
JobExecutionException {
JobKey key = context.getJobDetail().getKey();
JobDataMap dataMap = context.getMergedJobDataMap();
List<Future<Boolean>> pollerQueue = new
ArrayList<Future<Boolean>>();
final ExecutorService pollService =
Executors.newFixedThreadPool(20);
try {
mc.connect("", "", "172.23.0.146", "agent", 27017,
"false", 10, 10, null);
ArrayList<BasicDBOBject> acaList = getACAList();
Iterator acaIterator = acaList.iterator();
while(acaIter.hasNext()) {
BasicDBObject hostInfo = (BasicDBObject)
acaIter.next();
pollQueue.add(pollService.submit(new
hostPoller(hostInfo.getString("host"));
}
for (Future<Boolean> queueItem : pollQueue) {
result = queueItem.get();
if (result == true) {
// # do nothing.
} else {
// # log an error, the poll
failed for some reason.
}
}
} catch (Exception e) {
// # Log an error.
} finally {
// use shutdown now, just in case there are
threads out there hanging around still for some reason.. UE error or
something I didn't think of.
pollService.shutdownNow();
} // end try/catch block.
} // end Execute function.
public ArrayList<BasicDBObject> getACAList() {
ArrayList recordList = new ArrayList<BasicDBObject>();
DBCollection c =
mc.getDatabase().getCollection("aca_status");
// setup our date object to query date/time.
int minutesAgo = (0 - 1440);
Calendar cal = Calendar.getInstance();
cal.set(Calendar.MINUTE, minutesAgo);
Date activeSince = cal.getTime();
BasicDBObject query = new BasicDBObject();
query.append("stamp", new BasicDBObject("$gt",
activeSince));
/// DB Query should look like:
// { "stamp" : { "$gt" : { "$date" :
"2011-11-28T22:00:13Z"}}}
List hostListFromDB = c.distinct("host", query);
Iterator elIter = hostListfromDb.iterator();
// this can be more efficient. Fix this in version 2.
while(elIter.hasNext()) {
String host = (String) elIter.next();
BasicDBObject hostQuery = new BasicDBObject();
hostQuery.append("host", host);
DBCursor cursor = c.find(hostQuery).sort(new
BasicDBObject("stamp", -1)).limit(1);
while (cursor.hasNext()) {
BasicDBObject hostRecord = (BasicDBObject)
cursor.next();
acaRecordList.add(hostRecord);
} // end while.
} // end while.
return acaRecordList;
} // end getACAList
------------------ end quartzjob.class
------------------ begin MongoConnection.class
// DB Helper class. Adapted from code found with the Greylog project.
public class MongoConnection {
private static final org.apache.log4j.Logger log =
org.apache.log4j.Logger.getLogger(MongoConnection.class);
private static MongoConnection instance;
private Mongo m = null;
private DB db = null;
private DBCollection messagesCollection = null;
// dummy constructor.
private MongoConnection() {}
/**
* Singleton for the wrapper.
* @return
*/
public synchronized static MongoConnection getInstance() {
if(instance == null) {
instance = new MongoConnection();
}
return instance;
}
/**
* Connect to the db.
* @param username -- username to connect with.
* @param password -- password for user
* @param hostname -- Mongo server
* @param database -- Mongo database
* @param port -- mongo server port
* @param useAuth -- require authentication.
* @param maxConnections -- max number of connections allowed.
* @param threadsAllowedToBlockConnectionMultiplier -- # of
threads allowed to "block" the DB.
* @param replicaServers -- IP adddresses of replica servers
* @throws Exception
*/
public void connect(String username, String password, String
hostname, String database, int port, String useAuth,
int maxConnections, int
threadsAllowedToBlockConnectionMultiplier, List<ServerAddress>
replicaServers) throws Exception {
log.debug("connect() called.");
MongoOptions options = new MongoOptions();
options.connectionsPerHost = maxConnections;
options.threadsAllowedToBlockForConnectionMultiplier =
threadsAllowedToBlockConnectionMultiplier;
try {
if(replicaServers != null && replicaServers.size() > 0 ) {
log.debug("Creating connection to replica server
set.");
m = new Mongo(replicaServers, options);
} else {
log.debug("Creating connection to single server.");
ServerAddress address = new ServerAddress(hostname,
port);
m = new Mongo(address, options);
}
db = m.getDB(database);
// authenticate if required.
if(useAuth.equals("true")) {
if(!db.authenticate(username, password.toCharArray()))
{
throw new Exception("Could nto authenticate to
database '" + database + "' with user '" + username + "'.");
}
}
} catch (MongoException.Network e) {
throw new Exception("Could not connect to Mongo DB(" +
e.toString() + ")");
}
}
/**
* returns the raw connection.
* @return connection
*/
public Mongo getConnection() {
return m;
}
/**
* return raw db obj.
* @return database
*/
public DB getDatabase() {
return db;
Jeff
error:
MongoDB shell version: 1.8.3
connecting to: test
Tue Nov 29 23:14:57 MessagingPort recv() errno:104 Connection reset by
peer 127.0.0.1:27017
Tue Nov 29 23:14:57 SocketException: remote: error: 9001 socket
exception [1]
Tue Nov 29 23:14:57 DBClientCursor::init call() failed
exception: DBClientBase::findOne: transport error: 127.0.0.1 query:
{ whatsmyuri: 1 }
Exit 255
> ...
>
> read more »
Antoine Girbal
In theory the java driver will only use 11 connections by default per Mongo instance per host.
If you have 3 apps running that should be only 33, far from any limit.
Make sure that the Mongo instance is indeed a singleton (by printing object hashcode for example).
If you are getting 1 mongo instance per thread that would be the problem..
You may also want to check the setting on the OS server side.
To increase the number of fd allowed for the user running mongo, check out:
http://www.mongodb.org/display/DOCS/Too+Many+Open+Files
Jeff
and the connection count climbs to 816, then gets "stuck"
Interestingly, I did find a bug in the code, but that didn't seem to
solve the problem. The bug was in the Quartz job's first part of the
try{} block.
Old:
try {
mc.connect("", "", "172.23.0.146", "agent", 27017,
"false", 10, 10, null);
New:
try {
if(mc.getConnection() == null) {
mc.connect("", "", "172.23.0.146", "agent", 27017, "false", 10,
10, null);
} else {
log.debug("Already had an instance with a connection. Not
creating a new one."
}
Unfortunately, after 24 hours or so, I still see the # of connections
climb up to 816.
I'll try printing out the hash for the instance and compare to see if
I'm truly getting a singleton. I suspect that I'm not. It took me
_forever_ to wrap my brain around singletons, (and Java in general.
Java is just plain weird.) so I'm probably doing something wrong.
> ...
>
> read more »