2.02 hanging on find command (java driver)

[Steve Benjamin]

We're doing a multi-operation commit (where the commit has a number of
steps that are mostly related to maintaining db integrity on related
values, checking for required objects, etc etc).

for example
collection1.find ..
collection1.insert, ...
collection2.find...
collection2.insert, ...

What we're seeing now is that mongo will simply hang, not on the first
operation, but somewhere farther on down the line on one of the find
operations. Pretty sure this is unrelated to the actual form of the
queries where it hangs, because a) similar queries work properly
earlier in the sequence, and b) these have been working fine,
unchanged, for some time. Queries work fine in the shell.

e.g.
db.collection_1.find("_id" : { "$ne" : { "$oid" :
"4ecf38eee4b0c78c1a46111f"}} , "$or" : [ { "condition1" : "value1"},
{"condition2", "value2"]}
..
db.collection_2.find("_id" : { "$ne" : { "$oid" :
"4ecf38eee4b0c78c1a46222f"}} , "$or" : [ { "condition3" : "value3"}]}
com.mongodb.MongoException$Network: can't call something : server-name/
127.0.0.1:20001/db-name
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:227)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:
305)
at com.mongodb.DBCollection.findOne(DBCollection.java:647)

Setup. Setup is distributed, java driver 2.6.5. Slave reads have been
shut off while we're trying to figure this issue out.
wrapping the series of commands are calls to
_db.requestStart();
_db.requestEnsureConnection();
and
_db.requestDone();
although these have also been removed for the moment with no effect.

A number of things we've mucked with, to no effect
-updating the java driver to 2.7.2
-replacing find calls with find (... ReadPreference.PRIMARY)

current mongo options (although mostly trial and error, and mostly
added in testing - changes haven't had much of an effect, really)
options.autoConnectRetry = true;
options.connectionsPerHost = 60;
options.slaveOk = false;
options.connectTimeout = 30000;
options.socketTimeout = 60000;
options.threadsAllowedToBlockForConnectionMultiplier = 200;
options.fsync = false;

We're currently on a distributed setup on mongo 2.02rc1. We upgraded
from 2.00 after seeing errors similar to
https://jira.mongodb.org/browse/SERVER-4020

Running the driver with DB.TRACE=true doesn't seem to provide any
additional information - just prints out the queries.

[Scott Hernandez]

Do you always get this same exception? This indicates that there are
network connectivity/socket errors.

You may want to turn on keepAlives and see if that helps:
MongoOptions.socketKeepAlive

Where is this code running? Are there any firewalls or other NAT/Proxy
network hardware between the server/client?

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Steve Benjamin]

socketKeepAlive does not seem to affect the behavior, unfortunately.
All deployed on Amazon. Have you seen any similar behavior on amazon
deployments before?

[Scott Hernandez]

Yes, this is something which has been seen a good bit on AWS/EC2.
http://www.mongodb.org/display/DOCS/Amazon+EC2#AmazonEC2-Keepalive