Weird behavior: looks like the drivers is bypassing the mongos and go directly to the mongod

[Simon T.]

Hi again guys,

On our stress test environment we start to see some weird behavior: it seems like the java drivers is trying to bypass the mongos to connect directly to the mongod.

First here are the configurations:

- 2 mongos (altough Iam pretty sure only was use during the test)

- 2 shards of 2 servers + 1 arbiter

They're all on 2.2.0, java drivers are: 2.9.0

The end of the stack trace looks like this:

Caused by: com.mongodb.MongoException: socket exception [SEND_ERROR] for XX.YYY.ZZ.AAA:27018

        at com.mongodb.CommandResult.getException(CommandResult.java:100)

        at com.mongodb.CommandResult.throwOnError(CommandResult.java:134)

        at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:142)

        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:183)

        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:155)

        at com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:328)

        at com.mongodb.DBCollection.update(DBCollection.java:160)

        at com.mongodb.DBCollection.update(DBCollection.java:191)

        at com.mongodb.DBCollection.update(DBCollection.java:203)

The thing I don't understand is that XX.YYY.ZZ.AAA:27018 is a mongod (I had 2 differents error, one for each primary). Now how did the drivers manage to get that ip from the mongos ? Is that a server error that "bubled" is way up to the drivers ? I thought that what was underneath the mongos is suppose to be transparent to the drivers ?

Let's say that this behavior would be normal, those servers are up and working, I wouldn't be able to explain why there is a socket error.

Let me know if you need anything, logs, mms account, etc, etc...

Thank you very much,

Simon

[Simon T.]

Other weird error message I just saw from the java drivers:

"Unexpected end of file from server"

"Software caused connection abort: recv failed"

I never seen those before...

[Adam C]

Hi Simon,

For the original question on the exception being raised.  The mongod is the source of the error, because that is where the getLastError command (used to check for the success/fail of a write operation, in this case an udpate) is being sent by the mongos (over the same connection used for the original update).  This then gets passed back by the mongos to the driver.  I believe this is the preferred behavior, otherwise you would essentially obfuscate the source of the actual problem.

In terms of the other errors (and the root cause), we've got a separate thread going in Jira which we can use to dig deeper here (full stack trace, logs etc.).  Once we track down the problems there I can report back anything relevant for the rest of the group.

Adam.

[Simon T.]

Yeah I've seen the thread in Jira.

Thanks