Java Driver adjusting writeback timeout

[Christian Csar]

We're currently running Mongo 2.2.2 across 3 shards each a replicaset with 3 servers. Currently data is being written in by six instances of a java program using the 2.9.3 driver. We had the Mongos that these were talking to go unresponsive for some hours until we restarted it. While we still need to investigate the reasons for this in the logs of the java programs we have a series of the below exceptions spaced at 1 hour intervals with one for each thread that was doing work in each 1 hour cluster.
"com.mongodb.CommandResult$CommandFailure: command failed [getlasterror]: { "serverUsed" : "[removed mongoose host info]" , "errmsg" :
"exception: didn't get writeback for: 512bb9b30000000000000ea2 after: 3600006 ms from connection mongo-rs3-1:481072" , "code" : 13403 , "ok" : 0.0}
        at com.mongodb.CommandResult.getException(CommandResult.java:88)
        at com.mongodb.CommandResult.throwOnError(CommandResult.java:134)
        at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:142)
        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:183)
        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:155)
        at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:249)
        at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:205)
        at com.mongodb.DBCollection.insert(DBCollection.java:57)
        at com.mongodb.DBCollection.insert(DBCollection.java:86)"
In searching for this error code it appears that 100 seconds is substantially more common than 1 hour. How/can this value be tweaked to one that works better for our application? Here we are pulling jobs from a beanstalkd and it would be much easier to return the job to the beanstalkd if the job fails sooner. After we restarted the MongoS the java program began recording impressions again, though I am not certain if the hour time out had to expire first or not.

Christian