"command denied" showing up in the mongod logs a lot

[echin]

I'm running two replica sets as two shards on MongoDB 2.2 and I'm getting a lot of these messages in the logs of both replica sets (just the shards, not mongos):

Tue Sep 11 08:38:42 [conn22] command denied: { writebacklisten: ObjectId('5019a5bc16a3e8999751896e') }

The shard seems to be performing fine, inserting/reading etc.  Googling the error has failed to yield me any meaningful results.

All replica sets and config servers are using MongoDB 2.2, all using authentication with proper keyfiles.  Any ideas as to where this is coming from? 

[Gianfranco]

This is probably do to the fact that some servers are not being authenticated.

Please ensure that all MongoD and MongoS are version 2.2.0 and the correct keyfile is used on each host and and there are no permissions issues with accessing it?

If that is all verified, we can later check, with a simple script, what other applications are trying to access the replica sets.

[echin]

I can confirm all config servers, MongoS and MongoD are using version 2.2.0.  The same keyfile is on each host and is being used.  Each MongoS can log into every database and perform queries on all collections.

What kind of script are you referring to? Is it like a block of Javascript that runs some MongoDB commands to check connections?

Thanks for the help so far.

[Gianfranco]

Exactly. Run this commands on the admin db:

use admin

db.currentOp(true).inprog.forEach(function(z){if ( z.query.replSetGetStatus && ! z.query.$auth ){printjson(z);}})

Then check if you recognize those ip addresses.

[Eugene]

Ah ha, so there was a server that had been connecting with MongoS that hadn't yet been upgraded to version 2.2.0.  It was missed because it currently isn't in use for Mongo ops, but still had a MongoS active on it and connecting to the shards.  The command you provided aided in tracking it down.  The "command denied" message is no longer showing up in the logs.

Thanks for the help!

[Gianfranco]

no worries Eugene.

Glad to help