can't find plugin warning after upgrade to 2.2.1

[Mark Kwan]

I just upgraded my first Mongo cluster from 2.0.7 to 2.2.1

I noticed that when the mongod process starts, in the log file, there are a few warnings that say:

Tue Nov  6 17:22:16 [repl prefetch worker] warning: can't find plugin [stats.applied]

Tue Nov  6 17:22:16 [repl prefetch worker] warning: can't find plugin [stats.applied_success]

Tue Nov  6 17:22:16 [repl prefetch worker] warning: can't find plugin [stats.verified]

Tue Nov  6 17:22:16 [repl prefetch worker] warning: can't find plugin [stats.verified_success]

Anything to be concerned about?  I don't think I missed anything in the upgrade process.  All mongos, config server and mongod processes are now running 2.2.1.  Anything I need to do to get those plugins?

Thanks!

[Andre de Frere]

Hi Mark,

This could be caused by a sort in a query or some other operation that is not using the correct key names.  Do you have a particular time of the day that these messages appear, or can you recreate them at will?  If so, you can switch on database profiling level 2 and see what queries are using references to these names.

Look for anything suspicious in the profiler output which includes the [stats.applied], [stats.applied_success], [stats.verified] and [stats.verified_success].  If you have trouble finding what might be wrong with queries that include one of those keys, post it here and we will diagnose.

Cheers,

André

[Mark Kwan]

Hi Andre,

It happens every time, immediately after I restart the secondary.  We don't query the secondary that often, I'd be surprised if it had a chance to take a query by the time these messages are logged.

Wed Nov  7 11:31:43 [initandlisten] waiting for connections on port 27018

Wed Nov  7 11:31:43 [rsStart] replSet I am pitbull:27018

Wed Nov  7 11:31:43 [rsStart] replSet STARTUP2

Wed Nov  7 11:31:43 [rsHealthPoll] replSet member akita:27018 is up

Wed Nov  7 11:31:43 [rsHealthPoll] replSet member akita:27018 is now in state PRIMARY

Wed Nov  7 11:31:44 [rsSync] replSet SECONDARY

Wed Nov  7 11:31:45 [rsHealthPoll] replSet member lab:27020 is up

Wed Nov  7 11:31:45 [rsHealthPoll] replSet member lab:27020 is now in state ARBITER

Wed Nov  7 11:31:50 [rsBackgroundSync] replSet syncing to: akita:27018

Wed Nov  7 11:31:50 [repl prefetch worker] warning: can't find plugin [stats.applied]

Wed Nov  7 11:31:50 [repl prefetch worker] warning: can't find plugin [stats.applied_success]

Wed Nov  7 11:31:50 [repl prefetch worker] warning: can't find plugin [stats.verified]

Wed Nov  7 11:31:50 [repl prefetch worker] warning: can't find plugin [stats.verified_success]

Wed Nov  7 11:31:51 [rsSyncNotifier] replset setting oplog notifier to akita:27018

I'll see about turning on database profiling.  What's the significance of "can't find plugin"?

Thanks,

Mark

[Andre de Frere]

Hi Mark,

It might be easier to see this in an example.  If you were to execute a query that had a format like: 

db.collection.find().sort(1:"stats.applied") 

(where the "stats.applied" and 1 are in reversed positions). 

the query would return in the shell, but the logs would say "can't find plugin [stats.applied]".  This can also happen in other operations where the field and operator are reversed.  If you can profile one of these queries going through you should be able to find where it is coming from.  You might also want to check indexes on that collection and see if one of those has the incorrect form.

Depending on the operation, this shouldn't cause any specific harm.  The results would not be sorted correctly (in the case of sort) and this could cause further problems with performance if left unchecked.

Regards,

André

[Mark Kwan]

Ah, this makes sense now.  I don't think we're querying against those indexes, but we're querying the collection.  Somehow we managed to create a few indexes that looks like:

        {

                "v" : 1,

                "key" : [

                        "stats.applied"

                ],

                "ns" : "graph_saq.rules",

                "name" : "0_stats.applied"

        },

Apparently they were created a long time ago through PHP code.  We're fixing them now.

The "can't find plugin" message is a little confusing.  :)

Thank you Andre!

Mark