Tune query knobs based on storage engine?

[Igor Canadi]

Hi all,

There are some constants for query knobs defined in: https://github.com/mongodb/mongo/blob/master/src/mongo/db/query/query_knobs.h#L99. Would it make sense to make these configurable by storage engine? For Rocks and (I think) WiredTiger, (1) yielding is very expensive and (2) reading doesn't block writes, so it might be much better to yield less often than every 10ms.

Best,

Igor

[Geert Bosch]

> On Feb 12, 2015, at 2:55 PM, Igor Canadi <igor....@gmail.com> wrote:
>
> There are some constants for query knobs defined in: https://github.com/mongodb/mongo/blob/master/src/mongo/db/query/query_knobs.h#L99. Would it make sense to make these configurable by storage engine? For Rocks and (I think) WiredTiger, (1) yielding is very expensive and (2) reading doesn't block writes, so it might be much better to yield less often than every 10ms.
>

These can be set through server parameters either at startup or at run time, for example: mongod --setParameter internalQueryExecYieldIterations=1000 --setParameter internalQueryExecYieldPeriodMS=100.

Actually we have not found that increasing the defaults makes a significant difference in performance, but experimentation is easy now.

Regards,
-Geert

[David Murphy]

Using these I can create a 15% difference due to query plan caching on a heavily inserted system.

I had a patch that only clears the cache in index changes like adding an index or running and explain , this had huge gains in a 20% read/80% write system in 2.4 and 2.6

Sent from my iPhone

> --
> You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.
> To post to this group, send email to mongo...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-dev.
> For more options, visit https://groups.google.com/d/optout.

[Igor Canadi]

It makes a big difference for RocksDB. Currently aggregation/testshard1.js fails after 35min (timeout), while if I increase these values, we easily pass the test.

The problem is that restoreState() is doing a Seek(), which taking more than 10ms, so we basically yield on every getNext().

Here's where we seek: https://github.com/mongodb/mongo/blob/master/src/mongo/db/exec/index_scan.cpp#L322

It might also be useful to avoid doing the seek if the snapshot didn't change. I'm doing that optimization on the rocks level (repeated calls to locate() with exactly the same parameters and exactly the same snapshot)

Igor