Upgraded Replicaset from 2.6.7 to 3.0.2 with WiredTiger, performance suffered, looking for tips to tune WT

[Michael Shvartsman]

Good time of day @all!

Our production env consists of 3 R3.2Xlarge machines EC2 machines managed by MMS and the end user performance is being monitored via NewRelic.

Recently we've moved our production replicaset to 3.0.2 and converted it to Wired Tiger.

Comparing the application performance to 2.6.7 i see about 15-20% performance loss for typical requests load, and more than 100% performance drop under heavy load (running a 200k documents delete every 5 minutes). The application didn't change at all. Only change is WT and 3.0.2.

What i really am looking for is tips on how to approach the isso and figure out the bootlenecks.

The configuration is absolutely stock, i haven't changed any parameters in MMS except from storage engine change.

Thanks in advance! 

[Asya Kamsky]

Michael,

It's difficult to answer without knowing what your workload was limited by before.   For example, if previously your limiting factor was IOPS and your data is not very compressible then I'd expect you will still have IO limiting you with WT.

In this case it sounds like the heaviest load on the system is very large deletes?   It's actually very likely that with relatively large cache, deleting that many documents can take longer than 5 minutes to fully evict from the cache, which would slow down the next large request.  Note that I'm very much guessing in the dark here having not even seen your MMS stats, but that's something that is a possibility because of the way WiredTiger tracks its own cache where MMAP simply updates the memory mapped files.

3.0 mmap should have better performance than 2.6 mmap - did you happen to note how your performance was with 3.0 before you switched to MMAP?  It's possible that with your workload MMAP may perform better - it would be great to learn more as we are trying to understand and fully characterize all the different workloads that perform better on different storage engines.

Asya

 

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/7adc8c1f-a77e-4489-b4ef-94658997eba9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!

[Michael Shvartsman]

Asya, thanks a lot for your response.

To answer your questions :

 

Our system serves between 100 and 400 requests per second which is a constant load.

On top of that, we have a cronjob running every 10 minutes and deleting unneeded data (with a limit of 200k rows each 10 minutes) so as not to unnecessarily bloat the database.

Our storage on each note is 1TB SSD for mongo data files 

10GB partition for logs

10GB partition for journal

From the monitoring i've done to the IO system, i doubt it's our bottleneck.

I have not unfortunately monitored the performance of 3.0.2 mmap engine and directly switched to WT. I am in the process of converting the slaves to MMAP as we speak hoping to get our performance back to where it was, i will update here with the results.

I would happily share our environment with your team in case you really want to learn about our use case. I believe it will be beneficial to both parties :)

[PBS]

Hi Michael,

  Please share your findings on this! Nature of our requirement is also more or less similar to you and we are considering WT as an option but reading multiple posts on WT I am little skeptical. your input also would be helpful for me to take informed decision 

Thanks,

PBS