Using MongoDB with NFS storage

[Deepak Kumar]

Hi,

I wanted to compare the performance and response times with MongoDB server writing journal/data files on local disks vs NFS/Filer. Could you please let me know what additional tuning I need to do for using mongodb with NFS/Filer based storage? 

From http://docs.mongodb.org/manual/administration/production-notes/, " If you must use NFS, add the following NFS options to your /etc/fstab file: bg, nolock, and noatime." I don't see these specified in my /etc/fstab file currently and will really need strong reasons to convince our system administrator to add the same. Could you please help me understand how these NFS options help and if they are really necessary so that I can convince our System admins for the same?

Though I know use of NFS is not recommended for performance reasons, I wanted to compare and see what performance differences we see here. Are there any such benchmarks already available?

Are there any known issues with mongodb using memory mapped files over Filer/NFS based storage?

Any other gotchas that I should watch out for when using NFS with mongodb?

Thanks!

-Deepak

[Abhi]

Seems interesting -  running mongodb over NFS. I would also like to know any known pitfalls and gotchas for running mongodb over NFS.

Thanks,

Abhi

[William Zola]

You really really REALLY should NOT EVER run MongoDB using NFS for data or journal files.  You *will* run into the following problems:

1) Your performance will be horrible.  In particular, MongoDB does not use conventional I/O to access the data files: it uses mmap().  NFS is known to not perform well with the mmap() call, especially with the way that MongoDB uses it (re-mapping all of the data files 10 times per second).

2) You are likely to lose data.  I've dealt with dozens of cases where folks were using NFS for their MongoDB data files, and one or more data files just Went Away.  I'd never ever put any data I cared about on an NFS filesystem -- no matter what database I was using.

If you must use network storage, then anyone of iSCSI, FiberChannel, or FiberChannel over Ethernet are orders of magnitude preferable to using NFS.

 -William(and I've got the war stories to prove it)Z

[Deepak Kumar]

Thanks William.

The reason we were considering using NFS/Filer is because we don't have very high performance requirements and NFS/Filer performance may be just fine for our current use cases. Having said that, I am really concerned about the two points you mentioned -- though we are ok with slight performance compromise, we don't want to compromise on any functionality and reliability aspects. I didn't really understand your point on losing data using NFS. Could you please explain why we may lose data on NFS? Was this data loss you witnessed really due to NFS issue or other known cases of data loss in mongodb which I outlined in https://groups.google.com/forum/#!topic/mongodb-user/SDY82VKzif0?

Could you please also shed some more light on why mongodb's way of using  mmap works particularly bad with NFS and why does it have to re-map the data files 10 times per second?

Thanks,

Deepak

[Abhi]

Hi William,

I ran some quick tests to check the performance with filers. I was getting read performance same as local disks for the same queries. And inserts were faster than local disks. For inserts I ran 1000/5000 inserts in loop and time was less in case of filers. The time for bulk inserts were also better for filer.

In case of updates where 1000/5000 docs are updated the response times were better than local disks

In case removes the times were comparable.

I am not understanding how I am getting different results than what is expected. I am using mongodb v2.6.1 and javadriver 2.12.2.

Aside from performance, could you please tell me in what cases can data files just disappear, is it a known problem with NFS or is it because that mongodb isn't fit for use with NFS.

Thanks,

Abhi

On Thursday, June 26, 2014 10:46:31 PM UTC+5:30, William Zola wrote:

[Asya Kamsky]

It sounds like your reads were from RAM and not from disk, and this seems like way too small of a test to draw the conclusion you seem to be drawing.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/96299288-7140-457b-83f8-3993dee6691a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dwayne Campbell]

This is interesting. It sounds like you are using NetApp filers for your NFS filesystem. Depending on the filer that you have the performance might vary. However after seeing your update and the mention of the word filer. It came to my mind that most people that did NFS benchmarks did not test using enterprise hardware.

Can you do some more testing and post nice graphs for us ? Also mention the filer model

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

[Asya Kamsky]

Deepak,

We are not talking about a document or 10 "disappearing" - we are talking about an entire file (2GB of extents) disappearing or a directory.  Just within the last month:

https://groups.google.com/d/msg/mongodb-user/0u3YDJyXA9A/5I3A0jV393YJ

https://groups.google.com/forum/#!msg/mongodb-user/dbS1m890r6E/bViO6Px-x60J

Here is one where the journal directory disappeared:

https://groups.google.com/d/msg/mongodb-user/LJbmAv_tKzg/2Ttvchc-gK0J

It'll also cause very sporadic but massive performance fluctuations:

https://jira.mongodb.org/browse/SERVER-3034

You can also have very opaque failures that don't bring the host down:

https://jira.mongodb.org/browse/SERVER-10982

https://jira.mongodb.org/browse/SERVER-5429 is meant to track the possibility of making mongod more compatible with NFS, but it's really mmap - memory mapping files that doesn't work very well with NFS - this may be alleviated in the future when more different storage engines will be available for MongoDB - those that don't use mmap will probably not have as drastic performance issues with NFS, however, they may still perform not "as well" as on bare metal.     I've been discouraging users from putting database files on NFS almost as long as I've been working with databases (and that's a long time :) ).

Asya

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/7724eb52-64ec-4414-aa9f-6365952d63f4%40googlegroups.com.

[William Zola]

Hi Abhi and Deepak!

I want to echo and reenforce Asya's comments here.  Using NFS for database data files is dangerous; using it for MongoDB data or journal files is doubly dangerous.

1) The main reason is that NFS is problematic is that NFS is not a reliable protocol.  In every version of NFS prior to NFS 4, the protocol allows for writes to be lost (if you're using 'soft' mounts) or for your whole machine to hang indefinitely waiting for a write to complete (if you're using 'hard' mounts). 

Like Asya, I've been supporting databases for a couple of decades before signing on at MongoDB.  I've seen NFS filesystems responsible for dozens of corruptions of relational databases -- the data never makes it to the write-ahead journal, and the protocol allows NFS to lie when the OS asks if the data has been flushed to permanent storage.

2) If you want to have a reasonable performance test of using NFS storage, I'd recommend the following:

• A data set size that is at least 4x-8x the amount of physical RAM on the system  (e.g., if you have 16 GB of RAM on the system, use a data set that is 64 GB of data or more)
• At least 2 secondary indexes on each collection in addition to the _id index (4 would be better)
• A workload that is 50% read, 50% write, with the writes evenly split between inserts, updates, and deletes
• A workload of at least 1000 operations per second that lasts for at least 30 minutes
• A query load with the following properties: 
• uses all of the secondary indexes that you have defined
• accesses documents in a non-sequential order
• An update load with the following properies
• The updates use query strings to select individual documents for updating
• The update queries use all of the secondary indexes you've defined
• The updates access documents in a non-sequential order
• Journaling enabled

If you do that, you can watch your network traffic go through the roof, as all of your data files have to be re-shipped over the network 10 times a second.  

3) I cannot give specifics about any particular customer, but one time I handled an emergency customer situation.  This customer was using NFS to store their data files, had not load-tested the system, had put the system into production, and was, for the first time, taking significant write load.  (Technically, they weren't even a customer yet because they hadn't signed a contract, but their sales rep asked the Support team to help them as a show of goodwill ... )  

Why was this an emergency situation?  Because their system couldn't handle the load, and their network engineers were saying "Why are we transferring terabytes of data over our network each hour?".  (See above ... )

Let's just say that when the VP of <redacted> was on the phone with all of us, he was polite, but also was Extremely Not Happy that <redacted> had to throttle the number of operations per second they could handle, with significant impact to both immediate revenue and brand reputation.  We got them up-and-running through the initial crisis (using scotch tape and rubber bands), but the long-term fix (as in, the next day) was to stop using NFS.

4) What's the problem with NFS?  Well, and as I hope you know, MongoDB uses mmap() for all data file access.  As part of the journaling subsystem, MongoDB will remap the private view 10 times per second by default (more often if the journal and the data files are on different devices or if clients are using the {j:true} write concern).  Details of the journaling subsystem explaining this in detail are here:

 - http://www.kchodorow.com/blog/2012/10/04/how-mongodbs-journaling-works/

 - http://www.mongodb.com/presentations/mongosf-2012-journaling-and-storage-engine

It's been a long time since I had access to Unix kernel source,  I've never been intimately familiar with Linux internals, and I am by no means a networking expert.  However, and as I understand it, because NFS is stateless, every time MongoDB does a mremap(), all of the mapped data must be re-fetched from the NFS server.  Most programs use conventional I/O, so they don't have this limitation.  Most programs that use mmap() do *not* map files that are larger than physical RAM, and don't perform mremap() (much less doing a mremap() once per second), so they never hit these limits of NFS.  

5) I want to emphasise again the catastrophic nature of NFS failures -- entire 2 GB files go missing.  When this happens there's nothing to do but resync the node from another replica.  Asya's provided examples just from the past month -- if you go looking you can find scores more.  If you go looking in the other database groups (Postgres, mySQL, BerkeleyDB, SQLlite) you'll find scores more.

Or, you can try it yourself, and learn the hard way.  Maybe you'll get lucky.  I don't want to be you if your ops folks discover that your scaling plan was "Maybe I'll get lucky".   

6) If you're using a NetApp filer, both iSCSI and FCoE should be available to you.  Use them instead & save yourself a lot of heartache.  

 -William 

[Deepak Kumar]

Thanks for these really helpful and informative posts, Asya and William. I will check this out further and get back if I have any further queries.

-Deepak

[Deepak Kumar]

Hi Asya, William,

I had one more query on this thread.

Asya pointed us to https://jira.mongodb.org/browse/SERVER-5429 which is meant to track the possibility of making mongod more compatible with NFS. Do you think this would be implemented in the next couple of months -- are there any rough timelines on when this feature will be available?  

Though I am really considering not using NFS for my work, I am asking this out of curiosity since it looks like there are already some document store NoSQL databases like CouchDB that don't use mmap'ed files and may not suffer from the kind of issues that MongoDB may see with NFS/Filers (atleast I couldn't find post on the net talking about CouchDB issues with NFS). So, I was wondering if we could have such alternative storage engine that don't use mmap'ed files in MongoDB as well.

Thanks again for your informative and helpful replies.

-Deepak

[Tim Hawkins]

Is there any documentation on mongodb.org that outlines this, just dumping thus and aysa mail in the wiki would be a big help.

Its much easier to argue against an operations department policy, if there is an authoritive document available from the vendor.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/b5de941d-4092-462b-8229-f2650a695510%40googlegroups.com.

[Deepak Kumar]

Totally agree with Tim. Its much easier for new MongoDB users/beginners to understand limitations of using MongoDB and its mmaped approach with NFS and make proper hardware choices if there is such documentation/reasons on mongodb official documentation. I had tough time in searching through tons of posts and document pages and finally logging this ticket when I couldn't find any details there.  In other cases, users may not check this at all, proceed with using NFS and realize the issues later.

Thanks,

Deepak

[William Berkeley]

Hi Tim and Deepak. The MongoDB docs do have a warning about using NFS. The status of SERVER-5429 is "planned, but not scheduled", so there isn't a timeline for the feature at this point. Watching and upvoting the issue always helps raise its priority.

-Will