Read/Write performance in Mongo Db

[lahori moonda]

Hi,

Consider the following document which abstracts a real world scenario that we are facing:

{ name: John,
  holdings: { {name: 'MSFT', value: 2000},
                  {name: 'IBM, value, 3000} },
  totalValue: 5000
}

The collection in which this document resides contains millions of other documents. We can get updates for any document at a max frequency of every 31 seconds. The update always leads to an insertion into holdings, i.e., a new security is always added. As part of each update, totalValue needs to be updated as well. The peak throughput of the system needs to be around 2000 document updates per second (remember that the updates cause holdings to grow). And the max latency between the time an update arrives to the time it gets shown to the user needs to be in the range of 500 ms.

I am facing significant bottlenecks just writing into the database. My current understanding is that one of the most beautiful things about Mongo is it allows related data to be stored together. Thus, under no circumstances should holdings be separated out to another collection because if we want to read the entire document, we need to always manually specify the retrieval of the linked document.

What are the best practices for designing the collection and document in this case? In general, can you guys point me to design best practices that will help me design for performance?

Many thanks in advance for your help.

Regards,

Simon

[Chris Cunningham]

Hi Lahori,

Can you elaborate on the performance bottleneck you are seeing? What is your MongoDB version, which Storage Engine is utilized, memory usage and CPU statistics? Any additional information that you can provide about the performance bottleneck you are experiencing will be helpful.

As you know, schema design can be quite complex and careful consideration should be taken when implementing this initial step. Unlike relational database design where a normal form is recommended for many applications, MongoDB schema design depends entirely on your use case. What works for one case may not work for another case even though they look similar at a glance.

There are several facets of schema design that you may want to take into consideration. For example:

• What is the balance of the expected load with regard to the required performance numbers? ie: 50% read 50% write, 10% read 90% write, or 90% read 10% write? Different read/write balances may be best served with a different schema.
• Is the holdings field strictly append-only? e.g., data will never be deleted from that field?
• Is the holdings field need to be sorted in any way?

It might be worth investigating whether Sharding would be applicable to your use case. This could allow you to horizontally scale your implementation and take advantage of spreading the workload across multiple machines to improve the overall performance of your deployment.

The following links might also be helpful:

• MongoDB Best Practices - Performance
• MongoDB Best Practices - Operations
• 6 Rules of Thumb for Schema Design - Part 1
• 6 Rules of Thumb for Schema Design - Part 2
• 6 Rules of Thumb for Schema Design - Part 3

Regards,

Chris

​