Aggregation pipeline memory restrictions
Madhumitha psg
Kevin Adistambha
Hi Madhumitha,
Is there a way to configure/increase the memory limit(100MB) on the pipeline? What if I have a lot of RAM that can be used by MongoDB?
The reason for the 100MB limit is to ensure that aggregation queries cannot evict your working set from RAM. Allowing a single aggregation query to get too large to evict your working set from RAM will result in major performance penalty for all other operations on the database.
If your aggregation query requires more than 100MB of RAM at any stage, you can set the option allowDiskUse to true.
Best regards,
Kevin
Nikhil Naib
Madhumitha psg
On Monday, April 4, 2016 at 9:45:37 AM UTC+5:30, Nikhil Naib wrote:
The alliwdiskuse will obviously dump the data to disk which will be quite slower than RAM. You may want to look at in memory storage engine of MongoDb which keeps and processes data in memory only.
Madhumitha psg
Kevin Adistambha
Hi Nikhil, Madhumitha,
The alliwdiskuse will obviously dump the data to disk which will be quite slower than RAM.
Allowing pipeline stages to write data to temporary files will be slower than writing to RAM, however this is the only supported approach (as of MongoDB 3.2) if a stage is likely to exceed 100 MB of RAM.
Is this a theoretical question about attempting to use more memory, or do you need help optimizing a specific aggregation query?
You may want to look at in memory storage engine of MongoDb which keeps and processes data in memory only.
The aggregation pipeline memory restrictions are still in effect irrespective of storage engine, so this suggestion will not work around the RAM limit for pipeline stages.
Similarly, irrespective of storage engine, the allowDiskUse aggregation option currently writes data to the _tmp subdirectory in the dbPath directory.
does the in-memory storage engine persist the data to disk?
No, it doesn’t persist anything other than some metadata and diagnostic information to disk. If you shutdown a mongod process using the inMemory storage engine, data will have to be resync’d (if the inMemory mongod is part of a replica set) or reloaded (if the inMemory mongod is a standalone server).
Also please bear in mind that the inMemory storage engine is:
- Currently experimental and should not be used in production.
- Only available in MongoDB Enterprise Edition, which requires a commercial subscription.
Best regards,
Kevin