MongoError: too many namespaces/collections

[Freddy Vega]

We noticed this error last week when we turned on logging for the MongoDB while investigating why executionlogs where not being created. I'm sharing here in case anyone else encounters this error.

We have been running RedwoodHQ for about 1 year and a half and have run more than 50K executions (meaning over 500K test cases executed). Logs that the test engineers print via println, etc all windup in a collection in the DB that looks similar to executionlogs59232272a16b063d36b0d2c7. One of these is created for each locked execution.

The default setting for MongoDB maximum namespace size (nssize) is 16MB. This means that we can have 24,000 total collections with the default setting. This is what was causing the executionlogs not being created.

To mitigate we bumped the nssize to 500MB. In order to completely eradicate this limit we will be upgrading to Mongo 3 which supposedly does away with the nssize limit. The caveat here is that there is no way to just "increase the size". In order to achieve this we must destroy the database, start mongod with the appropriate nssize limit and recreate the database.

To increase the NSSIZE:

1. mongodump
2. drop database automationframework
3. reconfigure mongod with nssize option
4. start mongod
5. create database automationframework
6. mongorestore

By the way, as a practice, we have daily jobs that "prune" the database nightly keeping only the last 3 months of data. Which is probably why we were able to get by with the 24,000 collection limit default setting for so long. We prune one collection nightly so that we do not interfere with the testing jobs that are running every night via our RHQ server.

Freddy

[Mazen Bilbiesi]

Thanks Freddy for these valuable information on this case.

Mazen 

[Freddy Vega]

You're welcome Mazen. If you want to find out how many collections are currently in your namespace:

db.system.namespaces.count()

Freddy

[Mazen Bilbiesi]

HI Freddy,

Would you please share with us the script you're using to prune the DB, i want to use it too.

[Freddy Vega]

Yes of course; I will share it Monday when I'm in the office.

Freddy

On Jun 25, 2017 06:08, "Mazen Bilbiesi" <bilb...@gmail.com> wrote:

HI Freddy,

Would you please share with us the script you're using to prune the DB, i want to use it too.

--
You received this message because you are subscribed to a topic in the Google Groups "PrimaTest Automation" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/primatest-automation/EJFUO9yHFJQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to primatest-automation+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mazen Bilbiesi]

Thanks Freddy

[Freddy Vega]

Hi,

Attached is the dbcleanup script (rename it to JS) we use to prune our mongodb. I created a bat file for each of the executions so that I can schedule them on different days. In our case this process takes a little while so it made sense to prune a collection a day. With the exception of executionlogs which we clean twice a week.

Below is the contents of one of the bat files. We do mongodump first in case anything goes wrong (this is optional).

cd "C:\Program Files\RedwoodHQ\"
mongodump
mongo automationframework --eval "var goBackByCount=-90, cleanupNow = true, collsToCleanup = 'executions executiontestcases'" dbcleanup.js

Please let me know if, after looking at the script you have any questions. Basically the script takes three parameters:

goBackByCount: this is how many days you want to keep (we're keeping 90 days worth, with the exception of executionlogs which we keep 30 days)

cleanupNow: if this is set to false it will just inform you of what it will be deleting but not delete anything.

collsToCleanup: is a list of collections you want to cleanup

Hope that helps you and others!

Freddy

[Mazen Bilbiesi]

Thank you Freddy for this share, will apply it and let you know if any help required on it.

Appreciate it.

Mazen 

[psuth...@gmail.com]

Thanks Freddy,

We are started using redwood recently, we want to increase the size before we encounter the size problem., we will be running more than 500 tests a day. Is there limit to size for the database or can i increase to 10GB. Can you provide little more of these steps

[dmi...@primatest.net]

If you are using RedwoodHQ 2.5 or later you should not have any issues with DB. Let me know if you do.

[Freddy Vega]

You will only encounter this issue if you're on versions prior to 2.5

Freddy

[Mazen Bilbiesi]

Hi, Freddy

I would like to thank you for this script, i tried it and worked fine here.

Mazen

[Freddy Vega]

Hi Mazen,

Thanks for the feedback. Glad it was able to help you out.

Freddy

[Sujeet]

Hi Freddy

I have few suggestions to make this utility performant and to lower memory footprint while doing housekeeping,kindly let me know your view on this. I did these changes and looks good on my end. Thanks

Line #89 :     

Current:               var docs = db[coll].find({}).toArray();

Suggested:          var docs = db[coll].find({},{"executionID":1}).toArray();                                //Usage of projection to avoid holding entire document, in case of screenshots it holds the entire binary object which consumes at least few GB’s of memory.

 

Line #117:

Current :              db[coll].remove({"executionID": doc.executionID});

Suggested :        db[coll].remove({"_id": doc._id});                                            //Usage of index while looking up document for deletion

 

Line #145:

Current :              db[coll].remove({"executionID": doc.executionID});

Suggested :        db[coll].remove({"_id": doc._id});                                            //Usage of index while looking up document for deletion

Regards

Sujeet V

[Freddy Vega]

Hi Sujeet,

Thanks for the suggestions. Based on your comments I think it's logical and reasonable to make these changes. Thanks again!