New collection creation is slow and DB clients see primary as offline

36 views

Skip to first unread message

Home Hunter

unread,

Dec 13, 2017, 11:36:07 PM12/13/17

to mongodb-user

We started seeing that the mongodb clients are finding the primary to be down.

When i checked the mongodb log, i saw two different types of behavior/errors at different times:

1) simple INSERT into a (new non-existing) collection is taking very long time (~16 seconds) at times. The very same insert was running fine for years and nothing else changed

2) I saw the following errors:

warning: slow scan in allocFromFreeList (in write lock)

Sun Dec 10 13:00:07.698 [conn10363640] warning: newExtent 2223 scanned

insert xxxx ninserted:1 keyUpdates:0 locks(micros) w:10384820 10384ms

These two situations are causing an outage and should be fixed asap.

We are currently using 2.4.9 mongodb

I know that mongo upgrade is needed but this version was working fine till recently and nothing changed.

Any pointers will be highly appreciated at your earliest convenience.

Kevin Adistambha

unread,

Dec 19, 2017, 7:53:13 PM12/19/17

to mongodb-user

The very same insert was running fine for years and nothing else changed

Although perhaps your application doesn’t change, the data in your database definitely changed over the years. If you have many inserts and deletes, data inside the data files would eventually be fragmented and thus would slow down the database in some cases.

warning: slow scan in allocFromFreeList (in write lock)

This message typically shows up on ext3 filesystem and usually doesn’t show up when the underlying filesystem is XFS as recommended in the Production Notes for MongoDB 2.4. To ensure smooth operation, please make sure that your deployment follows all the recommendations put forth in the Production Notes.

Since you have a replica set, the best way forward is to perform an initial sync on the problematic node. This would remove the datafiles fragmentation. See https://docs.mongodb.com/v2.4/core/replica-set-sync/ for more details.

Performing initial sync is a major maintenance procedure. Please ensure you have backups and tested the procedure thoroughly before executing it in production.

We are currently using 2.4.9 mongodb

This version was released in January 2014 and is not supported anymore. Your issue could be caused by SERVER-14082 which was fixed. Note that you cannot upgrade from 2.4.9 straight into the latest version. That is, the upgrade procedure involve upgrading to successive major versions, so to upgrade to the latest MongoDB version (currently 3.6.0), you would have to do 2.4 -> 2.6 -> 3.0 -> 3.2 -> 3.4 -> 3.6. Note that this doesn’t involve upgrading to the WiredTiger storage engine, which is orthogonal to the versions upgrade.