Issues after Running Out of Space

[tmountain]

We've been using Mongo for tracking user feed data, and we
inadvertently ran out of space a while ago. This caused mongod to
complain that it couldn't allocate a data file, and we addressed the
issue by deleting some non-mongo data on the box and then restarting
mongod. Once we were back up and running, we ran our delete script
constantly in hopes of allowing Mongo to reuse large chunks of the
230GB of on disk storage that it allocated. Unfortunately it slowly
consumed the remaining 10GB of disk even though we deleted over
80,000,000 records. Is there a known issue regarding reusing
previously allocated space after filling the disk?

[Maciej Dziardziel]

from what i know disk space can be reused be either running
db.repairDatabase() or dumping/restoring.

[Eliot Horowitz]

What version are you running?
Can you run .stats() on all the collections (assuming its not
hundreds) and send that?

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[tmountain]

I believe the issue may lie in other collections continuing to grow
outside of the collection that we were deleting from. I've done a
bunch of testing locally, and Mongo seems to do a good job at reusing
available space, so I don't believe there's an issue here. We ended up
purging our feed data and starting over just to get things back
online, so I can't send the stats over. One thing I have noticed
though that I can reliably reproduce is as follows:

If mongo runs out of disk space and throws an assertion like this:

Tue Oct 5 13:14:16 [conn3] Caught Assertion in insert , continuing
Tue Oct 5 13:14:16 [conn3] insert test.bar exception 12520 file
allocation failure 16ms

It will not create the next datafile even if you free up disk space.
The only way to get it to continue operating normally is to shut it
down completely and start it back up. This is an issue for large
datasets where it takes the cache a while to warm up. One other issue
with deleting lots of data is that the resident memory usage never
goes down until you restart mongo. I'm not sure if this is intended,
but it seems like it should decrease if I'm deleting gigabytes of
data.