mongodb not fit for multi-time write-delete-rewrite operations?

[Moore]

does this mongodb will be slow down for bunch of documents write-delete many time?  does mongo is a Write Once Read Many DB ?

after many time write documents ,  delete documents ,  in monogdb.log , i get a long period operation log.

Tue Sep 25 14:57:06 [snapshotthread] cpu: elapsed:4000  writelock: 108%
           14:57:09 [conn70161] remove online.online query: { fullad: "ad54060064" } 3879ms
           14:57:10 [snapshotthread] cpu: elapsed:4000  writelock: 99%
           14:57:12 [conn70161] remove online.online query: { fullad: "ad48926103" } 2586ms
           14:57:14 [conn70161] insert online.online 2107ms
           14:57:14 [snapshotthread] cpu: elapsed:4000  writelock: 120%
           14:57:17 [conn70161] remove online.online query: { fullad: "ad58028172" } 2984ms
           14:57:18 [snapshotthread] cpu: elapsed:4000  writelock: 76%
           14:57:20 [conn70161] remove online.online query: { fullad: "ad50782741" } 2792ms
           14:57:22 [snapshotthread] cpu: elapsed:4000  writelock: 72%
           14:57:24 [conn70161] insert online.online 3707ms
           14:57:26 [conn70161] insert online.online 2454ms

[Sam Millman]

Mongo is optimised for write once read many that is true however that is a bit of a crazy time for some deletes.

What exactly are you deleting? How many are you deleting? What is the size of each document you are deleting? Maybe it is the find query on the remove taking the time?

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[Moore]

there are more than 4 millions documents,  from the begining , operation as lighting.

one portal do modify documents, write/delete something like that;

one python module as a tiny web-server do mongodb-query , 20 connections in concurre. hundreds of mongo-query per seconds.

sometime do have a full tables scan for another app.

for the mongodb server , box running centos-6.2 whith 32G ram and 300G lvm-hd.  mongod process occupied 3G and a bits more.

that's all the running-environment.

[Stephen Lee]

Per Sammaye's question, could you provide the .explain() for online.online { fullad: "ad54060064" }?  eg.

use online

db.online.find( { fullad: "ad54060064" } ).explain()

?

[Moore]

FYI ,  could you please explain what does these messages means?

> db.online.find( { fullad: "ad54060064" } ).explain()
{
    "cursor" : "BtreeCursor fullad_1",
    "nscanned" : 1,
    "nscannedObjects" : 1,
    "n" : 1,
    "millis" : 498,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "isMultiKey" : false,
    "indexOnly" : false,
    "indexBounds" : {
        "fullad" : [
            [
                "ad54060064",
                "ad54060064"
            ]
        ]

[Octavian Covalschi]

http://www.mongodb.org/display/DOCS/Explain 

--

[Moore]

I  do have build the index

> db.system.indexes.find()
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "online.online", "name" : "_id_" }
{ "v" : 1, "key" : { "ad" : 1 }, "ns" : "online.online", "name" : "ad_1" }
{ "v" : 1, "key" : { "ip" : 1 }, "ns" : "online.online", "name" : "ip_1" }
{ "v" : 1, "key" : { "fullad" : 1 }, "ns" : "online.online", "name" : "fullad_1" }
{ "v" : 1, "key" : { "ip" : 1, "st" : 1 }, "ns" : "online.online", "name" : "ip_1_st_1" }
{ "v" : 1, "key" : { "fullad" : 1, "st" : 1 }, "ns" : "online.online", "name" : "fullad_1_st_1" }
>

insert  query update delete getmore command flushes mapped  vsize    res faults locked%  idx miss%  qr|qw   ar|aw  netIn netOut  conn    time

   477      0      0    477       0       1       0  4.94g    13g  3.08g      0      106          0       0|0     0|1    91k   1k    22   08:50:54

[Rob Moore]

Could the slow operations have been during the "full table scan" you mentioned?  That could cause a backup of operations and the latency you are seeing.

The mongostat output you provided looked like things were running OK at that time.  Can you let that run and correlate a slow insert/delete log entry to the mongostat time?

If you are churning the documents in the database a lot and they are not already of a fairly uniform size you may want to look at using the "power of 2 allocator" with the collection.  That will cut down the amount of fragmentation you will see with MongoDB under heavy write/update/delete workloads.

http://docs.mongodb.org/manual/reference/command/collMod/#usePowerOf2Sizes

Rob.

[Moore]

> db.serverSa> db.serverS            tatus()
{
        "host" : "AAA-1",
        "version" : "2.2.0",
        "process" : "mongod",
        "pid" : 10830,
        "uptime" : 6880,
        "uptimeMillis" : NumberLong(6879549),
        "uptimeEstimate" : 6792,
        "localTime" : ISODate("2012-09-29T16:11:34.012Z"),
        "locks" : {
                "." : {
                        "timeLockedMicros" : {
                                "R" : NumberLong(42),
                                "W" : NumberLong(734429)
                        },
                        "timeAcquiringMicros" : {
                                "R" : NumberLong(17299239),
                                "W" : NumberLong(20)
                        }
                },
                "admin" : {
                        "timeLockedMicros" : {

                        },
                        "timeAcquiringMicros" : {

                        }
                },
                "local" : {
                        "timeLockedMicros" : {
                                "r" : NumberLong(5),
                                "w" : NumberLong(0)
                        },
                        "timeAcquiringMicros" : {
                                "r" : NumberLong(4),
                                "w" : NumberLong(0)
                        }
                },
                "online" : {
                        "timeLockedMicros" : {
                                "r" : NumberLong(80378888),
                                "w" : NumberLong("18451593434580")
                        },
                        "timeAcquiringMicros" : {
                                "r" : NumberLong(1941319341),
                                "w" : NumberLong(92725381)
                        }
                }
        },
        "globalLock" : {
                "totalTime" : NumberLong("6879549000"),
                "lockTime" : NumberLong(734429),
                "currentQueue" : {
                        "total" : 0,
                        "readers" : 0,
                        "writers" : 0
                },
                "activeClients" : {
                        "total" : 1,
                        "readers" : 0,
                        "writers" : 1
                }
        },
        "mem" : {
                "bits" : 64,
                "resident" : 3118,
                "virtual" : 7254,
                "supported" : true,
                "mapped" : 7102
        },
        "connections" : {
                "current" : 22,
                "available" : 234
        },
        "extra_info" : {
                "note" : "fields vary by platform",
                "heap_usage_bytes" : 25845000,
                "page_faults" : 9037
        },
        "indexCounters" : {
                "btree" : {
                        "accesses" : 27894592,
                        "hits" : 27894592,
                        "misses" : 0,
                        "resets" : 0,
                        "missRatio" : 0
                }
        },
        "backgroundFlushing" : {
                "flushes" : 89,
                "total_ms" : 4684642,
                "average_ms" : 52636.426966292136,
                "last_ms" : 411719,
                "last_finished" : ISODate("2012-09-29T16:08:34.316Z")
        },
        "cursors" : {
                "totalOpen" : 1,
                "clientCursors_size" : 1,
                "timedOut" : 4,
                "totalNoTimeout" : 1
        },
        "network" : {
                "bytesIn" : 133949678,
                "bytesOut" : 924862569,
                "numRequests" : 1397097
        },
        "opcounters" : {
                "insert" : 698074,
                "query" : 654,
                "update" : 0,
                "delete" : 698077,
                "getmore" : 221,
                "command" : 70
        },
        "asserts" : {
                "regular" : 9,
                "warning" : 1,
                "msg" : 0,
                "user" : 0,
                "rollovers" : 0
        },
        "writeBacksQueued" : false,
        "recordStats" : {
                "accessesNotInMemory" : 9570,
                "pageFaultExceptionsThrown" : 669,
                "online" : {
                        "accessesNotInMemory" : 9570,
                        "pageFaultExceptionsThrown" : 669
                }
        },
        "ok" : 1
}
> db.online.stats()
{
        "ns" : "online.online",
        "count" : 4763886,
        "size" : 530832796,
        "avgObjSize" : 111.42852620738616,
        "storageSize" : 651083776,
        "numExtents" : 24,
        "nindexes" : 6,
        "lastExtentSize" : 111796224,
        "paddingFactor" : 1,
        "systemFlags" : 0,
        "userFlags" : 1,
        "totalIndexSize" : 1604482768,
        "indexSizes" : {
                "_id_" : 256620112,
                "ad_1" : 288923488,
                "ip_1" : 250479936,
                "fullad_1" : 242083184,
                "ip_1_st_1" : 287051184,
                "fullad_1_st_1" : 279324864
        },
        "ok" : 1
}

[Max Schireson]

you might want to try the power of two space allocation option in 2.2.
It helps a lot in many cases with frequent deletes. I have not looked
at all the details of this situation so I can't confirm it will
resolve the issues.

-- Max

[Moore]

Somewhat tunning operations fixed this problem but I don't know which one was.

1. upgrade to 2.2,  for that OS is CentOS6.2
2. power of two space allocation setting.
3. tuned OS parameter from tuned.noarch parameters included in file named throughput-performance/sysctl.ktune
4. ulimit set OS Max open files more than default 1024 and others such as Max msgqueue size etc.

It is crucial that we know why we are doing what we are doing,

In God we trust , i don't know why i doing and i don't know how i doing.

[Max Schireson]

I will explain the power of two allocation.

When a document grows, it needs to be moved to a larger space. The old space it used becomes free. The system keeps track of these spaces and tries to reuse them. Depending on the exact pattern of document sizes, sometimes spaces can pile up and take a while to get reused. The movement, possible pileup, and associated bookkeeping his can slow the system down. Power of two allocation allocates storage to each document in fixed sizes of eg 512 bytes, 1024 bytes, etc. This greatly reduces the potential pileup/fragmentation as well as the move frequency. It does make the database somewhat less space efficient in the absence of frequent deletes, but from the quick description of your use case seemed like a better fit.

It is a new option with 2.2 to address just this type of issue, hence my recommendation to upgrade to take advantage of it.

Hope this helps,

-- Max

[Moore]

thanks Max,  it's very kind of you.

thanks with  the power of two.

[Moore]

Also  many thanks to Rob Moore.