Received: by 10.204.130.7 with SMTP id q7mr1077502bks.2.1349020160780; Sun, 30 Sep 2012 08:49:20 -0700 (PDT) X-BeenThere: mongodb-user@googlegroups.com Received: by 10.204.129.197 with SMTP id p5ls6212493bks.9.gmail; Sun, 30 Sep 2012 08:49:03 -0700 (PDT) Received: by 10.204.130.7 with SMTP id q7mr1077476bks.2.1349020143832; Sun, 30 Sep 2012 08:49:03 -0700 (PDT) Received: by 10.204.130.7 with SMTP id q7mr1077475bks.2.1349020143807; Sun, 30 Sep 2012 08:49:03 -0700 (PDT) Return-Path: Received: from mail-la0-f51.google.com (mail-la0-f51.google.com [209.85.215.51]) by gmr-mx.google.com with ESMTPS id k7si1309105bks.2.2012.09.30.08.49.03 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 30 Sep 2012 08:49:03 -0700 (PDT) Received-SPF: pass (google.com: domain of mschire...@gmail.com designates 209.85.215.51 as permitted sender) client-ip=209.85.215.51; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of mschire...@gmail.com designates 209.85.215.51 as permitted sender) smtp.mail=mschire...@gmail.com; dkim=pass header...@gmail.com Received: by mail-la0-f51.google.com with SMTP id m15so1820260lah.24 for ; Sun, 30 Sep 2012 08:49:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=T/8kOvIhxAVZcth8J6sfRA5+CySeN1n3tK+BDqn/MdE=; b=ojiMqkYacCNGj7ZeZuObxXsVYRZ1yaTY8AtniyTmwcOWxoEsKEHy8IQBz4dwOqfadV RvP8F4ZOgez0ikwO20yvfjBjM/+teG+eOLQ04YQqB6y4SqbHVob2rI7lX5xHAclkH62O n1sclQqx4vq10CVfCVqSyeVo97di/31tfyDEbxOPaIAPkek+OxW/zsWfud9jhl0wv9bc RYMcHlnzPbnrEuQb0/7boJrOoTbM7m5CXq93Av4MkpF9orM+Vs5H7I2WB+rKhJG/U+iE wQolLODgBVV8Y6k+g9G8yVdNTdOAsTzadInfdvyUR6zb7hemHdIJ29Cj2rVObIZh1K8k n6hw== MIME-Version: 1.0 Received: by 10.152.104.115 with SMTP id gd19mr957750lab.13.1349020143536; Sun, 30 Sep 2012 08:49:03 -0700 (PDT) Received: by 10.112.27.102 with HTTP; Sun, 30 Sep 2012 08:49:03 -0700 (PDT) Received: by 10.112.27.102 with HTTP; Sun, 30 Sep 2012 08:49:03 -0700 (PDT) In-Reply-To: References: <81c8bcc8-4a3f-47c3-adae-6df8506ef2aa@googlegroups.com> <37f720d0-a7d4-4ade-8cc8-bfbc408ab621@googlegroups.com> <19dfeefc-5da8-4816-8137-7ed665bd4f67@googlegroups.com> <3c0f1d23-14a0-4eeb-ac42-d527e6105d94@googlegroups.com> <12462f00-eb6c-4173-8f7d-d60773881043@googlegroups.com> <0e4c6cb7-f3e9-49f6-81c7-ecf4108e4a7f@googlegroups.com> Date: Sun, 30 Sep 2012 08:49:03 -0700 Message-ID: Subject: Re: [mongodb-user] mongodb not fit for multi-time write-delete-rewrite operations? From: Max Schireson To: mongodb-user@googlegroups.com Content-Type: multipart/alternative; boundary=f46d04088d23734b8a04caed3a77 --f46d04088d23734b8a04caed3a77 Content-Type: text/plain; charset=ISO-8859-1 I will explain the power of two allocation. When a document grows, it needs to be moved to a larger space. The old space it used becomes free. The system keeps track of these spaces and tries to reuse them. Depending on the exact pattern of document sizes, sometimes spaces can pile up and take a while to get reused. The movement, possible pileup, and associated bookkeeping his can slow the system down. Power of two allocation allocates storage to each document in fixed sizes of eg 512 bytes, 1024 bytes, etc. This greatly reduces the potential pileup/fragmentation as well as the move frequency. It does make the database somewhat less space efficient in the absence of frequent deletes, but from the quick description of your use case seemed like a better fit. It is a new option with 2.2 to address just this type of issue, hence my recommendation to upgrade to take advantage of it. Hope this helps, -- Max On Sep 29, 2012 4:21 PM, "Moore" wrote: > Somewhat tunning operations fixed this problem but I don't know which one > was. > > 1. upgrade to 2.2, for that OS is CentOS6.2 > 2. power of two space allocation setting. > 3. tuned OS parameter from tuned.noarch parameters included in file named > throughput-performance/sysctl.ktune > 4. ulimit set OS Max open files more than default 1024 and others such as > Max msgqueue size etc. > > It is crucial that we know why we are doing what we are doing, > > In God we trust , i don't know why i doing and i don't know how i doing. > > On Sunday, September 30, 2012 4:29:34 AM UTC+8, Max Schireson wrote: >> >> you might want to try the power of two space allocation option in 2.2. >> It helps a lot in many cases with frequent deletes. I have not looked >> at all the details of this situation so I can't confirm it will >> resolve the issues. >> >> -- > You received this message because you are subscribed to the Google > Groups "mongodb-user" group. > To post to this group, send email to mongodb-user@googlegroups.com > To unsubscribe from this group, send email to > mongodb-user+unsubscribe@googlegroups.com > See also the IRC channel -- freenode.net#mongodb > --f46d04088d23734b8a04caed3a77 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

I will explain the power of two allocation.

When a document grows, it needs to be moved to a larger space. The old s= pace it used becomes free. The system keeps track of these spaces and tries= to reuse them. Depending on the exact pattern of document sizes, sometimes= spaces can pile up and take a while to get reused. The movement, possible = pileup, and associated bookkeeping his can slow the system down. Power of t= wo allocation allocates storage to each document in fixed sizes of eg 512 b= ytes, 1024 bytes, etc. This greatly reduces the potential pileup/fragmentat= ion as well as the move frequency. It does make the database somewhat less = space efficient in the absence of frequent deletes, but from the quick desc= ription of your use case seemed like a better fit.

It is a new option with 2.2 to address just this type of issue, hence my= recommendation to upgrade to take advantage of it.

Hope this helps,

-- Max

On Sep 29, 2012 4:21 PM, "Moore" <<= a href=3D"mailto:lyx...@gmail.com">lyx...@gmail.com> wrote:
Somewhat tunning operations fixed this problem but I don't know which o= ne was.

1. upgrade to 2.2,=A0 for that OS is CentOS6.2
2. power o= f two space allocation setting.
3. tuned OS parameter from tuned.noarch = parameters included in file named throughput-performance/sysctl.ktune
4. ulimit set OS Max open files more than default 1024 and others such as M= ax msgqueue size etc.

It is crucial that we know why we are doing wh= at we are doing,

In God we trust , i don't know why i doing and= i don't know how i doing.

On Sunday, September 30, 2012 4:29:34 AM UTC+8, Max Schireson wrote:you might want to try the power of two= space allocation option in 2.2.
It helps a lot in many cases with frequent deletes. I have not looked
at all the details of this situation so I can't confirm it will
resolve the issues.

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongodb-user@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user+unsubscribe@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
--f46d04088d23734b8a04caed3a77--