OplogSize functionality

[Avise Sudhakar Rao]

Hi Team,

As we have huge read and writes in the mongoDB, like 2000 to 20000 and more reads per second and 2000 to 30000 and more writes per seconds
Changed opLogSize from 12GB to 20GB as our primary and secondary mongodb server total disk is of 250GB and the RAM is of 12GB.
And by the above changes Primary and Secondary replication was fast and better but don't somehow the data which was post from outside to MongoDB found some problem, as data not getting posted to MongoDB intently and its taking time to store data into primary mongodb server from outside and now it showing huge different of more the 6hours latency.

Do you know why this is causing?

Thanks,
Sudhakar

[Eliot Horowitz]

Not sure i follow.

Are you saying replication lag is 6 hours?

If there aren't simple issues, then it sounds like the primary is
overloaded and you need to shard.

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

[Avise Sudhakar Rao]

Let me clear.

Primary and Secondary replication is good and there is no latency after the change of oplogSize.
But data coming from outside to store in MongoDB Primary is taking time to store after this change.

Do you know the reason why?

[Eliot Horowitz]

Not without a lot more information...
Do you know what the bounds on the system are (cpu/disk, etc...)
Are there slow operations in the log?

On Thu, Jan 26, 2012 at 1:53 AM, Avise Sudhakar Rao

[Shekhar Vemuri]

Are you saying inserts have slowed down?

Sent from my iPhone.

[Avise Sudhakar Rao]

Yes

[Nat]

Can you post your iostat, mongostat?

---

From: Avise Sudhakar Rao <avisesud...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Thu, 26 Jan 2012 12:45:15 +0530

To: <mongod...@googlegroups.com>

ReplyTo: mongod...@googlegroups.com

Subject: Re: [mongodb-user] OplogSize functionality

[Avise Sudhakar Rao]

iostat -x 2

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.97    0.00   10.00   19.27    0.00   62.75

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.12     0.21    0.21    0.21     9.16     2.86    28.92     0.01   13.97   7.09   0.29
sdb               0.02   256.47   74.73   68.51  7156.74 15272.70   156.59     5.34   37.27   3.16  45.21
dm-0              0.00     0.00    0.30    0.33     8.97     2.66    18.35     0.01   14.88   4.53   0.29
dm-1              0.00     0.00    0.02    0.03     0.18     0.20     8.00     0.00   24.97   1.97   0.01

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          14.07    0.00    9.63   36.79    0.00   39.51

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  132.00   13.50  7148.00  2046.00    63.19     3.63   25.05   5.95  86.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.07    0.00   18.29   28.29    0.00   36.34

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.50  129.00   14.50  9292.00  2454.50    81.86     3.00   20.91   5.05  72.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.19    0.00   10.68   39.81    0.00   39.32

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  102.50   10.00  6160.00  2060.00    73.07     2.78   24.62   8.13  91.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.85    0.00   15.02   39.90    0.00   35.22

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00  3831.00   67.50  600.00  5364.00 70157.50   113.14     9.51   13.92   1.43  95.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.48    0.00    8.91   51.24    0.00   37.38

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00  3238.00   21.00  423.00  1132.00 57994.00   133.17    25.87   56.42   2.25 100.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.69    0.00    5.08   41.89    0.00   43.34

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00   52.00   15.50  4572.00  2614.00   106.46     3.99   74.74  13.85  93.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          18.25    0.00   22.63   25.06    0.00   34.06

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.50  144.50   15.50 10852.00  3064.50    86.98     2.33   14.59   4.78  76.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          25.54    0.00   21.93   18.80    0.00   33.73

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  142.50   14.00 11124.00  3388.00    92.73     2.04   13.00   3.90  61.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          25.25    0.00   21.57   16.91    0.00   36.27

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  126.00   15.00 10320.00  3228.00    96.09     2.02   14.36   4.68  66.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          13.12    0.00   12.87   32.43    0.00   41.58

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.50   67.50   10.50  4864.00  1758.50    84.90     2.83   36.03  10.90  85.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.20    0.00   10.45   37.56    0.00   41.79

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00   267.50   57.50  288.00  4540.00 40724.00   131.01     3.94   11.39   2.71  93.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.27    0.00    6.83   38.54    0.00   45.37

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00   55.00    9.00  4812.00  1610.00   100.34     1.61   25.47  13.98  89.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          22.84    0.00   14.42   25.48    0.00   37.26

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  106.00   14.00  7184.00  2540.00    81.03     1.43   11.54   5.12  61.50
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          13.40    0.00   14.89   33.00    0.00   38.71

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00    63.50   85.50  114.50  6688.00 10758.50    87.23     1.95    9.88   3.70  74.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

[Nat]

Your disk is overloaded. You would need to make your system more scalable by either

- move to faster disk

- add more memory

- move to sharding

[Avise Sudhakar Rao]

Thanks for the suggestion, but will be there any impact on the writes in Primary because of oplogSize increase? as before the change of oplogSize from 12GB to 20GB there was no latency in the writes, but after the change latency of writes observed.

--Sudhakar

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/WdfeBTg9qg0J.

[Barrie]

Hey Sudhakar,


There may be a slight increase in memory utilization with a bigger
oplog size, but judging from the information above your main issue in
terms of latency here is the stress on your hardware.  You can either
use sharding or vertically scale your hardware to alleviate the
stress.


Hope this helps.


Barrie

On Jan 26, 12:07 pm, Avise Sudhakar Rao <avisesudhakar...@gmail.com>
wrote:

[Avise Sudhakar Rao]

Hi,

Thanks for your responses, as the suggestions works but implementation will take time and because of that I changed the Secondary oplogsize to 15GB (300 GB harddisk, RAM 12 GB) and Primary oplogsize to 10GB (100 GB harddisk, RAM 12 GB), and checked the "Secondary" which again started moving to "Recovery", so Increased the RAM size of Secondary from 12 GB to 20GB and keep the oplogsize as 15GB and found that Secondary is not changing to "Recovery" state and also writes are ok, connections are getting established to 900 as earlier it was 300 (because of 300 writes where very very slow and observed lot of latency gap).

Now the current configuration of Secondary is :-
300 GB hard Disk
20GB RAM
Oplogsize is 15GB
And stack and nofile set in the /etc/security/limit.conf as below which will be useful for connection
mongodb soft nofile 65536
mongodb hard nofile 65536
mongodb hard stack 1024
mongodb soft stack 1024

And the current total connection established in the secondary is 18 only (Primary is acting ok, as current connection is 900) and the connection is not getting increased from 18, do you know any reason?

As I set oplogsize to 15GB then how much oplog will occupy the RAM? (I would like to understand more indepth about oplogsize)

Thanks,
Sudhakar

[Eliot Horowitz]

secondaries only get connections if you are doing slave ok reads.
Are you?

all memory is LRU - so increasing oplog generally doesn't impact ram much

[Avise Sudhakar Rao]

Hi Barrie,

Can you please help me to explain, if my oplogsize is 15GB, Hard Disk is 250GB and RAM is 20GB, how much memory oplog will utilize?

Thank you,
Sudhakar

On Fri, Jan 27, 2012 at 1:19 AM, Barrie <bar...@10gen.com> wrote:

[Barrie]

Hey Sudhakar,

The only part of the oplog kept in memory is that which is being
written to and read from by the replicas (which is a small part). The
size of the oplog doesn't at all relate to the amount of memory used
on the server.

Hope this helps,

Barrie

On Jan 31, 3:41 am, Avise Sudhakar Rao <avisesudhakar...@gmail.com>