readahead

[Doron Gutman]

I'm getting startup warnings regarding bad readahead settings:

** WARNING: Readahead for /data/mainRS is set to 1024KB",
**          We suggest setting it to 256KB (512 sectors) or less",
**          http://dochub.mongodb.org/core/readahead"

There's all sorts of conflicting or partial information for the below configuration (raid ebs on ec2) so I ask you for your advice about what should be the readahead configuration.

I've created a replica set using the following:

• 3 EC2 instances, running Amazon Linux
• 1 primary, 1 secondary, 1 arbiter
• I've installed the MMS agent and used MMS to configure (almost) everything.
• The primary and secondary servers use (each) a raid 10 for the mongodb data folder (log and journaling use other volumes), comprised from 4 EBS (gp2) drives - with volume size of 50 each, so the data folder is 100 in total.
• The file system is ext4

This is the outcome of blockdev --report (I removed the journal and log devices):

RO    RA   SSZ   BSZ   StartSec            Size   Device

rw   256   512  4096          0      8589934592   /dev/xvda

rw   256   512  4096       4096      8587820544   /dev/xvda1

rw   256   512  4096          0     53687091200   /dev/xvdz

rw   256   512  4096          0     53687091200   /dev/xvdy

rw   256   512  4096          0     53687091200   /dev/xvdx

rw   256   512  4096          0     53687091200   /dev/xvdw

rw  2048   512  4096          0    107306549248   /dev/md0

rw  2048   512  4096          0    107302879232   /dev/dm-0

This is the fstab entry for /data:

/dev/vg0/data /data ext4 defaults,auto,noatime,noexec 0 0

What should be the readahead configuration ?

Also - should I persist it as explained here: http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/ with a rules file in /etc/udev/rules.d/ or with a script that runs on boot ? 

Thanks.

[Doron Gutman]

Forgot to say I'm using mongodb 2.6.

Also - should I only set the readahead for the raid itself (md0) or for each of the raw EBS devices as well ? 

What should be the value of each of them ? 

[Dinesh Chander]

Hi Doron,

Readahead (RA) is a setting on a block device in linux (usually a storage device like a harddrive or NAS) that controls how much data is fetched whenever a read from that block device happens. This is done because disk seeks on spinning disks are very expensive, but accessing sequential blocks of data is relatively cheap. Setting it too high OR too low can cause excessive page faulting and increased disk utilization.

Generally speaking, our normal suggestion is to try tuning down to a RA value of 32 to start with and evaluating the improvement. This would mean that each pagefault would bring 16K of data (32*512B) per page fault rather than 128K (256*512B). Which means potentially bringing in 87.5% less data with each page fault.

If you are following our Amazon EC2 guidelines (http://docs.mongodb.org/ecosystem/platforms/amazon-ec2), and running with multiple EBS volumes in a software RAID configuration, then the recommendation is to set the RA to the appropriate size based on the stripe/chunk size of the RAID on the volumes that make up the RAID array while setting the RA to the value you want on the RAID volume itself.

Regarding persisting the values across system boot, you can choose whichever approach suits you.

Hope this answers your question. Let me know if you have further questions.

Thanks,

Dinesh Chander

[Doron Gutman]

Hi Dinesh,

Thank you for your help.

I'm afraid I did not understand what do you suggest regarding the RA of each of the volumes that comprise the RAID and the RA of the RAID volume itself.

Can you be a bit more specific or refer me to the appropriate documentation (as I said above - there isn't any clear, non-conflicting explanation on the matter).

I would appreciate that A LOT!

You can see above the volumes I have - both comprizing the raid and the raid itself.

I'm adding also the result of mdadm --detail on the relevant RAID:

# sudo mdadm --detail /dev/md0

/dev/md0:

        Version : 1.2

  Creation Time : Sun Feb  1 11:48:46 2015

     Raid Level : raid10

     Array Size : 104791552 (99.94 GiB 107.31 GB)

  Used Dev Size : 52395776 (49.97 GiB 53.65 GB)

   Raid Devices : 4

  Total Devices : 4

    Persistence : Superblock is persistent

    Update Time : Tue Aug 18 10:40:47 2015

          State : clean 

 Active Devices : 4

Working Devices : 4

 Failed Devices : 0

  Spare Devices : 0

         Layout : near=2

     Chunk Size : 256K

           Name : ip-10-0-0-10:0  (local to host ip-10-0-0-10)

           UUID : REDACTED

         Events : 246

    Number   Major   Minor   RaidDevice State

       0     202     5632        0      active sync   /dev/sdw

       1     202     5888        1      active sync   /dev/sdx

       2     202     6144        2      active sync   /dev/sdy

       3     202     6400        3      active sync   /dev/sdz

Thank you very much!

Doron

[Doron Gutman]

Hello Dinesh (or anybody else),

I'm still not clean on what should be the readahead settings.

Could anybody contribute his/her experience with it ? 

Thanks.