readahead on amazon ebs

[Leif Walsh]

Hi,

I came to office hours yesterday and had a nice talk with (I think, I'm terrible with names!) Scott and Jesse.  I'll hopefully have more interesting questions soon, but at the moment I'm curious about one thing you mentioned:

You said there were issues with a recent amazon upgrade to EBS, that increased readahead drastically.  I'm trying to replicate this situation but can't reproduce the behavior I'd expect even with a test program.  Is this a setting I can play with using blockdev --setra, or is it something else?

-- 

Cheers,

Leif

[Scott Hernandez]

That is it. And it was Jesse and I, yes.

You might also like to use lsblk to get a view of the device landscape.

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/k54OupDKmcQJ.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

[Leif Walsh]

Thanks!

Off hand, what's "filesystem readahead"?  --setfra says it's the same as --setra on 2.6 kernels but I don't actually have a clear picture of what else it could be.  Do you guys know?

> mongodb-user+unsubscribe@googlegroups.com.

[Jeff Lee]

Hey Scott,

Are your observations on changes to read-ahead with EBS something you can share with the group?

Thanks

To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/VaZYzLpX0ccJ.

To post to this group, send email to mongod...@googlegroups.com.

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[Scott Hernandez]

The short of it is that the settings on different OS/Distro/AMI
settings for the read-ahead are fairly large (megabytes) and in a
system with random access and small documents (in KB) this leads to
much more IO than is needed. If you couple this with the fact that EBS
is not a single spinning disk, and that a request is made on the
logical EBS volume, not physical underlying disk you are going to see
a larger number of iops having to be performed which will slow down
the IO operations from the host OS perspective.

Since each OS/Distro/AMI might set a different read-ahead value it is
best to test for yourself, but many people have seen much improvement
in latency, and reducing IO, by decreasing the read-ahead. This is
true on all disks, but seems especially true with EBS (RAID).

I don't have any numbers to share if that is what you are asking.

>>> > mongodb-user...@googlegroups.com.
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/mongodb-user/-/VaZYzLpX0ccJ.
>>
>> To post to this group, send email to mongod...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> mongodb-user...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.

[Leif Walsh]

Large readaheads also tend to kill cache performance on small random read workloads.

Sent from my iPhone

[Adam C]

Last but not least, large amounts of readahead with small random workloads will potentially hog an awful lot of RAM that could be put to better use elsewhere.

Adam

>>>>> mongodb-user+unsubscribe@googlegroups.com.

>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/mongodb-user?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "mongodb-user" group.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msg/mongodb-user/-/VaZYzLpX0ccJ.
>>>
>>> To post to this group, send email to mongod...@googlegroups.com.
>>> To unsubscribe from this group, send email to

>>> mongodb-user+unsubscribe@googlegroups.com.

>>> For more options, visit this group at
>>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongod...@googlegroups.com.
>> To unsubscribe from this group, send email to

>> mongodb-user+unsubscribe@googlegroups.com.

>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.

> To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.

[Leif Walsh]

Yeah, I meant page cache performance. You said it better, I think.

> > >>>>> mongodb-user...@googlegroups.com.

> > >>>>> For more options, visit this group at
> > >>>>> http://groups.google.com/group/mongodb-user?hl=en.
> > >>>
> > >>> --
> > >>> You received this message because you are subscribed to the Google
> > Groups
> > >>> "mongodb-user" group.
> > >>> To view this discussion on the web visit
> > >>> https://groups.google.com/d/msg/mongodb-user/-/VaZYzLpX0ccJ.
> > >>>
> > >>> To post to this group, send email to mongod...@googlegroups.com.
> > >>> To unsubscribe from this group, send email to

> > >>> mongodb-user...@googlegroups.com.

> > >>> For more options, visit this group at
> > >>> http://groups.google.com/group/mongodb-user?hl=en.
> > >>
> > >>
> > >> --
> > >> You received this message because you are subscribed to the Google
> > Groups
> > >> "mongodb-user" group.
> > >> To post to this group, send email to mongod...@googlegroups.com.
> > >> To unsubscribe from this group, send email to

> > >> mongodb-user...@googlegroups.com.

> > >> For more options, visit this group at
> > >> http://groups.google.com/group/mongodb-user?hl=en.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups "mongodb-user" group.
> > > To post to this group, send email to mongod...@googlegroups.com.
> > > To unsubscribe from this group, send email to

> > mongodb-user...@googlegroups.com.

> > > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
> > >
> >
>
>

--

Cheers,
Leif

[Jeff Lee]

Ah, Ok.

I think I might have mis-read the original post to imply that there had been some change to Amazon's EBS infrastructure.

Thanks for the info.

FWIW - Kristina also had a good blog entry on read-ahead a little while ago.

http://www.snailinaturtleneck.com/blog/2012/04/05/thursday-4-blockdev/