Cache Size

[Shawn Ligocki]

(Moved to a new email thread.)

File cache size: As a basic basic rule of thumb I think we usually suggest a cache twice the size of your website data. As Jonathan pointed out though, you may want to make it 3-5x the data since we will store both jpeg and webp versions of rewritten images (plus the original images).

Eviction algorithm: We actually evict the Least Recently Used (LRU) resources, so this should keep commonly used resources in preferentially. (Note: This is not true if your file-system is mounted with noatime. In that case we will evict the oldest resources first).

Note: If your cache is too small, we will not behave well, mod_pagespeed really does expect you to have a cache large enough to store the entire website data.

However there are a few things you could also try to reduce the load on your system:

* Mount file cache on tmpfs. (This should reduce load significantly)

* Integrate with memcached. (memcached may also reduce load spikes)

YMMV, but it may be better suited for your use-case than our standard file cache.

On Tue, Jun 18, 2013 at 5:54 PM, Daniel Bartholomew <dan...@squixa.com> wrote:

We struggle to right-size our caches as well.

We find sites with low cache hit ratios where the CPU gets smashed every cache purge, even when we have our cache set at 3GB. The documentation says that the cache clear evicts the oldest resources, not the least used resources - I was wondering if this might be a better solution?

Our strategy now is to just allocate a massive cache (have been thinking 100GB) because if the cache eviction happens to line up with a high traffic period then the users suffer.

[Daniel Bartholomew]

On memcached and file cache - these caches are not mutually exclusive right? If I turn on memcached  will the file cache continue to behave as it does?

--

Daniel Bartholomew

Technical Director & Founder

+61 488 215 220

dan...@squixa.com

[Shawn Ligocki]

memcached mostly replaces file cache. We still use file cache for very large files (because memcached has a filesize limit), but we expect most of the caching to be stored in memcached.

[Shawn Ligocki]

memcached mostly replaces file cache. We still use file cache for very large files (because memcached has a filesize limit), but we expect most of the caching to be done in memcached.

[Joshua Marantz]

Also please consider using the shared memory cache, particularly if you are using memcached.

[Jonathan Garbee]

There should be a way in this console to detect a small cache and alert the admin that it can be an issue. That alone could be a huge aid, on top of possibly adding a calculator to help estimate the size.

In our case, it is ~1.7GB in the folder of card images. That is most of our stock. Those images are displayed at least in five different sizes in seven or eight different places (assuming that images replicated in the same size should use the same cache, we really only need to stick with the five number.) So we have five sizes, two file types, plus the original all stored in cache.

Doing some strict math (way off) 1.7GB * five copies * two file types + originals (1.7GB) = 18.7GB of cache space. Now, of course since the copies are optimized we can take that 17GB and take off let's say 35% is a good rough median between webp and jpeg. That leaves 11.05 + 1.7GB for originals which means 12.75GB of cache space to start with.

The exact equation for a calculator could replicate something like what I just explained. [Input original size of items] * [input how many times items are shown in different sizes] * [type of conversions enabled, perhaps a checkbox-list] + [input of original size of items] to give a rough estimate. 

Either way, putting a calculator in the console or at least a help page explaining how to calculate a good cache size would be very helpful to administrators.