keeping the cache primed on a large site

[Marc Steel]

Hi Patrick (and new group!),

I'm continuing the discussion started in this thread:

http://wordpress.org/support/topic/plugin-w3-total-cache-keeping-the-cache-primed-on-a-large-site

The background is:

• WP 3.3 running in Network mode.
• Six main blogs on the network, 3 large, 3 small
• All six have DB and Object caching using APC (we don't have memcache)
• The three smalls are switched to disk caching for pages and running faster
• I have crons running for the three smalls

Now I'm going to bring the three big blogs in to the same set up. I tried before but I think I gave the server too much to do at once.

Right now I'm going to prime one large blog at a time and then set a cron for it.

My only question right now is if I should be sure to space the crons out at all. 

Patrick, your script and support are fantastic, thanks!

[Marc Steel]

p.s. the site is TechnologyTell.

[Patrick Mylund Nielsen]

Hey Marc,

Glad it's (partly) working. I'm curious where you're experiencing this
slowdown? Is it in the page response times when OCP is running, or the
system load--maybe the disk?

If you're using OCP in "local mode", i.e. specifying a directory it
should look in locally, then I'd recommend running just one OCP
process at a time because it's going to check for existing files as
fast as it can, and multiple processes doing that at the same time
might cause some slowdown. If you're not, then you could run as many
OCP as you want--just keep the -c (concurrency) parameter at a level
you're comfortable with (-c 1, the default, means each process only
loads one page, through one connection, at a time, and -c 20 means 20
pages through 20 connections. It keeps the same 20 connections open
for subsequent requests if your server has Keep-Alive enabled, so
you're saved the overhead of re-establishing connections all the
time.)

[Patrick Mylund Nielsen]

If it's the disk, then maybe it makes sense to add a parameter to OCP
that will let you space out the local file checks a little more. Right
now it's just doing as many as it can, but maybe you would see less of
an overhead if it checked for the existence of at most one cached
files at once per millisecond or something like that.

[Marc Steel]

The problem may have been that I tried to fully prime from scratch the
three big sections AND set crons on them while they were priming. Dumb
I know, got anxious I guess.

So tonight I started priming one of them, without a cron behind it.
The ocp process has been running for about 55 minutes so far. A few
minutes ago I got a 404 error, so it seems to be working. Also, the
number of items in the cache folder keeps increasing. Up to 7300 so
far.

So once I get this primed I figure I'll set a cron on it and watch it
for a while. If all goes well I'll bring up the other two the same
way.

[Patrick Mylund Nielsen]

Ah, that makes sense. Yes, the first time around, that would cause a
lot of load if you've set OCP to run e.g. every minute or so. If the
single process doesn't strain your server much, try to up -c to e.g. 3
or 4 (or some number a few under your number of PHP processes.) That
makes it a LOT faster.

You can still have OCP run every minute if you make sure only one copy
is running at once. If you're on Linux, you can use a script like
this:

#!/bin/sh
LOCKFILE=/tmp/ocp-lock.txt
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then
# already running
exit
fi

trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT
echo $$ > ${LOCKFILE}

ocp -c 4 -l /var/www/mysite/pgcache -ls _index.html
http://mysite.com/sitemap.xml

rm -f ${LOCKFILE}


The script will do nothing if OCP is already running. If you make a
single sitemapindex and run this every minute or so, I think it would
work to your satisfaction. It wouldn't kill the server, and once the
whole cache is primed (assuming you don't flush the whole cache at
once), every time it runs it'll just prime 50-100 pages or so.

I think I'll add this to the FAQ, or even add a --singleton switch to
OCP. It's hard to make a generalized solution though since Windows is
quite different.

> --
> You received this message because you are subscribed to the Google Groups "Optimus Cache Prime" group.
> To post to this group, send email to optimus-c...@googlegroups.com.
> To unsubscribe from this group, send email to optimus-cache-p...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/optimus-cache-prime?hl=en.
>

[Marc Steel]

That is simply awesome.

I think you've come up with quite a nice solution :)

[Marc Steel]

For the record, the process completed. It took a little over and hour
to generate roughly 9300 items.

[Marc Steel]

FTR, the process completed. It took little over an hour and about 9500
items.

[Patrick Mylund Nielsen]

Hey Marc,

Cool!

It should be significantly faster now (if the local file checking is
working). You can run ocp with the -v flag to check. If it finds an
existing (cached) page, it will say "Exists", and if not, "Get". If it
is still getting pages, then most likely there's an issue with the
paths.

On Fri, May 18, 2012 at 3:40 AM, Marc Steel <marc....@gmail.com> wrote:
> For the record, the process completed. It took a little over and hour
> to generate roughly 9300 items.
>

[Patrick Mylund Nielsen]

Great.

This actually seemed like a pretty neat thing, so I've made a more
generic version of the script: http://patrickmylund.com/projects/one/

I also added a reference to it on the OCP FAQ.

On Fri, May 18, 2012 at 3:26 AM, Marc Steel <marc....@gmail.com> wrote:
> That is simply awesome.
>
> I think you've come up with quite a nice solution :)
>

[Marc Steel]

I have a sitemap index of all my sitemap indexes, but is it possible
to do use once ocp cron script with it since each section needs its
own cache paths?

[Patrick Mylund Nielsen]

Sorry. You're right, it's pretty hard if the files are in different
root folders, too, and you'd need to be able to specify which domain
mapped to each folder. Maybe I can think of a way to do this without
changing the syntax, but for now it's easier to run one OCP per path.

I figure that would work rather well with the one-script (with one
lock for all OCP processes) from earlier, though.