APC Cache Mixing

[Pete]

Hi,

I've a couple of Apostrophe sites running on a server using Apache with Virtual Hosts and PHP 5.3.6 running as a module (mod_php).

I'm hitting a strange problem whereby some pages on site A are getting 500 errors, looking at the error log the messages are related to code in site B - none of which is used in site A. I think I've tracked this down to APC getting cache mixed up and serving site B code to site A. I would have assumed APC used the absolute location of files and could uniquely identify them.

Has anyone every heard of this? I found a Stackoverflow post on it, http://stackoverflow.com/questions/5691666/apc-and-php-broken-sites-due-to-cache-mixing

I've disabled APC on my server and both sites seem to be functioning correctly. Is my only option now to reconfigure the server and get PHP running as a FastCGI process. Does anyone know any easy way to prefix cache files so they can be uniquely identified?

Thanks in advance,
Pete

[Tom Boutell]

We run a lot of sites on a single staging server and we've never
encountered this. Have you tried disabling APC (yes, I know it's very
slow without APC) to see whether the behavior changes?

--
Tom Boutell
P'unk Avenue
215 755 1330
punkave.com
window.punkave.com

[Pete]

Hi Tom, thanks for your reply.

Yes APC is disabled on the server at the moment and the sites are now functioning correctly - it was my only option to stop the pages erroring.

I can't understand it as I increased the apc.shm_size so it wasn't anywhere near full or fragmenting.

Pete

[Tom Boutell]

Weird. I've never seen behavior like that from APC myself.

You could have problems caused by class files that aren't timestamped
as newer than the previously deployed version masquerading as this
other thing... we had that issue before we got the perfect rsync
options going in apostrophe:deploy (:

--

[Pete]

arr.. that's interesting as I have been using the symfony project:deploy command not the Apostrophe version!

What exactly does the Apostrophe deploy script do differently? Is it something I can debug to make sure this is the issue.

Thanks, Pete

[Tom Boutell]

apostrophe:deploy specifies rsync options that ensure every single
file is checksummed to determine if the local copy is newer than the
remote, rather than settling for timestamps. Timestamps can lie to you
in a variety of situations, especially if you have more than one
developer deploying. We have never had an APC "WTF" moment since
making this change (:

--

[Pete]

Thanks for your help Tom.. I'll have a fiddle with that and see if this is the problem :)

[Pete]

Hummm.. I'm not sure this will solve it & I can't risk trying it (at the moment) on the live server.

Looking at the deploy task you comment..
Don't preserve timestamp. That way we don't have to clear the APC cache after every sync, because APC can tell our code is new

I've cleared the APC cache numerous times whilst trying to fix the problem without deploying any files. Meaning the timestamp theory isn't the problem?

Pete

[Tom Boutell]

So clearing your APC cache doesn't help at all?

That's seriously screwy.

How old is your build of APC?

--

[Pete]

Clearing the APC cache seems to fix it for a while but once the cache builds up again the errors occur.

I also upgrade APC to the latest version, 3.1.9, that didn't fix it either!

The main error I'm getting is a SQL error.
In Site B I have extended the aPage table with some extra fileds:
aPage:
  columns:
    title_nav:
      type: string(255)
    title_h1:
      type: string(255)

Site A does not have these fields and dosen't use any code from Site B. Yet the error message that is causing Site A to give 500 errors is:
symfony [err] {Doctrine_Connection_Mysql_Exception} SQLSTATE[42S22]: Column not found: 1054 Unknown column 'a.title_nav' in 'field list'

.. as if Site A is using Site B's models! They must be coming from APC as it's all working with APC disabled.

Pete

[Jeremy Kauffman]

This is an issue of the Doctrine query cache being shared across projects. Set a per project APC prefix and you'll be good.

[Pete]

Thanks for your reply Jeremy..

Exactly how do I set a per project APC prefix? Is there a nice config setting within Symfony that I can tweak?
Sorry for the noobish question. I'm not familiar with any of this.

Thanks,
Pete

[Tom Boutell]

AHHH nice call Jeremy

But Pete you would have had to go out of your way to tell Doctrine to
use APC for query cache, did you do that?

--

[Pete]

Yes I used the guide at http://symfony-check.org/ 

config/ProjectConfiguration.class.php

public function configureDoctrine(Doctrine_Manager $manager)
{
  $manager->setAttribute(Doctrine::ATTR_QUERY_CACHE, new Doctrine_Cache_Apc());
}

Thanks so much for your help here guys.

Pete,

[Pete]

...so do I need to remove this line and disable APC with Doctrine? 

Or can I leave it enabled but set a prefix? How would I do this? I can't see any obvious options in the Doctrine_Cache_Apc class.

Thanks

Pete

[Tom Boutell]

Disabling APC with Doctrine will work. We've never enabled it here;
let us know if it makes much of a difference when turned off (:

I am certainly curious if the prefix thing exists as well.

--

[Jeremy Kauffman]

I thought I sent this yesterday afternoon discovered it in drafts this morning...

Pass in prefix as an option:

public function configureDoctrine(Doctrine_Manager $manager)
{
  $manager->setAttribute(Doctrine::ATTR_QUERY_CACHE, new Doctrine_Cache_Apc(array('prefix' => 'something_unique'));
}

something_unique can be whatever you want... hardcoded string, hashed path, sfConfig value, whatever...

The Doctrine query cache is a nice idea but it's implementation has a few flaws. For example, adding a column to a database will break all your SELECT * queries on that table unless you clear APC.

Jeremy Kauffman

267.571.9320

www.flickswitch.cc

[Tom Boutell]

Jeremy, is there a query cache at all by default? How much difference
does it make in practice?

--

[Tom Boutell]

You could write:

$ini = parse_ini_file(sfConfig::get('sf_config_dir').'/properties.ini', true);
$name = $ini['name'];
array('prefix' => $name);

To avoid having to hardcode this for every project.

Jeremy is right that APC has to be cleared on deployments if they
involve database migrations and you use the Doctrine APC cache option.

At one point we had APC clearing as part of apostrophe:deploy; it
tickled a special URL on the site whose purpose was simply to call
apc_clear_cache from a web action (command line PHP can't see the same
APC cache). We removed that code because improvements to our rsync
usage made it unnecessary for the purposes we originally added it for.
But the idea is still a viable one.

On Sat, Sep 3, 2011 at 10:21 AM, Jeremy Kauffman <jer...@flickswitch.cc> wrote:

--

[Tom Boutell]

Actually it would be $ini['symfony']['name'], but you get the idea.

[Pete]

I've disabled APC with Doctrine, I didn't want to risk the prefix because of the other flaws Jeremy mentions. Plus, if you don't use it then that's good enough for me.
The sites now seem to be running fast, can't notice any difference to when I had it enabled, and more importantly error free.

Thanks very much for your help Tom & Jeremy.
I've linked to this thread from the Stackoverflow question I referenced & post a comment to symfony-check.org so hopefully other people can benefit.

[Jeremy Kauffman]

In a quick informal test, I see about a 10% improvement with the query cache on. This is on our TopScore sites, which are relatively query heavy.

The only two issues I can remember running into with the query cache are both mentioned in this thread: identical queries across projects and migrations. Both are a little annoying, but easily solvable.

If you do choose to enable the query cache, I would recommend against using a page to clear the APC cache. You'll eventually run into catch-22 issues as queries happen on every page. Instead, I recommend an approach the does not require accessing Doctrine (such as this one I've outlined on StackOverflow).

Jeremy Kauffman

267.571.9320

www.flickswitch.cc

[Tom Boutell]

Thanks Jeremy, especially for pointing out the catch-22. You're right,
the script that clears the cache shouldn't be Symfony-based.

I think keeping that script around would be acceptable given the
localhost restriction and the essential harmlessness of clearing the
cache. You could also put a password in there instead of a localhost
restriction.

A 10% improvement is nothing to sneeze at, we should consider
integrating this into our practice.