duplicate compiled files in APC
191 views
Skip to first unread message
matterk
Nov 4, 2009, 11:54:12 AM11/4/09
to Smarty Developers
I'm not sure if this is an APC issue or a Smarty issue. I am seeing
multiple files in APC with different inode's but the exact same file
name, but this only appears to happen to smarty compiled template
files.
Steps to reproduce:
1. Load a page, allow smarty to compile all templates and APC to cache
those
2. Make a change to a template file
3. Reload the page now shows two files in APC with the same file name
but with different inode numbers, one is the updated compiled
template, the other is the old version.
The behavior we are seeing is that when you load a page, it seems to
randomly pick one of the files from APC to use as the compiled
template and you end up getting back different version of the template
for each page load.
My guess is this is happened because of the way compiled templates get
written to the file system, i.e. writing to a temp file then doing rm
of the original and a rename of the temp file to replace it.
APACHE: 2.2.9
PHP: 5.2.9
APC: 3.0.19
OS: debian (lenny)
multiple files in APC with different inode's but the exact same file
name, but this only appears to happen to smarty compiled template
files.
Steps to reproduce:
1. Load a page, allow smarty to compile all templates and APC to cache
those
2. Make a change to a template file
3. Reload the page now shows two files in APC with the same file name
but with different inode numbers, one is the updated compiled
template, the other is the old version.
The behavior we are seeing is that when you load a page, it seems to
randomly pick one of the files from APC to use as the compiled
template and you end up getting back different version of the template
for each page load.
My guess is this is happened because of the way compiled templates get
written to the file system, i.e. writing to a temp file then doing rm
of the original and a rename of the temp file to replace it.
APACHE: 2.2.9
PHP: 5.2.9
APC: 3.0.19
OS: debian (lenny)
Monte Ohrt
Nov 4, 2009, 12:06:15 PM11/4/09
to smarty-d...@googlegroups.com
That looks like an issue with APC, not removing the cache of no-longer existing php files. Does it persist for long? Is there a setting in APC that tells it how often to do garbage collection?
Monte Ohrt
Nov 4, 2009, 12:07:20 PM11/4/09
to smarty-d...@googlegroups.com
Also, is this new to Smarty 3, or did Smarty 2 also exhibit this APC behavior?
U.Tews
Nov 4, 2009, 12:09:25 PM11/4/09
to Smarty Developers
Another user did have similar problems with cache files a while ago.
It did turn out that it was a problem of APC version 3.0.19.
It did work with APC 3.0.17-dev
It did turn out that it was a problem of APC version 3.0.19.
It did work with APC 3.0.17-dev
matterk
Nov 4, 2009, 12:20:56 PM11/4/09
to Smarty Developers
We haven't seen it in smarty 2 at all. But these files stick around
until you explicitly flush the system cache in APC. We have also
tried the latest version of APC 3.1.3 and we see the same problem.
I did find this in the list of bugs for APC: http://pecl.php.net/bugs/bug.php?id=16736
But the long and short of it, it doesn't look like they are going to
change the way APC works.
"Sounds like with the inodes changing on every file update the cache
doesn't find the original entries in the cache (as they are key'd off
of
the inode #) and remove them on the new insert, so entries just stack
up.
Do you have a ttl set for your file cache? If not does setting one to
a
very low setting help? (entries will be removed if we come across them
or on a cache full scenario, rather than expunging the entire cache).
Another option here would be to upgrade to 3.1.1 or above, and use the
apc_delete_file($filename) before you delete the original file on
disk.
Another similar option would be to set apc.stat=0 and then use
apc_compile_file() to update the entries manually after you update the
files on disk. The LFU in APC-4.0.0 will expunge these entries more
efficiently, but that's not really ideal memory usage either. Perhaps
the proper fix here is to allow some combination of full paths but
with
stats as well as none of the above suggestions is really ideal."
until you explicitly flush the system cache in APC. We have also
tried the latest version of APC 3.1.3 and we see the same problem.
I did find this in the list of bugs for APC: http://pecl.php.net/bugs/bug.php?id=16736
But the long and short of it, it doesn't look like they are going to
change the way APC works.
"Sounds like with the inodes changing on every file update the cache
doesn't find the original entries in the cache (as they are key'd off
of
the inode #) and remove them on the new insert, so entries just stack
up.
Do you have a ttl set for your file cache? If not does setting one to
a
very low setting help? (entries will be removed if we come across them
or on a cache full scenario, rather than expunging the entire cache).
Another option here would be to upgrade to 3.1.1 or above, and use the
apc_delete_file($filename) before you delete the original file on
disk.
Another similar option would be to set apc.stat=0 and then use
apc_compile_file() to update the entries manually after you update the
files on disk. The LFU in APC-4.0.0 will expunge these entries more
efficiently, but that's not really ideal memory usage either. Perhaps
the proper fix here is to allow some combination of full paths but
with
stats as well as none of the above suggestions is really ideal."
Steve E
Nov 6, 2009, 3:13:49 AM11/6/09
to Smarty Developers
This is exactly the behaviour I had, but I had it on Windows using
Apache and APC, although it seemed isolated to one of my systems only.
It's working fine (it seems) on my production platform:
NGINX: 0.8.19
PHP: 5.2.11 using PHP-FPM
APC: 3.0.19
OS: Centos 5.3
I had never experienced this problem with any other php script. APC
kept alternating between the previous version and the new version, and
the only way to reset this (apart from removing the file itself from
the cache) was to restart APC.
Although it's not Smarty's fault, this could trip up a lot of users
and may be something you need to consider looking into further on how
to avoid. If nothing else, maybe a flag to change the behaviour from
doing remove/move to doing a copy over instead? Or detect if APC is in
use then delete the file from APC so only one version remains?
Steve
Apache and APC, although it seemed isolated to one of my systems only.
It's working fine (it seems) on my production platform:
NGINX: 0.8.19
PHP: 5.2.11 using PHP-FPM
APC: 3.0.19
OS: Centos 5.3
I had never experienced this problem with any other php script. APC
kept alternating between the previous version and the new version, and
the only way to reset this (apart from removing the file itself from
the cache) was to restart APC.
Although it's not Smarty's fault, this could trip up a lot of users
and may be something you need to consider looking into further on how
to avoid. If nothing else, maybe a flag to change the behaviour from
doing remove/move to doing a copy over instead? Or detect if APC is in
use then delete the file from APC so only one version remains?
Steve
matterk
Nov 6, 2009, 1:57:13 PM11/6/09
to Smarty Developers
Yeah, we actually had to turn off APC caching for opcodes in our dev
environments with apc.cache_by_default=0
Which is really less than ideal.
On Nov 6, 12:13 am, Steve E <st...@effectivemarketingtenerife.com>
wrote:
environments with apc.cache_by_default=0
Which is really less than ideal.
On Nov 6, 12:13 am, Steve E <st...@effectivemarketingtenerife.com>
wrote:
Monte Ohrt
Nov 6, 2009, 2:24:08 PM11/6/09
to smarty-d...@googlegroups.com
Although I don't like to fix other library problems in my own code, if this is a widespread problem, we could put a test in the write_file routine: if apc function exists, execute it to clear the cache. Do we know specifically what versions of APC this affects?
For the time being, you could patch the write_file routine yourself. Hopefully this problem will be short lived.
For the time being, you could patch the write_file routine yourself. Hopefully this problem will be short lived.
Steve E
Nov 6, 2009, 8:12:49 PM11/6/09
to Smarty Developers
Hmmm. I notice the apc_delete_file function doesn't exist in APC
3.0.19 which I think is a standard distribution for many systems, so
that doesn't help, unfortunately. It would have been so easy to remove
the old cached file after it was deleted...
Oh well.
I haven't had a chance to experiment with this myself, but if anyone
with this problem has time, try changing the file /libs/sysplugins/
smarty_internal_write_file.php from this:
// remove original file
if (file_exists($_filepath))
unlink($_filepath);
// rename tmp file
rename($_tmp_file, $_filepath);
to something like this (my apologies, haven't tested it so might very
well have errors in it):
copy($_tmp_file, $_filepath);
unlink($_tmp_file);
I would think this would work, as copying a file using 'cp' (or 'copy'
on Windows) has never caused this problem for me, as far as I am
aware.
Above this section there is a comment as follows:
// write to tmp file, then move to overt file lock race condition
I presume changing the code to the above reintroduces the potential of
a lock race condition, but I couldn't comment at all on the impact of
this. Does someone else have an idea?
Steve
On Nov 6, 7:24 pm, Monte Ohrt <mo...@ohrt.com> wrote:
> Although I don't like to fix other library problems in my own code, if this
> is a widespread problem, we could put a test in the write_file routine: if
> apc function exists, execute it to clear the cache. Do we know specifically
> what versions of APC this affects?
>
> For the time being, you could patch the write_file routine yourself.
> Hopefully this problem will be short lived.
>
3.0.19 which I think is a standard distribution for many systems, so
that doesn't help, unfortunately. It would have been so easy to remove
the old cached file after it was deleted...
Oh well.
I haven't had a chance to experiment with this myself, but if anyone
with this problem has time, try changing the file /libs/sysplugins/
smarty_internal_write_file.php from this:
// remove original file
if (file_exists($_filepath))
unlink($_filepath);
// rename tmp file
rename($_tmp_file, $_filepath);
to something like this (my apologies, haven't tested it so might very
well have errors in it):
copy($_tmp_file, $_filepath);
unlink($_tmp_file);
I would think this would work, as copying a file using 'cp' (or 'copy'
on Windows) has never caused this problem for me, as far as I am
aware.
Above this section there is a comment as follows:
// write to tmp file, then move to overt file lock race condition
I presume changing the code to the above reintroduces the potential of
a lock race condition, but I couldn't comment at all on the impact of
this. Does someone else have an idea?
Steve
On Nov 6, 7:24 pm, Monte Ohrt <mo...@ohrt.com> wrote:
> Although I don't like to fix other library problems in my own code, if this
> is a widespread problem, we could put a test in the write_file routine: if
> apc function exists, execute it to clear the cache. Do we know specifically
> what versions of APC this affects?
>
> For the time being, you could patch the write_file routine yourself.
> Hopefully this problem will be short lived.
>
Monte Ohrt
Nov 6, 2009, 8:42:31 PM11/6/09
to smarty-d...@googlegroups.com
Windows has had a problem with copying on top of existing files in the
past, hence the unlink first. I'm not sure about that any more.
past, hence the unlink first. I'm not sure about that any more.
mohrt
Nov 18, 2009, 9:39:20 PM11/18/09
to Smarty Developers
I am now running into this situation. However, it only seems to happen
when I use template inheritance, and I'm only changing the parent
template. The child template is recompiling fine, but APC isn't
getting the change. Sometimes it alternates between older versions,
sometimes it sticks to one old one. If I disable APC or restart
apache, all is good until I change the parent again.
> >>apcfunction exists, execute it to clear the cache. Do we know specifically
> >> what versions ofAPCthis affects?
> >>>> use then delete the file fromAPCso only one version remains?
> >>>>> I'm not sure if this is anAPCissue or a Smarty issue. I am seeing
> >>>>> multiple files inAPCwith different inode's but the exact same file
when I use template inheritance, and I'm only changing the parent
template. The child template is recompiling fine, but APC isn't
getting the change. Sometimes it alternates between older versions,
sometimes it sticks to one old one. If I disable APC or restart
apache, all is good until I change the parent again.
> >> what versions ofAPCthis affects?
>
> >> For the time being, you could patch the write_file routine yourself.
> >> Hopefully this problem will be short lived.
>
> >> On Fri, Nov 6, 2009 at 12:57 PM, matterk <matt.erkk...@gmail.com> wrote:
>
> >>> Yeah, we actually had to turn offAPCcaching for opcodes in our dev
> >> For the time being, you could patch the write_file routine yourself.
> >> Hopefully this problem will be short lived.
>
> >> On Fri, Nov 6, 2009 at 12:57 PM, matterk <matt.erkk...@gmail.com> wrote:
>
> >>> environments withapc.cache_by_default=0
>
> >>> Which is really less than ideal.
>
> >>> On Nov 6, 12:13 am, Steve E <st...@effectivemarketingtenerife.com>
> >>> wrote:
>
> >>>> This is exactly the behaviour I had, but I had it on Windows using
> >>>> Apache andAPC, although it seemed isolated to one of my systems only.
>
> >>> Which is really less than ideal.
>
> >>> On Nov 6, 12:13 am, Steve E <st...@effectivemarketingtenerife.com>
> >>> wrote:
>
> >>>> This is exactly the behaviour I had, but I had it on Windows using
>
> >>>> It's working fine (it seems) on my production platform:
>
> >>>> NGINX: 0.8.19
> >>>> PHP: 5.2.11 using PHP-FPM
> >>>>APC: 3.0.19
> >>>> OS: Centos 5.3
>
> >>>> I had never experienced this problem with any other php script.APC
> >>>> kept alternating between the previous version and the new version, and
> >>>> the only way to reset this (apart from removing the file itself from
> >>>> the cache) was to restartAPC.
>
> >>>> Although it's not Smarty's fault, this could trip up a lot of users
> >>>> and may be something you need to consider looking into further on how
> >>>> to avoid. If nothing else, maybe a flag to change the behaviour from
> >>>> doing remove/move to doing a copy over instead? Or detect ifAPCis in
> >>>> It's working fine (it seems) on my production platform:
>
> >>>> NGINX: 0.8.19
> >>>> PHP: 5.2.11 using PHP-FPM
> >>>>APC: 3.0.19
> >>>> OS: Centos 5.3
>
> >>>> I had never experienced this problem with any other php script.APC
> >>>> kept alternating between the previous version and the new version, and
> >>>> the only way to reset this (apart from removing the file itself from
> >>>> the cache) was to restartAPC.
>
> >>>> Although it's not Smarty's fault, this could trip up a lot of users
> >>>> and may be something you need to consider looking into further on how
> >>>> to avoid. If nothing else, maybe a flag to change the behaviour from
> >>>> use then delete the file fromAPCso only one version remains?
> >>>>> I'm not sure if this is anAPCissue or a Smarty issue. I am seeing
> >>>>> multiple files inAPCwith different inode's but the exact same file
> >>>>> name, but this only appears to happen to smarty compiled template
> >>>>> files.
>
> >>>>> Steps to reproduce:
>
> >>>>> 1. Load a page, allow smarty to compile all templates andAPCto cache
> >>>>> files.
>
> >>>>> Steps to reproduce:
>
> >>>>> those
> >>>>> 2. Make a change to a template file
> >>>>> 3. Reload the page now shows two files inAPCwith the same file name
> >>>>> 2. Make a change to a template file
> >>>>> but with different inode numbers, one is the updated compiled
> >>>>> template, the other is the old version.
>
> >>>>> The behavior we are seeing is that when you load a page, it seems to
> >>>>> randomly pick one of the files fromAPCto use as the compiled
> >>>>> template, the other is the old version.
>
> >>>>> The behavior we are seeing is that when you load a page, it seems to
Monte Ohrt
Nov 18, 2009, 9:53:37 PM11/18/09
to smarty-d...@googlegroups.com
I think I just figured out this problem and it's likely not APC's fault!
When a parent template changes, the child template gets recompiled by
Smarty. However, the timestamp does not get updated because the child
tpl source isn't changed. Therefore, APC does not pick up the change to
the compiled file.
So to fix this problem is the following: When a parent template changes
and a child template needs recompiled, the *timestamp* of the child
compiled template needs updated, as well as recompiled. I'm not sure how
well this will work, since Smarty compares the file timestamp to the
original child template source. We may have to revert to the Smarty 2
way of timestamping tests: if compiled file is same *or newer* than
source tpl, don't recompile it.
I'm hoping this is the source of APC creating multiple copies of the
same file with separate inodes: when you update a PHP file but do not
update the timestamp, APC has issues.
> --
>
> You received this message because you are subscribed to the Google Groups "Smarty Developers" group.
> To post to this group, send email to smarty-d...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/smarty-developers?hl=.
>
>
>
When a parent template changes, the child template gets recompiled by
Smarty. However, the timestamp does not get updated because the child
tpl source isn't changed. Therefore, APC does not pick up the change to
the compiled file.
So to fix this problem is the following: When a parent template changes
and a child template needs recompiled, the *timestamp* of the child
compiled template needs updated, as well as recompiled. I'm not sure how
well this will work, since Smarty compares the file timestamp to the
original child template source. We may have to revert to the Smarty 2
way of timestamping tests: if compiled file is same *or newer* than
source tpl, don't recompile it.
I'm hoping this is the source of APC creating multiple copies of the
same file with separate inodes: when you update a PHP file but do not
update the timestamp, APC has issues.
>
> You received this message because you are subscribed to the Google Groups "Smarty Developers" group.
> To post to this group, send email to smarty-d...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/smarty-developers?hl=.
>
>
>
Uwe.Tews
Nov 21, 2009, 7:38:52 AM11/21/09
to Smarty Developers
Timestamping of the compiled templates has been changed back to the
way how Smarty2 did it.
(the compiled template keeps the timestamp if its compilation time)
This solves the problems with extended templates.
way how Smarty2 did it.
(the compiled template keeps the timestamp if its compilation time)
This solves the problems with extended templates.
Reply all
Reply to author
Forward
0 new messages