Performance characteristics

[shabbyrobe]

Hello again,

I've noticed the performance of addendum seems to drop off rapidly
when dealing with many annotations in a single script.

If I run the below script with 'ab -n 2000 -c 5' with $fast set to
true, I get 363 requests per second. If I run it with $fast set to
true, I get 126 requests per second. It degrades significantly with
each new property I add.

Am I doing something wrong?

<?php
$fast = false;

class Table extends Annotation {}
class Column extends Annotation {
public $name; public $length; public $type='varchar';
}

/** @Table("country") */
class Country {
/** @Column(name="id", type="int") */
public $id;

/** @Column(name="name", length=30) */
public $name;

/** @Column(name="code", length=30) */
public $code;
}

$c = new ReflectionAnnotatedClass('Country');
var_dump($c->getAnnotations());
if (!$fast) {
foreach ($c->getProperties() as $p) {
var_dump($p->getAnnotations());
}
}

[Jan Suchal]

Hi,

As for performance. Addendum is mainly a parsing library, I do some
caching on per request level, but further caching is not included. There
are too many options to choose from if you want to share state between
requests. Redis/memcached or even plain files just to name a few. Even
more you have to manage cache invalidation. However it is not so hard to
plug in your own caching. Maybe refactoring cache from
AnnotationsBuilder to an adapter will be a good start
http://code.google.com/p/addendum/source/browse/trunk/annotations.php?r=74#148

johno

PS. Sorry for cross-posting this.

[shabbyrobe]

Cross posting is fine - it's better for both things have appropriate
answers!

I have been playing around with some caching of the parsing results to
APC, just hacking - your code is nice and lean so it's easy and fun to
hack on. I've been storing a dependency class alongside the parsing
results that checks the file mtime of the source on retrieval and
invalidates if it has changed. I've managed to go from the 130rps I
mentioned above to about 1000rps.

On Jan 8, 6:18 am, Jan Suchal <jo...@jsmf.net> wrote:
> Hi,
>
> As for performance. Addendum is mainly a parsing library, I do some
> caching on per request level, but further caching is not included. There
> are too many options to choose from if you want to share state between
> requests. Redis/memcached or even plain files just to name a few. Even
> more you have to manage cache invalidation. However it is not so hard to
> plug in your own caching. Maybe refactoring cache from

> AnnotationsBuilder to an adapter will be a good starthttp://code.google.com/p/addendum/source/browse/trunk/annotations.php...

[shabbyrobe]

Would you be interested in me packaging up some of my cache changes
and submitting a patch? If so, what would you like to see in there?
Did you have any thoughts on how you would like it to be done? I'm
happy to change anything

I think an adapter class is a good way to go, so you can plug any
cache you want in. Maybe we could throw in a simple no-frills cache so
people can use addendum at full speed out of the box, but provide a
wiki page demonstrating bridging to another library's cache?

[Jan Suchal]

Yeah, the first step is to decouple per-request cache to an adapter.
Then additional cache adapters can be used, maybe defining an
interface/abstract class for adapter is a good idea.

Do you want to get your hands dirty on this one?

[shabbyrobe]

I'd love to! I'll see what I can whip up tonight.

[shabbyrobe]

I've got a patch for this prepared, how would you like me to send it?

I spent a fair bit of time with kcachegrind trying to get it all in
there without affecting the style and structure of addendum, but I did
end up making a few tweaks - I hope that's OK, I promise they're
beneficial!

I did exactly what you suggested - factored out the cache bit of
AnnotationsBuilder into AddendumMemoryCache (which just masks an array
with no persistence between requests), which implements an
AddendumCache interface.

I added a mode setting to Addendum that determines how caching is
used. There are three possible values: debug, normal and performance.
"Debug" never caches anything, "normal" invalidates the cache if
either the reflection's file or any of the files containing the
annotations have been modified since the cache item was set (using the
file's mtime). "Performance" never checks mtimes. As a side note, I
added a test into AnnotationsBuilder that skipped all the mtime
malarkey when the cache was an instance of AddendumMemoryCache, but it
didn't improve performance at all so I left it out - I suppose if
users really cared they'd use a more persistent caching solution.

Speaking of more persistent caching solutions, included in the patch
are two simple classes that implement AddendumCache: AddendumFileCache
and AddendumAPCCache. AddendumFileCache is pretty braindead - it just
uses the md5 of the key (AnnotationsBuilder::createName()) as the file
name and serialises the parsed annotations array into it.
AddendumAPCCache is similarly braindead. I wanted to keep the caches
as lean as possible to encourage adapters. The caches I have added
aren't exactly comprehensive in their feature set; they do the
absolute bare minimum to allow users of addendum who don't have
another caching solution cooked up the option of benefiting from the
massive performance improvement without any significant effort.

My profiling experiments with this patch show that performance would
be improved substantially by caching the parsed annotation data for an
entire class in one hit. At the moment, the cache is hit for every
member for which we are requesting annotation data and easily 20 or
30% of the time is spent making many more calls to the caches than
would be made if it was only hitting for a full class. I didn't want
to tear the whole thing up trying to do that straight off the bat
though and I think this can evolve into something like that down the
line if you want.

Anyway, some basic numbers. I have a little program that has a 'mock
ORM' sort of thing (a fairly common use for annotations). There are 4
models and 6 annotations, many with @Targets, and a runner script that
iterates over the annotations for every property and method in all of
the models. I can zip this up and send it to you with the patch if you
like. I got the following numbers running ab -n 2000 -c 5 to get an
overall load figure:

Baseline PHP (echo "";): 2190.82 requests per second
Profiling test script, only models & annotation classes required, no
calls to addendum: 1072.01 rps
Addendum r77: 31.68 rps
Patched Addendum with no cache: 29.88 rps
Patched Addendum with APC cache in DEBUG mode: 20.43 rps, -31.62%
Patched Addendum with APC cache in NORMAL mode: 165.13 rps, +552.64%
Patched Addendum with APC cache in PERFORMANCE mode: 191.58 rps,
+641.16%

All tests pass on PHP 5.3.3 on OS X, PHP 5.2.11 Windows and PHP 5.3.?
Windows.

[shabbyrobe]

I realised I could just create an issue to attach the patch to:
https://code.google.com/p/addendum/issues/detail?id=29

Hope that's ok.

[Jan Suchal]

Hi,

you just earned yourself a commit bit to addendum repository. Feel free
to commit this.

I don't really have time to manage Addendum since I do most work in
ruby/rails now.

johno

> Profiling test script, only models& annotation classes required, no