how to avoid performance bottlenecks in phpcr-odm and how to test it
Christian Stocker
From time to time, we see a huge spike in jackrabbit request in our
phpcr-odm based project. To make it easier for checking how many
requests it does, I wrote a little script. You can get it here:
https://github.com/jackalope/jackalope-jackrabbit-meter
Put that for example into bin/jackalope-jackrabbit-meter into your
project copy config.inc.php-dest to config.inc.php and then run it it
"php meter.php" (see also the README.md for some more info)
and if everything works fine, you get something like this:
****
TEST for http://nzz.lo/app.php/
HTTP Code : 200
Body size : 27631 bytes
Request time : 394 ms
Total JR Requests : 22
GET : 10
POST : 3
REPORT : 7
SEARCH : 2
****
TEST for
http://nzz.lo/app.php/anotherpage
HTTP Code : 200
Body size : 26521 bytes
Request time : 554 ms
Total JR Requests : 25
GET : 12
POST : 3
REPORT : 7
SEARCH : 3
****
# Why can this happen?
The phpcr-odm sometimes does magic things. If you have for example a
$departement object, which has referenced articles in it like this
***
/**
* articles
* @var array
* @PHPCR\ReferenceMany(targetDocument="Article")
* @Serializer\XmlList(inline = false, entry = "article")
*/
***
These are WEAKREFERENCES in JCR terms in the Department node and is
basically a list of UUIDs. To get the articles, jackalope has to do a
REPORT request to jackrabbit to get the absolute path of that UUID and
then does a multi-get POST request to fetch all the articles. Makes
(n+1) requests for getting all articles of a department. There's a patch
in the making to make that only 1 request, no matter how many articles
there are, but that's not prime time ready yet (and jackrabbit itself
needs that patch, https://issues.apache.org/jira/browse/JCR-3245 if
you're interested)
until 2 weeks ago, the ODM did always do that, even if we didn't access
the articles property. We fixed that with lazy loading in the
ReferenceMany Class, see here
https://github.com/doctrine/phpcr-odm/blob/master/lib/Doctrine/ODM/PHPCR/ReferenceManyCollection.php
This brought down the amount of requests significantly (we don't need
the articles for example if we want a list of all departments)
Now what happened recently was, that we wanted to know, if there are any
articles at all in a department. This was basically done like this:
if (count($this->getArticles())) {....}
So the ODM again did fetch all the articles from jackrabbit, just to
check if there are any without actually needing them.
What we then did is adding an isEmpty() and count() method to the
ReferenceMany class, because we can decide that without having to fetch
the actual articles.
So now we're back to a decent amount of HTTP Requests
Maybe this is interesting for people outside of this actual project as well.
chregu
--
Liip AG // Feldstrasse 133 // CH-8004 Zurich
Tel +41 43 500 39 81 // Mobile +41 76 561 88 60
www.liip.ch // blog.liip.ch // GnuPG 0x0748D5FE
Henri Bergius
On Wed, Mar 7, 2012 at 11:27 AM, Christian Stocker
<christia...@liip.ch> wrote:
> From time to time, we see a huge spike in jackrabbit request in our
> phpcr-odm based project. To make it easier for checking how many
> requests it does, I wrote a little script. You can get it here:
> https://github.com/jackalope/jackalope-jackrabbit-meter
Sounds useful.
> The phpcr-odm sometimes does magic things.
Indeed. We were just discussing on #midgard how to ensure decent
performance with the different RDBMSs that the Midgard2 PHPCR provider
can talk to.
With SQLite for example, the performance difference for writing 100
nodes can be from 10 seconds to 0.25, depending how transactions are
used. The more we can wrap inside a single transaction in
$session->save(), the better the performance.
This is something to consider when writing PHPCR apps (and tests), in
that different providers may have different performance
characteristics based on how they're used.
/Henri
--
Henri Bergius
Motorcycle Adventures and Free Software
http://bergie.iki.fi/
Jabber: henri....@gmail.com
Microblogs: @bergie
Lukas Kahwe Smith
On Mar 7, 2012, at 05:27 , Christian Stocker wrote:
> Hi all
>
> From time to time, we see a huge spike in jackrabbit request in our
> phpcr-odm based project. To make it easier for checking how many
> requests it does, I wrote a little script. You can get it here:
> https://github.com/jackalope/jackalope-jackrabbit-meter
>
> Put that for example into bin/jackalope-jackrabbit-meter into your
> project copy config.inc.php-dest to config.inc.php and then run it it
> "php meter.php" (see also the README.md for some more info)
>
> and if everything works fine, you get something like this:
we also need to finally implement logging in Jackalope. but to make it truly portable, i guess it should rather be done inside PHPCR as part of Observation?
then we could write something like http://symfony.com/doc/2.0/cookbook/profiler/data_collector.html and add it to PHPCRBundle.
regards,
Lukas Kahwe Smith
m...@pooteeweet.org
David Buchmann
Hash: SHA1
hi,
dan and i are looking into observation right now. basically it will
allow you to have event listeners on NODE_ADDED, NODE_MOVED,
NODE_REMOVED, PERSIST, PROPERTY_ADDED, PROPERTY_CHANGED, ROPERTY_REMOVED
- - all generated on persist.
if this is enough for logging, it can do. but as i understand, its
mainly aimed at parallel jcr sessions going on, not about logging what
your own session is doing.
the data collector and what symfony can do sounds like more. there is no
log level or such. so i think we still need to implement real logging.
cheers,david
- --
Liip AG // Agile Web Development // T +41 26 422 25 11
CH-1700 Fribourg // PGP 0xA581808B // www.liip.ch
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk9Xk0YACgkQqBnXnqWBgIt6pgCfVpNfWKVCT7erYVzBhfdvLv7K
ggUAn2dd/3P2DorfwpUx5m57PHsMrXvd
=B0ig
-----END PGP SIGNATURE-----
Lukas Kahwe Smith
On Mar 7, 2012, at 11:56 , David Buchmann wrote:
> hi,
>
> dan and i are looking into observation right now. basically it will
> allow you to have event listeners on NODE_ADDED, NODE_MOVED,
> NODE_REMOVED, PERSIST, PROPERTY_ADDED, PROPERTY_CHANGED, ROPERTY_REMOVED
> - - all generated on persist.
>
> if this is enough for logging, it can do. but as i understand, its
> mainly aimed at parallel jcr sessions going on, not about logging what
> your own session is doing.
>
> the data collector and what symfony can do sounds like more. there is no
> log level or such. so i think we still need to implement real logging.
i guess it probably isnt .. since for this case its also about reads not just writes, which i assume are not part of PHPCR. maybe we can however add a few logging interfaces to phpcr-utils, which phpcr implementation can choose to use.
David Buchmann
Hash: SHA1
> i guess it probably isnt .. since for this case its also about reads not just writes, which i assume are not part of PHPCR.
genau.
> maybe we can however add a few logging interfaces to phpcr-utils,
> which phpcr implementation can choose to use.
+1
- --
Liip AG // Agile Web Development // T +41 26 422 25 11
CH-1700 Fribourg // PGP 0xA581808B // www.liip.ch
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk9XlhQACgkQqBnXnqWBgIvKegCfaWKbPZmssLVVNloK1vhHJaw1
me8An02eyEUVQDS6/WtQvzanxmpCZ7Hv
=z2az
-----END PGP SIGNATURE-----