performance costs of using internal requests
Alistair Miles
This is just a short note for the record to capture a few things I've just
learned about the performance trade-offs of using AtomBeat's ability to make
what I've been calling "internal protocol requests". This is where a custom
XQuery script or an AtomBeat plugin module makes a call to a function in
the atom-protocol.xqm module, e.g., atom-protocol:do-get-member(), or the
security-protocol.xqm module, e.g., security-protocol:do-get(). The effect of
this is to execute pretty much exactly what would be executed if an external
HTTP request (e.g., GET) was made. I.e., you can invoke AtomBeat's protocol
operations and make "virtual" http requests from within other XQueries,
but obviously without the networking overhead of actually having to make an
HTTP request.
One use of this feature is to implement side-effects to normal atom
protocol operations. For example, you could write a plugin that added
side-effects to creation of a member of collection /foo, which e.g. create
one or more members of other collections /bar and /baz. To make internal
protocol requests from within a plugin module, you cannot use the
atom-protocol.xqm module directly, because this causes an XQuery import
cycle. You can, however, use the plugin-util.xqm module, and call things
like plugin-util:atom-protocol-do-get-member().
The plugin-util.xqm module has functions that are essentially proxies
for functions in the atom-protocol.xqm and security-protocol.xqm modules'
functions, but which can be invoked without causing an import cycle. Making
this possible has required a bit of trickery, involving evaluation of
dynamically constructed XQueries using the eXist util:eval() function. One
form of this function takes a $cache-flag parameter, and I wasn't sure
what the consequences were of setting this to true or false. The other day
I happened on a bug in AtomBeat that arose when this flag was set to true,
but which went away when the flag was set to false, so I set it to false in
the AtomBeat trunk. However, I was afraid that this might impact performance,
by e.g. forcing the dynamically constructed query to be recompiled every time
it was invoked. However, I just did a bit of benchmarking with the flag set
both ways, and it didn't make any noticeable difference, so I'm leaving it
as false in the AtomBeat trunk for now. I'm still not sure exactly how eXist
deals with the queries passed to util:eval(), but it doesn't look like there
is any noticeable compilation step or overhead.
However, note that calling atom-protocol:do-get-member() (or
plugin-util:atom-protocol-do-get-member()) means that the internal request
will be passed through all before and after plugin functions, in addition
to the main protocol operation. Sometimes this is exactly what you want,
but this is computationally more expensive than making calls to lower-level
modules like atomdb.xqm and atom-security.xqm.
For example, I had written a plugin which added a total of 7 side-effect
operations to a create member operation. I.e., when a member of the collection
/project was created, 4 other collections were created and one of these
additional collections was populated with 3 new members. When I implemented
these side-effects as internal requests (using plugin-util:do-post-...) the
create member request took roughly 4 times longer than when I implemented the
side-effects using atomdb directly. In this case I'm sticking with using the
internal requests, because it makes managing security much easier, but it's
just something to bear in mind.
Finally, note that in AtomBeat 0.2-alpha-6 the link-expansion plugin was
modified to make use of internal requests instead of making direct calls
to atomdb.xqm and atom-security.xqm. This change may have a noticeable
performance cost for projects making heavy use of the link expansion module,
depending on the hardware you're running on of course (the limiting factor
is CPU). If this is a problem, let me know and I'll write an alternate
link-expansion-lite plugin module that works in the more limited but faster
way by calling atomdb and atom-security directly.
Funny how working on a laptop focuses your mind on performance issues :)
Cheers,
Alistair
--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health <http://cggh.org>
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Web: http://purl.org/net/aliman
Email: alim...@gmail.com
Tel: +44 (0)1865 287669