plugin performance issues

[Shane Caraveo]

Hello,

I've been running through a lot of code looking at performance issues
lately, and am wondering if there is a general document for plugin
authors that highlights potential/common performance problems and how to
avoid them. I'm not sure if I'm rediscovering common knowledge or not :)

From what I've seen so far, IRequestHandler.match_request and
ITemplateStreamFilter.filter_stream are two area's where performance can
(and is) hurt by plugins doing more than they should. I'm not certain
the issue is those mechanisms specifically, but rather how they get
used. Just one example, not picking on anyone as I have several
examples, the timing and estimation plugin is a major user of stream
filters, and in some cases more than doubled the request times.

If there is not such a document, I think it would be useful to try and
get one together. I'd be happy to help with the area's I've figured
out, but someone with longer experience working with genshi and trac
internals should, at a minimum, validate the information.

Shane

[John Hampton]

Shane Caraveo wrote:
> I've been running through a lot of code looking at performance issues
> lately, and am wondering if there is a general document for plugin
> authors that highlights potential/common performance problems and how to
> avoid them. I'm not sure if I'm rediscovering common knowledge or not :)

I don't believe that there is a document that contains that info. Other
than code, there isn't a huge exhaustive document on plugin development

> From what I've seen so far, IRequestHandler.match_request and
> ITemplateStreamFilter.filter_stream are two area's where performance can
> (and is) hurt by plugins doing more than they should. I'm not certain
> the issue is those mechanisms specifically, but rather how they get
> used. Just one example, not picking on anyone as I have several
> examples, the timing and estimation plugin is a major user of stream
> filters, and in some cases more than doubled the request times.

So, not quite sure what one would be doing in match_request to make it
take a long time (permission checks, perhaps?), but StreamFilters will
definitely affect performance, as you have discovered. This is due to
genshi. I know that some work was done on making some speed
improvements in this areas (or maybe it was just template includes).
However, it's just an expensive process.

> If there is not such a document, I think it would be useful to try and
> get one together. I'd be happy to help with the area's I've figured
> out, but someone with longer experience working with genshi and trac
> internals should, at a minimum, validate the information.

I think Noah probably has the best grasp of the performance bottlenecks.
I agree that it would be worth while getting some "best practices"
together for plugin developers.

-John

[Shane Caraveo]

On 7/1/09 5:05 PM, John Hampton wrote:
> Shane Caraveo wrote:

>> From what I've seen so far, IRequestHandler.match_request and
>> ITemplateStreamFilter.filter_stream are two area's where performance can
>> (and is) hurt by plugins doing more than they should. I'm not certain
>> the issue is those mechanisms specifically, but rather how they get
>> used. Just one example, not picking on anyone as I have several
>> examples, the timing and estimation plugin is a major user of stream
>> filters, and in some cases more than doubled the request times.
>
> So, not quite sure what one would be doing in match_request to make it
> take a long time (permission checks, perhaps?),

I've seen permission checks, uncompiled regex matching, and other more
minor things. It seems to me anything other than a
req.path_info.startswith(/mypath) is too much. I wonder if there would
be a better mechanism for this.

> but StreamFilters will
> definitely affect performance, as you have discovered. This is due to
> genshi. I know that some work was done on making some speed
> improvements in this areas (or maybe it was just template includes).
> However, it's just an expensive process.

There's definitely some improvement in genshi trunk, but it's still
slow. I've profiled some basic requests, genshi 5 takes around 25% of
the request time. With trunk it's down to probably 20%. It seems to be
primarily the xpath matching, but it's hard to tell, combining
kcachgrind with python profiling is sometimes odd. :) On my system, it
seems a large part of that is due to path matching from stream filters.

Shane

[Erik Bray]

On Wed, Jul 1, 2009 at 8:05 PM, John Hampton<paco...@pacopablo.com> wrote:
>
> Shane Caraveo wrote:
>> I've been running through a lot of code looking at performance issues
>> lately, and am wondering if there is a general document for plugin
>> authors that highlights potential/common performance problems and how to
>> avoid them.  I'm not sure if I'm rediscovering common knowledge or not :)
>
> I don't believe that there is a document that contains that info.  Other
> than code, there isn't a huge exhaustive document on plugin development
>
>>  From what I've seen so far, IRequestHandler.match_request and
>> ITemplateStreamFilter.filter_stream are two area's where performance can
>> (and is) hurt by plugins doing more than they should.  I'm not certain
>> the issue is those mechanisms specifically, but rather how they get
>> used.  Just one example, not picking on anyone as I have several
>> examples, the timing and estimation plugin is a major user of stream
>> filters, and in some cases more than doubled the request times.
>
> So, not quite sure what one would be doing in match_request to make it
> take a long time (permission checks, perhaps?), but StreamFilters will
> definitely affect performance, as you have discovered.  This is due to
> genshi.  I know that some work was done on making some speed
> improvements in this areas (or maybe it was just template includes).
> However, it's just an expensive process.

Though if handled with care, template stream filters shouldn't kill
you *that* much. Our batch ticket update plugin, for example, does
quite a number of transformations on report and query pages. But even
on very large reports with pagination turned off it usually only takes
an additional fraction of a second, at least on my development
workstation. Yes it slows things down, but I guess cases such as
large reports are already slow enough anyways that the extra penalty
isn't noticed ^^;