TW2 Benchmarks

[percious]

Hey guys. I am hoping that by profiling tw2 in the early stages, we
can get a good idea of what kind of performance we can get out of it.
I know that for many of you performance is not as much of a concern as
capability, but keep in mind that there are more than a few folks for
who the performance of TW is a hurdle to adoption.

Setup:

To keep things as fair as possible, I am using Mako as a templating
language for both TW and TW2. This means I had to extend the existing
tw2.forms classes I used because they only support Genshi currently.
The reason I chose to use mako is that most of the time the bottleneck
in rendering widgets or otherwise is Genshi, and mako is much faster
and therefore will show the raw differences between the inner workings
of the widget library, not just showing equal performance because the
hit from genshi is so large.

The variable in my benchmarks is # of widgets rendered. The easiest
way to do this is to create a form with multiple input forms. I have
found that the widget creation time increases exponentially with the
number text widgets in a form. I was hoping that tw2 would not have
this same behavior.

First, I am using TG2.1 branch from my http://www.bitbucket.org/percious
account . This has the modifications required to allow tw2 to
function. I am using a MBP Core 2 Duo 2.4ghz with 4 megs of ram.
Python 2.5.2 is the interpreter. I also added mako_utils.py from the
TW repo and added it to the TW2 repo to do attribute rendering. I
have not made a fork of tw2 yet, but one is probably forthcoming.

Here is the code for creating the TW2 widgets:

#mako-ize the templates
class MyTW2Form(twf.TableForm):
template = "mako:tw2test.templates.form"

class MyTW2TextField(twf.widgets.TextField):
template = "mako:tw2test.templates.input_field"

def TW2FormBuilder(n):
class NewForm(MyTW2Form):
children = [MyTW2TextField(id='input%s'%i) for i in range(n)]
return NewForm
MyTW2Form = TW2FormBuilder(50)

And for TW:
from tw.forms import TableForm
from tw.forms import TextField
def TWFormBuilder(n):
return TableForm(children=[TextField('input%s'%i) for i in range
(n)])
MyForm = TWFormBuilder(50)

Here are the controller methods:

@expose()
def tw(self, **kw):
tmpl_context.widget = MyForm
return MyForm()

@expose()
def tw2(self, **kw):
tmpl_context.widget = MyTW2Form
return MyTW2Form.display()


And here are the results:
(ab -n 1000 -c 3 http://127.0.0.1:8080/tw[2])
(r/sec)
n = number of input widgets inside of the form
n tw tw2
5 75.7 69.8
10 49.6 43.6
50 14.8 11.3
100 7.9 5.8
500 1.6 1.0

Here's a graph for those of you who are visually stimulated:

http://imagebin.ca/view/878uytwW.html

Conclusions:

tw2 is young enough that there lacks a certain hairiness to the
codebase and can be saved from a performance perspective. A large
part of the problem is the per-request object creation. If we can
somehow make this go away or be faster, tw2 will certainly be a good
next step to an already excellent widgeting system.

I will look into some profiling in the future and see _how_ we can do
better.

cheers.
-chris

[Christoph Zwerschke]

percious schrieb:

> The variable in my benchmarks is # of widgets rendered. The easiest
> way to do this is to create a form with multiple input forms. I have
> found that the widget creation time increases exponentially with the
> number text widgets in a form. I was hoping that tw2 would not have
> this same behavior.

Good idea to make a few performance checks (though I currently don't
have any performance critical apps).

Are you sure the problem is the widget creation time? I cannot see this
when I'm using TW outside of TG2.

Even the rendering time seems to be linear:

from timeit import Timer

from tw.forms import TableForm, TextField

def TWForm(n):
TableForm(children=[
TextField('input%s'%i) for i in range (n)]).render()

if __name__ == '__main__':
setup = "from __main__ import TWForm"
for n in (5, 10, 50, 100, 500):
print n, Timer("TWForm(%d)" % n, setup).timeit(5000/n)

-- Christoph

[percious]

I don't think widget creation time is overall the problem with widgets
(rendering is) but these benchmarks show that creating the object on
the fly is probably slower than once per server, as TW does it.

cheers.
-chris

[Diez B. Roggisch]

>
> And here are the results:
> (ab -n 1000 -c 3 http://127.0.0.1:8080/tw[2])
> (r/sec)
> n = number of input widgets inside of the form
> n tw tw2
> 5 75.7 69.8
> 10 49.6 43.6
> 50 14.8 11.3
> 100 7.9 5.8
> 500 1.6 1.0
>
> Here's a graph for those of you who are visually stimulated:
>
> http://imagebin.ca/view/878uytwW.html
>
> Conclusions:
>
> tw2 is young enough that there lacks a certain hairiness to the
> codebase and can be saved from a performance perspective. A large
> part of the problem is the per-request object creation. If we can
> somehow make this go away or be faster, tw2 will certainly be a good
> next step to an already excellent widgeting system.
>
> I will look into some profiling in the future and see _how_ we can do
> better.

So TW1 is currently faster than TW2. Has anybody ever tried and profiled
the former? Have there been attempts at optimization? Maybe it's better
time spent, than to re-write and re-educate a great deal of code and coders.

Diez

[Christoph Zwerschke]

percious schrieb:

> And here are the results:
> (ab -n 1000 -c 3 http://127.0.0.1:8080/tw[2])
> (r/sec)
> n = number of input widgets inside of the form
> n tw tw2
> 5 75.7 69.8
> 10 49.6 43.6
> 50 14.8 11.3
> 100 7.9 5.8
> 500 1.6 1.0
>
> Here's a graph for those of you who are visually stimulated:
>
> http://imagebin.ca/view/878uytwW.html

Btw, that graph does not help very much because it shows the r/sec. If
you draw the sec/r instead, you see that they both are pretty much
linear. You can use Wolfram Alpha for that, enter:

linear fit {{5,1/75.7},{10,1/49.6},{50,1/14.8},{100,1/7.9},{500,1/1.6}}

-- Christoph

[percious]

On Jun 26, 3:21 am, Christoph Zwerschke <c...@online.de> wrote:
> percious schrieb:
>
> > And here are the results:

> > (ab -n 1000 -c 3http://127.0.0.1:8080/tw[2])

> > (r/sec)
> > n = number of input widgets inside of the form
> > n  tw      tw2
> > 5  75.7    69.8
> > 10 49.6    43.6
> > 50 14.8    11.3
> > 100        7.9     5.8
> > 500        1.6     1.0
>
> > Here's a graph for those of you who are visually stimulated:
>
> >http://imagebin.ca/view/878uytwW.html
>
> Btw, that graph does not help very much because it shows the r/sec. If
> you draw the sec/r instead, you see that they both are pretty much
> linear. You can use Wolfram Alpha for that, enter:
>
> linear fit {{5,1/75.7},{10,1/49.6},{50,1/14.8},{100,1/7.9},{500,1/1.6}}
>
> -- Christoph

Sounds right, i should have done a scatter vs. linear plotting.

Btw, I used TG2 as a testing platform because both libraries use some
thread-locking to manage thread safety issues. I was hoping that
TW2's limited use of thread-locking would be more efficient, but it
seems that object creation is a real hurdle here. I'd love to hear
from PAJ on this subject.

cheers.
-chris

[percious]

Diez,

I spent about 25 hours during the need for speed profiling TW1. I was
able to improve it's speed by about 5% with that effort. The result
was 0.9.6 err... 0.9.7. I did not have time to do a formal write up
on the subject, as much as I wanted to. I have a bunch of kcachegrind
screenshots saved up if anyone wants to see them. Basically, there is
no room for optimization in TW without re-writing the way the
internals work. If you are using TW with genshi, you get about 30%
the performance that you do with mako. This is also why 0.9.6 chooses
mako for all widgets if you set default_view to mako (before it was
defaulting to genshi).

cheers.
-chris

[Paul Johnston]

Hi,

It's great to see you using tw2. Performance has never been a great
issue for me, and tw2 has never had any attempts at optimization. I
know Alberto was keen on instance-per-request as a performance
improvement, but my interest in tw2 has always been in making things
cleaner and better organised. I reckon that with just a little
tweaking, tw2 could probably be made more efficient - a template cache
would be a good start. If you'd like to contribute in this area,
that'd be fab.

BTW, I'm happy to take mako related patches in tw2.core. Mako support
is something I've always wanted, just not gotten around to doing it.

Happy coding,

Paul

[percious]

Hey,

So, I implemented the template caching you described, in fact I
"borrowed" some code from TG2 (much of which I wrote) and put it into
tw2. I was getting sort of poor results, so I dug deeper to find
TableLayout was a genshi template, so genshi was still rendering!!!
Once I got that outta there, the results were awesome. I was able to
achieve a 2x speedup, or 81 r/sec, for 10 textfields per form. This
is a marked improvement.

So, either TW is still rendering with genshi somewhere, or this is an
excellent improvement over TW1. Here is what my root controller
widget looks like:

class MyTableLayout(twf.TableLayout):
template = "mako:tw2test.templates.table_layout"

#mako-ize the templates
class MyTW2Form(twf.TableForm):
template = "mako:tw2test.templates.form"

child = twc.Variable(default=MyTableLayout)

class MyTW2TextField(twf.widgets.TextField):
template = "mako:tw2test.templates.input_field"

n = 10

def TW2FormBuilder(n):
class NewForm(MyTW2Form):
children = [MyTW2TextField(id='input%s'%i) for i in range(n)]
return NewForm

MyTW2Form = TW2FormBuilder(n)

and my TableLayout template:
<table>
<tr py:for="i,c in enumerate(w.children_non_hidden)" class="${(i %
2 and 'even' or 'odd') + ((c.validator and c.validator.required) and '
required' or '') + (c.error_msg and ' error' or '')}" title="$
{w.hover_help and c.help_text or None}" py:attrs="c.container_attrs"
id="${c.compound_id}:container">
<th py:if="c.label">$c.label</th>
<td py:attrs="None if c.label else dict(colspan='2')">
${c.display()}
<py:if test="not w.hover_help">$c.help_text</py:if>
<span id="${c.compound_id}:error" py:content="c.error_msg"/
>
</td>
</tr>
<tr class="error"><td colspan="2">
<py:for each="c in w.children_hidden">${c.display()}</py:for>
<span id="${w.compound_id}:error" py:content="w.error_msg"/>
</td></tr>
</table>

And the results:

ab -c 3 -n 1000 http://127.0.0.1:8080/tw2
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software: PasteWSGIServer/0.5
Server Hostname: 127.0.0.1
Server Port: 8080

Document Path: /tw2
Document Length: 2782 bytes

Concurrency Level: 3
Time taken for tests: 12.264 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 2985000 bytes
HTML transferred: 2782000 bytes
Requests per second: 81.54 [#/sec] (mean)
Time per request: 36.793 [ms] (mean)
Time per request: 12.264 [ms] (mean, across all concurrent
requests)
Transfer rate: 237.69 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.6 0 6
Processing: 10 36 19.2 33 146
Waiting: 1 29 16.0 26 131
Total: 10 37 19.2 33 146

Percentage of the requests served within a certain time (ms)
50% 33
66% 40
75% 46
80% 51
90% 61
95% 71
98% 85
99% 102
100% 146 (longest request)

I'm going to crank on this some more and maybe convert sprox over to
it. Nice work Paul.

cheers.
-chris

[percious]

Forgot to mention. Here is my fork @ bitbucket.

http://bitbucket.org/percious/tw2core-percious/

enjoy.
-chris

> ab -c 3 -n 1000http://127.0.0.1:8080/tw2

> This is ApacheBench, Version 2.3 <$Revision: 655654 $>

> Copyright 1996 Adam Twiss, Zeus Technology Ltd,http://www.zeustech.net/
> Licensed to The Apache Software Foundation,http://www.apache.org/

[percious]

Just found another small fix that gets us to the 90 r/sec mark. Rock
on!

cheers.
-chris

[percious]

If you turn off checking to see if there is a new file everytime
(read: production), you get 95+ r/sec!

[percious]

I added configuration to tg2 to pick up the makotemplate reload var
and apply it to tw2 by default. :) See my branches.

cheers.
-chris

[Paul Johnston]

Hi,

Excellent news! I will look over your changes and bring them into my
branch. At this point, it's probably best that we have our own
branches, and keep them fairly close.

As for sprox on tw2, go for it. I am planning my own twist on this,
tw2.sqlalchemy, which takes a slightly different approach to sprox. No
idea on timescales though.

All the best,

Paul

[percious]

I'm fine with keeping my own branch, but it is getting a tad
cumbersome. I now have changes to tw.core tw.forms, and soon
tw.devtools.

I added a few preliminary tests for tw.forms. To my surprise, there
was no way to add a "class" attribute (css) to the widget. I modified
the code to do this, it was not too bad, please chack my branch.

My next step is to implement "preferred_rendering_engines". This will
take a list of engines from the config and render the widgets with
those templates if they are available. If the template is not
available for that engine, engine selection will fall back
gracefully. The only challenge I have here is supporting template
engines that all use .html as an extension. I am not sure as of yet
how to handle this problem.

Lastly, I will modify how tw2 handles templates of different types.
If you know your Mako template can safely render other types of
widgets, there is really no reason to escape the HTML. If you use "$
{other_widget.display() | n}" throughout your mako templates, you
should not have a problem rendering without escaping. (escaping is
fairly expensive).

cheers.
-chris

[percious]

I added the ability to allow tw to select your template based on a
middleware config option. middleware.config grew two new options:

`preferred_rendering_engines`
List of rendering engines in order of preference. (default:
['mako','genshi','kid','cheetah'])

`strict_engine_selection`
If set to true, TW2 will only select rendering engines from
within your preferred_rendering_engines,
otherwise, it will try the default list if it does not find a
template within your preferred list.
(default: True)

Thus ends my tirade on this codebase. I've not added an option for
non-escaping, but I will probably in the future. If you are using all-
mako, it doesn't do any escaping anyway. Now I'm going to write some
tests that TW0.9 should have had... I'm going to write a test object
for the form widget that will make certain that no matter what
template language you use, the outcome is the same. This should help
us find some flaws in the future.

enjoy.
chris

[percious]

I got this working. The test class takes some basic parameters about
the widget, and then renders it for each of the template languages it
finds a template for. This is working for TestInputField, check it
out.

cheers.
-chris