benchmarks

[Massimo Di Pierro]

I am writing and article for the python magazin, almost done, and it
would be nice to have benchmarks about the ORM and about the
templating system.

Do I have a volunteer to run the ORM benchmarks? You need to have
sqlalchemy, storm, web2py and postgresql installed in a machine that
is not running other computing intensive processes.

Do I have a volunteer for the templating benchmarks? I would not know
how to go about this. I would like to test Make and Django templates
against web2py bytecode compiled templates.

Massimo

[Michael Wills]

Gotta see if I have a box available that can do that. I'd like to help in some way considering all the help I've been getting. :-) Checking...

[Michael Wills]

We should use web2py from source, correct?

[Michael Wills]

OK send on the details so I can estimate how much time needs to be set aside for it. Would you like system information like those here?

I am using the stable release of the precompiled windows binaries for psycopg2 provided here:

http://www.stickpeople.com/projects/python/win-psycopg/#Version2

specifically psycopg2-2.0.6.win32-py2.5-pg8.2.4-release.exe.

Testing templates has got to be challenging... if you would like to test it separately from the database part of it, perhaps you could use an arbitrary python set of data and have the template parse it. So each system would have to take the data and do different things like loops, conditional flow control, etc.

This page on Breve links to a big test and a basic test developed by Genshi.

[Michael Wills]

And some of the output from some of those tests from Genshi.

[Michael Wills]

These testing systems might be useful as well.

[Massimo Di Pierro]

yes

[Massimo Di Pierro]

I like all your ideas. I would start with ORM, then Templates. Let me know when you have a setup ready. I can send you benchmark code for the sqlalchemy, storm and  web2y orm. for templating I am not sure how to plugin into the testing framework you showed me.

[Massimo Di Pierro]

[Michael Wills]

will check in a bit

On Fri, Mar 14, 2008 at 12:02 PM, Massimo Di Pierro <mdip...@cs.depaul.edu> wrote:

Here is the benchmark code for the ORM. You may need to edit the postgres connection strings.

In benchmark_web2py.py if you do not pass a password it uses sqlite.

Massimo

[Michael Wills]

My apologies! I won't be able to get to the benchmark test so soon. My earliest estimated time is early Wednesday evening Chicago time. :-(

[Massimo Di Pierro]

I can live with it.

Massimo

[Michael Wills]

Sorry! I have been in transit for a while. There are some errors in the benchmark code that I will take a closer look at soon. I am trying to make sure that all the strings are unicode, i.e. the SQLAlchemy version. It's not throwing the errors as before so I may have gotten those now. In between runs for the web2py version I have to make sure I clear the .table files. The storm one seems OK now as well. I will try to clean up the code and post it here soon as well.

So with that, these are the *INITIAL* tests. Again I will try to clean up and post my current code and make sure I understand how this code needs to be run.

The test system is:
WinXP SP2
RAM: 512MB (-8MB shared video memory so 504MB)
CPU: P4 3GHz (shows as 2.99GHz in computer properties)
HD: Seagate ST340014A

Web2py
Ran step_1_create_tables in: 0:00:00.156000
Ran step_1a_populate in: 0:00:00.031000
Ran step_2_insert in: 0:00:00.078000
Ran step_3_Properties in: 0:00:00.703000
Ran step_4_Expressions in: 0:00:01.704000
Ran step_5_Aggregates in: 0:00:00.765000
Ran step_6_Editing in: 0:00:00.485000

Storm
Ran step_1_create_tables in: 0:00:00.235000
Ran step_1a_populate in: 0:00:00.047000
Ran step_2_insert in: 0:00:00.015000
Ran step_3_Properties in: 0:00:00.469000
Ran step_4_Expressions in: 0:00:26.547000
Ran step_5_Aggregates in: 0:00:00.640000
Ran step_6_Editing in: 0:00:01.204000

SQLAlchemy
Ran step_1_create_tables in: 0:00:00.234000
Ran step_1a_populate in: 0:00:00.047000
Ran step_2_insert in: 0:00:00.250000
Ran step_3_Properties in: 0:00:02.312000
C:\Python25\lib\site-packages\sqlalchemy-0.4.4-py2.5.egg\sqlalchemy\sql\expression.py:1257: SADeprecationWarning: passing in_ arguments as varargs is deprecated, in_ takes a single argument that is a sequence or a selectable
  return self._in_impl(operators.in_op, operators.notin_op, *other)
Ran step_4_Expressions in: 0:00:10.406000
Ran step_5_Aggregates in: 0:00:01.235000
Ran step_6_Editing in: 0:00:01.094000
Ran step_7_Multiview in: 0:00:01.906000

[Michael Wills]

Some version information:

Postgresql 8.3
Python 2.5.1
SQLAlchemy 0.4.4
Storm 0.12
Web2Py 1.26

[Massimo Di Pierro]

Thank you Michael,

this is wonderful present. Just one thing. I am not sure I trust benchmarks with times so low. Could you increase the ITERTIONS parameter to 10000 from 100? It may take 1h to run storms and sqlalchemy tests but it is worth it. Could you also try both the version you have and the latest version in trunk (only need to replace the sql.py file for the purpose of testing)? The latest should be even faster but I am not completely sure, I can still revert to the old one.

Happy Easter!

Massimo

[Michael Wills]

Yeah I was wondering about those numbers, too. :-P 10000 would definitely take a while to run. SQLAlchemy's run for step 4 was 26 seconds for the current 100 iterations!!! :-)

I'll let you know when I get to those tests.

And you're welcome!

[Massimo Di Pierro]

It feels great to know web2py is between 2 and 20 times faster than the competition!

Look forward to the new benchmarks.

Massimo

[Massimo Di Pierro]

Any news? It would be great to post this on the new web site.

Massimo

On Mar 22, 2008, at 10:12 PM, Michael Wills wrote:

[Michael Wills]

Yeah it would but I'll need a good chunk of time for the longer benchmarks. :-( I am planning to start those on early Friday evening Chicago time when that box is freed up. I have to go check out the new site, too! :-)

[Zoom.Quiet]

On Fri, Mar 28, 2008 at 6:21 AM, Michael Wills <mcw...@gmail.com> wrote:
> Yeah it would but I'll need a good chunk of time for the longer benchmarks.
> :-( I am planning to start those on early Friday evening Chicago time when
> that box is freed up. I have to go check out the new site, too! :-)
>

Y! great try!
in fact my team,just monent usage Strom replace web2py's ORM for
better ORM feeling ;)
not problem, but not test like this!
suggest test SQLObject too, as comparison , for help people choise
best ORM in web2py;

btw:
us how to usage Strom in web2py ,will sharing by lvyi

--
'''Time is unimportant, only life important!
过程改进乃是开始催生可促生靠谱的人的组织!
PI keeps evolving organizations which promoting people be good!
'''http://zoomquiet.org
博 @ http://blog.zoomquiet.org/pyblosxom/
维 @ http://wiki.woodpecker.org.cn/moin/ZoomQuiet
豆 @ http://www.douban.com/group/zoomquiet/
看 @ http://zoomq.haokanbu.com/
作 4 http://trac-hacks.org/wiki/TracChineseTranslation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Pls. usage OOo to replace M$ Office. http://zh.openoffice.org
Pls. usage 7-zip to replace WinRAR/WinZip. http://7-zip.org
You can get the truely Freedom 4 software.

[Massimo Di Pierro]

Mind that storm does not work with SQLFORM and SQLTABLE and appadmin.

Massimo

[Michael Wills]

Yeah there are some nice things about Storm's ORM that I like as well. As Zoom mentioned, it has a different feel to it. However, because of web2py's flexibility, I could imagine there being a contrib package that helps bring optional functionality to web2py's ORM, perhaps even a limited emulation layer. Storm actually has a SQLObject emulation layer though I haven't tried that.

One thing though, it is really easy to use Storm with web2py, though you do lose SQLForm, SQLTable and appadmin. I was just surprised how easy it is to start working with a different ORM as needed. Formencode is the closest thing I have seen so far to help bring that functionality into web2py in an ORM agnostic way. I haven't tested that thoroughly though.

I'll let you know when I get through those benchmarks though.

Michael

[Zoom.Quiet]

2008/3/28 Michael Wills <mcw...@gmail.com>:

> Yeah there are some nice things about Storm's ORM that I like as well. As
> Zoom mentioned, it has a different feel to it. However, because of web2py's
> flexibility, I could imagine there being a contrib package that helps bring
> optional functionality to web2py's ORM, perhaps even a limited emulation
> layer. Storm actually has a SQLObject emulation layer though I haven't tried
> that.
>

COOL! like TurboGears , big platform 's idea is best than all in one;
so if web2py can opening, make easy mashup others great package for
help happy usage web2py..
that is good thing ;)

> One thing though, it is really easy to use Storm with web2py, though you do
> lose SQLForm, SQLTable and appadmin. I was just surprised how easy it is to
> start working with a different ORM as needed. Formencode is the closest
> thing I have seen so far to help bring that functionality into web2py in an
> ORM agnostic way. I haven't tested that thoroughly though.
>

Mnnn can we through JSON as middle form define data,
with one FormAgent model to free all kinds of ORM ?

[Michael Wills]

I understand Massimo's desire to create a new ORM. I see is taking good ideas from different places but with a focus on simplicity and speed. So that is good.

As is, I think web2py is already very good at connecting with other projects so it's not a big problem. I have limited experience though. :-) You would lose some of the automatic ORM things but if you already have legacy code, like your team (Zoom) already has Storm code, then it's good to be able to continue to use that.

Hmm... JSON as a data format. Hmm... I am not sure if it is needed. I don't have enough experience in this area to say either. Formencode just uses standard Python data types. Of course if you are working with forms then your data is fairly restricted already. But it is something to think about. There are a lot of long-term things that could be done with web2py. :-)

2008/3/27 Zoom. Quiet <zoom....@gmail.com>:

[Massimo Di Pierro]

What are the features of storm that you like over web2py?

Massimo

[Michael Wills]

This is a personal preference so it's not a reflection against web2py or anything. I am not a major developer (perhaps one day) so this is really subjective stuff for me.

I like being able to have tables that are classes. The Storm tutorial shows good examples of it. That way, say for the cookbook, I can refer to

Recipe
Recipe.id
and also for foreign keys

Recipe.category.name

I really like that for foreign key relations. I can do

In [35]: [r.category.name for r in recs if r.category is not None]
Out[35]: [u'test 1', u'test 1']
In [36]: recs=store.find(Recipe).order_by(Recipe.id)
In [37]: [r.category.name for r in recs if r.category is not None]
Out[37]: [u'test 1', u'test 1']

Finding things then become very clean visually for me. I find having a lot of 'db' makes it more cluttered for me yet I understand why it's there and why it is that way. That's part of why I looked to use Classes with the web2py ORM.

Also I have not gone through the helper functions that are available for web2py's ORM. Things like first, last, one, get(by id), min, avg, first, etc. Those seem to be convenience functions and so to me a lot of this can be added later as the whole package develops. To be fair, I haven't gone through all of web2py's available functions though!

And, with all these things I am still learning! :-D

2008/3/27 Massimo Di Pierro <mdip...@cs.depaul.edu>:

[Massimo Di Pierro]

notice that in web2py you can do:

    db=SQLDB('sqlite://db.db')

    category=db.define_table('cetegory',SQLField('name'))

    recipe=db.define_table('recipe',SQLField('name'),SQLField('category',category))

    rows=db(recipe.category==category.id).select(orderby=recipe.id) # trasparent JOIN

    for r in rows: print r.category.name, r.recipe.name

and no db(s) anymore.

Massimo

[Michael Wills]

Doh! I never checked if define_table returned a value! That's a nice idiom there. I like that... hmm...

2008/3/27 Massimo Di Pierro <mdip...@cs.depaul.edu>:

notice that in web2py you can do:

[Massimo Di Pierro]

it is the same as 

     category=db.category

you can have shortcuts for fields too

     a=db.recipe.name

     db().select(a,orderby=a)

web2py does not use classes in oder to allow this.

Massimo

[Massimo Di Pierro]

You can do so for queries and sets too

     query=category.id==recipe.category

     recipes_and_categories=db(query)

     for row in recipes_and_categories.select():

         print row.category.name, row.recipe.name

     for recipe in db(query&category.name=="dessert").select(recipe.name):

         print recipe.name

     recipes_and_categories.delete()

Massimo

[Michael Wills]

Just tried that out in the shell (did I say I like that shell?!) and... wow... yeah I like this.

[voltron]

Hmm, I do not think that there are many advantages over Web2pys ORM
apart from the support of some PostgreSQL fields . I have used Storm
and SQLAlchemy. Web2pys syntax is clean and simple, and I can write
code against it without constantly asking questions or reading the
manual.

After a while I just tossed my skepticism away, because as an ORM
newbie, the advantages of coding fast and things like SQLFORM, and
Database admin are just to good, just reading the FormEncode manual
made me want to tear my hair out sometimes, and has anybody tried
using SQLALchemy 0.3 and Pylons? Not something for newbies I tell you,
some of my hair went gray because of it. Dont get me wrong, I really
like Storm, but Web2py rocks harder IMHO :-)

On Mar 28, 3:55 am, Massimo Di Pierro <mdipie...@cs.depaul.edu> wrote:
> What are the features of storm that you like over web2py?
>
> Massimo
>
> On Mar 27, 2008, at 8:52 PM, Michael Wills wrote:
>
> > Yeah there are some nice things about Storm's ORM that I like as
> > well. As Zoom mentioned, it has a different feel to it. However,
> > because of web2py's flexibility, I could imagine there being a
> > contrib package that helps bring optional functionality to web2py's
> > ORM, perhaps even a limited emulation layer. Storm actually has a
> > SQLObject emulation layer though I haven't tried that.
>
> > One thing though, it is really easy to use Storm with web2py,
> > though you do lose SQLForm, SQLTable and appadmin. I was just
> > surprised how easy it is to start working with a different ORM as
> > needed. Formencode is the closest thing I have seen so far to help
> > bring that functionality into web2py in an ORM agnostic way. I
> > haven't tested that thoroughly though.
>
> > I'll let you know when I get through those benchmarks though.
>
> > Michael
>
> > On Thu, Mar 27, 2008 at 6:16 PM, Massimo Di Pierro
> > <mdipie...@cs.depaul.edu> wrote:
>
> > Mind that storm does not work with SQLFORM and SQLTABLE and appadmin.
>
> > Massimo
>
> > On Mar 27, 2008, at 8:00 PM, Zoom.Quiet wrote:
>

> > > On Fri, Mar 28, 2008 at 6:21 AM, Michael Wills <mcwi...@gmail.com>

> > > 作 4http://trac-hacks.org/wiki/TracChineseTranslation
> > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > Pls. usage OOo to replace M$ Office.http://zh.openoffice.org
> > > Pls. usage 7-zip to replace WinRAR/WinZip.http://7-zip.org

[limodou]

2008/3/28 voltron <nhy...@googlemail.com>:

>
> Hmm, I do not think that there are many advantages over Web2pys ORM
> apart from the support of some PostgreSQL fields . I have used Storm
> and SQLAlchemy. Web2pys syntax is clean and simple, and I can write
> code against it without constantly asking questions or reading the
> manual.
>
> After a while I just tossed my skepticism away, because as an ORM
> newbie, the advantages of coding fast and things like SQLFORM, and
> Database admin are just to good, just reading the FormEncode manual
> made me want to tear my hair out sometimes, and has anybody tried
> using SQLALchemy 0.3 and Pylons? Not something for newbies I tell you,
> some of my hair went gray because of it. Dont get me wrong, I really
> like Storm, but Web2py rocks harder IMHO :-)
>

Very interesting!

--
I like python!
UliPad <<The Python Editor>>: http://code.google.com/p/ulipad/
meide <<wxPython UI module>>: http://code.google.com/p/meide/
My Blog: http://www.donews.net/limodou

[Massimo Di Pierro]

Thank you Zoom and Voltron!

[Massimo Di Pierro]

web2py also support seamlessly arbitrary database connections

db1=SQLDB(...)
db2=SQLDB(...)
db3=SQLDB(...)

most of the other frameworks cannot do this because they have a
configuration file that can handle a single connection.

Massimo

[voltron]

Anothere great advantage! Seamless arbitrary database connections!
Thanks for bring that point up too Massimo :-) Here is an article
about Python frameworks in general, the points mentioned in the
article apply in this thread too.

http://adam.gomaa.us/blog/frameworks-exist-for-conceptual-integrity/