Chameleon.Genshi template engine ?

[Damien Accorsi]

Hello all,

I face dramatic performance issue with a page rendering a table with around 1000 lines and around 10 columns, certainly because of the content of the cells which is made of forms, images, etc. I'm using genshi. If I remove the genshi.XML() call (which I believe makes the XML validation), then the rendering duration becomes user-friendly.

For that reason, I'd like to try the chameleon genshi template engine which is base on a C library for the XML stuff. I didn't find documentation (there is a page on the TG2.1 online doc, unfortunately it's "empty").

The first step I did was to install chameleon.core 1.0.3 (the last release seems to be broken ?!), and to configure the app_cfg.py with the following lines:

base_config.renderers.append('genshi')
base_config.renderers.append('chameleon_genshi')
base_config.default_renderer = 'chameleon_genshi'

but it seems to work the same way (I mean... really slowly, while the chameleon.genshi doc says it should be around as fast as mako template engine).

Any idea, link? Any information so that I can add some stuff to the official documentation?

Damien

[Damien Accorsi]

Any idea, link? Any information so that I succeed in configuring chameleon.genshi (I would be able to add some stuff to the official documentation in that case)?

Damien

[Christoph Zwerschke]

Am 29.06.2011 00:51 schrieb Damien Accorsi:
> I face dramatic performance issue with a page rendering a table with
> around 1000 lines and around 10 columns, certainly because of the
> content of the cells which is made of forms, images, etc. I'm using
> genshi. If I remove the genshi.XML() call (which I believe makes the XML
> validation), then the rendering duration becomes user-friendly.

Where and why are you using genshi.XML()? You don't need it in normal TG
usage. If you want to include an already rendered HTML table in a
template, you use Markup().

-- Christoph

[Michael Pedersen]

As I read this, I find myself wondering what you're actually testing. The reason I ask is because you're discussing the rendering speed without telling us how you're measuring it.

You could be measuring how long it takes for the HTML to be sent to the browser, *or* how long it takes before the page is viewable by the user. If you're measuring the latter, then I have to point out that the last time I tried to render a table with over a thousand rows, it pretty much choked the browsers entirely. Pagination was required.

If you're measuring how long it takes for the page to be received (for instance, via wget), then yes, there's a whole different question.

For that, double check how your templates are being @expose'd. Make sure that it looks like this:

@expose('chameleon_genshi:dotted.template.name')

and not this:

@expose('genshi:dotted.template.name')

That's all I've got right now, anyway.

--
You received this message because you are subscribed to the Google Groups "TurboGears" group.
To post to this group, send email to turbo...@googlegroups.com.
To unsubscribe from this group, send email to turbogears+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/turbogears?hl=en.

--
Michael J. Pedersen
My IM IDs: Jabber/pede...@icelus.tzo.com, AIM/pedermj022171
          Yahoo/pedermj2002, MSN/pederm...@hotmail.com
My LinkedIn Profile: http://www.linkedin.com/in/michaeljpedersen
Twitter: pedersentg

[Diez B. Roggisch]

AFAIK the latter is a synonym for the former - isn't it?

Diez

[Damien Accorsi]

Ok, I didn't know. Actually I'm using this (and now Markup) with the idea of making my app faster by exporting some base templates functions into python code functions, from which the result is base html nodes. (these functions are basic but used a lot for each "button" I add into my web interface)

-- Christoph

[Damien Accorsi]

Le 29/06/2011 06:48, Michael Pedersen a écrit :

As I read this, I find myself wondering what you're actually testing. The reason I ask is because you're discussing the rendering speed without telling us how you're measuring it.

You could be measuring how long it takes for the HTML to be sent to the browser, *or* how long it takes before the page is viewable by the user. If you're measuring the latter, then I have to point out that the last time I tried to render a table with over a thousand rows, it pretty much choked the browsers entirely. Pagination was required.

You're right, as I started by testing the whole system (database access + server + rendering in the browser). Actually, I added some time logging for the database part, I make queries using wget, and I also added some time logging at the beginning/end of my controller function.

The huge part of time used when processing a query is "out of my python code", meaning it's in the framework or in my template code (and I know it's in my templates;) I tried removing some parts of my templates and it fixes the performance issue (but this is only test as I need what I developped).

If you're measuring how long it takes for the page to be received (for instance, via wget), then yes, there's a whole different question.

That's it.

For that, double check how your templates are being @expose'd. Make sure that it looks like this:

@expose('chameleon_genshi:dotted.template.name')

and not this:

@expose('genshi:dotted.template.name')

Actually, I believe we missunderstood.
* I'm currently using Genshi. I face performance issues explained in my first message. => any help welcome.
* Then, I tried to use Chameleon.genshi in order to see if there are performance increase. => I failed using it. => any help welcome.

For the Genshi part, I use a lot of <py:def function="myFunctionWithParameters(bla,bla2,bla3...)"></py:def> which are called in loops of loops (that's why, I believe, it's not efficient).

I'm thinking about migrating to Mako if :
* I can't improve genshi performance
* I can't succeed in setting up chameleon.genshi

(I'd like not to migrate to mako as I like the way genshi guarantee the XML strucutre)

Damien

[Alessandro Molina]

On Wed, Jun 29, 2011 at 9:57 AM, Damien Accorsi <damien....@free.fr> wrote:

For the Genshi part, I use a lot of <py:def function="myFunctionWithParameters(bla,bla2,bla3...)"></py:def> which are called in loops of loops (that's why, I believe, it's not efficient).

I'm thinking about migrating to Mako if :
* I can't improve genshi performance
* I can't succeed in setting up chameleon.genshi

(I'd like not to migrate to mako as I like the way genshi guarantee the XML strucutre)

To be sure if ChameleonGenshi is enabled you can check if your app_globals have a chameleon_genshi_loader variable. If they do not have it then you didn't correctly install or enable Chameleon.

as Michael said ChameleonGenshi can actually be enabled by using  @expose('chameleon_genshi:dotted.template.name') in the page where you want to use it. If it is slow or you still have problems enabling it you can try to use kajiki which is fast and genshi-like.

[Damien Accorsi]

Le 29/06/2011 10:59, Alessandro Molina a écrit :

On Wed, Jun 29, 2011 at 9:57 AM, Damien Accorsi <damien....@free.fr> wrote:

For the Genshi part, I use a lot of <py:def function="myFunctionWithParameters(bla,bla2,bla3...)"></py:def> which are called in loops of loops (that's why, I believe, it's not efficient).

I'm thinking about migrating to Mako if :
* I can't improve genshi performance
* I can't succeed in setting up chameleon.genshi

(I'd like not to migrate to mako as I like the way genshi guarantee the XML strucutre)

To be sure if ChameleonGenshi is enabled you can check if your app_globals have a chameleon_genshi_loader variable. If they do not have it then you didn't correctly install or enable Chameleon.

I believe this is not correctly enabled. I have the same problem with kajiki.

Here is the way I configure the template engine when not using genshi :

In app_cfg.py:

> base_config.renderers.append('chameleon_genshi')
> base_config.default_renderer = 'chameleon_genshi'

(idem for kajiki).

If I add the following code in the controller:
> from myproject.lib import app_globals
> print app_globals.chameleon_genshi_loader

then I get the following error:
AttributeError: 'module' object has no attribute 'chameleon_genshi_loader'

This is exactly the same for chameleon_genshi and for kajiki. Is there something else to configure ?! It looks like this is enough for mako to be used.

Damien

[Christoph Zwerschke]

Am 29.06.2011 09:45 schrieb Diez B. Roggisch:
>> Where and why are you using genshi.XML()? You don't need it in
>> normal TG usage. If you want to include an already rendered HTML
>> table in a template, you use Markup().
>
> AFAIK the latter is a synonym for the former - isn't it?

No, Markup() only marks a string as being for inclusion in the output
without being escaped, which is very quick (no parsing involved), while
XML() parses a string as XML into a markup stream which is much slower.

-- Christoph

[Christoph Zwerschke]

Am 29.06.2011 09:46 schrieb Damien Accorsi:
> Ok, I didn't know. Actually I'm using this (and now Markup) with the
> idea of making my app faster by exporting some base templates functions
> into python code functions, from which the result is base html nodes.
> (these functions are basic but used a lot for each "button" I add into
> my web interface)

You should consider using Genshi's py:def or ToscaWidgets for this.

-- Christoph

[Diez B. Roggisch]

Whao. I didn't know that. And should be able to optimize quite a bit of our code in the future ;)

Diez

[Damien Accorsi]

Actually I was using py:def and it was really slow.

[Damien Accorsi]

I finally "resetted" all my templates with mako.

My opinion about that choice :
+ I'm really happy with the speed;
+ I feel ok with the templating syntax (only developpers work on the application, even for the css/templating)
+ nothing special to setup in order to use in TG2.1
- I like the "full XML" style of Genshi, and the associated validation
- the syntax of mako is clearly for python dev.

Damien

[Christoph Zwerschke]

Sorry for rehashing and old thread. When trying to improve the render_genshi function with Alessandro today, we noticed that render_chameleon_genshi was actually the same as render_genshi. On setup the Chameleon.Genshi template loader was in fact imported, but then render_chameleon_genshi used the normal Genshi template loader anyway. That would explain your experience.

Has anybody actually ever used Chameleon.Genshi with TurboGears and knows the state of the project? Seems it has been abandoned and not kept up with the chameleon core. It is also not fully compatible with Genshi, particularly concerning match templates.

I have fixed the problem with the loader for now in the trunk, but if nobody uses it, we might as well remove it from TG2. Or support plain Chameleon (ZPT) instead.

-- Christoph

[Michael Pedersen]

On a related note: We do need to do something with Chameleon.genshi. I haven't attempted to fix it yet, but it is downlevel, and will not pass the tests when fully upgraded. IOW, chameleon.genshi is old, and must remain old as of now, or TG won't work properly with it.

Add in that chameleon.genshi is the only reason for LXML (a library I strongly dislike, as it makes getting a fully functional development environment on a 64 bit Linux system into a guessing game, as "STATIC_DEPS=true" is not enough normally), and booting it is something I would be okay with.

Just haven't sat down to clear it out one way or the other.

--
You received this message because you are subscribed to the Google Groups "TurboGears" group.

To view this discussion on the web visit https://groups.google.com/d/msg/turbogears/-/19cOqYtpiFgJ.

To post to this group, send email to turbo...@googlegroups.com.
To unsubscribe from this group, send email to turbogears+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/turbogears?hl=en.

[Damien Accorsi]

Le 09/08/2011 01:53, Christoph Zwerschke a écrit :

Sorry for rehashing and old thread. When trying to improve the render_genshi function with Alessandro today, we noticed that render_chameleon_genshi was actually the same as render_genshi. On setup the Chameleon.Genshi template loader was in fact imported, but then render_chameleon_genshi used the normal Genshi template loader anyway. That would explain your experience.

Has anybody actually ever used Chameleon.Genshi with TurboGears and knows the state of the project? Seems it has been abandoned and not kept up with the chameleon core. It is also not fully compatible with Genshi, particularly concerning match templates.

That may be the reason why it was not working. I don't know about the amount of work for Chameleon.Genshi to be working nice, but here are the reasons why I would have used it if performance was here :

• Valid XML documents "by design" (like Genshi)
• Inheritage between templates (like Mako)
• Genshi syntax is largely more readable than Mako one
• Genshi syntax may even be "web site designer" readable

In my case I finally decided to use Mako because of complex templates and speed requirements, but I'm not satisfied because document syntax may be invalid.

The last release of Chameleon is the 2.2 and was out on 2011-07-28.
The last release if Chameleon.Genshi is the 1.0.4 and was out on 2009-03-05. It depends on Chameleon 1.0b21 :-o

Chameleon.Genshi repository has moved from svn://svn.repoze.org/svn/chameleon.genshi to git://github.com/Pylons/pyramid_chameleon_genshi

The trunk of pyramid_chameleon_genshi seems still to be maintained, but it's "locked on Chameleon<2" : the latest release is the 0.6 (from april) and a release note says : Added dependency on Chameleon<1.999 (Genshi support is not present in Chameleon 2).

As Michael said : "Chameleon.Genshi is old"... it's even not supported anymore. pyramid_chameleon_genshi may replace it, but Chameleon.Genshi itself is dead.

Damien

[Michael Pedersen]

On Tue, Aug 9, 2011 at 4:22 AM, Damien Accorsi <damien....@free.fr> wrote:

As Michael said : "Chameleon.Genshi is old"... it's even not supported anymore. pyramid_chameleon_genshi may replace it, but Chameleon.Genshi itself is dead.

With this in mind, I'm thinking we should mark chameleon.genshi as deprecated, possibly for the 2.1.2 release, and drop it entirely by the 2.2 release.

At least, unless someone wants to review getting support for it in place for pyramid_chameleon_genshi. Even then, though, I'm not sure it's a viable option as a replacement, since there could be incompatibilities. Note: I'm not saying that there are, just that I'm not sure.

What does everybody else think?

--

[Christoph Zwerschke]

Am 09.08.2011 10:22 schrieb Damien Accorsi:
> That may be the reason why it was not working. I don't know about the
> amount of work for Chameleon.Genshi to be working nice, but here are the
> reasons why I would have used it if performance was here :

Same reasons why I'm using Genshi. Performance was never an issue for me
- maybe only because my projects are not performance critical anyway and
because database performance is much more important.

-- Christoph

[Christoph Zwerschke]

Am 10.08.2011 04:35 schrieb Michael Pedersen:
> With this in mind, I'm thinking we should mark chameleon.genshi as
> deprecated, possibly for the 2.1.2 release, and drop it entirely by the
> 2.2 release.
>
> At least, unless someone wants to review getting support for it in place
> for pyramid_chameleon_genshi. Even then, though, I'm not sure it's a
> viable option as a replacement, since there could be incompatibilities.
> Note: I'm not saying that there are, just that I'm not sure.

pyramid_chameleon_genshi is actually only a wrapper for Pyramid, it uses
the chameleon.genshi module available in Chameleon < 2, which is a bit
more recent than the old separate chameleon.genshi 1.0b4 that TG2 is
using. Or rather pretends to use, in reality it always used the Genshi
engine. So since chameleon.genshi support never worked anyway, we could
just as well drop it immediately. But I'll try to fix it so that it
works at least with Chameleon < 2 as pyramid_chameleon_genshi is doing.

-- Christoph

[Michael Pedersen]

I'm okay with either of these options. Does anybody else have a distinct preference?

If nobody speaks up, I'm going to vote for killing it for the 2.1.2 release.

--
You received this message because you are subscribed to the Google Groups "TurboGears" group.

To post to this group, send email to turbo...@googlegroups.com.

To unsubscribe from this group, send email to turbogears+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/turbogears?hl=en.

[Alessandro Molina]

On Wed, Aug 10, 2011 at 5:40 AM, Michael Pedersen <m.ped...@icelus.org> wrote:
> I'm okay with either of these options. Does anybody else have a distinct
> preference?
>
> If nobody speaks up, I'm going to vote for killing it for the 2.1.2 release.
>

I'm +1 for killing it.
Currently I don't see any reason to use Chameleon Genshi as Kajiki is
similar enough, fast enough and well maintained, so I don't see
reasons to keep it.

I would issue a warning and use genshi renderer even when Chameleon is
enabled for people that already used it (removing it in the successive
release), even thought as it was broken and none complained probably
there is no one out there using it.

[Florent Aide]

+1 for the kill.

I originally worked on this and I stumbled upon problem which was
never fixed in chameleon.genshi: it does not support filters. Which
means you cannot use i18n filters with chameleon.genshi

Apart from this as it was said earlier in this thread it adds the lxml
dependency to TG which is really a hard thing to setup on some
machines if you don't have the proper development tools installed.

Florent

> --
> You received this message because you are subscribed to the Google Groups "TurboGears" group.
> To post to this group, send email to turbo...@googlegroups.com.

> To unsubscribe from this group, send email to turbogears+...@googlegroups.com.

[Christoph Zwerschke]

Am 10.08.2011 09:29 schrieb Florent Aide:
> +1 for the kill.
>
> I originally worked on this and I stumbled upon problem which was
> never fixed in chameleon.genshi: it does not support filters. Which
> means you cannot use i18n filters with chameleon.genshi
>
> Apart from this as it was said earlier in this thread it adds the lxml
> dependency to TG which is really a hard thing to setup on some
> machines if you don't have the proper development tools installed.

Just talked with Malthe and he said that it's planned to move
chameleon.genshi back into Chameleon 2. So it seems it has a future and
therefore I think we should continue to support Chameleon.Genshi. I have
already fixed the development branch so that it uses the
Chameleon.Genshi in Chameleon 1.3rc1 for the time being instead of the
older chameleon.genshi 1.0b4.

lxml dependency is not such a big problem because we don't require
chameleon.genshi anywhere, that is left to the developer who wants to
replace Genshi with Chameleon.Genshi. The bigger problems are lack of
filters and incompatibilites concerning XPath and MatchTemplate support,
but for people who don't need this it may be acceptable.

So I'm for leaving it in now, just as an option that does no harm, but
not advertizing it until it is ported to Chameleon 2 and improved.

-- Christoph

[Michael Pedersen]

On Wed, Aug 10, 2011 at 12:49 PM, Christoph Zwerschke <ci...@online.de> wrote:

Just talked with Malthe and he said that it's planned to move chameleon.genshi back into Chameleon 2. So it seems it has a future and therefore I think we should continue to support Chameleon.Genshi. I have already fixed the development branch so that it uses the Chameleon.Genshi in Chameleon 1.3rc1 for the time being instead of the older chameleon.genshi 1.0b

My only upset there is that I wish I'd thought of doing that. Thank you for doing it. And thank you again for updating the code to use a newer version. It will be good to have that fully functional.
 

lxml dependency is not such a big problem because we don't require chameleon.genshi anywhere, that is left to the developer who wants to replace Genshi with Chameleon.Genshi. The bigger problems are lack of filters and incompatibilites concerning XPath and MatchTemplate support, but for people who don't need this it may be acceptable.

We don't require it until it comes time to run the test suite. And then in comes lxml. I've mostly sussed out the gotchas in the 64 bit Linux land with it, but it's still not perfect. I really do wish that chameleon.genshi did not have that dependency.
 

--

[Christoph Zwerschke]

Am 11.08.2011 05:37 schrieb Michael Pedersen:
> We don't require it until it comes time to run the test suite. And then
> in comes lxml. I've mostly sussed out the gotchas in the 64 bit Linux
> land with it, but it's still not perfect. I really do wish that
> chameleon.genshi did not have that dependency.

Had a look into this. The dependency was actually dropped in
chameleon.core 1.0b32 and it now is only needed for MatchTemplates in
Chameleon.Genshi which are somewhat broken or incompatible anyway.
So I have removed lxml from our test_requirements. The test for
cahmeleon.genshi page templates still works, but I skip the test with
master templates when lxml is not installed, because it did not really
work anyway. We can revisit and improve this when Chameleon.Genshi has
been ported to Chameleon 2.

-- Christoph

[Michael Pedersen]

Excellent. Thank you!

It'll be nice being able to tell people that they can just do simple straightforward commands. The whole process just got made much easier by this, and I appreciate it a lot.

-- Christoph

--
You received this message because you are subscribed to the Google Groups "TurboGears" group.
To post to this group, send email to turbo...@googlegroups.com.

To unsubscribe from this group, send email to turbogears+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/turbogears?hl=en.

[Michael Pedersen]

And I spoke too soon. Seems we have a small problem with that cset, Christoph:

http://jenkins.turbogears.org/view/TurboGears%20Hosted/job/tg-2.1-py2.4/99/console

Also affects Python 2.6, and prevents everything from being blue. I'm still working through the last doc changes, so can I ask you to fix this tomorrow?

On Thu, Aug 11, 2011 at 1:42 AM, Christoph Zwerschke <ci...@online.de> wrote:

-- Christoph

--
You received this message because you are subscribed to the Google Groups "TurboGears" group.
To post to this group, send email to turbo...@googlegroups.com.

To unsubscribe from this group, send email to turbogears+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/turbogears?hl=en.

[Christoph Zwerschke]

Am 12.08.2011 05:24 schrieb Michael Pedersen:
> And I spoke too soon. Seems we have a small problem with that cset,
> Christoph:
>
> http://jenkins.turbogears.org/view/TurboGears%20Hosted/job/tg-2.1-py2.4/99/console
>
> Also affects Python 2.6, and prevents everything from being blue. I'm
> still working through the last doc changes, so can I ask you to fix this
> tomorrow?

I think think this is because Chameleon 1.3rc1 is only available on
PyPI, but PyPI has been blocked for Jenkins. I'll add it to our TG
download files, then it should work again.

-- Christoph

[Christoph Zwerschke]

Am 12.08.2011 10:29 schrieb Christoph Zwerschke:
> I think think this is because Chameleon 1.3rc1 is only available on
> PyPI, but PyPI has been blocked for Jenkins. I'll add it to our TG
> download files, then it should work again.

Yes, that was it, and I was also able to remove the lxml installation
from the Jenkins script without breaking it again.

-- Christoph