some thoughts on tg + javascript
Daniel Fetchinson
With the release of tg2 in the near future tg has evolved into
a robust, reliable, well-designed and easy to use web development
framework. It greatly reduces the work of developers on the server
side of web applications by automating common tasks as well as
providing sane defaults.
However web applications these days are not server side applications
but rather client/server applications. Traditionally web developers
did not need to worry about client side issues because the clients,
web browsers, were assumed to do their job on their own. These days
client side development is just as an integral part of web development
as writing server side code. Unfortunately tg does little to ease
this part of the development. (Apart from a couple of ajaxish forms,
which will soon be gone and moved into toscawidgets as far as I know.
Even if they stayed this would not really address the issue of a
full stack client application framework component for tg.)
I believe it would be a major boost to tg and would elevate it to
the position of *the* best web application framework if it contained
a comprehensive component for client side application development of the
same great quality as the rest of tg. From what I'll say below I hope
it will be clear that tg widgets + some javascript here and there is
not what I have in mind but something much more comprehensive.
You might ask why should it be part of tg and why shouldn't it be
up to the developer to choose the tools he/she wishes for his/her
client side needs? I think the success of tg is largely due to the
fact that there are default components with sane defaults while
it allows advanced developers to bypass these defaults and use their
own toolsets if flexibility is required. Secondly, tg2 is not an
uber-flexible web framework for just about anything, this role is
played by pylons. The flexibility naturally gets limited in tg
simply because otherwise it would be duplicating functionality
found in pylons. Due to this I believe a new client side application
framework should be an integral part of tg but implemented such
of course that users should easily be able to bypass it if they
desired.
Client side applications in our case mean -- unfortunately -- javascript.
I looked into what would be the best approach for tg and found these 3
choices:
1. Let's write everything in python, not a single line of javascript.
The pypy project has a fully functional RPython -> javascript translator
complete with a very basic client/server framework. Although this is
a very interesting direction and one that actually works (I used it)
the huge overhead that comes with pypy probably doesn't worth it at
this time. http://codespeak.net/pypy/dist/pypy/doc/js/using.html
2. Using the pyjamas project. This is also an interesting approach,
basically porting the Google Web Toolkit from java to python. We
would use a subset of python which would get translated to javascript.
Unfortunately pyjamas is not a very mature project and development
is extremely slow. http://code.google.com/p/pyjamas/
3. Let's pick a well-designed, robust and reliable javascript library
and add functionally to tg that helps integrating it to any tg project.
Users would have to write their javascript themselves but tg would
automate some or most of the typical tasks that one needs to do
in order to really make sure the client and server plays nicely together.
This means that only minimal javascript code has to be maintained, only
the basic API for client/server communication.
I think the best choice would be the third one and as I have mentioned
in an earlier post the Ext.js would be an excellent choice because it's
a mature project, 2.0 was released some time ago. It's 100% documented
and the documentation is top quality. It's very well-designed, unlike
most of the other js libraries. It helps create maintainable and
scalable code. It contains widgets for just about any use case and makes
it easy to extend these widgets to create reusable new ones. The core
size is not very big and it's possible to easily include only those
modules which are actually needed. And last but not least it does not
interfere with other js libraries.
Choosing a js library is of course just one bit the bigger bit is
having tg components which will allow users to integrate the server
side code with the client side. The following list is a (partial) list
of what I think tg should provide as an interface between the python
tg application server and the javascript client:
1. A decorator for controller methods that will only be used from javascript.
This has slightly different requirements then other controller methods even
beyond @expose('json'). For example the Ext.js library likes to attach
certain GET variables to requests which we don't want to declare as
keyword arguments in each and every controller method. Another example
is that if authentication is required and authorization fails we don't
want to redirect to the login page in these controller methods. Instead,
an error should be sent in JSON. There are certainly other peculiarities
associated with the fact that the controller method is exclusively used
from javascript which warrant a special decorator for these methods.
2. Tg exception API in JSON. Whenever a server side exception
is raised it should be sent in a well-defined JSON object to the requesting
javascript client.
3. Configuration sharing between server and client. There should be a
standard way to expose some of the configuration options that users
set for their tg app to the client. For example configuration options
that control the visual appearance of the app like paging lengths,
number of columns, etc, stuff which is not CSS and is
relevant for both the server and client and is set by the user in
one of the configuration files like config/app.cfg.
(the above were stuff which needs work in python as really part of tg,
the stuff that comes below should be implemented in javascript)
4. A separate 'tg' namespace in javascript should be available. This
namespace should have out-of-the-box objects for creating a client
instance, widgets, etc, with a well-defined API for both inter-javascript
communication as well as communicating with the tg server.
I'm sure there are tons of other issues too. I have implemented some of
the stuff above in a very preliminary way and if there is interest I
would like to contribute it somehow but more importantly I'm looking
for the opinion of tg users and developers alike.
Does any of this make any sense?
Cheers,
Daniel
Alberto Valverde
Daniel Fetchinson wrote:
> (....)
>
> 3. Let's pick a well-designed, robust and reliable javascript library
> and add functionally to tg that helps integrating it to any tg project.
> Users would have to write their javascript themselves but tg would
> automate some or most of the typical tasks that one needs to do
> in order to really make sure the client and server plays nicely together.
> This means that only minimal javascript code has to be maintained, only
> the basic API for client/server communication.
>
> I think the best choice would be the third one and as I have mentioned
> in an earlier post the Ext.js would be an excellent choice because it's
> a mature project, 2.0 was released some time ago. It's 100% documented
> and the documentation is top quality. It's very well-designed, unlike
> most of the other js libraries. It helps create maintainable and
> scalable code. It contains widgets for just about any use case and makes
> it easy to extend these widgets to create reusable new ones. The core
> size is not very big and it's possible to easily include only those
> modules which are actually needed. And last but not least it does not
> interfere with other js libraries.
Thanks for you elaborate post. I for one would like TG to choose a
"standard" js library to build a heavier js infrastructure around it to
support rich clients.
I've used ExtJS too and I think it would be the best choice being very
well designed and modular; the core is not too big and integrates well
with other js libs. I'm +1 on having ExtJS as the recommended lib for TG
add-ons.
There's a ToscaWidgets ExtJS egg which packages version 1.something
(would be nice to upgrade it to v2) which could be used to dynamically
include the needed parts of ExtJS in the page as TW can take care of
tracking dependences between JS static files and include only the ones
needed, this is done server-side in python and could become more clever
(such and concatenating into a big file and jsminifying it on the fly
for performance)
> Choosing a js library is of course just one bit the bigger bit is
> having tg components which will allow users to integrate the server
> side code with the client side. The following list is a (partial) list
> of what I think tg should provide as an interface between the python
> tg application server and the javascript client:
>
> 1. A decorator for controller methods that will only be used from javascript.
> This has slightly different requirements then other controller methods even
> beyond @expose('json'). For example the Ext.js library likes to attach
> certain GET variables to requests which we don't want to declare as
> keyword arguments in each and every controller method. Another example
> is that if authentication is required and authorization fails we don't
> want to redirect to the login page in these controller methods. Instead,
> an error should be sent in JSON. There are certainly other peculiarities
> associated with the fact that the controller method is exclusively used
> from javascript which warrant a special decorator for these methods.
TG's exposed methods are designed to be reusable for different output
types, so for example, if the client requests html, the dict controllers
returns will feed a template and if it requests json the dict will be
jsonified. This can, or should be able to, be extended so when dealing
with json output a 401 (authentication required) becomes an object that
is transformed client-side into a particular js exception. The point I'm
trying to make is that ideally there needn't be methods used exclusively
for javascript, with a little more thought TG could do this
automatically behind the scenes (sort-of like expose does now)
>
> 2. Tg exception API in JSON. Whenever a server side exception
> is raised it should be sent in a well-defined JSON object to the requesting
> javascript client.
>
> 3. Configuration sharing between server and client. There should be a
> standard way to expose some of the configuration options that users
> set for their tg app to the client. For example configuration options
> that control the visual appearance of the app like paging lengths,
> number of columns, etc, stuff which is not CSS and is
> relevant for both the server and client and is set by the user in
> one of the configuration files like config/app.cfg.
>
> (the above were stuff which needs work in python as really part of tg,
> the stuff that comes below should be implemented in javascript)
>
> 4. A separate 'tg' namespace in javascript should be available. This
> namespace should have out-of-the-box objects for creating a client
> instance, widgets, etc, with a well-defined API for both inter-javascript
> communication as well as communicating with the tg server.
>
> I'm sure there are tons of other issues too. I have implemented some of
> the stuff above in a very preliminary way and if there is interest I
> would like to contribute it somehow but more importantly I'm looking
> for the opinion of tg users and developers alike.
>
> Does any of this make any sense?
I like the ideas, looking forward to see TG2 move in this direction.
Alberto
Mark Ramm
ext.js+turbogears integration patterns, I would be very willing to add
some of this as a TG dependency eventually. Particularly if there
were some working code ;)
I do think that the future of web development is going to be
http:htlm+ where the + is javascript rich client libraries (ext,
dojo, yui, etc), Flex, Silverlight and whatever other "rich" client
toolkits come into vogue over the next couple of years. TurboGears 2
has to provide rich support for this new style of hybrid web/client
server applications.
So, I'm +1 on your suggestion, though if there were some enterprising
yui, dojo, or whatever, user out there who was able to make that
happen first I'd be willing to entertain the possibility of another
library.
For the moment I would suggest that none of this go into the core, but
live in an tg-ext add on library. This allows us to keep a small,
easy to understand core, with a stable API, and it allows the tg-ext
library to more closely track with updates to Ext.js. We can
certainly add a dependency to tg-ext as we go, and I'm open to making
some changes to the tg core to better accomodate ext.js integration,
but I have to admit to not quite knowing what those changes might be
up-front. I think most of the things suggested below, can be done
without changing the core at all.
--Mark Ramm
--
Mark Ramm-Christensen
email: mark at compoundthinking dot com
blog: www.compoundthinking.com/blog
Mark Ramm
> (would be nice to upgrade it to v2) which could be used to dynamically
> include the needed parts of ExtJS in the page as TW can take care of
> tracking dependences between JS static files and include only the ones
> needed, this is done server-side in python and could become more clever
> (such and concatenating into a big file and jsminifying it on the fly
> for performance)
There is apparently some code which does this kind of thing (the
concatination and minification) in the pylons helpers package (as of a
couple weeks ago).
I'm still trying to figure out how the helpers library will fit in
with TurboGears users, but most of them should be usable (though some
depend on routes). And others duplicate things we get from
ToscaWidgets.
--Mark Ramm
Alberto Valverde
>> There's a ToscaWidgets ExtJS egg which packages version 1.something
>> (would be nice to upgrade it to v2) which could be used to dynamically
>> include the needed parts of ExtJS in the page as TW can take care of
>> tracking dependences between JS static files and include only the ones
>> needed, this is done server-side in python and could become more clever
>> (such and concatenating into a big file and jsminifying it on the fly
>> for performance)
>
> There is apparently some code which does this kind of thing (the
> concatination and minification) in the pylons helpers package (as of a
> couple weeks ago).
I'm aware of that code and that's more or less the idea, the difference
being that since TW can generate the js and css links that are injected
in the html it can take advantage of this and only insert a single link
to the whole ball on production and separate links on debugging. AFAIK
webhelpers only takes care of the minification of a single file or
concatenation of some but doesn't deal with the links in the html.
Alberto
Mark Ramm
> being that since TW can generate the js and css links that are injected
> in the html it can take advantage of this and only insert a single link
> to the whole ball on production and separate links on debugging. AFAIK
> webhelpers only takes care of the minification of a single file or
> concatenation of some but doesn't deal with the links in the html.
I think you use some special helper functions to put the links in the
HTML, and then it either just adds the links (in development mode) or
created (in production mode) adds a single link to the
concatinated-minified file.
But actually haven't used the code at all, that's just the impression
I got from looking at an example someone showed me.
--Mark Ramm
Christoph Zwerschke
> If we can get some people (at least 2 preferably 3) to work on
> ext.js+turbogears integration patterns, I would be very willing to add
> some of this as a TG dependency eventually. Particularly if there
> were some working code ;)
I played a bit with ext.js, but I soon felt like hitting a wall because
the documentation was too meager. Though the API reference is excellent,
it was not sufficient for me. For instance, I could not get a simple
AutocompleteField working. My idea was to use a Combobox that pulls data
from a TG controller via a JsonStore. With static data it was fine, but
it became cumbersome to access the remote data, and even though I got it
running, it does not really behave as expected. Maybe I need to work
with it some more, or maybe I'm missing something. But I found it much
easier to use YUI - which looks like a huge colorful and commerical
mess, but there are so much more examples and docs.
Anyway, I'd like to hear the experiences of others.
-- Chris
iain duncan
> Hi Daniel,
>
> Daniel Fetchinson wrote:
> > (....)
> >
> > 3. Let's pick a well-designed, robust and reliable javascript library
> > and add functionally to tg that helps integrating it to any tg project.
> > Users would have to write their javascript themselves but tg would
> > automate some or most of the typical tasks that one needs to do
> > in order to really make sure the client and server plays nicely together.
> > This means that only minimal javascript code has to be maintained, only
> > the basic API for client/server communication.
> >
> > I think the best choice would be the third one and as I have mentioned
> > in an earlier post the Ext.js would be an excellent choice because it's
> > a mature project, 2.0 was released some time ago. It's 100% documented
> > and the documentation is top quality. It's very well-designed, unlike
> > most of the other js libraries. It helps create maintainable and
> > scalable code. It contains widgets for just about any use case and makes
> > it easy to extend these widgets to create reusable new ones. The core
> > size is not very big and it's possible to easily include only those
> > modules which are actually needed. And last but not least it does not
> > interfere with other js libraries.
I've been very impressed lately with jquery. I remember reading
something about an ext jquery project, is that a possibility? I'm not up
on ext really, but jquery is very well documented and has good community
support. And I think the author has a very TG compatible attitude.
Iain
Daniel Fetchinson
> > and add functionally to tg that helps integrating it to any tg project.
> > Users would have to write their javascript themselves but tg would
> > automate some or most of the typical tasks that one needs to do
> > in order to really make sure the client and server plays nicely together.
> > This means that only minimal javascript code has to be maintained, only
> > the basic API for client/server communication.
> >
> > I think the best choice would be the third one and as I have mentioned
> > in an earlier post the Ext.js would be an excellent choice because it's
> > a mature project, 2.0 was released some time ago. It's 100% documented
> > and the documentation is top quality. It's very well-designed, unlike
> > most of the other js libraries. It helps create maintainable and
> > scalable code. It contains widgets for just about any use case and makes
> > it easy to extend these widgets to create reusable new ones. The core
> > size is not very big and it's possible to easily include only those
> > modules which are actually needed. And last but not least it does not
> > interfere with other js libraries.
>
> Thanks for you elaborate post. I for one would like TG to choose a
> "standard" js library to build a heavier js infrastructure around it to
> support rich clients.
>
> I've used ExtJS too and I think it would be the best choice being very
> well designed and modular; the core is not too big and integrates well
> with other js libs. I'm +1 on having ExtJS as the recommended lib for TG
> add-ons.
I maybe didn't emphasize enough but there are 2 issues here: one is
the choice of js library and the other is all the helper tg python
code and js code that makes it easy to interface the client and
server. The secod issue is js library agnostic and should be so.
> There's a ToscaWidgets ExtJS egg which packages version 1.something
> (would be nice to upgrade it to v2) which could be used to dynamically
> include the needed parts of ExtJS in the page as TW can take care of
> tracking dependences between JS static files and include only the ones
> needed, this is done server-side in python and could become more clever
> (such and concatenating into a big file and jsminifying it on the fly
> for performance)
Yes, minifying should be supported out-of-the-box. The C code at
http://javascript.crockford.com/jsmin.html has a python version at
http://javascript.crockford.com/jsmin.py.txt
Maybe it's just me but the way I usually set up my tg app is that I
have methods which are really 'views' meaning there is a template
attached to them and there is an api controller whose methods only
return JSON. This api controller is then used by the 'view'
controllers to fetch their data. But I agree tg already knows how to
deal with such a situation and only a couple of helpers are needed.
Kevin Horn
I also think that tg2 needs work in other areas first, more than it needs this right now.
I'd really like to see something like this eventually, but there are deeper issues ATM, in my opinion.
Kevin Horn
Daniel Fetchinson
> ext.js+turbogears integration patterns, I would be very willing to add
> some of this as a TG dependency eventually. Particularly if there
> were some working code ;)
I have some working code (very alpha of course), what is the good
place to post it to?
> I do think that the future of web development is going to be
> http:htlm+ where the + is javascript rich client libraries (ext,
> dojo, yui, etc), Flex, Silverlight and whatever other "rich" client
> toolkits come into vogue over the next couple of years. TurboGears 2
> has to provide rich support for this new style of hybrid web/client
> server applications.
>
> So, I'm +1 on your suggestion, though if there were some enterprising
> yui, dojo, or whatever, user out there who was able to make that
> happen first I'd be willing to entertain the possibility of another
> library.
Maybe I didn't emphasize it enough but I think there are two issues
(1) the choice of js library and code that uses this library (2) js
library agnostic infrastructure for client/server communication.
> For the moment I would suggest that none of this go into the core, but
> live in an tg-ext add on library. This allows us to keep a small,
> easy to understand core, with a stable API, and it allows the tg-ext
> library to more closely track with updates to Ext.js. We can
> certainly add a dependency to tg-ext as we go, and I'm open to making
> some changes to the tg core to better accomodate ext.js integration,
> but I have to admit to not quite knowing what those changes might be
> up-front. I think most of the things suggested below, can be done
> without changing the core at all.
I agree, most (if not all) of the python code in tg that would support
js library agnostic client/server communication would take the form of
helpers on top of the core. So it's really independent of the core and
can come in a separate package which would also include some js
library agnostic javascript code for talking to the server.
Daniel Fetchinson
> > (would be nice to upgrade it to v2) which could be used to dynamically
> > include the needed parts of ExtJS in the page as TW can take care of
> > tracking dependences between JS static files and include only the ones
> > needed, this is done server-side in python and could become more clever
> > (such and concatenating into a big file and jsminifying it on the fly
> > for performance)
>
> There is apparently some code which does this kind of thing (the
> concatination and minification) in the pylons helpers package (as of a
> couple weeks ago).
Great, I didn't know that :)
Concatenation and minification is one of those things that everyone
needs if using js heavily.
Daniel Fetchinson
> > ext.js+turbogears integration patterns, I would be very willing to add
> > some of this as a TG dependency eventually. Particularly if there
> > were some working code ;)
>
> I played a bit with ext.js, but I soon felt like hitting a wall because
> the documentation was too meager. Though the API reference is excellent,
> it was not sufficient for me.
There are loads of examples http://extjs.com/deploy/dev/examples/
which are really useful in combination with the API reference.
> For instance, I could not get a simple
> AutocompleteField working. My idea was to use a Combobox that pulls data
> from a TG controller via a JsonStore. With static data it was fine, but
> it became cumbersome to access the remote data, and even though I got it
> running, it does not really behave as expected. Maybe I need to work
> with it some more, or maybe I'm missing something.
Well, there is certainly a learning curve but it's true with anything
isn't it? :)
> But I found it much
> easier to use YUI - which looks like a huge colorful and commerical
> mess, but there are so much more examples and docs.
I personally haven't tried that.
Daniel Fetchinson
> execution. I'd definitely prefer to see it in a seperate package from tg2
> proper, as a lot of people won't really need this and it's a shame to force
> it on them.
Fully agreed. A separate package is the the right place as the whole
thing will not add anything to the core but will only build helpers on
top of it.
Christoph Zwerschke
> There are loads of examples http://extjs.com/deploy/dev/examples/
> which are really useful in combination with the API reference.
Yes, but these operate with static data, and do not cover the whole
problems of shuffling the data from the controller to the widget via
various Data-Stores, -Proxies and -Readers.
> Well, there is certainly a learning curve but it's true with anything
> isn't it? :)
I like when I am able to learn something, but when I can only experiment
and guess or delve into the source to understand how things are intended
to work, that's not so much fun any more (or maybe even fun, but it
takes a lot of time that you cannot put into your actual work).
-- Chris
Daniel Fetchinson
>
> Daniel Fetchinson schrieb:
> > There are loads of examples http://extjs.com/deploy/dev/examples/
> > which are really useful in combination with the API reference.
>
> Yes, but these operate with static data, and do not cover the whole
> problems of shuffling the data from the controller to the widget via
> various Data-Stores, -Proxies and -Readers.
Here are (some of) the fully functional examples which do cover remote
data access of all shapes and sizes using Stores, Proxies, Readers,
etc, etc via JSON and/or XML:
http://extjs.com/deploy/dev/examples/form/custom.html
http://extjs.com/deploy/dev/examples/form/xml-form.html
http://extjs.com/deploy/dev/examples/form/forum-search.html
http://extjs.com/deploy/dev/examples/grid/edit-grid.html
Christoph Zwerschke
> Here are (some of) the fully functional examples which do cover remote
> data access of all shapes and sizes using Stores, Proxies, Readers,
> etc, etc via JSON and/or XML:
>
> http://extjs.com/deploy/dev/examples/form/custom.html
> http://extjs.com/deploy/dev/examples/form/xml-form.html
> http://extjs.com/deploy/dev/examples/form/forum-search.html
> http://extjs.com/deploy/dev/examples/grid/edit-grid.html
Thanks. This helps somewhat, but for me it does not replace a systematic
documentation of the concepts and ideas behind the data handling, when
and why and how to use the different stores and proxies and readers. Can
you point me to anything helpful here? Also, some classes such as
JsonStore aren't covered in these examples.
-- Chris
Daniel Fetchinson
The best way I think is first starting with
http://extjs.com/learn/Tutorial:Introduction_to_Ext_2.0 General
concepts of data handling (Stores, Proxies, Readers, etc) are here
http://extjs.com/learn/Manual:Data and when you got through all of
that the API documentation at http://extjs.com/deploy/dev/docs/ is not
that frightening any more :)
For example it will be clear that JsonStore is just a Store with a
reader that knows how to read JSON. You could write your Store and
Reader by yourself but since it's such a common case you have
JsonStore for convenience.
There is of course a learning curve but in this case I think it's
fully justified because the design is great with several (necessary)
layers of abstraction. More layers, more learning, but it pays off.
The alternative would be a "It's so easy to use" framework with no
abstraction at all but that would not scale and would be difficult to
maintain.
Luciano G. Panaro
http://inside.glnetworks.de/2008/01/18/announcing-ext-scaffold-generator-plugin-for-rails/
I just saw this and haven't tried it yet but it reminded me of this
thread.
On Jan 19, 10:12 pm, "Daniel Fetchinson" <fetchin...@googlemail.com>
wrote:
> > > data access of all shapes and sizes using Stores, Proxies, Readers,
> > > etc, etc via JSON and/or XML:
>
> > >http://extjs.com/deploy/dev/examples/form/custom.html
> > >http://extjs.com/deploy/dev/examples/form/xml-form.html
> > >http://extjs.com/deploy/dev/examples/form/forum-search.html
> > >http://extjs.com/deploy/dev/examples/grid/edit-grid.html
>
> > Thanks. This helps somewhat, but for me it does not replace a systematic
> > documentation of the concepts and ideas behind the data handling, when
> > and why and how to use the different stores and proxies and readers. Can
> > you point me to anything helpful here? Also, some classes such as
> > JsonStore aren't covered in these examples.
>
> The best way I think is first starting withhttp://extjs.com/learn/Tutorial:Introduction_to_Ext_2.0General
> that the API documentation athttp://extjs.com/deploy/dev/docs/is not
Mark Ramm
http://extjs.com/blog/2008/01/22/ext-growth-and-server-side-community-projects/
I know a lot of us would love to see some good server-side+widget side
stuff happen in TurboGears ;)
I just haven't (and probably won't for a while) had time to look much
into doing something interesting in that area. But as I said
earlier, I'll definitely support those who do.
--Mark Ramm
--