web3py wish list

[Massimo DiPierro]

0) keeps dal.py, html.py, template.py and markmin2html.py

1) based on gevent

2) instead of

def index():
id = request.args(0)
return 'hello'+'world'

uses

def index(id):
yield 'hello'
yield 'world'

to allow for efficient realtime

3) built on top of a thread local object like current (never passing, request, response, session, T).

4) rewrite SQLFORM as SMALFORM (or other name) with the ability to receive and generate input/output in json/xml as well as regular forms with post backs. No need to call process(). Would not build the TAG tree unless needed to serialize in HTML.

5) no need to explicitly use cache ever. Objects would automatically cache everything and expire cache intelligently. Some caching to be client side.

[Jonathan Lundell]

On 17 Aug 2012, at 11:47 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
>
> 4) rewrite SQLFORM as SMALFORM (or other name) with the ability to receive and generate input/output in json/xml as well as regular forms with post backs. No need to call process(). Would not build the TAG tree unless needed to serialize in HTML.

Related: RPC requests are first-class. In particular, error responses are appropriate to the expected response format.

[Massimo DiPierro]

This logic can be written and tested with web2py, then made central in the new framework.

We should also avoid

request.env

and user

current.environ

Moreover

a=Storage()
a.x=5
accessing a.x takes 2e-6 seocnds

this is done in so many places that it is the biggest bottle neck in web2py after the database. This would be much faster:

class A:
def __getattr__(self,key):
return getattr(self,key,None)
a.x=5
reading a.x takes 2e-7 seconds

> --
> -- mail from:GoogleGroups "web2py-developers" mailing list
> make speech: web2py-d...@googlegroups.com
> unsubscribe: web2py-develop...@googlegroups.com
> details : http://groups.google.com/group/web2py-developers
> the project: http://code.google.com/p/web2py/
> official : http://www.web2py.com/
>
>

[Anthony]

I think this is an interesting new web app architecture to consider as well: http://derbyjs.com/. Much of the app code is pushed to client side Javascript, including all the templates and rendering. The app is initially loaded via an HTTP request, but then a comet or websocket connection is established, and all subsequent data exchange with the server happens over that connection. Everything is realtime, and the templates are reactive (a change in one browser automatically propagates to the server and then to other browsers).

Because it runs on Node.js, the JS code can run on the server as well as the browser. That's important for search engine indexing, because you need a way to render HTML on the server for crawlability. Rendering on the server also speeds things up for the initial page load of the app, as well as for slower mobile devices. Client/server JS also makes it easy to share other code between the browser and server, such as model and validation code.

There are some cool demos here and here -- most impressive if you open the same page in two browser windows so you can see the changes in one instantly propagate to the other. Here's a video showing what it can do: http://www.youtube.com/watch?v=akMqTuiYn0g. Another similar framework that's gotten a lot of attention (and funding) recently is http://www.meteor.com/.

Another interesting new platform to consider is vert.x. It takes inspiration from Node.js (event loop, async IO), but runs on the JVM (with support for multiple JVM languages,  including Javascript via Rhino and Python via Jython). Could be an interesting way to combine server-side Javascript and Python on a platform that can handle async IO for realtime communication.

Anthony

[Massimo DiPierro]

I would still want the server side to be python and not node js.

[Anthony]

I would still want the server side to be python and not node js.

vert.x allows that if you'll settle for Jython. Anyway, doesn't have to be node.js -- it's more the general app architecture that's interesting. For the reactive UI, you need client-side templates, but with client-side templates, it's best to also have a way of rendering on the server, which could be a Python version of the template engine (I think that's what Planet does), or maybe server-side JS just for the templates.

Anthony

[Massimo DiPierro]

The attached file contains a class NewStorage. As you can see it is 10x faster than out storage. This and not parsing os.environ into request.env would add lots of speedup to web2py. The problem is that this works for storage not for NewStorage and I do not know how to define ** for a non-dict object.

s = NewStorage()
f(**s)

Massimo

On Aug 17, 2012, at 3:02 PM, Jonathan Lundell wrote:

[Jonathan Lundell]

On 17 Aug 2012, at 3:38 PM, Massimo DiPierro <massimo....@gmail.com> wrote:
>
> The attached file contains a class NewStorage. As you can see it is 10x faster than out storage. This and not parsing os.environ into request.env would add lots of speedup to web2py. The problem is that this works for storage not for NewStorage and I do not know how to define ** for a non-dict object.
>
> s = NewStorage()
> f(**s)

You should be able to follow the example of UserDict: http://docs.python.org/library/userdict.html

[Bruno Rocha]

worth a try in this logic for LazyDAL ?

    def __getattr__(self, tablename):

        return getattr(self, tablename) if tablename in self else define_table(.....)

I still dont know how to do it in a nested level for db.table.<field>

But I am still trying..

[Massimo DiPierro]

I believe that is slower than the current implementation and does not support the ** syntax because does not extend dict.

[Massimo DiPierro]

Internally it would be called everywhere anyway...

[Bruno Rocha]

On Fri, Aug 17, 2012 at 8:10 PM, Massimo DiPierro <massimo....@gmail.com> wrote:

Internally it would be called everywhere anyway...

But in a model with 50+ lazy_defined tables, it will be called to real_define only those who has direct references. In a context where only two tables are called, it may be a great gain of performance. (or maybe I am missing the point and the bottleneck is another)

 

[Massimo DiPierro]

Let's try. I may be wrong.

[Massimo DiPierro]

Here is a proof of concept design of web sockets on top of wsgi on top of gevent. The user program can be a normal web app (like web2py) but can read the input stream and field multiple data.

import uuid

class Connection(object):

    CLIENTS = {}

    APPLICATIONS = {}

    def __call__(self, environ, start_response):

        self.environ = environ

        self.uuid = str(uuid.uuid4())

        headers = [('Content-Type', 'text/html')]

        k = None

        app = environ['PATH_INFO']

        try:

            for k,data in enumerate(Connection.APPLICATIONS[app](self)):

                if k==0:

                    start_response('200 OK', headers)

                    Connection.CLIENTS[self.uuid] = self

                if isinstance(data,dict):

                    yield render(self.view,context=data)

                elif isinstance(data,str):

                    yield data

                elif isisnatnce(data,file):

                    # stream data                                                                                                         

                    raise NotImplementedError

                else:

                    raise NotImplementedError

        except Exception, e:

            if k is None:

                start_response('500 INTERNAL ERROR',

                               [('Content-Type', 'text/html')])

                yield str(e)

        if self.uuid in Connection.CLIENTS:

            del Connection.CLIENTS[self.uuid]

    @staticmethod

    def register(path):

        def __register(f,path=path):

            Connection.APPLICATIONS[path] = f

            return f

        return __register

### USER PROGRAM ####################                                                                                                     

@Connection.register(path='/')

def main(connection):

    yield '<html><body>'

    yield 'Hello %s' % connection.environ['REMOTE_ADDR']

    yield '</body></html>'

### END USER PROGRAM #################                                                                                                    

if __name__=='__main__':

    import gevent.wsgi

    gevent.wsgi.WSGIServer(('', 8000),Connection()).serve_forever()

On Aug 17, 2012, at 6:13 PM, Bruno Rocha wrote:

[Massimo DiPierro]

Seems to work. Check trunk.

massimo

On Aug 17, 2012, at 6:13 PM, Bruno Rocha wrote:

[Massimo DiPierro]

Seems to work. Check trunk.

On Aug 17, 2012, at 6:06 PM, Bruno Rocha wrote:

[Jonathan Lundell]

On 17 Aug 2012, at 4:09 PM, Massimo DiPierro <massimo....@gmail.com> wrote:
>
> I believe that is slower than the current implementation and does not support the ** syntax because does not extend dict.

That's not what I'm suggesting.

UserDict supports the ** syntax, not because it extends dict (it does not), but because it conforms the the requirements of a mapping type. My guess is that you're missing a method or two, so Python isn't seeing NewStorage as a mapping type.

[Jonathan Lundell]

On 17 Aug 2012, at 5:05 PM, Jonathan Lundell <jlun...@pobox.com> wrote:
>
> On 17 Aug 2012, at 4:09 PM, Massimo DiPierro <massimo....@gmail.com> wrote:
>>
>> I believe that is slower than the current implementation and does not support the ** syntax because does not extend dict.
>
> That's not what I'm suggesting.
>
> UserDict supports the ** syntax, not because it extends dict (it does not), but because it conforms the the requirements of a mapping type. My guess is that you're missing a method or two, so Python isn't seeing NewStorage as a mapping type.

Actually, I just tried it, and ** works fine with NewStorage.

[Michele Comitini]

the web2py_component() should be extended/substituted to simplify
communication back and forth from web2py to client and viceversa.
This would benefit from websockets, but could be useful even in a
client event driven setup.
The idea is allowing something similar to the component, behavior,
event of some web frameworks (JSF, wicket, wt, tapestry), and
Javascript aggregation, with the benefits of being pythonic. There
are many ways to do it, although every one uses a set of interfaces in
the API to expose those features in the components.
Seems to me that on most frameworks of this kind the component is
equivalent to a complex state machine. We should avoid complexity so
it should be added only if can be made simple.

mic




2012/8/18 Jonathan Lundell <jlun...@pobox.com>:

[Osman Masood]

What about offloading some platform-independent functionality to modules written in C? Such as the Storage class?

[Michael Toomim]

One of my favorite things about web2py is that you can edit code -- even database schema -- without restarting the web server.

Now, I've noticed some fears that this won't be preserved when moving to a modules-esque approach, and that perhaps the reload() functionality in python won't work. I have faith that this will be investigated thoroughly.

In the process, I suggest a design decision—can we require ZERO restarts of the web server, for any change?

Right now we still need to restart the web server for:

  - changing routes.py

  - or adding a new application?

  - changes to modules/

This would be a great design goal to achieve! We have all run into the confusing situation of finding routes.py not working... because you forgot to restart, because you so rarely have to restart the server in the first place!

[Niphlod]

issue here is that code evaluating at every request is surely a +1 for development but an absolute performance-wise downsider for production apps that do not change often.
One of the things that keep users away web2py is response times (even when faced with the ultra-explained-with-proofs that most of the times for normal apps the difference between, e.g., web2py and django is unnoticeable).

[Alec Taylor]

Thanks, two major feature requests:

- Decoupled views, automatically generated; within an MVC JavaScript framework such as EmberJS or AngularJS
- Continue efforts to keep components decoupled; so I can switch in different backends (e.g.: different logging engines) and turn off unneeded high-overhead features (e.g.: implicit caching)

And a minor feature for support of Flask-style decorators for routing and HTTP verb restrictions.