web3py - important!

1,796 views
Skip to first unread message

Massimo DiPierro

unread,
Oct 25, 2012, 12:21:06 PM10/25/12
to web2py-d...@googlegroups.com
Hello everybody,

Time to start thinking about web3py. And as I think about it, I hear this in my mind: http://www.youtube.com/watch?v=HofoK_QQxGc

Here is what I have in mind and I am building a prototype:

1) eliminate every computation done when the request arrives (for example parsing of url, headers, accept_languages, etc) and make everything lazy.
2) eliminate request, response, session, T, cache objects and have a single "current" object. It should not be a thread local to avoid locking issues, but it should be passed around.
3) eliminate all the exec/execfiles but preserve some version of the current multi-applications folder structures. Each app will have an entry point (main.py) where actions are registered to allow a freer structure at the app level. 
4) A global routes to configure "only" application prefixes. Then each app is in charge of its own internal routing, perhaps using decorators as Flask does.
5) No more global environment. Apps will do "from web3py import *" (see below)
6) No more file-system write unless to install/edit apps. We store session in cookies (if small) or db (if large) and tickets always in db (so they can be searched and cleared more easily). They will not go in the app db but in a dedicated common ticket db.
7) Remove all the helpers except TAG, CAT, XML, BEAUTIFY, MENU and renamed them tag, cat, safe,  beautify, menu (with simpler logic)
8) Replace SQLFORM.factory with Form and SQLFORM with DALForm. They will have a simpler logic that separates the HTML representation from the input processing, thus it will be faster and will be able to handle non-html form submission (ajax, xml) while using a syntax similar to the current one.
9) No more appadmin. There should be a global application to  manage the databases of all the apps.
10) should work with python 2.7 and 3.x
11) A compatibility layer that will allow to run most existing web2py applications when using python 2.7.
12) no more cache.disk but better cache.ram (renamed as cache) with actual expiration (current cache.ram only expires when accessed).
13) automatic reverse routing
14) keep only dal.py and template.py from existing web2py.

I have most of this done and the code is smaller and cleaner than current web2py is. If is 20x faster on hello world. The compatibility layer is imperfect as I am still working on BEAUTIFY, MENU, sessions, db transactions, and a many nasty details. Tools.py (auth) is missing.

Here is an example of code I got working:

<begin file apps/welcome/main.py>
import time
from web3py import DAL, Field, expose, Form, DALForm, IS_NOT_EMPTY, HTTP, tag

db.define_table('person',Field('name',requires=IS_NOT_EMPTY()))

@expose() # /appname/index
def index(current):
    form = Form(Field('name',requires=IS_NOT_EMPTY())).process(current)
    if form.submitted and form.processed and form.accepted: print form.vars
    elif form.submitted: print form.errors
    message = form.vars.name or ''
    return locals()

@expose('/test_args/<str:a>(/<int:b>)?') # /appname/test_args/hello/1
def test_args(current,a,b=9): # args are parsed, validated and passed to action
    return repr(dict(a=a,b=b)) 

@expose(template='index.html')
def test_form(current):
    form = DALForm(db.person, record_id = None).process(current)
    message = tag.div(tag.h1(form.vars.name))
    return locals()

@expose(cache_expire=10)
def timer(current):
    return time.ctime()

@expose()
def try_redirect(current):
    HTTP.redirect(current.url('index')) # does reverse routing
<end>

As you can see the syntax is similar for an easier transition. To benefit from speed apps will have to be rewritten.

Legacy web2py welcome app, without tools and session also works. I will add tools and session support asap.

The error tickets also need work but I just discovered inspect.stack and had an epiphany!

Am I on the right track?

Want to help? Comments suggestions? What can we do better? When should apps be reloaded (since they are no longer executed we must trigger reload)? What would you like to see changed? Should we preserve validators as they are? Should we have a built-in realtime event handling for websockets in every app? What do we do with the LOAD helper? what do we do with web2py.js? Do we still need response.files of should we make more use of template {{block…}} to inject stuff in <head/>? Should we backport some of new stuff (Form, cache, helpers) to web2py?

If we take this route, we need to have a permanent feature freeze on web2py, except on DAL and Template which are the only shared modules.

I will share my code next week, as soon as I get sessions to work.
You will appreciate the clarity compare with the current web2py. In particular the helpers and form handling logic.

Massimo

Jonathan Lundell

unread,
Oct 25, 2012, 12:57:33 PM10/25/12
to web2py-d...@googlegroups.com
On 25 Oct 2012, at 9:21 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
Hello everybody,

Time to start thinking about web3py. And as I think about it, I hear this in my mind: http://www.youtube.com/watch?v=HofoK_QQxGc

Here is what I have in mind and I am building a prototype:

1) eliminate every computation done when the request arrives (for example parsing of url, headers, accept_languages, etc) and make everything lazy.
2) eliminate request, response, session, T, cache objects and have a single "current" object. It should not be a thread local to avoid locking issues, but it should be passed around.

Are there locking issues? My impression is that there are not. The object is locked very briefly, but the lock is released immediately, is it not? I'm doubtful that there would be a measurable performance gain from avoiding the lock altogether, and it's a considerable PITA to have to pass an extra argument everywhere.

A single object for sure.

Related: maybe a well-defined persistent global per-application state.

3) eliminate all the exec/execfiles but preserve some version of the current multi-applications folder structures. Each app will have an entry point (main.py) where actions are registered to allow a freer structure at the app level. 
4) A global routes to configure "only" application prefixes. Then each app is in charge of its own internal routing, perhaps using decorators as Flask does.

Routing by decorator could be a real rat's nest in a big app. At the very least it should be possible to do central (per-app) routing.

5) No more global environment. Apps will do "from web3py import *" (see below)
6) No more file-system write unless to install/edit apps. We store session in cookies (if small) or db (if large) and tickets always in db (so they can be searched and cleared more easily). They will not go in the app db but in a dedicated common ticket db.
7) Remove all the helpers except TAG, CAT, XML, BEAUTIFY, MENU and renamed them tag, cat, safe,  beautify, menu (with simpler logic)
8) Replace SQLFORM.factory with Form and SQLFORM with DALForm. They will have a simpler logic that separates the HTML representation from the input processing, thus it will be faster and will be able to handle non-html form submission (ajax, xml) while using a syntax similar to the current one.
9) No more appadmin. There should be a global application to  manage the databases of all the apps.
10) should work with python 2.7 and 3.x
11) A compatibility layer that will allow to run most existing web2py applications when using python 2.7.
12) no more cache.disk but better cache.ram (renamed as cache) with actual expiration (current cache.ram only expires when accessed).
13) automatic reverse routing
14) keep only dal.py and template.py from existing web2py.

15) First-class support for RPC-oriented apps or functions, including proper error handling (no redirecting of a JSON request to an HTML error page).

16) Built-in support for long-running processes of some kind, independent of requests. Not sure what--maybe the current scheduler model, or the GAE task model?


--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 


Massimo DiPierro

unread,
Oct 25, 2012, 1:07:22 PM10/25/12
to web2py-d...@googlegroups.com
On Oct 25, 2012, at 11:57 AM, Jonathan Lundell wrote:

On 25 Oct 2012, at 9:21 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
Hello everybody,

Time to start thinking about web3py. And as I think about it, I hear this in my mind: http://www.youtube.com/watch?v=HofoK_QQxGc

Here is what I have in mind and I am building a prototype:

1) eliminate every computation done when the request arrives (for example parsing of url, headers, accept_languages, etc) and make everything lazy.
2) eliminate request, response, session, T, cache objects and have a single "current" object. It should not be a thread local to avoid locking issues, but it should be passed around.

Are there locking issues? My impression is that there are not. The object is locked very briefly, but the lock is released immediately, is it not? I'm doubtful that there would be a measurable performance gain from avoiding the lock altogether, and it's a considerable PITA to have to pass an extra argument everywhere.

A single object for sure.

Related: maybe a well-defined persistent global per-application state.

I hear you. We can make current be an imported thread local.

3) eliminate all the exec/execfiles but preserve some version of the current multi-applications folder structures. Each app will have an entry point (main.py) where actions are registered to allow a freer structure at the app level. 
4) A global routes to configure "only" application prefixes. Then each app is in charge of its own internal routing, perhaps using decorators as Flask does.

Routing by decorator could be a real rat's nest in a big app. At the very least it should be possible to do central (per-app) routing.

Basically expose (in the example below) is the app-level router. It stores info in expose.routes_in and expose.routes_out. We can add an interface to pre-define them all in main.py. You would still need to use the @expose decorator, although no needs to parse arguments. Would this be ok?


5) No more global environment. Apps will do "from web3py import *" (see below)
6) No more file-system write unless to install/edit apps. We store session in cookies (if small) or db (if large) and tickets always in db (so they can be searched and cleared more easily). They will not go in the app db but in a dedicated common ticket db.
7) Remove all the helpers except TAG, CAT, XML, BEAUTIFY, MENU and renamed them tag, cat, safe,  beautify, menu (with simpler logic)
8) Replace SQLFORM.factory with Form and SQLFORM with DALForm. They will have a simpler logic that separates the HTML representation from the input processing, thus it will be faster and will be able to handle non-html form submission (ajax, xml) while using a syntax similar to the current one.
9) No more appadmin. There should be a global application to  manage the databases of all the apps.
10) should work with python 2.7 and 3.x
11) A compatibility layer that will allow to run most existing web2py applications when using python 2.7.
12) no more cache.disk but better cache.ram (renamed as cache) with actual expiration (current cache.ram only expires when accessed).
13) automatic reverse routing
14) keep only dal.py and template.py from existing web2py.

15) First-class support for RPC-oriented apps or functions, including proper error handling (no redirecting of a JSON request to an HTML error page).

Strongly agree.

16) Built-in support for long-running processes of some kind, independent of requests. Not sure what--maybe the current scheduler model, or the GAE task model?

I like the current scheduler. It could work very much the same as far as DB is concerned but it would need some rewriting.

Martín Mulone

unread,
Oct 25, 2012, 1:21:21 PM10/25/12
to web2py-d...@googlegroups.com
I like it!.

2012/10/25 Massimo DiPierro <massimo....@gmail.com>

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 



--
http://martinmulone.com.ar

Massimo DiPierro

unread,
Oct 25, 2012, 1:43:47 PM10/25/12
to web2py-d...@googlegroups.com
How about this but making it a thread-local?

class Current(threading.thread_local):

    def initialize_local(self,environ):
        self.environ = environ
        self.status_code = 200
        self.scheme = 'https' if \
            environ.get('wsgi.url_scheme','').lower() == 'https' or \
            environ.get('HTTP_X_FORWARDED_PROTO','').lower() == 'https' or \
            environ.get('HTTPS','') == 'on' else 'http'
        self.name = '<app>.<main>.<func>' # defined in expose.run                       
        self.hostname = environ['HTTP_HOST']
        self.method = environ.get('HTTP_METHOD', 'get').lower()
        self.path_info = environ['PATH_INFO']
        self.input = environ.get('wsgi.input')
        self.now = environ['w3p.now']
        self.application = environ['w3p.application']
        self.extension = (self.path_info.rsplit('.',1)+['html'])[1]
        self.response_cookies = SimpleCookie()
        self.response_headers = {'Content-Type':contenttype(self.path_info,'text/html')\

    @property
    def get_vars(self):
        " lazily parse the query string into get_vars "
        if not hasattr(self,'_get_vars'):
            self._parse_get_vars()
        return self._get_vars

    @property
    def post_vars(self):
        " lazily parse the request body into post_vars "
         if not hasattr(self,'_post_vars'):
            self._parse_post_vars()
        return self._post_vars

    @property
    def session(self):
        " lazily create the session "
         if not hasattr(self,'_session'):
            self._session = Session(self)
        return self._session

    @property
    def request_cookies(self):
        " lazily parse the request cookies "
         if not hasattr(self,'_request_cookies'):
            self._request_cookies = SimpleCookie()
            self._request_cookies.load(self.environ.get('HTTP_COOKIE',''))
        return self._request_cookies

    @property
    def T(self):
        " lazily allocate the T object "
         if not hasattr(self,'_t'):
            self._t = translator(os.path.join('apps',self.application,'languages'),
                                 self.environ.get('HTTP_ACCEPT_LANGUAGE'))
        return self._t

On Oct 25, 2012, at 11:57 AM, Jonathan Lundell wrote:

Jonathan Lundell

unread,
Oct 25, 2012, 1:44:08 PM10/25/12
to web2py-d...@googlegroups.com
On 25 Oct 2012, at 10:07 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
Routing by decorator could be a real rat's nest in a big app. At the very least it should be possible to do central (per-app) routing.

Basically expose (in the example below) is the app-level router. It stores info in expose.routes_in and expose.routes_out. We can add an interface to pre-define them all in main.py. You would still need to use the @expose decorator, although no needs to parse arguments. Would this be ok?


I think so, yes.

Two more things.

* Have you given any thought to a non-blocking event-driven structure? 

* Ajax. Are we satisfied with the current mechanisms for interacting with client-side JavaScript? 

Massimo DiPierro

unread,
Oct 25, 2012, 1:47:17 PM10/25/12
to web2py-d...@googlegroups.com
On Oct 25, 2012, at 12:44 PM, Jonathan Lundell wrote:

On 25 Oct 2012, at 10:07 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
Routing by decorator could be a real rat's nest in a big app. At the very least it should be possible to do central (per-app) routing.

Basically expose (in the example below) is the app-level router. It stores info in expose.routes_in and expose.routes_out. We can add an interface to pre-define them all in main.py. You would still need to use the @expose decorator, although no needs to parse arguments. Would this be ok?


I think so, yes.

Two more things.

* Have you given any thought to a non-blocking event-driven structure? 

No but the python folks are discussing adding async to python 3. It may be premature to take that route.

* Ajax. Are we satisfied with the current mechanisms for interacting with client-side JavaScript? 

No. What do you propose?

Jonathan Lundell

unread,
Oct 25, 2012, 1:53:35 PM10/25/12
to web2py-d...@googlegroups.com
On 25 Oct 2012, at 10:47 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
* Ajax. Are we satisfied with the current mechanisms for interacting with client-side JavaScript? 

No. What do you propose?


I don't use it enough to have a view (my current work is mostly service-oriented JSON-RPC). But I've been seeing a steady stream of questions on the user list, and find it really confusing. 

Niphlod

unread,
Oct 25, 2012, 3:02:25 PM10/25/12
to web2py-d...@googlegroups.com
I see why you're busy now!!!
some random thoughts....
- session handling with cookies or db is a good idea, but locking must be considered for the single session (or we drop the idea of locking the session?). having a separate db just for session is ok, I guess (I use it already), but a sqlite db will choke sooner than file-based sessions (as long as WAL isn't activated by default http://code.google.com/p/web2py/issues/detail?id=984). A single session creation (insert or update) will block reads for every other session, that is, the entire site blocked for a while.
- LOAD is good, given that almost all modern apps seems to support some kind of "component" reloading instead of reloading all the page. the only bit missing is pjax-style history management, in order to let only a part of the page to be reloaded and have consistent urls.
- what I find in every project really really hard is some kind of "shared" initialization of my own classes and modules. Having something persistent that is not related to cache.ram is hard (managed right now with singletons, but that logic is not linear). e.g., for w2p_tvseries I'd like to use the same requests object (the "modern" urllib) to interact with various webservices, without istantiating a connection everytime. For another project, e.g. storing fixtures and/or settings to be executed on load or at the 1st request only is a PITA. Yes, there is cache, but not everything is serializable.
- I think the main "difference" in web2py vs others is the ability to have everything re-executed at run-time. Pros: faster development, cons: slower in production. Since I see a "move" towards having all loaded at start and never read again, I'm eager to see what will come up (restarting webserver each time is not really nice). Scheduler can benefit from a rewrite, for sure: e.g. now it's just a web2py shell executing all over again whatever the code is. Pros: you can change all your app and never stop the scheduler, cons: slower in production. Other implementations load everything at the start and execute only the function (faster). The main example is, you have to send an email. You could easily skip large chunks of code of your app just to send that email, and currently is faster to run the Scheduler in "external" mode than having all modules reloaded for every mail you'd like to send. However, keeping execution environment "separated" and pass all the objects in multiprocessing.Queues can be painful (i.e., db connections objects)
- new routing: I like the flexibility right now, but the code is cumbersome, and users just can't face it (don't know why, django urls are not that different). Don't know if flask syntax is more understandable, but it seems to grip better into users heads (not in mine, though, I'll learn )
- I like 8) a lot. All the "moves" between sqlhtml, validators, dal, html is hard to understand. I got a grip on that just after a few rounds of debugging. Idea to consider: a simple way to code widgets, with html, js, validation, presentation layers separated but with a "linear" flow logic. I recently had to come up with handling different timezones and had to write more code than I expected: then I realized it was better to pack it as a plugin, then I quit the idea because web2py plugins are really not first class citizens... I don't think that currently many users watch them
- 12) in multiprocessing environments cache.disk is the only "embedded" cache a user can lean towards. I use redis a lot and I'm happy with that, but memcache and redis are not really spread amongst Windows developers.
- adding a remote 17)... supporting nosql dbs is hard, but current implementation lacks for any hope of stability in the future (needs more testers without doubts). Any "internal" module (auth, scheduler, wiki, sessions) needs some "hook" to intercept store/retrieve in a de-normalized form. If we're going to rewrite them from scratch (in Italy we say, "fatto 30, facciamo 31", in English I think is "in for a penny, in for a pound"), we should avoid implicit foreign keys
- a remote 18) that is a wish for DAL: support for nested transactions (or contexts, we're talking about 2.7 to 3.0, where context are available for a while). I found contexts in python really useful for handling transactions.... Right now to be safe a developer is "forced" to do a db.commit() to save the current status, update/insert/delete and then again db.commit().

Niphlod

unread,
Oct 25, 2012, 3:23:12 PM10/25/12
to web2py-d...@googlegroups.com
on 12) .... an LRU cache is quite fine, and it's ready in standard library (functools) . We'd need also something like cache.update(key) to close the deal off the "thundering herd" problem (lock is used to avoid that right now, but, e.g. I'd like to compute the most 10 new articles on my blog and cache that info, and recalculate that only when I insert a new article, current cache kinda of gets in the way (need to clear() and cache(), where I'd really like to serve the stale object instead as long as the fresh one is not in there)).

Massimo DiPierro

unread,
Oct 25, 2012, 3:33:28 PM10/25/12
to web2py-d...@googlegroups.com
I agree. I see it has been back ported to 2.6

Is the backport good enough? What about 2.5?
Is it better to have a max number of objects or have an expiration time?

Massimo

On Oct 25, 2012, at 2:23 PM, Niphlod wrote:

on 12) .... an LRU cache is quite fine, and it's ready in standard library (functools) . We'd need also something like cache.update(key) to close the deal off the "thundering herd" problem (lock is used to avoid that right now, but, e.g. I'd like to compute the most 10 new articles on my blog and cache that info, and recalculate that only when I insert a new article, current cache kinda of gets in the way (need to clear() and cache(), where I'd really like to serve the stale object instead as long as the fresh one is not in there)).

Niphlod

unread,
Oct 25, 2012, 4:12:04 PM10/25/12
to web2py-d...@googlegroups.com
backports are somewhat "certified". Supporting 2.5 will get us in trouble? I though for web3py min was 2.7 and I was really happy about that.
Max number of objects make you cache of a somewhat fixed size (ok, depends on what you put in cache but at least it won't blow up storing undefinitely things you really don't need anymore).
Expiration time forces you to use an external thread to check all keys for expiration and remove them or scan all the keys everytime you access a single key. Yes, the 2nd can be optimized (e.g. pick 10 random keys and check only those), but it's an operation nonetheless on every access.

Massimo DiPierro

unread,
Oct 25, 2012, 4:16:40 PM10/25/12
to web2py-d...@googlegroups.com
So far I have a class cache that maintains a heap sorted by expiration. Inserting is O(log(size)). When a new object is inserted if deletes those that expired without a thread. Yet is O(log(size)) acceptable?

The reason I want with this is that we can implement the same interface on top of redis and memcache (because they seem to support the same type of expiration).

Massimo

Michele Comitini

unread,
Oct 25, 2012, 4:48:25 PM10/25/12
to web2py-d...@googlegroups.com
Hi Massimo + all,

Forgive me if what I write is old.  I started replying, but I was interrupted many times so my reply is old since it is not update with all the other's replies.

mic


2012/10/25 Massimo DiPierro <massimo....@gmail.com>

Hello everybody,

Time to start thinking about web3py. And as I think about it, I hear this in my mind: http://www.youtube.com/watch?v=HofoK_QQxGc

Here is what I have in mind and I am building a prototype:

1) eliminate every computation done when the request arrives (for example parsing of url, headers, accept_languages, etc) and make everything lazy.
I'd look into something to add interceptors or something similar to the WSGI shells, but cleaner, dynamic and simpler. 
2) eliminate request, response, session, T, cache objects and have a single "current" object. It should not be a thread local to avoid locking issues, but it should be passed around.
What about using Borg(s)?  Maybe looking in multiprocessing module could bring fresh ideas. 
3) eliminate all the exec/execfiles but preserve some version of the current multi-applications folder structures. Each app will have an entry point (main.py) where actions are registered to allow a freer structure at the app level. 
Should the app be able to manage any stage of the request/response trail or that is done exclusively at framework level? 
4) A global routes to configure "only" application prefixes. Then each app is in charge of its own internal routing, perhaps using decorators as Flask does.
This partially answer to my question above.  The router could be implemented in term of listeners, an app hooks as a listener on a URI. 
5) No more global environment. Apps will do "from web3py import *" (see below)
6) No more file-system write unless to install/edit apps. We store session in cookies (if small) or db (if large) and tickets always in db (so they can be searched and cleared more easily). They will not go in the app db but in a dedicated common ticket db.
Shared mem can do fine and fast with medium sized sessions, without resorting to external db or stores.
7) Remove all the helpers except TAG, CAT, XML, BEAUTIFY, MENU and renamed them tag, cat, safe,  beautify, menu (with simpler logic)
+1 
8) Replace SQLFORM.factory with Form and SQLFORM with DALForm. They will have a simpler logic that separates the HTML representation from the input processing, thus it will be faster and will be able to handle non-html form submission (ajax, xml) while using a syntax similar to the current one.
+1 
9) No more appadmin. There should be a global application to  manage the databases of all the apps.
+1 
10) should work with python 2.7 and 3.x
+1 
11) A compatibility layer that will allow to run most existing web2py applications when using python 2.7.
Not a priority, could start as a test bed for the new framework. 
12) no more cache.disk but better cache.ram (renamed as cache) with actual expiration (current cache.ram only expires when accessed).
Shared mem for cache ram if possible.  How expiration is managed?  A dedicated process? 
13) automatic reverse routing
+1 
14) keep only dal.py and template.py from existing web2py.
Separate adapters from the core functionalities of DAL, if there is need to distribute as a single file, solutions can be found with some  assembling of child modules in a single file.


--

Niphlod

unread,
Oct 25, 2012, 4:50:56 PM10/25/12
to web2py-d...@googlegroups.com
O(log n) is good (very very good). I'm not a big fan of microbenchmarks details, but the "real problem" relies on what takes to delete, e.g., 100 expired keys at the time 1 is inserted.
Assuming no external thread is needed, than the "deletion" time has to be accounted for, because you must delete before returning the value of the cached function. I don't know how the functools LRU performs in terms of Big-O, but for the architecture of the LRU itself the worst case scenario of "added time" (between inserting/updating and inserting when the stack is full) is the time taken to delete one key.

Michele Comitini

unread,
Oct 25, 2012, 5:08:41 PM10/25/12
to web2py-d...@googlegroups.com
Ajax.

LOAD or something similar for me would be fine, non a perfect solution, but simple enough and clean.
web3py should allow the programmer to retain much of the common practices of GUI programming and being in this aspect more similar to a toolkit such Gtk or QT.  This requires defining a simple wire protocol (json) to streamline events, dom objects, js code object, back and forth.
That means that code on both server and client should be written to handle the encoding, decoding, collecting and dispatching of events and instances.

mic


2012/10/25 Niphlod <nip...@gmail.com>

Martín Mulone

unread,
Oct 25, 2012, 5:49:37 PM10/25/12
to web2py-d...@googlegroups.com
About cache I like the way of wheezy.web in CacheProfile. http://packages.python.org/wheezy.http/userguide.html#cache-profile

2012/10/25 Massimo DiPierro <massimo....@gmail.com>



--
http://martinmulone.com.ar

Massimo DiPierro

unread,
Oct 25, 2012, 6:25:19 PM10/25/12
to web2py-d...@googlegroups.com
Can you show an example?

> just throwing ideas.....
> If every function is executed in a class with contexts support, you can define an __exit__ method where you can "restore" the original variables > to get a clean environment for the next request. 
> I saw back in the days that there are a lot of events hooks available on other frameworks, something like "before_request, after_request, etc" > > and lots of decorators. Maybe those decorators and hooks are configured to restore the original env to its initial state.


On Oct 25, 2012, at 3:12 PM, Niphlod wrote:

Massimo DiPierro

unread,
Oct 25, 2012, 6:27:13 PM10/25/12
to web2py-d...@googlegroups.com
We can cap the max number of deletes.  We do not have to delete all that we find at every request. Although the probability that many expire at once is low.

Massimo DiPierro

unread,
Oct 25, 2012, 6:35:15 PM10/25/12
to web2py-d...@googlegroups.com
On Oct 25, 2012, at 3:48 PM, Michele Comitini wrote:

Hi Massimo + all,

Forgive me if what I write is old.  I started replying, but I was interrupted many times so my reply is old since it is not update with all the other's replies.

mic


2012/10/25 Massimo DiPierro <massimo....@gmail.com>
Hello everybody,

Time to start thinking about web3py. And as I think about it, I hear this in my mind: http://www.youtube.com/watch?v=HofoK_QQxGc

Here is what I have in mind and I am building a prototype:

1) eliminate every computation done when the request arrives (for example parsing of url, headers, accept_languages, etc) and make everything lazy.
I'd look into something to add interceptors or something similar to the WSGI shells, but cleaner, dynamic and simpler. 

Can you show an example?

2) eliminate request, response, session, T, cache objects and have a single "current" object. It should not be a thread local to avoid locking issues, but it should be passed around.
What about using Borg(s)?  Maybe looking in multiprocessing module could bring fresh ideas. 

The problem is concurrency. Borg and Singletons and may have problems in multithreaded environment. We should also imagine an async server. Thread.local is usually patched to work with those. 

Massimo DiPierro

unread,
Oct 25, 2012, 9:18:12 PM10/25/12
to web2py-d...@googlegroups.com
On Oct 25, 2012, at 2:02 PM, Niphlod wrote:

I see why you're busy now!!!
some random thoughts....
- session handling with cookies or db is a good idea, but locking must be considered for the single session (or we drop the idea of locking the session?). having a separate db just for session is ok, I guess (I use it already), but a sqlite db will choke sooner than file-based sessions (as long as WAL isn't activated by default http://code.google.com/p/web2py/issues/detail?id=984). A single session creation (insert or update) will block reads for every other session, that is, the entire site blocked for a while.
- LOAD is good, given that almost all modern apps seems to support some kind of "component" reloading instead of reloading all the page. the only bit missing is pjax-style history management, in order to let only a part of the page to be reloaded and have consistent urls.
- what I find in every project really really hard is some kind of "shared" initialization of my own classes and modules. Having something persistent that is not related to cache.ram is hard (managed right now with singletons, but that logic is not linear). e.g., for w2p_tvseries I'd like to use the same requests object (the "modern" urllib) to interact with various webservices, without istantiating a connection everytime. For another project, e.g. storing fixtures and/or settings to be executed on load or at the 1st request only is a PITA. Yes, there is cache, but not everything is serializable.
- I think the main "difference" in web2py vs others is the ability to have everything re-executed at run-time. Pros: faster development, cons: slower in production. Since I see a "move" towards having all loaded at start and never read again, I'm eager to see what will come up (restarting webserver each time is not really nice).

This is the main issue. 


Scheduler can benefit from a rewrite, for sure: e.g. now it's just a web2py shell executing all over again whatever the code is. Pros: you can change all your app and never stop the scheduler, cons: slower in production. Other implementations load everything at the start and execute only the function (faster). The main example is, you have to send an email. You could easily skip large chunks of code of your app just to send that email, and currently is faster to run the Scheduler in "external" mode than having all modules reloaded for every mail you'd like to send. However, keeping execution environment "separated" and pass all the objects in multiprocessing.Queues can be painful (i.e., db connections objects)
- new routing: I like the flexibility right now, but the code is cumbersome, and users just can't face it (don't know why, django urls are not that different). Don't know if flask syntax is more understandable, but it seems to grip better into users heads (not in mine, though, I'll learn )
- I like 8) a lot. All the "moves" between sqlhtml, validators, dal, html is hard to understand. I got a grip on that just after a few rounds of debugging. Idea to consider: a simple way to code widgets, with html, js, validation, presentation layers separated but with a "linear" flow logic. I recently had to come up with handling different timezones and had to write more code than I expected: then I realized it was better to pack it as a plugin, then I quit the idea because web2py plugins are really not first class citizens... I don't think that currently many users watch them
- 12) in multiprocessing environments cache.disk is the only "embedded" cache a user can lean towards. I use redis a lot and I'm happy with that, but memcache and redis are not really spread amongst Windows developers.
- adding a remote 17)... supporting nosql dbs is hard, but current implementation lacks for any hope of stability in the future (needs more testers without doubts). Any "internal" module (auth, scheduler, wiki, sessions) needs some "hook" to intercept store/retrieve in a de-normalized form. If we're going to rewrite them from scratch (in Italy we say, "fatto 30, facciamo 31", in English I think is "in for a penny, in for a pound"), we should avoid implicit foreign keys
- a remote 18) that is a wish for DAL: support for nested transactions (or contexts, we're talking about 2.7 to 3.0, where context are available for a while). I found contexts in python really useful for handling transactions.... Right now to be safe a developer is "forced" to do a db.commit() to save the current status, update/insert/delete and then again db.commit().


Mariano Reingart

unread,
Oct 25, 2012, 10:13:30 PM10/25/12
to web2py-d...@googlegroups.com
On Thu, Oct 25, 2012 at 1:21 PM, Massimo DiPierro
<massimo....@gmail.com> wrote:
> Hello everybody,
>
> Time to start thinking about web3py. And as I think about it, I hear this in
> my mind: http://www.youtube.com/watch?v=HofoK_QQxGc
>
> Here is what I have in mind and I am building a prototype:
>
> 1) eliminate every computation done when the request arrives (for example
> parsing of url, headers, accept_languages, etc) and make everything lazy.

web2py will be a microframework?


> 2) eliminate request, response, session, T, cache objects and have a single
> "current" object. It should not be a thread local to avoid locking issues,
> but it should be passed around.

nice but "current" wording can be misleading, why not a more explicit
request and response?
(like now...)

> 3) eliminate all the exec/execfiles but preserve some version of the current
> multi-applications folder structures. Each app will have an entry point
> (main.py) where actions are registered to allow a freer structure at the app
> level.

how will it manage "in-the-fly" modifications?

are you saying good by to the online IDE?

that kind of dynamism is what makes web2py different and easy to learn/use

> 4) A global routes to configure "only" application prefixes. Then each app
> is in charge of its own internal routing, perhaps using decorators as Flask
> does.

this will be hard to explain to beginners and can cause a mess in the
app structure

a standard directory layout about where to put each file (MVC) is a
good approach now

yes, it is rigid, but it works for the majority of webapps made with web2py

> 5) No more global environment. Apps will do "from web3py import *" (see
> below)

for the permormance perspective, this is equal or worse than the
current approach

also, import * should be discouraged as it chokes IDE and static checkers

a shorter namespace would be useful, like

import w3p

db = w3p.DAL(....)

or better:

from web2py import dal

db = dal.DAL(
dal.Field('foo', dal.STRING,
requires=dal.IS_IN_SET([])
)

(btw, I think this is currently achievable nowadays)

> 6) No more file-system write unless to install/edit apps. We store session
> in cookies (if small) or db (if large) and tickets always in db (so they can
> be searched and cleared more easily). They will not go in the app db but in
> a dedicated common ticket db.

nice, but a better fine-grained ticket system should be necessary to
further enhance performance and avoid some pitfalls in production
(a bunch of tickets without enough detail was not very useful in my experience)

in development, a more interactive inspection would be better than
storing the error snapshot, like paste.evalexception.middleware

http://tools.cherrypy.org/wiki/PasteEvalException

> 7) Remove all the helpers except TAG, CAT, XML, BEAUTIFY, MENU and renamed
> them tag, cat, safe, beautify, menu (with simpler logic)

why not a namespace?

from web2py import tag

tag.DIV

or better:

from web2py.tag import A, IMG, ...

IMHO helpers are part of web2py differentiation for easiness, why
remove or complicate them?

> 8) Replace SQLFORM.factory with Form and SQLFORM with DALForm. They will
> have a simpler logic that separates the HTML representation from the input
> processing, thus it will be faster and will be able to handle non-html form
> submission (ajax, xml) while using a syntax similar to the current one.

good
again, namespaces maybe useful here
why this cannot be backported to web2py?

> 9) No more appadmin. There should be a global application to manage the
> databases of all the apps.

good

> 10) should work with python 2.7 and 3.x

bad, wasn't web3py py3k only?
having two active versions could create confusion in users, be a
maintenance nightmare and will not enable to take some advantages from
python 3 new features
or are you planing to discontinue web2py?

> 11) A compatibility layer that will allow to run most existing web2py
> applications when using python 2.7.

ah, ok, you are planning to discontinue web2py...

IMHO they should be two different projects to avoid any misunderstanding

> 12) no more cache.disk but better cache.ram (renamed as cache) with actual
> expiration (current cache.ram only expires when accessed).

cache.disk is useful in some places, surely I'm missing why this is evil

> 13) automatic reverse routing
> 14) keep only dal.py and template.py from existing web2py.

why not separate them as proper packages?
what about gluino?

> I have most of this done and the code is smaller and cleaner than current
> web2py is. If is 20x faster on hello world. The compatibility layer is
> imperfect as I am still working on BEAUTIFY, MENU, sessions, db
> transactions, and a many nasty details. Tools.py (auth) is missing.
>
> Here is an example of code I got working:
>
> <begin file apps/welcome/main.py>
> import time
> from web3py import DAL, Field, expose, Form, DALForm, IS_NOT_EMPTY, HTTP,
> tag
>
> db = DAL('sqlite://storage.test')
> db.define_table('person',Field('name',requires=IS_NOT_EMPTY()))
>
> @expose() # /appname/index

decorators, although not an easy thing to explain to beginners, may be
a good idea, but sometimes they are hard to understan/follow
if this simple case should be explained, it may be not so good
explicit is better than implicit ;-)
maybe they could have more info about URL, or are they limited to
function names?

> def index(current):
> form = Form(Field('name',requires=IS_NOT_EMPTY())).process(current)
> if form.submitted and form.processed and form.accepted: print form.vars
> elif form.submitted: print form.errors
> message = form.vars.name or ''
> return locals()
>
> @expose('/test_args/<str:a>(/<int:b>)?') # /appname/test_args/hello/1
> def test_args(current,a,b=9): # args are parsed, validated and passed to
> action

arggh, regex or whatever it looks like... :(

why not use python 3 annotations?

def test_args(current,a: str, b: int=9):

http://www.python.org/dev/peps/pep-3107/

> return repr(dict(a=a,b=b))
>
> @expose(template='index.html')
> def test_form(current):
> form = DALForm(db.person, record_id = None).process(current)
> message = tag.div(tag.h1(form.vars.name))

div is a function, method or a class here?

it should be Div() ?
why not DIV ?

anyway, we'll suffer from explicit boilerplate code :-)

> return locals()
>
> @expose(cache_expire=10)
> def timer(current):
> return time.ctime()
>
> @expose()
> def try_redirect(current):
> HTTP.redirect(current.url('index')) # does reverse routing
> <end>
>
>
> As you can see the syntax is similar for an easier transition. To benefit
> from speed apps will have to be rewritten.
>
> Legacy web2py welcome app, without tools and session also works. I will add
> tools and session support asap.
>
> The error tickets also need work but I just discovered inspect.stack and had
> an epiphany!
>
> Am I on the right track?

maybe from the pythonic standpoint, but who will care?
we'll become a common framework like everyone else
why will anyone use web3py instead flask+sqlachemy, django or whatever?

> Want to help?

no, as I say in earlier mails, I don't see the point here, so sorry

> Comments suggestions?

IMHO stick with web2py, split it in standalones modules and add
alternative implementations with current structure, so it can be
comparable in minimal benchmarks :-)

> What can we do better?

enhance web2py to a better 3.0 version

> When should apps be reloaded (since they are no longer executed we must trigger reload)?

you can't, you'll have to restart the whole server

reload is one of the missing pieces in py3k

> What would you like to see changed?

not much, I'm quite happy with web2py
I think we should have more complex app to do a deeper analysis

I've learned a lot of things this year with web2conf for PyCon
Argentina 2012 conference, it is "advanced" as Pinax Symposion (and
even it may have more features), and still it is simple (in the web2py
sense), easy to understand and it could be optimized to reach
acceptable performance without compromising dynamism.

> Should we preserve validators as they are?

we should preserve as most as possible to not break backward compatibility

> Should we have a built-in realtime event handling for websockets in every
> app? What do we do with the LOAD helper? what do we do with web2py.js? Do we
> still need response.files of should we make more use of template {{block…}}
> to inject stuff in <head/>? Should we backport some of new stuff (Form,
> cache, helpers) to web2py?

yes to them all (excepting the complex {{block}} thing and the
realtime event handling)

> If we take this route, we need to have a permanent feature freeze on web2py,
> except on DAL and Template which are the only shared modules.

this is the part I dislike
I think we can keep improving web2py and make alternatives or further
works possible

> I will share my code next week, as soon as I get sessions to work.
> You will appreciate the clarity compare with the current web2py. In
> particular the helpers and form handling logic.

IMHO that we should clean up the code is not a reason to start a
massive rewrite from scratch.
I argued in previous discussions that this seems like the second
system effect (and according the literature, it has not been good for
any project...):

http://en.wikipedia.org/wiki/Second-system_effect

This is a nice article explaining why:

http://www.joelonsoftware.com/articles/fog0000000069.html

For example, DAL has more important problems like the id/keyed table
and could be optimized in the id/sequence/autonumeric handling, bool
logic, left joins, etc.
Fixed that it should be taken more seriously to be used in standalone,
gui and other web applications/frameworks.
They are in my TODO list, or wish list as this year I didn't have enough time :(

About the odd benchmark, please read the discussion in django-dev about this.

For me, just because web2py isn't faster that ever
minimicrosmallpythonframework out in the wild is not a an enough
excuse for a big re-design.
That's why I think gluino idea was amazing, if we could take the best
of both worlds (full-stack and micro-framework), why not?

Sorry for my lack of optimism

Best regards

Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com

Niphlod

unread,
Oct 25, 2012, 11:30:17 PM10/25/12
to web2py-d...@googlegroups.com, mar...@tecnodoc.com.ar
have you actually tried it ? that seems to be an easy wrapper to set for cache headers, not a cache of the application values ^_^


On Thursday, October 25, 2012 11:49:39 PM UTC+2, Martin.Mulone wrote:
About cache I like the way of wheezy.web in CacheProfile. http://packages.python.org/wheezy.http/userguide.html#cache-profile

--
http://martinmulone.com.ar

Massimo DiPierro

unread,
Oct 25, 2012, 11:33:14 PM10/25/12
to web2py-d...@googlegroups.com
On Oct 25, 2012, at 9:13 PM, Mariano Reingart wrote:

On Thu, Oct 25, 2012 at 1:21 PM, Massimo DiPierro
<massimo....@gmail.com> wrote:
Hello everybody,

Time to start thinking about web3py. And as I think about it, I hear this in
my mind: http://www.youtube.com/watch?v=HofoK_QQxGc

Here is what I have in mind and I am building a prototype:

1) eliminate every computation done when the request arrives (for example
parsing of url, headers, accept_languages, etc) and make everything lazy.

web2py will be a microframework?

No. Just faster. Also smaller because better code.



2) eliminate request, response, session, T, cache objects and have a single
"current" object. It should not be a thread local to avoid locking issues,
but it should be passed around.

nice but "current" wording can be misleading, why not a more explicit
request and response?
(like now…)

There will be a compatibility layer so you can still use it as now.
I just think there should be a single thread local. Anther issue is to minimize the number of objects being created for speed.



3) eliminate all the exec/execfiles but preserve some version of the current
multi-applications folder structures. Each app will have an entry point
(main.py) where actions are registered to allow a freer structure at the app
level.

how will it manage "in-the-fly" modifications?

Not sure hot to implement it. But it will work as today from user prospective.

are you saying good by to the online IDE?

Not at all but I think it should be simpler.

that kind of dynamism is what makes web2py different and easy to learn/use

I agree. We will keep it. But there are some hurdles.


4) A global routes to configure "only" application prefixes. Then each app
is in charge of its own internal routing, perhaps using decorators as Flask
does.

this will be hard to explain to beginners and can cause a mess in the
app structure

a standard directory layout about where to put each file (MVC) is a
good approach now

yes, it is rigid, but it works for the majority of webapps made with web2py

This is what I have in mind:

apps/
  myapp/
     main.py # entry point for some app all they need
     templates/
     static/
     databases/
     uploads/
     languages/
     plugins/
     modules/

Legacy apps will also have

    controllers/
    views/

This is negotiable.


5) No more global environment. Apps will do "from web3py import *" (see
below)

for the permormance perspective, this is equal or worse than the
current approach

also, import * should be discouraged as it chokes IDE and static checkers

a shorter namespace would be useful, like

import w3p

db = w3p.DAL(....)

or better:

from web2py import dal

db = dal.DAL(
      dal.Field('foo', dal.STRING,
                   requires=dal.IS_IN_SET([])
      )

(btw, I think this is currently achievable nowadays)

I agree. It was just an example.


6) No more file-system write unless to install/edit apps. We store session
in cookies (if small) or db (if large) and tickets always in db (so they can
be searched and cleared more easily). They will not go in the app db but in
a dedicated common ticket db.

nice, but a better fine-grained ticket system should be necessary to
further enhance performance and avoid some pitfalls in production
(a bunch of tickets without enough detail was not very useful in my experience)

in development, a more interactive inspection would be better than
storing the error snapshot, like paste.evalexception.middleware

http://tools.cherrypy.org/wiki/PasteEvalException

7) Remove all the helpers except TAG, CAT, XML, BEAUTIFY, MENU and renamed
them tag, cat, safe,  beautify, menu (with simpler logic)

why not a namespace?

from web2py import tag

tag.DIV

or better:

from web2py.tag import A, IMG, ...

IMHO helpers are part of web2py differentiation for easiness, why
remove or complicate them?

Because we do not need them. tag would generate them on demand.


8) Replace SQLFORM.factory with Form and SQLFORM with DALForm. They will
have a simpler logic that separates the HTML representation from the input
processing, thus it will be faster and will be able to handle non-html form
submission (ajax, xml) while using a syntax similar to the current one.

good
again, namespaces maybe useful here
why this cannot be backported to web2py?

9) No more appadmin. There should be a global application to  manage the
databases of all the apps.

good

10) should work with python 2.7 and 3.x

bad, wasn't web3py py3k only?
having two active versions could create confusion in users, be a
maintenance nightmare and will not enable to take some advantages from
python 3 new features
or are you planing to discontinue web2py?

I am planning to freeze web2py. web3py should have one version that works on both 2.7 and 3.x. On 2.7 will run also the compatibility layer.


11) A compatibility layer that will allow to run most existing web2py
applications when using python 2.7.

ah, ok, you are planning to discontinue web2py...

IMHO they should be two different projects to avoid any misunderstanding

12) no more cache.disk but better cache.ram (renamed as cache) with actual
expiration (current cache.ram only expires when accessed).

cache.disk is useful in some places, surely I'm missing why this is evil

I agree. We can have is an option along with memcache and redis. I would just not make it primary citizen as cache.ram.


13) automatic reverse routing
14) keep only dal.py and template.py from existing web2py.

why not separate them as proper packages?

That is a distribution issue. We can distribute them separately as well as combined. I still want the binary versions.

what about gluino?

Nobody uses it. Has no reason to exist. web3py should be usable as gluino.
I had not thought about it. It would rule out 2.7. I am not sure we can handle it.


   return repr(dict(a=a,b=b))

@expose(template='index.html')
def test_form(current):
   form = DALForm(db.person, record_id = None).process(current)
   message = tag.div(tag.h1(form.vars.name))

div is a function, method or a class here?
it should be Div() ?
why not DIV ?

tag.ANYTHING is <ANYTHING….>. tag is the same as web2py TAG.
You can make tag.Div, tag.DIV, tag.diV, etc.

anyway, we'll suffer from explicit boilerplate code :-)

   return locals()

@expose(cache_expire=10)
def timer(current):
   return time.ctime()

@expose()
def try_redirect(current):
   HTTP.redirect(current.url('index')) # does reverse routing
<end>


As you can see the syntax is similar for an easier transition. To benefit
from speed apps will have to be rewritten.

Legacy web2py welcome app, without tools and session also works. I will add
tools and session support asap.

The error tickets also need work but I just discovered inspect.stack and had
an epiphany!

Am I on the right track?

maybe from the pythonic standpoint, but who will care?
we'll become a common framework like everyone else
why will anyone use web3py instead flask+sqlachemy, django or whatever?

I think it will still look a lot like web2py but faster.


Want to help?

no, as I say in earlier mails, I don't see the point here, so sorry

Comments suggestions?

IMHO stick with web2py, split it in standalones modules and add
alternative implementations with current structure, so it can be
comparable in minimal benchmarks :-)

What can we do better?

enhance web2py to a better 3.0 version

You vision is not much different than mine. I want to simply the source code. It is messy.
Moreover there are so many caveats that it is getting hard to teach.
I also learned many new Python APIs that web2py is not taking advantage of.

In my view the final product should be 95% compatible with current web2py. We have to break backward compatibility anyway to go to python 3.x. 

When should apps be reloaded (since they are no longer executed we must trigger reload)?

you can't, you'll have to restart the whole server

reload is one of the missing pieces in py3k



What would you like to see changed?

not much, I'm quite happy with web2py
I think we should have more complex app to do a deeper analysis

I've learned a lot of things this year with web2conf for PyCon
Argentina 2012 conference, it is "advanced" as Pinax Symposion (and
even it may have more features), and still it is simple (in the web2py
sense), easy to understand and it could be optimized to reach
acceptable performance without compromising dynamism.


Should we preserve validators as they are?

we should preserve as most as possible to not break backward compatibility

Should we have a built-in realtime event handling for websockets in every
app? What do we do with the LOAD helper? what do we do with web2py.js? Do we
still need response.files of should we make more use of template {{block…}}
to inject stuff in <head/>? Should we backport some of new stuff (Form,
cache, helpers) to web2py?

yes to them all (excepting the complex {{block}} thing and the
realtime event handling)

If we take this route, we need to have a permanent feature freeze on web2py,
except on DAL and Template which are the only shared modules.

this is the part I dislike
I think we can keep improving web2py and make alternatives or further
works possible

I am not saying "the only shared APIs". In the compatibility mode, the APIs will be the same and it will use exec as it does not. The actual modules will be rewritten and organized better.


I will share my code next week, as soon as I get sessions to work.
You will appreciate the clarity compare with the current web2py. In
particular the helpers and form handling logic.

IMHO that we should clean up the code is not a reason to start a
massive rewrite from scratch.
I argued in previous discussions that this seems like the second
system effect (and according the literature, it has not been good for
any project...):

http://en.wikipedia.org/wiki/Second-system_effect

Excellent point.

This is a nice article explaining why:

http://www.joelonsoftware.com/articles/fog0000000069.html

For example, DAL has more important problems like the id/keyed table
and could be optimized in the id/sequence/autonumeric handling, bool
logic, left joins, etc.
Fixed that it should be taken more seriously to be used in standalone,
gui and other web applications/frameworks.
They are in my TODO list, or wish list as this year I didn't have enough time :(

About the odd benchmark, please read the discussion in django-dev about this.

For me, just because web2py isn't faster that ever
minimicrosmallpythonframework out in the wild is not a an enough
excuse for a big re-design.
That's why I think gluino idea was amazing, if we could take the best
of both worlds (full-stack and micro-framework), why not?

Sorry for my lack of optimism

I understand the skepticism. I am glad we will be able to talk more about this in person in two weeks.

Massimo DiPierro

unread,
Oct 25, 2012, 11:37:23 PM10/25/12
to web2py-d...@googlegroups.com, mar...@tecnodoc.com.ar
I am leaning toward a design like this:


<begin>
import inspect

class Cleaner(object):
def at_start(self): pass
def at_success(self): pass
def at_failure(self): pass

class expose(object):
def __init__(self, cleaners=[]):
self.cleaners = cleaners
def __call__(self, f):
def wrap(f, cleaner):
def g(*a,**b):
try:
cleaner.at_start()
output = f(*a,**b)
cleaner.at_success()
return output
except:
cleaner.at_failure()
raise
return g
for cleaner in self.cleaners:
if isinstance(cleaner,Cleaner):
print 'wrapping cleaner'
f = wrap(f, cleaner)
return f

### user code

class Database(Cleaner):
def __init__(self): print 'connecting'
def at_start(self): print 'pool connection'
def at_success(self): print 'commit'
def at_failure(self): print 'rollback'
def insert(self,**data): print 'inserting %s' % data

db1 = Database()
db2 = Database()
db3 = Database()

@expose(cleaners=(db1,db2))
def action(x):
db1.insert(key=1/x)
return

def rec_pickle(obj):
try:
return = pickle.dumps(obj)
else:

try:
a = action(1)
a = action(1)
a = action(0)
except:
# log error
pass
</end>


The critical point is this:

@expose(cleaners=(db1,db2))
def action(x): ….

The cleaners are objects like db (which needs to be checked, reset, committed and rolled back) or sessions (which also needs to be checked, saved, etc.

developers will be able to register common cleaners (all actions need db, all actions need session) or per action (this action needs a db connection or a redis connection).



Massimo DiPierro

unread,
Oct 26, 2012, 12:02:36 AM10/26/12
to web2py-d...@googlegroups.com
How about this design:

import inspect

class Error(object):
    def __init__(self,stack):
        self.stack = stack

def expose(f):
    def h(f,obj):
        def e(*a,**b):
            try:
                obj.at_start()
                output = f(*a,**b)
                obj.at_success()
            except:
                obj.at_failure()
                output = Error(inspect.stack)
            obj.at_end()
            return output
        return e
    for obj in f.func_globals.itervalues():
        if isinstance(obj,Clean):
            f=h(f,obj)
    return f

class Clean(object):
    def at_start(self): pass
    def at_success(self): pass
    def at_failure(self): pass
    def at_end(self): pass

class Database(Clean):
    def __init__(self): print 'connecting'
    def at_start(self): print 'pool connection'
    def at_success(self): print 'commit'
    def at_failure(self): print 'rollback'
    def at_end(self): print 'recycle connection'
    def insert(self,**data): print 'inserting %s' % data

db = Database()

@expose
def action(x):
    db.insert(key=1/x)
    return

a = action(1)
a = action(1)
a = action(0)
print isinstance(a,Error)



On Oct 25, 2012, at 2:02 PM, Niphlod wrote:

szimszon

unread,
Oct 26, 2012, 3:25:54 AM10/26/12
to web2py-d...@googlegroups.com, mar...@tecnodoc.com.ar
Just one thing, I do  not know the py3k but in py2k the code like


class expose(object):
   
def __init__(self, cleaners=[]):
       
self.cleaners = cleaners


can be really nasty. cleaners=[] is computed only one time. So all expose instances have cleaners pointing to the some address. If there is a goal than ok.

I'm not a big python programmer just had a bad day once with this problem :-o

I used to do

class expose(object):
   
def __init__(self, cleaners=None):
       
if not cleaners:
           
self.cleaners = []


now.

Jonathan Lundell

unread,
Oct 26, 2012, 9:37:45 AM10/26/12
to web2py-d...@googlegroups.com
On 26 Oct 2012, at 12:25 AM, szimszon <szim...@gmail.com> wrote:
Just one thing, I do  not know the py3k but in py2k the code like

class expose(object): 
    def __init__(self, cleaners=[]): 
        self.cleaners = cleaners 


can be really nasty. cleaners=[] is computed only one time. So all expose instances have cleaners pointing to the some address. If there is a goal than ok.

The problem isn't so much that they're pointing to the same object as that the object they're pointing to is mutable. If an expose object mutates its self.cleaners, they all get mutated (well, the ones that aren't otherwise initialized, anyway).


I'm not a big python programmer just had a bad day once with this problem :-o

I used to do

class expose(object): 
    def __init__(self, cleaners=None):
        if not cleaners: 
            self.cleaners = [] 


now.

self.cleaners = [] if cleaners is None else cleaners

or

self.cleaners = cleaners or []

szimszon

unread,
Oct 26, 2012, 10:58:39 AM10/26/12
to web2py-d...@googlegroups.com


2012. október 26., péntek 15:38:06 UTC+2 időpontban Jonathan Lundell a következőt írta:
On 26 Oct 2012, at 12:25 AM, szimszon <szim...@gmail.com> wrote:
Just one thing, I do  not know the py3k but in py2k the code like

class expose(object): 
    def __init__(self, cleaners=[]): 
        self.cleaners = cleaners 


can be really nasty. cleaners=[] is computed only one time. So all expose instances have cleaners pointing to the some address. If there is a goal than ok.

The problem isn't so much that they're pointing to the same object as that the object they're pointing to is mutable. If an expose object mutates its self.cleaners, they all get mutated (well, the ones that aren't otherwise initialized, anyway).

Right.

I'm not a big python programmer just had a bad day once with this problem :-o

I used to do

class expose(object): 
    def __init__(self, cleaners=None):
        if not cleaners: 
            self.cleaners = [] 


now.

self.cleaners = [] if cleaners is None else cleaners

or

self.cleaners = cleaners or []

+1

Mariano Reingart

unread,
Oct 26, 2012, 11:27:06 AM10/26/12
to web2py-d...@googlegroups.com
On Fri, Oct 26, 2012 at 12:33 AM, Massimo DiPierro
<massimo....@gmail.com> wrote:
>
> On Oct 25, 2012, at 9:13 PM, Mariano Reingart wrote:
>
> On Thu, Oct 25, 2012 at 1:21 PM, Massimo DiPierro
> <massimo....@gmail.com> wrote:
>
> Hello everybody,
>
>
> Time to start thinking about web3py. And as I think about it, I hear this in
>
> my mind: http://www.youtube.com/watch?v=HofoK_QQxGc
>
>
> Here is what I have in mind and I am building a prototype:
>
>
> 1) eliminate every computation done when the request arrives (for example
>
> parsing of url, headers, accept_languages, etc) and make everything lazy.
>
>
> web2py will be a microframework?
>
>
> No. Just faster. Also smaller because better code.
>
>
>
> 2) eliminate request, response, session, T, cache objects and have a single
>
> "current" object. It should not be a thread local to avoid locking issues,
>
> but it should be passed around.
>
>
> nice but "current" wording can be misleading, why not a more explicit
> request and response?
> (like now…)
>
>
> There will be a compatibility layer so you can still use it as now.
> I just think there should be a single thread local. Anther issue is to
> minimize the number of objects being created for speed.

Does it make any difference if its create 1 object instead of three?
It should be faster if we use python dicts directly (or namedtuples)

This could speed up a micro-benchmark, but I doubt it would be helpful
in a real-world app.

>
> 3) eliminate all the exec/execfiles but preserve some version of the current
>
> multi-applications folder structures. Each app will have an entry point
>
> (main.py) where actions are registered to allow a freer structure at the app
>
> level.
>
>
> how will it manage "in-the-fly" modifications?
>
>
> Not sure hot to implement it. But it will work as today from user
> prospective.

I don't know if it would be possible without restarting the whole
server as django and others do when they detect a file modification.

Using decorators and removing exec will have a side-effect that apps
would have to be initialized, maintaining some state/settings in the
imported modules.
This will not be easy to follow and a minimal syntax error could down
the entire application or have secondary nasty effects (i.e. weird
import errors)

I doubt it will work like today from the user perpective, I think that
it would require a more standard python "command-line" approach

> are you saying good by to the online IDE?
>
>
> Not at all but I think it should be simpler.
>
> that kind of dynamism is what makes web2py different and easy to learn/use
>
>
> I agree. We will keep it. But there are some hurdles.

Yes, mostly because of python, but we could address the issues instead
of moving to a different approach.

>
> 4) A global routes to configure "only" application prefixes. Then each app
>
> is in charge of its own internal routing, perhaps using decorators as Flask
>
> does.
>
>
> this will be hard to explain to beginners and can cause a mess in the
> app structure
>
> a standard directory layout about where to put each file (MVC) is a
> good approach now
>
> yes, it is rigid, but it works for the majority of webapps made with web2py
>
>
> This is what I have in mind:
>
> apps/
> myapp/
> main.py # entry point for some app all they need
> templates/
> static/
> databases/
> uploads/
> languages/
> plugins/
> modules/
>
> Legacy apps will also have
>
> controllers/
> views/
>
> This is negotiable.
>

Ok

plugins will have a statics folder too?

The main.py approach is similar to what I've been thinking ;-)

You could use almost the same code as today just

- encapsulating the models in a def
- passing around request, response, and so on (or using current)

For example, in main.py:

from controllers.default import index

def main(request, response, session, ...):
return response.render(index(request, response))

in models/db.py

def get_db(request, response, session):
db = DAL(...)
db.define_table("my_table", Field("my_field"))
return db

in controllers/default.py

def index(request, response, session):
from models.db import ANY_OTHER_CONSTANT, my_function, get_db
db = get_db(request, response, session)
return dict(...)


I think this will be a minor cosmetic change compared with the web3py approach.
In fact, most of the changes in models and in controllers (like adding
defs or adding parameters to functions) can be automatically
implemented (like a 2to3 tool)

The other advantage is that we could implement an alternative way if
no main.py is present (the current exec behaviour)
That is, the "compatibilty layer" would be very simple and
straightforward to create (even without the 2to3-like tool, I think)

So, this approach would be useful for advanced user that need
performance, and it will simple enough for begginers
It will be cleaner and pythonic ;-)
we could generate them on demand and importing them as usual
they cause a bit of namespace pollution but it has been nice to work
with tags directly, without too much explanation or OO knowledge

also, creating them on deman will choke with IDE and static checkers,
and I no see a huge improvement in performance
Creating a class on the fly is not faster than just import it
(maybe I'm missing the new implementation...)
but it will not be web2py :-)

>
> Want to help?
>
>
> no, as I say in earlier mails, I don't see the point here, so sorry
>
> Comments suggestions?
>
>
> IMHO stick with web2py, split it in standalones modules and add
> alternative implementations with current structure, so it can be
> comparable in minimal benchmarks :-)
>
> What can we do better?
>
>
> enhance web2py to a better 3.0 version
>
>
> You vision is not much different than mine. I want to simply the source
> code. It is messy.
> Moreover there are so many caveats that it is getting hard to teach.
> I also learned many new Python APIs that web2py is not taking advantage of.

Surely, in fact I do not use nor teach many of the new improvements.
Maybe if we list all the known issues and workaronund it is easier to
get convinced about such big change.

> In my view the final product should be 95% compatible with current web2py.
> We have to break backward compatibility anyway to go to python 3.x.

I don't know, internally there will be many changes (mostly because
unicode), but for the end user most code could be run with minimal
changes

I don't see any new py3k killer feature that worth to justify this
(beside annotations, that are not available in py2)

> When should apps be reloaded (since they are no longer executed we must
> trigger reload)?
>
>
> you can't, you'll have to restart the whole server
>
>
> reload is one of the missing pieces in py3k
>
>
> Are you saying python 3 has no reload?
> http://docs.python.org/dev/py3k/library/imp.html?highlight=reload#imp.reload
>

Python 3 has no reload because it is evil :-)
However, it has been moved to the imp library, and could be emulated
with some python internals (with some care because it has more nasty
secondary effects than exec...)
In fact, AFAIK the import machinery will be / is a pure python
implementation in py3.3 (importlib), you can talk with Brett Cannon in
PyCon Argentina about that ;-)
Me too

BTW, for the web2py sprint, there are around 25 people registered!!!!!!
(right now, more than any other sprint in the conference)

http://ar.pycon.org/2012/schedule

Massimo DiPierro

unread,
Oct 26, 2012, 11:42:20 AM10/26/12
to web2py-d...@googlegroups.com
The fact is web2py uses exec and reload. In fact i uses custom imports for reload. Without it users would not be able to track changes in modules. Lots of people rely on the custom import because of Bruno modelless approach.

I think we can forfeit reload for only reload. What I have in mind is that each app is a package. The package register routes. If when a route in the package is called and any py file in the package has changed reload the entire package.

The compatibility mode would continue use exec.

Massimo

Massimo DiPierro

unread,
Oct 26, 2012, 11:47:27 AM10/26/12
to web2py-d...@googlegroups.com

Agreed. Two more issues:

1) 

Do you like better:
    
    @expose('/index/<str:a>(/<int:b>)?')
    def index(a,b=3)

or this

    @expose('/index)
    def index(a = Arg('str'), b = Arg('int',default=3))

we cannot use the python 3 notation because it would not work with 2.7 and it not reach enough to specify validators for args.

2)
 
The cleaners in my previous examples are like wsgi middlewere but they see current instead of environ. Is that a good idea?
Should we allow the use of any existing WSGI middleware a the expense of syntax from creating new middleware?

So:

    class Example(Cleaner):
        def __init__(self): print 'init'
        def on_start(self): print 'start' # can access surrent
        def on_success(self): print 'success'
        def on_failure(self): print 'failure' 

    @expose(cleaners = [Example()])
    def index()

or

    class Example(Middleware):
        def __init__(self, app): 
             self.app = app
             print 'init'
        def __call__(self, environ,start_response):
              try:
                    print 'start' # no access to current
                    output = self.app( environ,start_response)
                    print 'success'
              except:
                    print 'failure'
                    raise 

    @expose(middleware = [Example()])
    def index()

The latter can use any existing middleware and is more general but syntax is terrible in my view.

Massimo



Martín Mulone

unread,
Oct 26, 2012, 12:05:53 PM10/26/12
to web2py-d...@googlegroups.com
@expose('/index/<str:a>(/<int:b>)?')
    def index(a,b=3)
this one

2012/10/26 Massimo DiPierro <massimo....@gmail.com>



--
http://martinmulone.com.ar

Jonathan Lundell

unread,
Oct 26, 2012, 12:06:17 PM10/26/12
to web2py-d...@googlegroups.com
On 26 Oct 2012, at 8:47 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
Do you like better:
    
    @expose('/index/<str:a>(/<int:b>)?')
    def index(a,b=3)

or this

    @expose('/index)
    def index(a = Arg('str'), b = Arg('int',default=3))

we cannot use the python 3 notation because it would not work with 2.7 and it not reach enough to specify validators for args.


The second version seems more flexible.

Massimo DiPierro

unread,
Oct 26, 2012, 12:26:40 PM10/26/12
to web2py-d...@googlegroups.com
The first version is similar to Flask. It allows arbitrary URL expressions using regex.

The second version is limited in the sense that you cannot specify regular expressions and therefore to handle some off legacy URL you would not be able to re-route it. 

The second could allow specify validators for the Args(…,requires).

None is perfect. For now I have the former implemented, the one Martin likes.

Massimo DiPierro

unread,
Oct 26, 2012, 12:33:44 PM10/26/12
to web2py-d...@googlegroups.com
On a second thought…. why not both?


On Oct 26, 2012, at 11:06 AM, Jonathan Lundell wrote:

Jonathan Lundell

unread,
Oct 26, 2012, 12:48:16 PM10/26/12
to web2py-d...@googlegroups.com
On 26 Oct 2012, at 9:26 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
> The first version is similar to Flask. It allows arbitrary URL expressions using regex.
>
> The second version is limited in the sense that you cannot specify regular expressions and therefore to handle some off legacy URL you would not be able to re-route it.
>
> The second could allow specify validators for the Args(…,requires).
>
> None is perfect. For now I have the former implemented, the one Martin likes.

I overlooked part of the logic in claiming flexibility for the second version. But sure, why not both, if that works?

I'm all for regexes, but I'm also aware that they sow lots of confusion among users. One of the aspects of the current regex routing that bothers me is the treatment of trailing slashes. Do they get rstrip'd before the match?

Unless I'm missing something, it seems like option 1 would be a lot easier to collect into a single routing table. And speaking of routing tables, what determines the order of evaluation? And is there a syntax for lists of arguments? How would the equivalent of request.args be handled for variable number of args?

Massimo DiPierro

unread,
Oct 26, 2012, 12:57:13 PM10/26/12
to web2py-d...@googlegroups.com

On Oct 26, 2012, at 11:48 AM, Jonathan Lundell wrote:

> On 26 Oct 2012, at 9:26 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
>> The first version is similar to Flask. It allows arbitrary URL expressions using regex.
>>
>> The second version is limited in the sense that you cannot specify regular expressions and therefore to handle some off legacy URL you would not be able to re-route it.
>>
>> The second could allow specify validators for the Args(…,requires).
>>
>> None is perfect. For now I have the former implemented, the one Martin likes.
>
> I overlooked part of the logic in claiming flexibility for the second version. But sure, why not both, if that works?
>
> I'm all for regexes, but I'm also aware that they sow lots of confusion among users. One of the aspects of the current regex routing that bothers me is the treatment of trailing slashes. Do they get rstrip'd before the match?

it is not obvious to me how this should be done. I will post my code soon and you can fix it.

>
> Unless I'm missing something, it seems like option 1 would be a lot easier to collect into a single routing table. And speaking of routing tables, what determines the order of evaluation? And is there a syntax for lists of arguments? How would the equivalent of request.args be handled for variable number of args?

Some arges should be optional but do we want arbitrary number of args?

As it is now this case would be handled byt the action from the raw path_info.

>
>
>>
>> On Oct 26, 2012, at 11:06 AM, Jonathan Lundell wrote:
>>
>>> On 26 Oct 2012, at 8:47 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
>>>> Do you like better:
>>>>
>>>> @expose('/index/<str:a>(/<int:b>)?')
>>>> def index(a,b=3)
>>>>
>>>> or this
>>>>
>>>> @expose('/index)
>>>> def index(a = Arg('str'), b = Arg('int',default=3))
>>>>
>>>> we cannot use the python 3 notation because it would not work with 2.7 and it not reach enough to specify validators for args.
>>>>
>>>
>>> The second version seems more flexible.
>>>
>>
>
>

Mariano Reingart

unread,
Oct 26, 2012, 1:08:16 PM10/26/12
to web2py-d...@googlegroups.com
On Fri, Oct 26, 2012 at 12:42 PM, Massimo DiPierro
<massimo....@gmail.com> wrote:
> The fact is web2py uses exec and reload. In fact i uses custom imports for reload. Without it users would not be able to track changes in modules. Lots of people rely on the custom import because of Bruno modelless approach.
>
> I think we can forfeit reload for only reload. What I have in mind is that each app is a package. The package register routes. If when a route in the package is called and any py file in the package has changed reload the entire package.

Yes, and custom import has some glitches too

In particular, I don't see anything wrong with normal python module
just of functions, but you'll see side effects if the module has
classes and do any other initialization (like registering routes).
This could be ever dangerous if the module is a package, as it has to
reload a lot of subsidiary modules (that, by the way, will not happen
unless you tweak the custom import even further)

The downside is that if you reload, the previous instances are not
deleted properly and references are not rebound automatically (so they
may be floating around a long time, it can cause memory leaks and
wrong behaviour with other python modules like pickle)

I think that's why django restarts the whole server instead of
reloading (IIRC, they had this problem before, and they didn't found a
clever workaround, so I doubt we will)

> The compatibility mode would continue use exec.

And so import uses exec internally too ;-)

The exec glitches are well known and more manageable IIRC

Best regards,

Jonathan Lundell

unread,
Oct 26, 2012, 1:10:30 PM10/26/12
to web2py-d...@googlegroups.com
On 26 Oct 2012, at 9:57 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
>
> On Oct 26, 2012, at 11:48 AM, Jonathan Lundell wrote:
>
>> On 26 Oct 2012, at 9:26 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
>>> The first version is similar to Flask. It allows arbitrary URL expressions using regex.
>>>
>>> The second version is limited in the sense that you cannot specify regular expressions and therefore to handle some off legacy URL you would not be able to re-route it.
>>>
>>> The second could allow specify validators for the Args(…,requires).
>>>
>>> None is perfect. For now I have the former implemented, the one Martin likes.
>>
>> I overlooked part of the logic in claiming flexibility for the second version. But sure, why not both, if that works?
>>
>> I'm all for regexes, but I'm also aware that they sow lots of confusion among users. One of the aspects of the current regex routing that bothers me is the treatment of trailing slashes. Do they get rstrip'd before the match?
>
> it is not obvious to me how this should be done. I will post my code soon and you can fix it.
>
>>
>> Unless I'm missing something, it seems like option 1 would be a lot easier to collect into a single routing table. And speaking of routing tables, what determines the order of evaluation? And is there a syntax for lists of arguments? How would the equivalent of request.args be handled for variable number of args?
>
> Some arges should be optional but do we want arbitrary number of args?

We have an arbitrary number of args now: we just split the args part of the url into request.args.

How about adopting the Python *args notation somehow? Aside from the fact that it conflicts with regex notation, that is...

Mariano Reingart

unread,
Oct 26, 2012, 2:05:37 PM10/26/12
to web2py-d...@googlegroups.com
+1 to *args **kwargs notation!
(but if we pass request I doubt what the usefulness will be)

-1 to Args(…,requires), I would prefer explicit checks and actions, a
mutable default could be dangerous, hard to explain/implement, and it
will proliferate custom validators

-10 to regex or custom notation, it is not easy at all (advanced users
could use them, but being mandatory may raise some
concerns/difficulties on newbies)

Best regards

Massimo DiPierro

unread,
Oct 26, 2012, 2:15:50 PM10/26/12
to web2py-d...@googlegroups.com
So...

@expose('….')
def index(*args, **kwargs)

wat should be in '…'. The point is that we still want to specify the url in, how to map into args and kwargs, and possibly filtering the args.

Massimo DiPierro

unread,
Oct 26, 2012, 3:09:52 PM10/26/12
to web2py-d...@googlegroups.com
What do people think about Mariano's proposal of 3 thread locals (request, response, session)? We also need to pass T (a fourth thread local?) and what if the developers wants to pass around additional variables?

I like the idea of one (let's say "current") but now session.counter becomes current.session.counter. Moreover people will attempt to do

session = current.session

and they will run into threading issues.

Any way out?

On Oct 26, 2012, at 1:05 PM, Mariano Reingart wrote:

szimszon

unread,
Oct 26, 2012, 3:44:27 PM10/26/12
to web2py-d...@googlegroups.com
What about to trim current to cur? :) or to c...

Massimo DiPierro

unread,
Oct 26, 2012, 3:45:36 PM10/26/12
to web2py-d...@googlegroups.com
from current import current as c

Jonathan Lundell

unread,
Oct 26, 2012, 4:06:41 PM10/26/12
to web2py-d...@googlegroups.com
On 26 Oct 2012, at 12:09 PM, Massimo DiPierro <massimo....@gmail.com> wrote:
>
> What do people think about Mariano's proposal of 3 thread locals (request, response, session)? We also need to pass T (a fourth thread local?) and what if the developers wants to pass around additional variables?
>
> I like the idea of one (let's say "current") but now session.counter becomes current.session.counter. Moreover people will attempt to do
>
> session = current.session
>
> and they will run into threading issues.
>
> Any way out?

I like the idea of one as well. But I think we could separate it into lifetime-related objects. request would contain everything that's specific to the current request (and that includes response!); session is another object with a longer lifetime (granted, part of its life is in suspended animation, but it's still the same object); and maybe one for the application itself.

Rolling now...the application-life object would contain some indication of a restart. A detail, but convenient. Some kind of backing store, but an indicator for first-run. (The problem with the application object is that it'd have to be locked, and that would be expensive in a multiprocess environment.)

Finally, we'd have a convention for an app-local namespace in each of these objects, where the app developer could put items without risking a namespace collision.

Jonathan Lundell

unread,
Oct 26, 2012, 4:13:57 PM10/26/12
to web2py-d...@googlegroups.com
On 26 Oct 2012, at 11:15 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
>
> So...
>
> @expose('….')
> def index(*args, **kwargs)
>
> wat should be in '…'. The point is that we still want to specify the url in, how to map into args and kwargs, and possibly filtering the args.

Just for the record: we're getting rid of the idea that URLs are (necessarily) /a/c/f? They could be, of course, and we do need a top-level way of steering to an app, but it needn't be contained in the path.

How badly do we need to support multiple top-level apps? Especially if the admin+appadmin architecture is changing? How about pushing that whole question out to the server, and running an entire web2py instance for each app? Seems to me that supporting multiple apps in a single web2py instance unnecessarily complicates web2py for 95% of all installations (ignoring admin), and for the other 5% (I'm making these numbers up, of course) the burden of multiple web2py instances isn't that great. Simplify!

Massimo DiPierro

unread,
Oct 26, 2012, 4:14:24 PM10/26/12
to web2py-d...@googlegroups.com
can you show an example?

Jonathan Lundell

unread,
Oct 26, 2012, 4:24:07 PM10/26/12
to web2py-d...@googlegroups.com
On 26 Oct 2012, at 1:14 PM, Massimo DiPierro <massimo....@gmail.com> wrote:
>
> can you show an example?

Three global objects: request, session, application. request includes response. request & session are thread-local; access to session is serialized as now. Each of them has a top-level Storage member called 'app' (request.app, etc) that serves as the app's private namespace.

The application object lives in the database (sort of a global session). But access to it can't be serialized, so we use database semantics to guarantee atomic updates (need to have atomic read-modify-write) (can we require memcached?).

Massimo DiPierro

unread,
Oct 26, 2012, 4:48:22 PM10/26/12
to web2py-d...@googlegroups.com

So here is what I have so far:


start it with

python runme.py

Try visit:

defined in apps/welcome/main.py


defined in apps/welcome_legacy/ (a normal web2py app + main.py)

There are many new files:
web3py/__init__.py web3py/menu.py
web3py/aes.py web3py/pbkdf2.py
web3py/beautify.py web3py/portalocker.py
web3py/cache.py web3py/recipe-578078-1.py
web3py/cfs.py web3py/rocket.py
web3py/cleaners.py web3py/sanitizer.py
web3py/contenttype.py web3py/session.py
web3py/current.py web3py/storage.py
web3py/dal.py web3py/stream.py
web3py/expose.py web3py/template.py
web3py/fancy_pickle.py web3py/utf8.py
web3py/forms.py web3py/utils.py
web3py/helpers.py web3py/validators.py
web3py/http.py web3py/web2py.py
web3py/languages.py web3py/wsgi.py

You should look into
- helpers.py (a simpler and nicer version of web2py's html.py)
- forms.py (a simpler and nicer version of SQLFORM and SQLFORM.factory, could also be backported)

- wsgi.py (the main wsgi handler, much cleaner)
- expose.py and current.py work together to provide the @expose and current objects.
- http.py is the HTTP object, same purpose of web2py's one but rewritten
- dal, languages, validators, storage, template, sanitize, contenttype are the same as before
- cleaners.py and session.py implement the middleware discussed here
- menu.py and beautify.py are just a port of the web2py ones and should be refactored for clarity.

- web2py.py is the compatibility handler. It is called by apps/welcome_legacy/main.py

runme.py calls wsgi.py which calls scan_apps which scans the apps/ folder and imports all the main.py.

I have not ported tools yet nor SQLTABLE and SQLFORM.grid but that should be pendantic but easy.
I am unhappy with cache and it should be rewritten in terms of recipe-578078-1.py.

The URLs are always [prefix]/[app specific]
The [prefix] is set to /application-name but can be overwritten.
The [app-specific] is set by @expose(path='/index') and defaults to the name of the function being exposed.

Nothing here is set in stone but it is an experiment that may help us improve web2py.

Comments? throw stones?

Massimo





Niphlod

unread,
Oct 26, 2012, 5:14:16 PM10/26/12
to web2py-d...@googlegroups.com
1st stone, then it's bed-time, sorry.... there's no runme.py in the repo :P
I see little holes here and there (but I acknowledge the "work in progress" status, don't mind) but I'm amazed nonetheless by the readability of the flow logic. Will take very little time tomorrow to fully grasp all the bits.

Niphlod

unread,
Oct 26, 2012, 5:15:28 PM10/26/12
to web2py-d...@googlegroups.com
whoopsy, spoken too early, (I'm really in the need of resting). runme is there. sorry for the false alarm

Massimo DiPierro

unread,
Oct 26, 2012, 7:18:06 PM10/26/12
to web2py-d...@googlegroups.com
https://github.com/mdipierro/web3py/blob/master/runme.py

web2py got a little convoluted over time.

For example SQLFORM creates the HTML representation before processing the input and delegates to the helpers the processing. 
SQLFORM.facotry was built on top of SQLFORM by faking a no-name table. All of this is non-sensical. Looked like a good idea at the time.

So goes for all the helpers A,DIV, SPAN, etc. They should all be defined in terms of TAG.  There are many places where we check hasattr(obj, 'xml') instead we really should check hasattr(obj, TAG).

wsgibase does everything including streaming static files. It is better to separate the different functionalities in many wsgi chained calls.
The way web2py handle streaming is different than the way is handles static files. In web3py the latter is simply delegated to the former code (most of which you wrote, including handling of zipped content and version of static files).

All of the above could be backported to web2py without much trouble/

Massimo

On Oct 26, 2012, at 4:14 PM, Niphlod wrote:

1st stone, then it's bed-time, sorry.... there's no runme.py in the repo :P
I see little holes here and there (but I acknowledge the "work in progress" status, don't mind) but I'm amazed nonetheless by the readability of the flow logic. Will take very little time tomorrow to fully grasp all the bits.


Massimo DiPierro

unread,
Oct 26, 2012, 7:47:01 PM10/26/12
to web2py-d...@googlegroups.com
perhaps as an intermediate step we should do what Mariano suggest. Improve web2py by backporting some of these new modules to web2py while keeping backward compatibility. Leave request, response, session, alone but make some of they attributes lazy (as done in web3py/current.py) and the new streamer handler and error handler.

Massimo


Martín Mulone

unread,
Oct 26, 2012, 7:51:04 PM10/26/12
to web2py-d...@googlegroups.com
I can believe so many changes in such short time. The code is very clean, for what I see.

there are a: db.commit() in welcome app. You are planning to do this way, or in a future there are going to be a context manager like web2py?.

2012/10/26 Massimo DiPierro <massimo....@gmail.com>



--
http://martinmulone.com.ar

massimo....@gmail.com

unread,
Oct 26, 2012, 10:06:41 PM10/26/12
to web2py-d...@googlegroups.com
No. The commit should be removed


From my Android phone on T-Mobile. The first nationwide 4G network.


-------- Original message -------- Subject: Re: [web2py-dev] web3py - important! From: Martín Mulone To: web2py-d...@googlegroups.com CC:

Massimo Di Pierro

unread,
Oct 27, 2012, 2:26:13 AM10/27/12
to web2py-d...@googlegroups.com
There was a typo in the commit. Runs in 2.7 but not yet in 3.3. The new code is mostly 3.3 compatible but some of the old code is not. For example the code in utils.py, languages.py, and dal.py. The issues is that I do not always know when a string should be bytes or utf8 (therefore converted to unicode).

On Saturday, October 27, 2012 12:40:48 AM UTC-5, David Marko wrote:
Running with Python 3.3 getting this ...

c:\java\web3py>c:\python3\python.exe runme.py
Traceback (most recent call last):
  File "runme.py", line 2, in <module>
    from web3py import run
  File "c:\java\web3py\web3py\__init__.py", line 1, in <module>
    from .wsgi import run
  File "c:\java\web3py\web3py\wsgi.py", line 9, in <module>
    from .cleaners import smart_traceback
  File "c:\java\web3py\web3py\cleaners.py", line 52
    def __init__(self): print 'connecting'
                                         ^
SyntaxError: invalid syntax 

Martín Mulone

unread,
Oct 27, 2012, 6:18:50 AM10/27/12
to web2py-d...@googlegroups.com
Why not call it gluino framework, at least at this stage, is too different and there a long way to be similar to web2py (in functions)?

2012/10/26 Mariano Reingart <rein...@gmail.com>
On Fri, Oct 26, 2012 at 12:33 AM, Massimo DiPierro
<massimo....@gmail.com> wrote:
>
> On Oct 25, 2012, at 9:13 PM, Mariano Reingart wrote:
>
> On Thu, Oct 25, 2012 at 1:21 PM, Massimo DiPierro
> <massimo....@gmail.com> wrote:
>
> Hello everybody,
>
>
> Time to start thinking about web3py. And as I think about it, I hear this in
>
> my mind: http://www.youtube.com/watch?v=HofoK_QQxGc
>
>
> Here is what I have in mind and I am building a prototype:
>
>
> 1) eliminate every computation done when the request arrives (for example
>
> parsing of url, headers, accept_languages, etc) and make everything lazy.
>
>
> web2py will be a microframework?
>
>
> No. Just faster. Also smaller because better code.
>
>
>
> 2) eliminate request, response, session, T, cache objects and have a single
>
> "current" object. It should not be a thread local to avoid locking issues,
>
> but it should be passed around.
>
>
> nice but "current" wording can be misleading, why not a more explicit
> request and response?
> (like now…)
>
>
> There will be a compatibility layer so you can still use it as now.
> I just think there should be a single thread local. Anther issue is to
> minimize the number of objects being created for speed.

Does it make any difference if its create 1 object instead of three?
It should be faster if we use python dicts directly (or namedtuples)

This could speed up a micro-benchmark, but I doubt it would be helpful
in a real-world app.

>
> 3) eliminate all the exec/execfiles but preserve some version of the current
>
> multi-applications folder structures. Each app will have an entry point
>
> (main.py) where actions are registered to allow a freer structure at the app
>
> level.
>
>
> how will it manage "in-the-fly" modifications?
>
>
> Not sure hot to implement it. But it will work as today from user
> prospective.

I don't know if it would be possible without restarting the whole
server as django and others do when they detect a file modification.

Using decorators and removing exec will have a side-effect that apps
would have to be initialized, maintaining some state/settings in the
imported modules.
This will not be easy to follow and a minimal syntax error could down
the entire application or have secondary nasty effects (i.e. weird
import errors)

I doubt it will work like today from the user perpective, I think that
it would require a more standard python "command-line" approach

> are you saying good by to the online IDE?
>
>
> Not at all but I think it should be simpler.
>
> that kind of dynamism is what makes web2py different and easy to learn/use
>
>
> I agree. We will keep it. But there are some hurdles.

Yes, mostly because of python, but we could address the issues instead
of moving to a different approach.

>
> 4) A global routes to configure "only" application prefixes. Then each app
>
> is in charge of its own internal routing, perhaps using decorators as Flask
>
> does.
>
>
> this will be hard to explain to beginners and can cause a mess in the
> app structure
>
> a standard directory layout about where to put each file (MVC) is a
> good approach now
>
> yes, it is rigid, but it works for the majority of webapps made with web2py
>
>
> This is what I have in mind:
>
> apps/
>   myapp/
>      main.py # entry point for some app all they need
>      templates/
>      static/
>      databases/
>      uploads/
>      languages/
>      plugins/
>      modules/
>
> Legacy apps will also have
>
>     controllers/
>     views/
>
> This is negotiable.
>

Ok

plugins will have a statics folder too?

The main.py approach is similar to what I've been thinking ;-)

You could use almost the same code as today just

- encapsulating the models in a def
- passing around request, response, and so on (or using current)

For example, in main.py:

from controllers.default import index

def main(request, response, session, ...):
     return response.render(index(request, response))

in models/db.py

def get_db(request, response, session):
    db = DAL(...)
    db.define_table("my_table", Field("my_field"))
    return db

in controllers/default.py

def index(request, response, session):
    from models.db import ANY_OTHER_CONSTANT, my_function, get_db
    db = get_db(request, response, session)
    return dict(...)


I think this will be a minor cosmetic change compared with the web3py approach.
In fact, most of the changes in models and in controllers (like adding
defs or adding parameters to functions) can be automatically
implemented (like a 2to3 tool)

The other advantage is that we could implement an alternative way if
no main.py is present (the current exec behaviour)
That is, the "compatibilty layer" would be very simple and
straightforward to create (even without the 2to3-like tool, I think)

So, this approach would be useful for advanced user that need
performance, and it will simple enough for begginers
It will be cleaner and pythonic ;-)

> 5) No more global environment. Apps will do "from web3py import *" (see
>
> below)
>
>
> for the permormance perspective, this is equal or worse than the
> current approach
>
> also, import * should be discouraged as it chokes IDE and static checkers
>
> a shorter namespace would be useful, like
>
> import w3p
>
> db = w3p.DAL(....)
>
> or better:
>
> from web2py import dal
>
> db = dal.DAL(
>       dal.Field('foo', dal.STRING,
>                    requires=dal.IS_IN_SET([])
>       )
>
> (btw, I think this is currently achievable nowadays)
>
>
> I agree. It was just an example.
>
>
> 6) No more file-system write unless to install/edit apps. We store session
>
> in cookies (if small) or db (if large) and tickets always in db (so they can
>
> be searched and cleared more easily). They will not go in the app db but in
>
> a dedicated common ticket db.
>
>
> nice, but a better fine-grained ticket system should be necessary to
> further enhance performance and avoid some pitfalls in production
> (a bunch of tickets without enough detail was not very useful in my
> experience)
>
> in development, a more interactive inspection would be better than
> storing the error snapshot, like paste.evalexception.middleware
>
> http://tools.cherrypy.org/wiki/PasteEvalException
>
> 7) Remove all the helpers except TAG, CAT, XML, BEAUTIFY, MENU and renamed
>
> them tag, cat, safe,  beautify, menu (with simpler logic)
>
>
> why not a namespace?
>
> from web2py import tag
>
> tag.DIV
>
> or better:
>
> from web2py.tag import A, IMG, ...
>
> IMHO helpers are part of web2py differentiation for easiness, why
> remove or complicate them?
>
>
> Because we do not need them. tag would generate them on demand.

we could generate them on demand and importing them as usual
they cause a bit of namespace pollution but it has been nice to work
with tags directly, without too much explanation or OO knowledge

also, creating them on deman will choke with IDE and static checkers,
and I no see a huge improvement in performance
Creating a class on the fly is not faster than just import it
(maybe I'm missing the new implementation...)

> 8) Replace SQLFORM.factory with Form and SQLFORM with DALForm. They will
>
> have a simpler logic that separates the HTML representation from the input
>
> processing, thus it will be faster and will be able to handle non-html form
>
> submission (ajax, xml) while using a syntax similar to the current one.
>
>
> good
> again, namespaces maybe useful here
> why this cannot be backported to web2py?
>
> 9) No more appadmin. There should be a global application to  manage the
>
> databases of all the apps.
>
>
> good
>
> 10) should work with python 2.7 and 3.x
>
>
> bad, wasn't web3py py3k only?
> having two active versions could create confusion in users, be a
> maintenance nightmare and will not enable to take some advantages from
> python 3 new features
> or are you planing to discontinue web2py?
>
>
> I am planning to freeze web2py. web3py should have one version that works on
> both 2.7 and 3.x. On 2.7 will run also the compatibility layer.
>
>
> 11) A compatibility layer that will allow to run most existing web2py
>
> applications when using python 2.7.
>
>
> ah, ok, you are planning to discontinue web2py...
>
> IMHO they should be two different projects to avoid any misunderstanding
>
> 12) no more cache.disk but better cache.ram (renamed as cache) with actual
>
> expiration (current cache.ram only expires when accessed).
>
>
> cache.disk is useful in some places, surely I'm missing why this is evil
>
>
> I agree. We can have is an option along with memcache and redis. I would
> just not make it primary citizen as cache.ram.
>
>
> 13) automatic reverse routing
>
> 14) keep only dal.py and template.py from existing web2py.
>
>
> why not separate them as proper packages?
>
>
> That is a distribution issue. We can distribute them separately as well as
> combined. I still want the binary versions.
>
> what about gluino?
>
>
> Nobody uses it. Has no reason to exist. web3py should be usable as gluino.
>
>
> I have most of this done and the code is smaller and cleaner than current
>
> web2py is. If is 20x faster on hello world. The compatibility layer is
>
> imperfect as I am still working on BEAUTIFY, MENU, sessions, db
>
> transactions, and a many nasty details. Tools.py (auth) is missing.
>
>
> Here is an example of code I got working:
>
>
> <begin file apps/welcome/main.py>
>
> import time
>
> from web3py import DAL, Field, expose, Form, DALForm, IS_NOT_EMPTY, HTTP,
>
> tag
>
>
> db = DAL('sqlite://storage.test')
>
> db.define_table('person',Field('name',requires=IS_NOT_EMPTY()))
>
>
> @expose() # /appname/index
>
>
> decorators, although not an easy thing to explain to beginners, may be
> a good idea, but sometimes they are hard to understan/follow
> if this simple case should be explained, it may be not so good
> explicit is better than implicit ;-)
> maybe they could have more info about URL, or are they limited to
> function names?
>
> def index(current):
>
>    form = Form(Field('name',requires=IS_NOT_EMPTY())).process(current)
>
>    if form.submitted and form.processed and form.accepted: print form.vars
>
>    elif form.submitted: print form.errors
>
>    message = form.vars.name or ''
>
>    return locals()
>
>
> @expose('/test_args/<str:a>(/<int:b>)?') # /appname/test_args/hello/1
>
> def test_args(current,a,b=9): # args are parsed, validated and passed to
>
> action
>
>
> arggh, regex or whatever it looks like... :(
>
> why not use python 3 annotations?
>
> def test_args(current,a: str, b: int=9):
>
> http://www.python.org/dev/peps/pep-3107/
>
>
> I had not thought about it. It would rule out 2.7. I am not sure we can
> handle it.
>
>
>    return repr(dict(a=a,b=b))
>
>
> @expose(template='index.html')
>
> def test_form(current):
>
>    form = DALForm(db.person, record_id = None).process(current)
>
>    message = tag.div(tag.h1(form.vars.name))
>
>
> div is a function, method or a class here?
>
> it should be Div() ?
> why not DIV ?
>
>
> tag.ANYTHING is <ANYTHING….>. tag is the same as web2py TAG.
> You can make tag.Div, tag.DIV, tag.diV, etc.
>
> anyway, we'll suffer from explicit boilerplate code :-)
>
>    return locals()
>
>
> @expose(cache_expire=10)
>
> def timer(current):
>
>    return time.ctime()
>
>
> @expose()
>
> def try_redirect(current):
>
>    HTTP.redirect(current.url('index')) # does reverse routing
>
> <end>
>
>
>
> As you can see the syntax is similar for an easier transition. To benefit
>
> from speed apps will have to be rewritten.
>
>
> Legacy web2py welcome app, without tools and session also works. I will add
>
> tools and session support asap.
>
>
> The error tickets also need work but I just discovered inspect.stack and had
>
> an epiphany!
>
>
> Am I on the right track?
>
>
> maybe from the pythonic standpoint, but who will care?
> we'll become a common framework like everyone else
> why will anyone use web3py instead flask+sqlachemy, django or whatever?
>
>
> I think it will still look a lot like web2py but faster.

but it will not be web2py :-)

>
> Want to help?
>
>
> no, as I say in earlier mails, I don't see the point here, so sorry
>
> Comments suggestions?
>
>
> IMHO stick with web2py, split it in standalones modules and add
> alternative implementations with current structure, so it can be
> comparable in minimal benchmarks :-)
>
> What can we do better?
>
>
> enhance web2py to a better 3.0 version
>
>
> You vision is not much different than mine. I want to simply the source
> code. It is messy.
> Moreover there are so many caveats that it is getting hard to teach.
> I also learned many new Python APIs that web2py is not taking advantage of.

Surely, in fact I do not use nor teach many of the new improvements.
Maybe if we list all the known issues and workaronund it is easier to
get convinced about such big change.

> In my view the final product should be 95% compatible with current web2py.
> We have to break backward compatibility anyway to go to python 3.x.

I don't know, internally there will be many changes (mostly because
unicode), but for the end user most code could be run with minimal
changes

I don't see any new py3k killer feature that worth to justify this
(beside annotations, that are not available in py2)

> When should apps be reloaded (since they are no longer executed we must
> trigger reload)?
>
>
> you can't, you'll have to restart the whole server
>
>
> reload is one of the missing pieces in py3k
>
>
> Are you saying python 3 has no reload?
> http://docs.python.org/dev/py3k/library/imp.html?highlight=reload#imp.reload
>

Python 3 has no reload because it is evil :-)
However, it has been moved to the imp library, and could be emulated
with some python internals (with some care because it has more nasty
secondary effects than exec...)
In fact, AFAIK the import machinery will be / is a pure python
implementation in py3.3 (importlib), you can talk with Brett Cannon in
PyCon Argentina about that ;-)

> What would you like to see changed?
>
>
> not much, I'm quite happy with web2py
> I think we should have more complex app to do a deeper analysis
>
> I've learned a lot of things this year with web2conf for PyCon
> Argentina 2012 conference, it is "advanced" as Pinax Symposion (and
> even it may have more features), and still it is simple (in the web2py
> sense), easy to understand and it could be optimized to reach
> acceptable performance without compromising dynamism.
>
>
>
> Should we preserve validators as they are?
>
>
> we should preserve as most as possible to not break backward compatibility
>
>
> Should we have a built-in realtime event handling for websockets in every
>
> app? What do we do with the LOAD helper? what do we do with web2py.js? Do we
>
> still need response.files of should we make more use of template {{block…}}
>
> to inject stuff in <head/>? Should we backport some of new stuff (Form,
>
> cache, helpers) to web2py?
>
>
> yes to them all (excepting the complex {{block}} thing and the
> realtime event handling)
>
> If we take this route, we need to have a permanent feature freeze on web2py,
>
> except on DAL and Template which are the only shared modules.
>
>
> this is the part I dislike
> I think we can keep improving web2py and make alternatives or further
> works possible
>
>
> I am not saying "the only shared APIs". In the compatibility mode, the APIs
> will be the same and it will use exec as it does not. The actual modules
> will be rewritten and organized better.
>
>
> I will share my code next week, as soon as I get sessions to work.
>
> You will appreciate the clarity compare with the current web2py. In
>
> particular the helpers and form handling logic.
>
>
> IMHO that we should clean up the code is not a reason to start a
> massive rewrite from scratch.
> I argued in previous discussions that this seems like the second
> system effect (and according the literature, it has not been good for
> any project...):
>
> http://en.wikipedia.org/wiki/Second-system_effect
>
>
> Excellent point.
>
> This is a nice article explaining why:
>
> http://www.joelonsoftware.com/articles/fog0000000069.html
>
> For example, DAL has more important problems like the id/keyed table
> and could be optimized in the id/sequence/autonumeric handling, bool
> logic, left joins, etc.
> Fixed that it should be taken more seriously to be used in standalone,
> gui and other web applications/frameworks.
> They are in my TODO list, or wish list as this year I didn't have enough
> time :(
>
> About the odd benchmark, please read the discussion in django-dev about
> this.
>
> For me, just because web2py isn't faster that ever
> minimicrosmallpythonframework out in the wild is not a an enough
> excuse for a big re-design.
> That's why I think gluino idea was amazing, if we could take the best
> of both worlds (full-stack and micro-framework), why not?
>
>
> Sorry for my lack of optimism
>
>
> I understand the skepticism. I am glad we will be able to talk more about
> this in person in two weeks.

Me too

BTW, for the web2py sprint, there are around 25 people registered!!!!!!
(right now, more than any other sprint in the conference)

http://ar.pycon.org/2012/schedule
--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details    : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official    : http://www.web2py.com/





--
http://martinmulone.com.ar

Marin Pranjić

unread,
Oct 27, 2012, 6:34:22 AM10/27/12
to web2py-d...@googlegroups.com
+1 for name change
web2py is cool but i don't like web2py


Is it possible, using the current translation system, to make a translation management page that will list all the translation strings and their locations in source code?

Sometimes I miss it. Semantics can be lost in translation and sometimes one string in default language should/could be translated to more then one in another language.

Marin Pranjić

unread,
Oct 27, 2012, 6:34:56 AM10/27/12
to web2py-d...@googlegroups.com
typo

On Sat, Oct 27, 2012 at 12:34 PM, Marin Pranjić wrote:
+1 for name change
web2py is cool but i don't like web3py

Massimo DiPierro

unread,
Oct 27, 2012, 3:22:01 PM10/27/12
to web2py-d...@googlegroups.com
We can have a name contest. A requirement is that nobody else must have the trademark. Web2py has that names because I was threatened to be sued when it was called "gluon".

Massimo DiPierro

unread,
Oct 27, 2012, 5:11:41 PM10/27/12
to web2py-d...@googlegroups.com
I am more and more convinced that we should merge existing web2py with web3py. We can do it in this way:

1) complete the new modules so that they run on both 2.7 and 3.x and implement basic functionality that we want

2) one by one replace existing we2py modules with web3py modules and check it behaves the same. For example:

SQLFORM = DALForm
SQLFORM.factory = Form
for name in 'A,B,BODY,BR,CENTER,CLEANUP,CRYPT,DAL,DIV,EM,EMBED,FIELDSET,FORM,H1,H2,H3,H4,H5,H6,HEAD,HR,HTML,I,IFRAME,IMG,INPUT,LABEL,LEGEND,LI,LINK,LOAD,MARKMIN,META,OBJECT,OL,ON,OPTGROUP,OPTION,P,PRE,SCRIPT,SELECT,SPAN,STYLE,TABLE,TBODY,TD,TEXTAREA,TFOOT,TH,THEAD,TITLE,TR,TT,UL'.split(','): environment[name] = tag[name]

etc.

3) Use new logic of save exception in DB vs filesystem (so they can more easily be searched, organized and deleted). Also the new logic saves a partial serialization of stack trace as opposed to a BEAUTIFIcation of it.

4) move the existing routing logic down the stack to that we can make it optional. Developers will be given the option to by-pass the routing+exec mechanism and user the alternative @expose mechanism.

I am not sure about 3 and 4.

What do you think?



On Oct 27, 2012, at 5:34 AM, Marin Pranjić wrote:

guruyaya

unread,
Oct 28, 2012, 5:04:20 AM10/28/12
to web2py-d...@googlegroups.com
It's not a deal breaker for me, but I really don't like the @expose idea. One thing I like about web2py is the short distance between idea and something on the screen. I have an idea, I build a function that show's it - we're done. I want it in json - I using the url with .json - poff it's here. No SQL hassle, not much thinking about security. You think
Now, @expose is cool for security perpose, don't get me wrong. But it makes my work a little but less spontanious. I'm not sure how many of you agree with me, but if you do - raise your voice.


On Thursday, October 25, 2012 6:21:12 PM UTC+2, Massimo Di Pierro wrote:
Hello everybody,

Time to start thinking about web3py. And as I think about it, I hear this in my mind: http://www.youtube.com/watch?v=HofoK_QQxGc

Here is what I have in mind and I am building a prototype:

1) eliminate every computation done when the request arrives (for example parsing of url, headers, accept_languages, etc) and make everything lazy.
2) eliminate request, response, session, T, cache objects and have a single "current" object. It should not be a thread local to avoid locking issues, but it should be passed around.
3) eliminate all the exec/execfiles but preserve some version of the current multi-applications folder structures. Each app will have an entry point (main.py) where actions are registered to allow a freer structure at the app level. 
4) A global routes to configure "only" application prefixes. Then each app is in charge of its own internal routing, perhaps using decorators as Flask does.
5) No more global environment. Apps will do "from web3py import *" (see below)
6) No more file-system write unless to install/edit apps. We store session in cookies (if small) or db (if large) and tickets always in db (so they can be searched and cleared more easily). They will not go in the app db but in a dedicated common ticket db.
7) Remove all the helpers except TAG, CAT, XML, BEAUTIFY, MENU and renamed them tag, cat, safe,  beautify, menu (with simpler logic)
8) Replace SQLFORM.factory with Form and SQLFORM with DALForm. They will have a simpler logic that separates the HTML representation from the input processing, thus it will be faster and will be able to handle non-html form submission (ajax, xml) while using a syntax similar to the current one.
9) No more appadmin. There should be a global application to  manage the databases of all the apps.
10) should work with python 2.7 and 3.x
11) A compatibility layer that will allow to run most existing web2py applications when using python 2.7.
12) no more cache.disk but better cache.ram (renamed as cache) with actual expiration (current cache.ram only expires when accessed).
13) automatic reverse routing
14) keep only dal.py and template.py from existing web2py.

I have most of this done and the code is smaller and cleaner than current web2py is. If is 20x faster on hello world. The compatibility layer is imperfect as I am still working on BEAUTIFY, MENU, sessions, db transactions, and a many nasty details. Tools.py (auth) is missing.

Here is an example of code I got working:

<begin file apps/welcome/main.py>
import time
from web3py import DAL, Field, expose, Form, DALForm, IS_NOT_EMPTY, HTTP, tag

db.define_table('person',Field('name',requires=IS_NOT_EMPTY()))

@expose() # /appname/index
def index(current):
    form = Form(Field('name',requires=IS_NOT_EMPTY())).process(current)
    if form.submitted and form.processed and form.accepted: print form.vars
    elif form.submitted: print form.errors
    message = form.vars.name or ''
    return locals()

@expose('/test_args/<str:a>(/<int:b>)?') # /appname/test_args/hello/1
def test_args(current,a,b=9): # args are parsed, validated and passed to action
    return repr(dict(a=a,b=b)) 

@expose(template='index.html')
def test_form(current):
    form = DALForm(db.person, record_id = None).process(current)
    message = tag.div(tag.h1(form.vars.name))
    return locals()

@expose(cache_expire=10)
def timer(current):
    return time.ctime()

@expose()
def try_redirect(current):
    HTTP.redirect(current.url('index')) # does reverse routing
<end>

As you can see the syntax is similar for an easier transition. To benefit from speed apps will have to be rewritten.

Legacy web2py welcome app, without tools and session also works. I will add tools and session support asap.

The error tickets also need work but I just discovered inspect.stack and had an epiphany!

Am I on the right track?

Want to help? Comments suggestions? What can we do better? When should apps be reloaded (since they are no longer executed we must trigger reload)? What would you like to see changed? Should we preserve validators as they are? Should we have a built-in realtime event handling for websockets in every app? What do we do with the LOAD helper? what do we do with web2py.js? Do we still need response.files of should we make more use of template {{block…}} to inject stuff in <head/>? Should we backport some of new stuff (Form, cache, helpers) to web2py?

If we take this route, we need to have a permanent feature freeze on web2py, except on DAL and Template which are the only shared modules.

I will share my code next week, as soon as I get sessions to work.
You will appreciate the clarity compare with the current web2py. In particular the helpers and form handling logic.

Massimo

Massimo DiPierro

unread,
Oct 28, 2012, 7:31:56 AM10/28/12
to web2py-d...@googlegroups.com
We could have something.

@expose.all_function()

and you would not be able to decorate each function in the file. I was planning to keep the fact that .json gives you json output (and other serialiazation types as well) for both the compatibility mode and the new web3py mode. I think we should actually improve it. As it is is only serializes output but does not handle json input.

guruyaya

unread,
Oct 28, 2012, 9:00:49 AM10/28/12
to web2py-d...@googlegroups.com
You know how to keep me happy!

Massimo DiPierro

unread,
Oct 28, 2012, 9:20:35 AM10/28/12
to web2py-d...@googlegroups.com
I hope my wife does not read this comment or she will misunderstand it. ;-)

Massimo DiPierro

unread,
Oct 29, 2012, 1:58:30 PM10/29/12
to web2py-d...@googlegroups.com
I have changed the current structure so that has:

current.request
current.response
current.session
current.T

They are not identical to before but similar. I want to minimize the computed attributes since they take time. Some of them can be turned into properties. Moreover they are not Storage objects, for speed.

I am considering backporting the new helpers and forms to web2py but it is more complex than anticipated.

we can define A = helpers.tag.A but the old A contains logic used by FORM and used by SQLFORM.
we can rewrite SQLFORM on top of forms.Form and it would work great but user will no longer be able to do

form = SQLFORM(…..)
form.element(….)[attribute] = value

this is because the new Form is more abstract and does not have a representation in terms of object until later, when it is serialized by {{=form}}.
This was the point for creating the new forms.From, so that it could be used with other input (for example json) without the helpers overhead. Moreover the old SQLFORM has lots of logic to "fix" forms after processing. The new Form is leaner. It is only generated after processing so no logic for fixing.

Anyway… on suggestions are appreciated.




Jonathan Lundell

unread,
Oct 29, 2012, 8:00:23 PM10/29/12
to web2py-d...@googlegroups.com
I'd embed response in request, on the grounds that it's part of the request-specfic state.

Another idea from earlier discussions: replace routes.py (in this case as used either for global app routing, or legacy rewrite logic) with a generic Pythonic configuration file. Overall structure TBD, but maybe sections for global and app-specific configuration.

Massimo DiPierro

unread,
Oct 29, 2012, 10:40:19 PM10/29/12
to web2py-d...@googlegroups.com
Isn't session also part of the request. Isn't current what we mean by request?

Jonathan Lundell

unread,
Oct 29, 2012, 11:51:00 PM10/29/12
to web2py-d...@googlegroups.com
On 29 Oct 2012, at 7:40 PM, Massimo DiPierro <massimo....@gmail.com> wrote:
>
> Isn't session also part of the request. Isn't current what we mean by request?

Logically, the lifetime of a session extends across multiple requests, while the lifetime of a response corresponds exactly to a request.

And yes, my first choice of a name for 'current' was 'request', but it was already taken...

LightDot

unread,
Nov 1, 2012, 9:08:39 AM11/1/12
to web2py-d...@googlegroups.com
Two questions in regards to web3py/web2py subject in general:

- what will happen with experimental features in web2py? Having web2py in feature freeze seems logical, but what are the plans for existing experimental features and their documentation?

- web3py will require python 3.3 and 2.7 as a minimum, no 2.6 support..? I'm not saying I'm for or against this, just want to confirm I'm understanding correctly...

Regards,
Ales

Massimo DiPierro

unread,
Nov 1, 2012, 9:31:04 AM11/1/12
to web2py-d...@googlegroups.com
On Nov 1, 2012, at 8:08 AM, LightDot wrote:

Two questions in regards to web3py/web2py subject in general:

- what will happen with experimental features in web2py? Having web2py in feature freeze seems logical, but what are the plans for existing experimental features and their documentation?

They need to be documented. The many experimental features are SQLCustomType, the Scheduler, and aith.wiki. They will all stay.


- web3py will require python 3.3 and 2.7 as a minimum, no 2.6 support..? I'm not saying I'm for or against this, just want to confirm I'm understanding correctly…

Yes.


Regards,
Ales


On Tuesday, October 30, 2012 4:51:06 AM UTC+1, Jonathan Lundell wrote:
On 29 Oct 2012, at 7:40 PM, Massimo DiPierro <massimo....@gmail.com> wrote:
>
> Isn't session also part of the request. Isn't current what we mean by request?

Logically, the lifetime of a session extends across multiple requests, while the lifetime of a response corresponds exactly to a request.

And yes, my first choice of a name for 'current' was 'request', but it was already taken...

Alec Taylor

unread,
Oct 27, 2012, 6:20:24 PM10/27/12
to web2py-d...@googlegroups.com
Looking very good!

Just one thing to add to the list:

- Better RBAC support

(or maybe just better documentation, see:
https://groups.google.com/d/topic/web2py/OhlQ_7KhC-Q)

Also it would be good to have full OAuth1 and OAuth2 support with
example code for popular providers; out of the box.

(maybe using the requests and/or the sanction and/or the oauth2 [for
OAuth 1] libraries)

Nathan Morris

unread,
Oct 28, 2012, 5:48:30 PM10/28/12
to web2py-d...@googlegroups.com
Changing the name of the framework could have some downsides from a marketing / business perspective for those of us who are consulting. Some clients who have invested in web2py might not make the jump to "web3py" (or whatever new name becomes) based on the name change alone. 

For example, in Rails version 3.0 had some major differences from the previous version 2.3 and broke backwards compatibility. Rails 3 was basically a merging of the Merb framework and Rails 2.   


If they had renamed the Rails framework, I'm not sure as many clients / business customers would have agreed to try a new framework but they were willing to move to Rails 3.0 even though their apps required changes to run. 

Recommending a client upgrade to "Web2py (version 3.0.0) which has some breaks in backwards compatibility but is 20 times faster and has some amazing new features" would be an easier sell than recommending a framework with a different name. Non-technical clients can be very sensitive to the small amount of information they have to make a decision. Things like changes to the name, the quality of the logo, and design of the project page can make an unreasonable impact when they select a software for their project.

Martín Mulone

unread,
Nov 7, 2012, 6:37:58 AM11/7/12
to web2py-d...@googlegroups.com
I'm getting 'invalid action', the trunk is broken?


2012/10/27 David Marko <dmar...@gmail.com>
Updated from git but still having the problem, this is what I can see on console when accessing index action:

INFO:Rocket.Requests:127.0.0.1 - "GET /favicon.ico HTTP/1.1" - 200 14
welcome.main.index
ERROR:root:Traceback (most recent call last):
  File "c:\java\web3py\web3py\wsgi.py", line 64, in error_handler
    return static_handler(environ, start_response)
  File "c:\java\web3py\web3py\wsgi.py", line 50, in static_handler
    data = dynamic_handler(environ, start_response)
  File "c:\java\web3py\web3py\wsgi.py", line 25, in dynamic_handler
    expose.run_dispatcher()
  File "c:\java\web3py\web3py\expose.py", line 163, in run_dispatcher
    output = obj.func(**match.groupdict())
  File "c:\java\web3py\web3py\cleaners.py", line 20, in g
    output = cleaner.wrap_call(f)(*a,**b)
  File "c:\java\web3py\web3py\cleaners.py", line 19, in g
    cleaner.on_start()
  File "c:\java\web3py\web3py\session.py", line 11, in on_start
    request_cookies = current.request_cookies
  File "c:\java\web3py\web3py\current.py", line 94, in request_cookies
    self._request_cookies.load(self.environ.get('HTTP_COOKIE',''))
  File "c:\python27\lib\Cookie.py", line 632, in load
    self.__ParseString(rawdata)
  File "c:\python27\lib\Cookie.py", line 665, in __ParseString
    self.__set(K, rval, cval)
  File "c:\python27\lib\Cookie.py", line 585, in __set
    M.set(key, real_value, coded_value)
  File "c:\python27\lib\Cookie.py", line 460, in set
    raise CookieError("Illegal key value: %s" % key)
CookieError: Illegal key value: ti:mid

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 

guruyaya

unread,
Nov 7, 2012, 6:38:51 AM11/7/12
to web2py-d...@googlegroups.com
I would like to suggest something I miss from, belive it or not - php
First, I would like to see that:
<input name="example[a][b][c] value="yaya" /> 
will return to the server side something like
request.vars.example.a.b.c = yaya.

After this is done, the create SQLFORM will name the vars diffrently then current web2py does. It'll output something like
<input name="tablename[1][title] />
to update the title of record number 1 in table tablename
or 
<input name="tablename[new1][title] />
to create new record in the table.

One advantage of this idea, is allowing to update 2 tables that has fields with the same name in one row. Today, if you have 2 tables with, say, a title field, you cannot combine them both into one big happy form. 

The main thing I hope it to see, is a big form that allow you to update several rows. I imagine a grid that works "live", where you can update several records on the grid itself, then submit them to be reviewed on the server side (security issues must be delt with, ofcourse). This, ofcourse, is not compatible with the old web2py schema, therefor can't be done in the old framework, but it's a vision worth considering for the new build, can make these changes. 

What do you guys think?

villas

unread,
Nov 7, 2012, 1:17:07 PM11/7/12
to web2py-d...@googlegroups.com
Hi guruyaya
I guess you mean a '_filter_fields' method which also works for tables which have similarly named fields.
In which case, yes, that would be useful.

guruyaya

unread,
Nov 8, 2012, 3:22:08 AM11/8/12
to web2py-d...@googlegroups.com
Acctually, this is the less useful feature I see for this feature. As I said, allowing update of several rows in the same table, inside one form. I'm acctually implementing it now myself, in an app I'm doing, but it's a patch for my own needs, not a generic solution you can work with.
Message has been deleted

Massimo DiPierro

unread,
Dec 2, 2012, 9:23:17 AM12/2/12
to web2py-d...@googlegroups.com
Yes but this would not be able to discriminate if a parameter is required or optional.

On Nov 28, 2012, at 11:59 AM, VP wrote:

> On Friday, October 26, 2012 11:06:22 AM UTC-5, Jonathan Lundell wrote:
>> On 26 Oct 2012, at 8:47 AM, Massimo DiPierro <massimo....@gmail.com> wrote:
>> Do you like better:
>>
>> @expose('/index/<str:a>(/<int:b>)?')
>> def index(a,b=3)
>>
>>
>> or this
>>
>>
>>
>> @expose('/index)
>> def index(a = Arg('str'), b = Arg('int',default=3))
>>
>>
>> we cannot use the python 3 notation because it would not work with 2.7 and it not reach enough to specify validators for args.
>>
>>
>> The second version seems more flexible.
>
>
> How about this:
>
>> @expose('/index)
>> def index(a = '', b = 3):
>> pass
>
>
>
> Since you don't "exec" this but rather import and can afford preprocessing, you can inspect the default values and derive the types for each argument. This way type checking can be done cleanly. Additional validations should be done inside the controller. I think it is unconventional and perhaps unnecessary to do other checking and validations other than types at this level.
>
> It looks like web3py will be quite interesting.
>
>
> Things I would vote "off the island":
>
> + The online IDE
> + Exec (although default=request.now in a table is really nice with exec, but not worth it)

Massimo DiPierro

unread,
Dec 2, 2012, 9:24:24 AM12/2/12
to web2py-d...@googlegroups.com
We can. The issue is how? If we store them in db that is easy but if we store them in fs I think we need an external process.
Message has been deleted

Martín Mulone

unread,
Dec 11, 2012, 3:43:24 PM12/11/12
to web2py-d...@googlegroups.com
+ pip
+ and a way to execute the app directly main.py (no apps) wsgi application, like bottle, flask etc.

This two thing are great to make it friendly in production enviroment.


2012/12/2 Massimo DiPierro <massimo....@gmail.com>

Giovanni Barillari

unread,
Aug 5, 2013, 4:56:11 AM8/5/13
to web2py-d...@googlegroups.com
Hi all,
seems that this topic is quite old and the development of web3py is paused right now, but I have an idea about the controllers.

I don't have a real knowledge of all the web2py flow for requests, but If I understood correctly, in web2py you use exec to load the controllers. The expose() idea is interesting, because implement the regexes for routing. But the sintax for the user became quite different. So what about keeping controller but loads them as modules?

Pratically we need a Base controller in the app which would be something like this:
class BaseController:
   
def __init__(self, fn=''):
       
## loads some vars
       
self.request = current.request
       
....
       
## load the function requested by url
       
self.__getattribute__(fn)()


Then in controllers/mycontroller.py we can put the users controller which would be something like this:
class MyController(BaseController):
   
def index(self):
       
## some actions


   
def user(self):
       
## some other actions

I don't know if it's achievable, but It's just another way :)


/Giovanni

Massimo DiPierro

unread,
Aug 5, 2013, 10:42:39 AM8/5/13
to web2py-d...@googlegroups.com
It is achievable but I personally do not like the idea o starting a controller with a class.

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Giovanni Barillari

unread,
Aug 5, 2013, 4:40:51 PM8/5/13
to web2py-d...@googlegroups.com
Good to know :)

Probably I've missed that, but you're thinking of have controllers in web3py or totally remove them? Because i was thinking on the main.py in app, and if I have an app with 4-5 controllers in web2py, should i write thousands lines in main.py on web3py? Or you were thinking on modules or something to call from main.py like that?

(web3py is interesting ^^)

/Giovanni

Marek Mollin

unread,
Aug 6, 2013, 6:25:07 AM8/6/13
to web2py-d...@googlegroups.com
web3py seems like a sweetpoint between web2py and flask...
would use it day and night ;]

Giovanni Barillari

unread,
Jun 20, 2014, 11:34:18 AM6/20/14
to web2py-d...@googlegroups.com
Hi all,

during these days I'm working on the code of web3py, trying to make it usable for a new project.
I've made a PR on massimo's github repository with my first edits on the code, planning to made another PR in the next few days: I'm porting the auth module while I'm typing.

Hope to rise up some interest by the developers.

Massimo, let me know if you notice something you don't like.


Keep in touch,
Reply all
Reply to author
Forward
0 new messages