web2py architecture

[nihil]

Hi to all
Sometimes I modified web2py sources for better performance and i think
now is the time to share with you my modifications and my ideas about
web2py architecture.

Web2py runs application code into a "private" environment (i think
this is good) but notice web2py builds and destroys this envirnoment
for each request (and this is no good).

I tried to let theese environments in memory for next user request and
this speeds up execution of wsgibase() function from an average of
28ms with DAL(migrate=False) and code compiled to an average of 3ms in
prebuilt envirnoments.

So my question is "there is a reason to build and destroy environments
for each request?"

[pierreth]

On 16 mai, 06:10, nihil <nihil...@gmail.com> wrote:
> Hi to all

>
> So my question is "there is a reason to build and destroy environments
> for each request?"

Maybe there is no reason. No one thought about reuse. What you did
seems to be a great improvement.

[Massimo Di Pierro]

I am not sure I understand the question. What environment are you talking about? the environment in which models/controllers are executed?

This env can only live within a request because:
- it is not serializable
- we do not want it shared, for security
- there is no overhead in creating the environment

Anyway, I am not sure I understand the question.

> --
> mail from:GoogleGroups "web2py-developers" mailing list
> make speech: web2py-d...@googlegroups.com
> unsubscribe: web2py-develop...@googlegroups.com
> details : http://groups.google.com/group/web2py-developers
> the project: http://code.google.com/p/web2py/
> official : http://www.web2py.com/

[pbreit]

Maybe it would help us to understand the change if we saw the code?

[nihil]

I think don't run models for every request is a good improvement, (why
redefine data models in every request? especially if for complex
models in which are defined many tables with many validators), and
don't read and compile code files (models,controllers and views) is a
good inprovement too. I notice in my computer theese optimization and
some other reduces an average of 25ms in executing every request, so
time spent to build and destroy environments is about 25ms for each
request.
Considering that application code is generally the execution of a
query and building an html output, and this requires about 2-3ms I
say: "why i have to spend 25ms for framework activities and only 2-3
for my code?", it's possible reduce it?
the answer is Yes.

[nihil]

On 18 Mag, 18:59, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

> I am not sure I understand the question. What environment are you talking about? the environment in which models/controllers are executed?

Yes, exactely.

> This env can only live within a request because:
> - it is not serializable
> - we do not want it shared, for security

theese environments will not be shared between different requests from
different sessions (there is no reason)

> - there is no overhead in creating the environment

the overhead is not directly connected to building environment but in
executing what is inside this environment (ie: db, auth, crud and
everithing is declared inside models).

[nihil]

sure,
the code for now is only for testing pourpouse and has some bug (i
know) but i think it coud be a good starting point for critics.

import globals as glob
Session = glob.Session
from random import randrange
from datetime import datetime, timedelta
#from compiler import compile
from compileapp import build_environment,restricted,parse_template
from time import time as tim
from dal import DAL
from storage import Storage
from copy import copy
import gc
import sys
import traceback
import os

def scan_dir(d):
for a in os.listdir(d):
a = os.path.join(d,a)
if os.path.isdir(a):
for b in scan_dir(a):
yield b
else:
yield a


def timed(name = ''):
n = name
def t(func):
name = n + '.' + func.__name__
def w(*a,**b):
t = tim()
ret = func(*a,**b)
sys.stderr.write( ' --> %s() eseguita in %f secondi.\n' %
(name,tim() - t))
return ret
return w
return t


class SessionManager(dict):
@timed('SessionManager')
def __init__(self,on_delete,sess_name =
'nihil_session',expire_time = 600):
self.expire_time = timedelta(0,expire_time)
self.expire_count = 0
self.session_name = sess_name
self.on_delete = on_delete

@timed('SessionManager')
def unique(self):
u = ''.join(map(chr,(randrange(65,120) for a in xrange(32))))
if u in self:
return self.unique()
return u

@timed('SessionManager')
def connect(self,request,response):
self.expire_count += 1
if self.expire_count == 10:
self.gc()
self.expire_count = 0
if self.session_name in request.cookies:
sess = request.cookies[self.session_name].value
else:
sess = None
if not sess in self:
s = Session()
if not sess:
sess = self.unique()
self[sess] = s
response.cookies[self.session_name] = sess
else:
s.connect(request,response)
self[sess] = s
s.SESSION_ID = sess
else:
s = self[sess]
if '_last_timestamp' in s and request.now -
s._last_timestamp > self.expire_time:
self.free_session(s)
s = self.connect(request, response)
sys.stderr.write('Sessione scaduta : ricostruisco l
\'ambiente e distruggo %d oggetti.\n' % gc.collect(2))
response.flash = 'Sessionse scaduta'
return s

s._last_timestamp = request.now
return s

def free_session(self,sess):
self.on_delete(sess.SESSION_ID)
sess.forget()
del self[sess.SESSION_ID]
del sess

@timed('SessionManager')
def gc(self):
now = datetime.now()
for expired in (a for a in self.itervalues() if
(a._last_timestamp - now) > self.expire_time):
self.free_session(expired)

def read_file(fn):
f = open(fn,'r')
ret = f.read()
f.close()
return ret

comp = compile
def compile(*a):
sys.stderr.write( 'sto compilando %s ...\n' % a[1])
return comp(*a)

class Application:
environments = {}
applications = {}

@timed('Application')
def __init__(self,name):
self.name , self.databases = name , []
dir = ('applications/%s/models/' % name).replace('/',os.sep)
self.models = dict((a[:-3],compile(read_file(dir +
a),os.path.abspath(dir + a),'exec')) for a in os.listdir(dir) if
a.endswith('.py'))
dir = ('applications/%s/controllers/' %
name).replace('/',os.sep)
self.controllers = dict((a[:-3],compile(read_file(dir +
a),os.path.abspath(dir + a),'exec')) for a in os.listdir(dir) if
a.endswith('.py'))
dir = ('applications/%s/views/' % name).replace('/',os.sep)
self.views = dict(('/'.join(a.split(os.sep)
[3:]),compile(parse_template(os.sep.join(a.split(os.sep)
[3:]),dir),os.path.abspath(os.sep.join(a.split(os.sep)[3:])),'exec'))
for a in scan_dir(dir) if a[-3:] in ('tml','son','pdf'))
self.sm = SessionManager(self.delete_environmet,'sion_id_' +
name)
self.applications[name] = self

def __str__(self):return 'Application(%s) with %d environments
loaded.' % (self.name,len(self.environments))

@timed('Application')
def delete_environmet(self,sess):
del self.environments[sess]

@timed('Application')
def execute_models(self,env):
for model in sorted(self.models):
exec self.models[model] in env
if not len(self.databases):
for k,v in env.iteritems():
if isinstance(v,DAL):
self.databases.append(k)

@timed('Application')
def __call__(self,request,response):
session = self.sm.connect(request, response)
sess_id = session.SESSION_ID
if sess_id in self.environments:
environment = self.environments[sess_id]

environment.update(dict(request=request,response=response,session=session))
else:
environment =
Storage(build_environment(request,response,session))
environment.App =
Application.applications[request.application]
self.execute_models(environment)
# self.environments[sess_id] = environment.copy()
self.environments[sess_id] = environment
environment['request'] , environment['response'] =
request,response
cont,func = request.controller, request.function
if not cont in self.controllers:
raise HTTP(200,'<html><body>function (%s/%s/%s) not found</
body></html>' % (self.name,cont,func))
if environment.controller != cont:
sys.stderr.write('\nsto eseguendo il controller %s ...\n'
% cont)
exec self.controllers[cont] in environment
environment['controller'] = cont
sys.stderr.write( 'eseguo %s/%s/%s/%s per %s\n' %
(self.name,request.controller,request.function,'/'.join(map(str,request.args)),sess_id))
sys.stderr.write( ''.join(('\n--R -- %30s = %s' % a for a in
request.vars.items())))
if not func in environment:
return '<html><body>function (%s/%s/%s) not found</body></
html>' % (self.name,cont,func)
page = eval (func + '()',environment)
if isinstance(page, dict):
response._view_environment = environment.copy()
response._view_environment.update(page)
response._vars = page
view = response.view
if not view in self.views:
view = 'generic' + os.path.splitext(view)[1]
# try:
exec self.views[view] in response._view_environment
# except Exception, e:
# sys.stderr.write( '\nErrore : ' + e.message + '\n')
if 'flash' in session:
del session['flash']
page = response.body.getvalue()
for db in self.databases:
environment[db].commit()
return page

@timed('Application')
def recompile(self,layer,which):
which,ext = os.path.splitext(which)
sys.stderr.write('\n' + which)
sys.stderr.write('\n' + ext)
if layer == 'controllers':
dir = ('applications/%s/controllers/' %
self.name).replace('/',os.sep)
self.controllers[which] = compile(read_file(dir + which +
ext),os.path.abspath(dir + which + ext),'exec')
for env in self.environments.itervalues():
if env.controller == which:
env.controller = None
elif layer == 'models':
# sys.stderr.write('.')
dir = ('applications/%s/models/' %
self.name).replace('/',os.sep)
sys.stderr.write('\n' + which)
sys.stderr.write('\n' + os.path.abspath(dir + which + ext))
self.models[which] = compile(read_file(dir + which +
ext),os.path.abspath(dir + which + ext),'exec')
# sys.stderr.write('.')
for env in self.environments.itervalues():
# sys.stderr.write('.')
self.execute_models(env)
elif layer == 'views':
dir = ('applications/%s/views/' %
self.name).replace('/',os.sep)
self.views[which.replace(os.sep,'/') + ext] =
compile(parse_template(which + ext,dir),os.path.abspath(dir + which +
ext),'exec')


glob.Application = Application

[nihil]

about the code:
this is a new module so don't touch web2py structures, but some
editing i did in main.py to call this module and use it in correct
way, i also tried to modify "admin" application for calling
Application.recompile after a file editing, and someting in other
places.

[Massimo Di Pierro]

Thank you nihil. I will take a close look asap.

[nihil]

yes, i know it's not a good develop environment and i know theese
structural modification could forbid execution in CGI, FastCGI
environment, but i think we can build a good production web2py version
that runs on a good production anvironment and consider this
production environment as officially best environment (IE: cherokee
with uWSGI or apache with WSGI) so a user can develop his application
in develop environment knowing his application will not run as fast as
he think because his production environment is not best environment so
a user can consider to develop his application where he can, for a
small number of user connected simultaniusly but he must know that
when his users will be more than a certain number he have to migrate
to a provider who is supporting our official production environment to
allow his application to be used by a large amount of users
simultaneusly.

[pbreit]

This is certainly over my head. Did you see the new features in trunk where you can put your models in folders in order to minimize how many models are executed on each request?

[Anthony]

On Thursday, May 19, 2011 2:19:37 PM UTC-4, pbreit wrote:

This is certainly over my head. Did you see the new features in trunk where you can put your models in folders in order to minimize how many models are executed on each request?

 

With the new custom importer and 'current' object, it is also now easier to move code that might have been in models into a module and just import/call as needed.

 

Anthony

[nihil]

I just reed new web2py trunk "features" like 'current =
threading.local()' and models splitted following controller structure
and i think it's a very good way to prevent useless table defining at
runtime, but i don't see a strong improvement.

maybe I'm wrong but if we want to mantain CGI/FCGI compatibility this
is better then nothing, but as You know, better performaces can be
reached in a WSGI environment in which you can let someething in RAM
memory for future use, similar to java servlets.

So I think web2py could have a good production environment similar to
servlets in parallel with actual one.

May be I'm the only one who think performance is a requirement?

[Massimo Di Pierro]

On May 19, 2011, at 5:40 PM, nihil wrote:

>
> May be I'm the only one who think performance is a requirement?
>

Not at all. I am on the road with limited ability to check emails and code. I will look at your proposal closer during the week end.

[Stifan Kristi]

you are not the only one nihil, for me performance is the most important thing

[Michele Comitini]

I think we all care about speed here... I think nihil ideas are valuable

Just one question @nihil on the porposed code. It is common to use
models as equivalent of jee filters or interceptors, so that
the incoming request is manipulated before it arrives to the handling
function in the controller. For instance I could
put the following in db.py (just a stupid example never use it!):

superuser = False
if auth.user.id < 100 and (session.login_time -
datetime.datetime.now()).seconds < 3600:
superuser = True

Would something like the above still work?

mic

2011/5/20 Stifan Kristi <steve.van...@gmail.com>:

[Martín Mulone]

I care a lot of speed, but if this improvements doesn't affect security!.

2011/5/20 Michele Comitini <michele....@gmail.com>

--

 http://martin.tecnodoc.com.ar

[nihil]

ok Michele, I don't think this code is usefull in a model on a mvc
paradigm, and does not affect any behavure in 'my' code.
If you want something is executing for a certain number of requests
you ca define simply a decorator in controller and code this in
decorator (this the python way ;) )

On 20 Mag, 14:53, Michele Comitini <michele.comit...@gmail.com> wrote:
> I think we all care about speed here... I think nihil ideas are valuable
>
> Just one question @nihil on the porposed code.  It is common to use
> models as equivalent of jee filters or interceptors, so that
> the incoming request is manipulated before it arrives to the handling
> function in the controller.  For instance I could
> put the following in db.py (just a stupid example never use it!):
>
> superuser = False
> if auth.user.id < 100 and (session.login_time -
> datetime.datetime.now()).seconds < 3600:
>    superuser = True
>
> Would something like the above  still work?
>
> mic
>

> 2011/5/20 Stifan Kristi <steve.van.chris...@gmail.com>:

[nihil]

On 20 Mag, 15:09, Martín Mulone <mulone.mar...@gmail.com> wrote:
> I care a lot of speed, but if this improvements doesn't affect security!.
>

> 2011/5/20 Michele Comitini <michele.comit...@gmail.com>

>
>
>
> > I think we all care about speed here... I think nihil ideas are valuable
>
> > Just one question @nihil on the porposed code.  It is common to use
> > models as equivalent of jee filters or interceptors, so that
> > the incoming request is manipulated before it arrives to the handling
> > function in the controller.  For instance I could
> > put the following in db.py (just a stupid example never use it!):
>
> > superuser = False
> > if auth.user.id < 100 and (session.login_time -
> > datetime.datetime.now()).seconds < 3600:
> >   superuser = True
>
> > Would something like the above  still work?
>
> > mic
>

> > 2011/5/20 Stifan Kristi <steve.van.chris...@gmail.com>:

[Anthony]

On Thursday, May 19, 2011 4:55:19 AM UTC-4, nihil wrote:

I think don't run models for every request is a good improvement, (why
redefine data models in every request?

 

But models can (and I think often do) include code that is conditional on the request. For example:

 

auth = Auth(globals(),db)

 

or

 

Field('created_on', 'datetime', default=request.now)

 

 

So, wouldn't we need the capability to define at least *some* model files that run on every request (and possibly define others that are fixed and do not need to run on every request)?

 

Anthony

 

[Michele Comitini]

2011/5/20 nihil <nihi...@gmail.com>:

> ok Michele, I don't think this code is usefull in a model on a mvc
> paradigm, and does not affect any behavure in 'my' code.
> If you want something is executing for a certain number of requests
> you ca define simply a decorator in controller and code this in
> decorator (this the python way ;) )

IMHO that's the java way: you would end up putting decorators all over
the code ;-)
You cannot put a decorator for the whole controller, but having
dynamic models you
can do the following:

if request.controller=='mycontroller':
do stuff

MVC as intended in most jee frameworks have a static model hence limited.
Of course 90% of the code in /models under web2py is made of constants so
it is silly to run that code each time. For instance table definition
can be skipped under "normal"
conditions...
So the question is: how do we split constant from dynamic environment?
Could we add a new dir contaning constant models?

mic

[nihil]

Yes Anthony You're right, infact I modified some Auth method in order
to store logged user informations in session so an auth object is
reads and writes the session object so i run successifully.

for something like default = request.now or other dynamically built
values you can use a lambda function if I remember correctly.

but I think also we can write an other application layer that will
runs for every request (but it's not implemented yet).

On 20 Mag, 21:17, Anthony <abasta...@gmail.com> wrote:
> On Thursday, May 19, 2011 4:55:19 AM UTC-4, nihil wrote:
>
> > I think don't run models for every request is a good improvement, (why
> > redefine data models in every request?
>
> But models can (and I think often do) include code that is conditional on
> the request. For example:
>

> auth = Auth(globals(),db)p

[nihil]

just to make a lot of brightness on this thread:

I use web2py since version 1.54 when I abbandoned Ruby on Rails
because its slowness and learned python language.
I was fashinating from web2py simplicity and i used it for a long time
but sometimes I had to modify some behavioure in web2py internals.
A cople of week ago i was debugging my application and i noticed
web2py runs a big amount of code before executing my application code
so i studied deepely its main architecture and i tryed to create an
environment in which web2py can run faster.

first of all i developed SessionManager class and a primeval form of
class Application than ran models once to testing its potetial speed
up; my tests sed that web2py can runs 4-5 time faster if precompiled
with a session manager and preexecuted models. "Wow, It's a good job"
I sed, but when i tested a real application in which were auth's
decorators i understood importance to execute even controllers within
same environment of models.

Now It runs successifully models, controllers and views. It has few
bugs (bigger one is in sessions with masterapp=something).

I know there are a lot of possible problems with this architecture (ie
scalability on a multi-core/multi-processor computer (i know how fix
it by multiprocessing module or I have a indea about it) or some
problems relating to everything is usually defined in models like AUTH
and CRUD).

I think if I could modify Auth class in order to run on this
optimizated web2py version I can do the same for other things that are
in web2py yet but to make this I need your help.

out of misundertandings: I don't think this code above is best
optimization solution. i think a very good solution can be another one
in which there are no persistent private environments (who eats a lot
of ram) but the one in which only session objects are private and
persistent (with a lower ram allocation).

so now the question is: "in your opinion is this way walkable?"

I'm very sorry for my english. :(

[nihil]

i'm sorry Michele for previous posts but i'm writing from a mobile
phone and sometimes doesn't upload posts entirely.
in second part of previous post i suggested exactely your solution to
have another layer to run for each request (a sort of MVCR where R
means Request) maybe one for each controller to do this kind of stuff.

I apologize again :-(

On 21 Mag, 00:22, Michele Comitini <michele.comit...@gmail.com> wrote:
> 2011/5/20 nihil <nihil...@gmail.com>:

[Pierre Thibault]

2011/5/20 Michele Comitini <michele....@gmail.com>

2011/5/20 nihil <nihi...@gmail.com>:

MVC as intended in most jee frameworks have a static model hence limited.--

Java has a simple and very efficient model. The fact that it is possible to plug something like JBoss Seam on top of Java Servlet is a good proof of that.

I believe an interceptor mechanism in web2py would be very welcome for a lot of tasks: every that must run at the global level.

The solution I have in mind is a new custom importer that will enable AOP, aspects oriented programming, in Python. I think this will be both simple and very powerful. Just like the previous importer I did to reload code changes it will be universal: it works with Python and not just with web2py.

As for speed, the actual custom importer makes it easy to put things in modules (the directory). Think about putting everything that does not need to be run at each request in modules. With this feature I don't think we need a special mechanism for things that must be run at each request because we already have it. It is trunk at the moment.

A+

-------------
Pierre
My blog and profile (http://pierrethibault.posterous.com)
YouTube page (http://www.youtube.com/user/tubetib)
Twitter (http://twitter.com/pierreth2)

[nihil]

hi Pierre can i ask you what your custom importer do?

[Pierre Thibault]

2011/5/20 nihil <nihi...@gmail.com>

hi Pierre can i ask you what your custom importer do?

--

Hi Nihil,

It enables the usage of modules and packages in the 'modules' directory of each applications. Before that, we were force to use local_import. Now, it just works like if the modules directory was in the Python path.

Also, if you edit gluon/custom_import.py, we will see a DEBUG variable. If you set this variable to True before starting web2py, the importer will track the modification made to module files, all modules not just the ones in 'modules', and will reload the last version available when import statement is made.

This is available in trunk.

--

[nihil]

Oh ok, that's good in my opinion.

In pratiice it means i can write my data model and everything i consider a static value in modules directory and i can import it where i need to? (for each request)

this can speed up web2py application.

Good job.

[Pierre Thibault]

2011/5/21 nihil <nihi...@gmail.com>

Yes, this is the idea. So special stuff needed.

[nihil]

Erery one who is interested in how much it can speed up an application can try downloading this.

If you're running Linux or MacOS you can put a symlink in applications directory that links to a real web2py app before launcing web2py as usual.

You can test performances inspecting elements in chrome browser at network tab. or similar tools (Firebug on firefox or dragnofly under opera).

Better reliable tests are done using 2 separeate computers (one for browser and one for server).

repeat this test with standard web2py implementation.

I know there are bugs, especially in making stardad menu because it's defined in a model but you can fix it looking at installed "init" app. You can use auth.settings.login_onaccept and auth.settings.logout_onlogout to modifying session menus.

Admin interface works (not very well, but it shall be disabled in a real production environment). you can change application code as usual but changing in views does'nt affects subviews while you don't save explicitely subview.

IE: if you change layout.html and default/index extends layout you must save layout and default/index.

[Massimo Di Pierro]

I finally was able to take a look at your code. What you did is interesting and you clearly put a lot of effort into it. Unfortunately this code cannot go into web2py.

First of all it is true that it does speed up web2py because the models are executed once per session. But that is also one of the reasons teh code cannot go into web2py. It causes a major break of backward compatibility. There is code in the models that is expected to be called at every request.

There is a reason the code is written the way it is. One of the reasons is that for scalability different requests could be executed on different servers. In your case the environment only lives in ram and cannot be shared among processes.

Another issue is database connection. If you do not call DAL(...) at every request, connection pooling does not work as it should.

There is a value in what you are proposing and we can think about creating a new framework that borrows some ideas from web2py (dal, template, html, sqlhtml, tools), some from other frameworks (I like bottle, it is fast) and it could have what you suggest (where basically environment is the session). Yet something like this would require a lot of work.

I have been postponing this discussion to when python 3 is more mature.

Massimo

[Pierre Thibault]

From the web2py book:

Efficiency Tricks

web2py application code is executed on every request, so you want to minimize this amount of code. Here is what you can do:

• Run once with migrate=True then set all your tables to migrate=False.
• Bytecode compile your app using admin.
• Use cache.ram as much as you can but make sure to use a finite set of keys, or else the amount of cache used will grow arbitrarily.
• Minimize the code in models: do not define functions there, define functions in the controllers that need them or - even better - define functions in modules, import them and use those functions as needed.
• Do not put many functions in the same controller but use many controllers with few functions.
• Call session.forget(response) in all controllers and/or functions that do not change the session.
• Try to avoid web2py cron, and use a background process instead. web2py cron can start too many Python instances and cause excessive memory usage.

 
But why do we have to manually call forget on the session? web2py does not know if the session has changed?

[Pierre Thibault]

I did an article about AOP on by blog. I think this can inspired Massimo for AOP in web2py or for having an interceptor like mechanism like Java EE has.

http://pierrethibault.posterous.com/python-aop-do-it-yourself


--

[nihil]

Thank You Massimo for your consideration,

I've in mind a way to fix issues above:

I'm yet designing a better solution with follows ideas:
One only environment for each application created executing every
db*.py (models with name starting with 'db') in model tree at start
up.
Other model files (IE menu.py) is executed for each request as now
(It could simplify migration).
Controllers will be compiled and executed once at start up,
The same for views,

the trick is in environment variable where request is executed in (not
dict instance, but a new class).

I think Massimo is right about backward compatibility breaks and
calability issues. As i wrote some posts above I know, and theese are
my suggestion to fix them:

- Retrocompatibility can be guarteed by an option in commandline or
options.py that swithes over behaviors (IE -precompiled -
multiprocessing -relaynetwork)

- Scalability can be guaranteed by multiprocessing python module for a
single machine whith more cores/processors.
- Scalability can be guaranteed in a server network in two possible
ways simple:(coupling Client-IP and Server-IP in router function, who
demands server which executed that client in prevous client request),
and hard way (making a simple overlay nework)

-Database connection when a sole intance of DAL is referred by anyone
who is using an appllication? (I don't know, :( but proposals are
always welcome).
(maybe considering reimplents some parts of DAL to work with a
connection pool? =)


On 21 Mag, 23:24, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

[Massimo Di Pierro]

How about having a folder called models/static and everything in there is executed once and only one? that would prepare the environment for following requests. It would case no conflict or backward compatibility issues since there can be no controller called static.py.

Mind that DAL already does connection pooling. The issue is what triggers the pooling. Right now a call to DAL(...).

Even without pooling if db=DAL(...) is executed only once there are two problems:
- if there are no requests for a long period of time, the db engine closes the connection. Who restarts it?
- if there are two concurrent connections and there is one db connection the requests have to be serialized. This would move the handling of concurrent database requests from the database to web2py. Not a good idea. db=DAL(...) and table definitions should not in the pre-defined environment.

[Anthony]

On Sunday, May 22, 2011 10:02:15 AM UTC-4, Massimo Di Pierro wrote:

How about having a folder called models/static and everything in there is executed once and only one? ... db=DAL(...) and table definitions should not in the pre-defined environment.

 

Aren't table definitions the main thing you'd want to put in models/static? Without table definitions, how useful would models/static be?

 

Anthony

[Michele Comitini]

Ciao Massimo,

2011/5/22 Massimo Di Pierro <massimo....@gmail.com>:

> Even without pooling if db=DAL(...) is executed only once there are two problems:
> - if there are no requests for a long period of time, the db engine closes the connection. Who restarts it?

I solved this some time ago, I sent the patch. If you can take time
to look into it. It does not add overhead in case the connection
is already up (i.e. it does not add extra sql instructions). It also
manages postgresql better...

mic

[Michele Comitini]

Table definitions are static objects under normal circustamces.
Static here means that they are shared among all sessions and requests.
Also the pool is a static object already also its manager can be static.
Parameters like number of connections are also static.

The pool manager must be called to action when:
1) tables are created/migrated
2) transaction is opened
3) transaction is closed

1,2,3 are not care of average application code.

mic

2011/5/22 Anthony <abas...@gmail.com>:

[Michele Comitini]

What you do with jee filters can be done very well using WSGI
wrapping. web2py does not allow simple WSGI extending by design. The
idea is to
simplify the work of developers, but you can still use it if you want.

Talking about WSGI... IMHO part of DAL work (pooling + transaction
management) would fit nicely as a WSGI shell around web2py

mic

2011/5/21 Pierre Thibault <pierre.t...@gmail.com>:

[Massimo Di Pierro]

you did? I do not have it. Perhaps I lost it.

[Massimo Di Pierro]

Do you have any benchmark numbers?
I always thought that the type to bytecode compile is negligible compared to the time of execution.

Massimo

On May 22, 2011, at 8:19 AM, nihil wrote:

[Martín Mulone]

Yesterday I made some test of the dal in bottle, and I'm very surprise how well dal was, very fast response also with migrate true all the time. So I start to think is not all the dal fault :P. 

2011/5/22 Massimo Di Pierro <massimo....@gmail.com>

Do you have any benchmark numbers?

--

 http://martin.tecnodoc.com.ar

[nihil]

so web2py can be execute in a sole environment for each application in
wich use DAL as Michele's version and use object proxies for request,
response and session objects

this is perfectly compatible with auth decorators, crud objects and
maybe all of web2py objects usually created in models

a new kind of session could be created to fit new environment like in
my SessionManager

I thnk this chabges can be base for a major version numbering (web2py
2.0) so we can create a document with all theese deadlines/proposals
and then discuss about this.

On 23 Mag, 00:26, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

> you did? I do not have it. Perhaps I lost it.
>
> On May 22, 2011, at 3:37 PM, Michele Comitini wrote:
>
> > Ciao Massimo,
>

> > 2011/5/22 Massimo Di Pierro <massimo.dipie...@gmail.com>:

[Massimo Di Pierro]

Nihil,

I would like to see a test case and some benchmarks. I am not convinced yet this is worth the trouble but could just be I do not fully understand what you propose.

Massimo

[nihil]

There is no code implemented about this changes, It's only on my mind
for now and I have not many time in this period to make this kind of
stuff. I will do an implementation and test case when i can (i hope in
next weeks).

To do this stuff I need Michele's DAL implementation.


On 23 Mag, 16:19, Massimo Di Pierro <massimo.dipie...@gmail.com>

[Massimo Di Pierro]

Before you spend more time on this. Can you provide an example from the user prospective. It does not need to be working.
Just show us how you expect people to change the way the code in web2py.

massimo

[nihil]

From user point of view there will be no differences except they were
not forced to follow optimization tricks because code will perform
better just as is

Differences from now are web2py internal not in user application and
are follows:

- Code compiled permanently in RAM (no file read needed)
- static and application-wide models (no table definition needed)
- ? session-wide translator object (i dont'know if will make sense).
- request,response, and session will be proxied to current thread
request,response, session (a little slower but for compatibility
reasons)
- session object lives in RAM(no marshaling ,file read and write
needed)

perhaps session could be a new class for compatibility reason.

I don't know if static model can make sense or if every model could be
static, condiering proxy's dynamicity.

IE: in static models:
db.create_table('pippo',
Field('name',default = request.now),
Field('image','upload',uploadfolder = request.folde + 'bla/bla'),
)
auth = Auth(globals(),db)

will do exactely what we expect in both system as the same

when you execute someting like:
db(db.pippo.id == request.args[0]).select()

the same as in previous example.
It will perform same operations as now.

because request.args[0] refers allways the same request.args object,
db (Michele's version) will runs same code in as before.

maybe i'm wrong, some little aspect in secundary objects couldn't runs
correctly, we can fix bugs.


On 23 Mag, 17:55, Massimo Di Pierro <massimo.dipie...@gmail.com>

[Massimo Di Pierro]

> - Code compiled permanently in RAM (no file read needed)

I once benchmarked this. I did not see any improvement. Do you have data?

> - request,response, and session will be proxied to current thread
request,response, session (a little slower but for compatibility
reasons)

How is this different then current.request which is already in trunk?

> - session object lives in RAM(no marshaling ,file read and write
needed)

This is not a good idea. web2py must be able to handle multiple concurrent processes and they cannot share objects in ram.
As should allow sessions in cookies eventually (digitally signed, like Flask) but not in ram.

- static and application-wide models (no table definition needed)

This is the part I do not fully understand because I would not know how to make it work myself without breaking lots of stuff, including connection pooling, and dealing with database timeouts and I am not convinced there is a real benefit. There is the part that needs a working proof of concept, perhaps with bottle.

[nihil]

On 23 Mag, 19:39, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

> > - Code compiled permanently in RAM (no file read needed)
>
> I once benchmarked this. I did not see any improvement. Do you have data?
>

I haven't; I know restricted() function or run models_in()
run_controllers_in and run_views_in() takes time not very long time
but allways useless time especially in aprodutione where code
doesn'change from a request to another

> > - request,response, and session will be proxied to current thread
>
> request,response, session (a little slower but for compatibility
> reasons)
>
> How is this different then current.request which is already in trunk?
>

i didn't get time to look into it but maybe the same.

> > - session object lives in RAM(no marshaling ,file read and write
>
> needed)
>
> This is not a good idea. web2py must be able to handle multiple concurrent processes and they cannot share objects in ram.
> As should allow sessions in cookies eventually (digitally signed, like Flask) but not in ram.

as i said some posts ago this is not a must. It's only a choice a
developer can choose by parameters in options.py or in commandline
options (no prescrictivity).

>
> - static and application-wide models (no table definition needed)
>
> This is the part I do not fully understand because I would not know how to make it work myself without breaking lots of stuff, including connection pooling, and dealing with database timeouts and I am not convinced there is a real benefit. There is the part that needs a working proof of concept, perhaps with bottle.

I was referring what Michele said about his DAL.
If I'm not wrong Michele just fixed theese stuff.

[Pierre Thibault]

I hope you are not forgetting the concerns about having an interceptor mechanism or a hook for aspects oriented programming.
--

[nihil]

i think you're right but i didn't undertand how.
please suggest.

On 23 Mag, 22:37, Pierre Thibault <pierre.thibau...@gmail.com> wrote:
> I hope you are not forgetting the concerns about having an interceptor
> mechanism or a hook for aspects oriented programming.
> --
>
> A+
>
> -------------
> Pierre
> My blog and profile

> (http://pierrethibault.posterous.com)<http://pierrethibault.posterous.com>
> YouTube page (http://www.youtube.com/user/tubetib)<http://www.youtube.com/user/tubetib>
> Twitter (http://twitter.com/pierreth2) <http://twitter.com/pierreth2>

[pbreit]

Nihil, are you able to propose any smaller changes that would be beneficial? It sounds like you have some major changes in mind which might be difficult to introduce all at once.

[nihil]

no, I haven't (for now). ;)
but theese things can be coded once a time because theese are not
strongly related.

i also think when you find a code that is executed many times and
returns allways the same result i think this code could be cached.
and caching much code as we can is a good practice.

[Pierre Thibault]

2011/5/23 nihil <nihi...@gmail.com>

i think you're right but i didn't undertand how.
please suggest.

On 23 Mag, 22:37, Pierre Thibault <pierre.thibau...@gmail.com> wrote:
> I hope you are not forgetting the concerns about having an interceptor
> mechanism or a hook for aspects oriented programming.
> --
>

I'll give an example. Suppose you have a web2py server running 12 apps and your boss say something like "It would be good to know from which web sites people are coming from.". How are you going to implement the collect of this information? Are you going to modify 12 apps? For sure, you want a way to do it only once. And here comes the idea of filters and interceptors from the Java world. It offers a place to run code and treat the server as a whole entity.

As I know, web2py is not offering an easy solution to this problem. I feel that it would be great to address the issue at some point in time.

--


A+

-------------
Pierre
My blog and profile (http://pierrethibault.posterous.com)

[nihil]

i undertand behavior i undestand what is AOP, but not understand how
you can put it into web2py.
baybe a sort of supermodel (a model that acts in each applications)?

On 24 Mag, 00:54, Pierre Thibault <pierre.thibau...@gmail.com> wrote:
> 2011/5/23 nihil <nihil...@gmail.com>

[Massimo Di Pierro]

I do not feel so good about this type of supermodels.

Massimo

[Pierre Thibault]

2011/5/23 nihil <nihi...@gmail.com>

i undertand behavior i undestand what is AOP, but not understand how
you can put it into web2py.
baybe a sort of supermodel (a model that acts in each applications)?

I thought about a folder called 'request_filters' running like 'models' but for each request.


2011/5/23 Massimo Di Pierro <massimo....@gmail.com>

I do not feel so good about this type of supermodels.

Do you have something else in mind for solving the kind of problems that I described?

--


A+

-------------
Pierre
My blog and profile (http://pierrethibault.posterous.com)

[pbreit]

I can't imagine that being a feature of the framework. I'd use Google Analytics or write a small amount of code to do it.

[Michele Comitini]

You can write a WSGI layer that logs incoming connections.  Easier than Java bloat ...

Mic

Il giorno 24/mag/2011 02:38, "pbreit" <pbreit...@gmail.com> ha scritto:
> I can't imagine that being a feature of the framework. I'd use Google
> Analytics or write a small amount of code to do it.
>

[Pierre Thibault]

2011/5/24 Michele Comitini <michele....@gmail.com>

You can write a WSGI layer that logs incoming connections.  Easier than Java bloat ...

Mic

This only works if running under WSGI.

But it seems that the feature is less useful that I thought it was.

[Massimo Di Pierro]

web2py is a WSGI application. All requests go over wsgi.