Fwd: My thoughts on web2py

99 views
Skip to first unread message

Massimo Di Pierro

unread,
Aug 2, 2009, 2:21:49 PM8/2/09
to web...@googlegroups.com
Thank you Armin, I am passing this to the list and will address them asap.

Begin forwarded message:

From: Armin Ronacher <armin.r...@active-4.com>
Date: August 2, 2009 1:10:29 PM CDT
To: "DiPierro, Massimo" <MDiP...@cs.depaul.edu>
Subject: My thoughts on web2py

Hi Massimo,

First a word of warning.  I send you this mail in private because I want
to share my thoughts on the web2py code and because I hope I can help
with that somehow.  You're free to publish this mail if you want.

As you might remember I came in contract with web2py a long ago when you
were evaluating pygments as a syntax highlighter for the web2py
documentation or something and you decided to implement your own because
you were unable to "ship" pygments as part of web2py.  As far as I
remember py2app or py2exe were not able to pick up the on-demand imports
in the lexer module.

I have been watching the development ever since because (And please
don't think bad of me because of that) it appeared to me that web2py was
a joke.  I remember reading some very early revisions of your PDF
documentation where I remember a code that looks something like that:

   db.define_table('something', db.Field('meh', 'date',
                   default=date.today()))

That line freaked me out.  This line alone showed me that web2py is
doing something very, very weird.  From this line I could see that it
means one of the following two things:

- either the code has a bug for long running applications (in this case
 when the application code runs longer than a day in which case the
 default would have to change dynamically)

- or the global scope has different semantics and is reevaluated each
 request.

Turns out the latter is what happens in web2py.  There still is a race
condition but only a small one.  The bigger problem with this however is
that this means the (not so) static definitions are reevaluated each
request.

Especially for the database table definitions (which cause many function
calls) this means an enormous per-request penality if you have many of
them.  I even remember reading a case like this in the web2py newsgroup
about someone who had twenty table definitions or something and the
majority of the request time was spent setting up the tables.

However that also means another thing.  Standard Python optimizations no
longer apply.  In Python usually modules are cached in an interpreter
bound dictionary for reusing.  This of course does not apply for web2py.
However the second problem here is that only the local scope gets
certain optimizations in cpython such as fast-local lookup.  Instead of
storing the variables in a dictionary they are instead stored in an an
object where items are looked up by index (fast locals) rather than by a
string.

Now of course one can argue that this is the way it's intended.  That's
nice but for that, the documentation is lacking.  There is no
explanation of what implications this has on the code.  And in
comparison with "enterprise ready" that gives me a strange feeling about
the project.

I'm not even exactly sure what kills web2py for me in the end, but I'm
pretty sure that part of the reason is the way the project is advertised.

web2py does not have worse code than many popular libraries out there.
BeautifulSoup comes to mind.  That library does some really ugly things
there including monkeypatching of the standard library.  It does
different things depending on the version of Python it's running on and
in the code are some really ugly idioms like a unicode subclass that has
a reference to the parsed tree that results in memory leaks very easily
because it's nearly impossible to get rid of the reference (actually
just by exploiting some weird Python internals that are mostly
undocumented).  But the author of BeautifulSoup does not advertise it as
a backwards-compatible, enterprise ready HTML parser.

The web2py code on the other hand has some really problematic idioms as
well.  The execution of models, views and whatnot in a made-up namespace
comes to mind.  Due to the decision of being backwards compatible it
will be nearly impossible to "fix" those things.  However there are so
many implications nobody yet knows about which is terribel for large
projects.

For example the database table redfinition problem cited above.  Of
course that particular problem can be worked around with making the
definition code internally lazy or something but there can be more
nobody yet knows about.

Especially side-effects of code execution can do some terrible damage.
Decorators that would register code would instantly leak for example.
And some libraries provide auto-registration that you would not
instantly know about.

Also libraries like pickle do not support weird situations like this.

So I guess in the end I'm not (only) complaining about the web2py code
or decisions there, but mainly how web2py is advertised as enterprise
ready when nobody knows how these idioms scale or behave.

Another part is that web2py (and this is something web2py has in common
with so many other web frameworks) is the craze to reimplement
everything.  I know you don't want to pull depedencies in but from what
I can see you're providing one-click installers for the framework which
could ship external dependencies.  Both py2exe and py2app provide
support for external dependencies and I can't see what "not depending on
webob/werkzeug" gives you there.

The reason why I'm addressing this is that there are so many
limitations, problems and weird aspects in WSGI and HTTP that are hard
to tackle and many people get wrong.  I'm not saying you're getting it
wrong, I haven't even studied the code in question, but I'm pretty sure
it would make sense to collaborate on that.

About the code quality itself, not the (IMO) broken concepts I picked
some of the snippets I found quickly searching with the help of ack:

- Missing `close()` calls with files (ignored matches from compileapp):

 gluon/fileutils.py:160:    f.write(open(tarname, 'rb').read())
 gluon/fileutils.py:246:        data = open(filename, 'rb').read()
 gluon/highlight.py:313:    data = open(sys.argv[1]).read()
 gluon/languages.py:37:    lang_text = open(filename, 'r').read()...
 gluon/languages.py:153:        data = open(file, 'r').read()
 gluon/main.py:107:web2py_version = open(os.path.join(web2py_path, ...
 gluon/rewrite.py:26:        exec open('routes.py', 'r').read() ...
 gluon/template.py:104:            text = open(os.path.join(path, ...
 gluon/template.py:118:            parent = open(t, 'rb').read()
 gluon/template.py:134:            child = open(t, 'rb').read()
 gluon/tools.py:1934:        return urllib.urlopen(url).read()
 gluon/widget.py:55:ProgramVersion = open('VERSION', ...

- Same in streamer.py for checking for file existence
 (open(static_file))

- Template engine seems to be missing proper escapes.  Having ''' in the
 template should be enough to break it.

- Whitespace between "else" and colon in the template engine is unlikely
 to work with the current parsing rules as far as I can see.

- xmlrpc system sends text/xml content type which is with a missing
 charset property set to us-ascii.  No charset is set so it has to be
 specified explicitly (proper header would be application/xml;
 charset=utf-8)

- Fileutils has a listdir method that does very inefficient deleting
 from the list.  Current code.

 50         for dir in dirs[:]:
 51             if dir.startswith('.'):
 52                 dirs.remove(dir)

 That creates a shallow copy, for each item it performs an O(n) lookup
 for the item and the remove is a O(n) operation as well.  Even
 unecessary because you have the position if you enumerate over the
 list.

 However you can do that with a list comprehension:

 50         dirs[:] = [x for x in dirs if not x.startswith('.')]

- Having classes in local scope (fileutils.py for the tarfile) makes it
 incredible hard to reuse code.  Not a good idea, move that into the
 global scope instead.  Will also help performance.

- There is no bug in shutil.copyfileobj.  That function works perfetly
 and does what the documentation says it should do.  Maybe you were
 copying from the wsgi input stream.  That of course won't work that
 easy because there is no end-of-file marker in the input stream.  See
 werkzeug's LimitedStream for ways how to fix that problem.  A similar
 class also exists in WebOb but I don't remember the name.

- You should not inherit from BaseException unless you are dealing with
 special exceptions that should not be catched by an "except Exception"

- there are realtive imports all over the place.  This is a deprecated
 unfeature, switch to "from gluon.html import blah" instead of the
 current "from html import blah".

- the email code in tools is vulnerable to header injections if the
 input is unfiltered.  If this is intentional it should be documented
 in the code.

- There is no proper undecoding of the path info in main.py but spaces
 are transformed into underscores.  However a plus is converted into a
 space as well which is against the RFC.  Plus is only an alias for %20
 in the querystring of a URL.

- request.vars is either a dict of lists or a dict of plain strings.
 This means the user has to perform instance checks each access, not
 very clever.  Other systems fix that by providing multi-value dicts.

- time.strftime should not be used for header-date formatting because
 it's affected by locales.  That means if a user is using locales it
 will break all headers sent that deal with dates.

This list was created in ~15 minutes from quickly checking some files.
If one does a check on the code more things will come up.  Some things
are also underspecified or unclear in the code.  For example how the
code is supposed to deal with unicode (it does that in very different
ways) and how headers are quoted/unquoted, how URLs are decoded etc.
Also the code architecture is suboptimal in places.  Hardcoded support
for Google appengine in places where you wouldn't expect it (like
validators etc.)

For an "enterprise" framework this is a bit too much smelly code that is
found in no time.

What would I fix?  Stop advertising the framework as enterprise ready
until it matured more.  And do less advertising in general.  And if you
can, fix some of the things in the code.  Those that are caused by
design choices won't be fixed anyway.


Regards,
Armin


Pynthon

unread,
Aug 2, 2009, 2:41:50 PM8/2/09
to web2py-users
Nice, I know I think who this is.. However, do you all going to fix
this and when do you think the new version will be available?

Thanks

On 2 aug, 20:21, Massimo Di Pierro <MDiPie...@cs.depaul.edu> wrote:
> Thank you Armin, I am passing this to the list and will address them  
> asap.
>
> Begin forwarded message:
>

Fran

unread,
Aug 2, 2009, 3:05:27 PM8/2/09
to web2py-users
On Aug 2, 7:21 pm, Massimo Di Pierro <MDiPie...@cs.depaul.edu> wrote:
> > From: Armin Ronacher <armin.ronac...@active-4.com>
> > - or the global scope has different semantics and is reevaluated each
> >  request.
> > Turns out the latter is what happens in web2py.  There still is a race
> > condition but only a small one.  The bigger problem with this  
> > however is
> > that this means the (not so) static definitions are reevaluated each
> > request.
> > Especially for the database table definitions (which cause many  
> > function
> > calls) this means an enormous per-request penality if you have many of
> > them.  I even remember reading a case like this in the web2py  
> > newsgroup
> > about someone who had twenty table definitions or something and the
> > majority of the request time was spent setting up the tables.

Easily fixed: In Production set migrate=False
This can be done globally by setting all tables to use migrate=migrate
& having migrate=True during development & migrate=False for
Production.
No big issue at all...

> > Now of course one can argue that this is the way it's intended.  
> > That's
> > nice but for that, the documentation is lacking.

Seems clear enough to me in the Manual: pg 89

Thanks for the other suggestions - I'm sure that some of those can be
easily incorporated to make the toolkit better :)

F

mdipierro

unread,
Aug 2, 2009, 3:21:27 PM8/2/09
to web2py-users
I am about to catch a plane so I will be short. We can talk more about
this in a couple of days.

> > First a word of warning.  I send you this mail in private because I  
> > want
> > to share my thoughts on the web2py code and because I hope I can help
> > with that somehow.  You're free to publish this mail if you want.

This is constructive criticism so I appreciate it.
Correct.

> There still is a race
> > condition but only a small one.

No there is not. Why do you think there is a race condition?

> The bigger problem with this  
> > however is
> > that this means the (not so) static definitions are reevaluated each
> > request.

Correct. A small price to pay for a lot of flexibility.
We are working on lazy evaluation of tables.
There is also a speedup if you press the "compile" button in admin.

> > Especially for the database table definitions (which cause many  
> > function
> > calls) this means an enormous per-request penality if you have many of
> > them.  I even remember reading a case like this in the web2py  
> > newsgroup
> > about someone who had twenty table definitions or something and the
> > majority of the request time was spent setting up the tables.

There was a slow down problem that has been fixed by Alexey. I agree
there is still room for improvement.

> > However that also means another thing.  Standard Python  
> > optimizations no
> > longer apply.  In Python usually modules are cached in an interpreter
> > bound dictionary for reusing.  This of course does not apply for  
> > web2py.
> > However the second problem here is that only the local scope gets
> > certain optimizations in cpython such as fast-local lookup.  Instead  
> > of
> > storing the variables in a dictionary they are instead stored in an an
> > object where items are looked up by index (fast locals) rather than  
> > by a
> > string.
>
> > Now of course one can argue that this is the way it's intended.  
> > That's
> > nice but for that, the documentation is lacking.

Yes it is true that documentation is lacking.

>  There is no
> > explanation of what implications this has on the code.  And in
> > comparison with "enterprise ready" that gives me a strange feeling  
> > about
> > the project.
>
> > I'm not even exactly sure what kills web2py for me in the end, but I'm
> > pretty sure that part of the reason is the way the project is  
> > advertised.

We prefer to talk about "advocating" not "advertising" since we do not
sell it.
Mistakes were made on my side.

> > web2py does not have worse code than many popular libraries out there.

Thanks. I take is as a compliment.

> > BeautifulSoup comes to mind.  That library does some really ugly  
> > things
> > there including monkeypatching of the standard library.  It does
> > different things depending on the version of Python it's running on  
> > and
> > in the code are some really ugly idioms like a unicode subclass that  
> > has
> > a reference to the parsed tree that results in memory leaks very  
> > easily
> > because it's nearly impossible to get rid of the reference (actually
> > just by exploiting some weird Python internals that are mostly
> > undocumented).  But the author of BeautifulSoup does not advertise  
> > it as
> > a backwards-compatible, enterprise ready HTML parser.
>
> > The web2py code on the other hand has some really problematic idioms  
> > as
> > well.  The execution of models, views and whatnot in a made-up  
> > namespace
> > comes to mind.

Yes. It was a design decision to be closer to Rails than Django in
this respect. I think this is what is making web2py popular, more than
anything else.

> Due to the decision of being backwards compatible it
> > will be nearly impossible to "fix" those things.

We think of them as features, not bugs.

> However there are so
> > many implications nobody yet knows about which is terribel for large
> > projects.
>
> > For example the database table redfinition problem cited above.  Of
> > course that particular problem can be worked around with making the
> > definition code internally lazy or something but there can be more
> > nobody yet knows about.

Alexey has already done this. It achieves a 10-20% speedup. I am
rewriting the DAL completely to make it easier to port on other
backends and faster. The new one may support lazy evaluations of
tables. Time permitting,

> > Especially side-effects of code execution can do some terrible damage.
> > Decorators that would register code would instantly leak for example.
> > And some libraries provide auto-registration that you would not
> > instantly know about.

Can you provide an example of what you mean? I am pretty sure there is
no leak.

> > Also libraries like pickle do not support weird situations like this.
> > So I guess in the end I'm not (only) complaining about the web2py code
> > or decisions there, but mainly how web2py is advertised as enterprise
> > ready when nobody knows how these idioms scale or behave.

I disagree with "nobody knows how these idioms scale or behave"?
We have been doing this for 2 years. We know very well how it works
and scales.
It actually works very well because as soon as a request is completed
the environment is deleted, this is clean, avoids conflicts when
multiple apps are running, and prevents memory leaks.

> > Another part is that web2py (and this is something web2py has in  
> > common
> > with so many other web frameworks) is the craze to reimplement
> > everything.  I know you don't want to pull depedencies in but from  
> > what
> > I can see you're providing one-click installers for the framework  
> > which
> > could ship external dependencies.  Both py2exe and py2app provide
> > support for external dependencies and I can't see what "not  
> > depending on
> > webob/werkzeug" gives you there.

web2py started as a teaching tool. It was important (and still is)
that I can go over every line of code with my students know exactly
why it is there.

> > The reason why I'm addressing this is that there are so many
> > limitations, problems and weird aspects in WSGI and HTTP that are hard
> > to tackle and many people get wrong.

I agree.

>  I'm not saying you're getting it
> > wrong, I haven't even studied the code in question, but I'm pretty  
> > sure
> > it would make sense to collaborate on that.

I like the idea. Some conditions have to be met:
1) we have a new contributor license agreement on web2py-developers
2) patches must not break backward compatibility
3) patches must either make the code smaller/faster, or add a needed
functionality.

> > About the code quality itself, not the (IMO) broken concepts I picked
> > some of the snippets I found quickly searching with the help of ack:
>
> > - Missing `close()` calls with files (ignored matches from  
> > compileapp):
>
> >  gluon/fileutils.py:160:    f.write(open(tarname, 'rb').read())
> >  gluon/fileutils.py:246:        data = open(filename, 'rb').read()
> >  gluon/highlight.py:313:    data = open(sys.argv[1]).read()
> >  gluon/languages.py:37:    lang_text = open(filename, 'r').read()...
> >  gluon/languages.py:153:        data = open(file, 'r').read()
> >  gluon/main.py:107:web2py_version = open(os.path.join(web2py_path, ...
> >  gluon/rewrite.py:26:        exec open('routes.py', 'r').read() ...
> >  gluon/template.py:104:            text = open(os.path.join(path, ...
> >  gluon/template.py:118:            parent = open(t, 'rb').read()
> >  gluon/template.py:134:            child = open(t, 'rb').read()
> >  gluon/tools.py:1934:        return urllib.urlopen(url).read()
> >  gluon/widget.py:55:ProgramVersion = open('VERSION', ...
>
> > - Same in streamer.py for checking for file existence
> >  (open(static_file))

I do not see the problem. Files are automatically closed when the file
object goes out of scope.
Can you provide an example when it does not?

> > - Template engine seems to be missing proper escapes.  Having ''' in  
> > the
> >  template should be enough to break it.

I do not believe that is the case but I may be wrong. Can you provide
an example of what you say?

> > - Whitespace between "else" and colon in the template engine is  
> > unlikely
> >  to work with the current parsing rules as far as I can see.

I agree it will not work.

> > - xmlrpc system sends text/xml content type which is with a missing
> >  charset property set to us-ascii.  No charset is set so it has to be
> >  specified explicitly (proper header would be application/xml;
> >  charset=utf-8)

This has been fixed by user Mateusz. The patch will be incorporated
soon.

> > - Fileutils has a listdir method that does very inefficient deleting
> >  from the list.  Current code.
>
> >  50         for dir in dirs[:]:
> >  51             if dir.startswith('.'):
> >  52                 dirs.remove(dir)
>
> >  That creates a shallow copy, for each item it performs an O(n) lookup
> >  for the item and the remove is a O(n) operation as well.  Even
> >  unecessary because you have the position if you enumerate over the
> >  list.
> >  However you can do that with a list comprehension:
>
> >  50         dirs[:] = [x for x in dirs if not x.startswith('.')]

Thanks. We will do so.

> > - Having classes in local scope (fileutils.py for the tarfile) makes  
> > it
> >  incredible hard to reuse code.  Not a good idea, move that into the
> >  global scope instead.  Will also help performance.

Thanks. We will do so.

> > - There is no bug in shutil.copyfileobj.  That function works perfetly
> >  and does what the documentation says it should do.  Maybe you were
> >  copying from the wsgi input stream.  That of course won't work that
> >  easy because there is no end-of-file marker in the input stream.  See
> >  werkzeug's LimitedStream for ways how to fix that problem.  A similar
> >  class also exists in WebOb but I don't remember the name.

I am not sure what you are referring to here. Is this about a comment
in the code?

> > - You should not inherit from BaseException unless you are dealing  
> > with
> >  special exceptions that should not be catched by an "except  
> > Exception"
>
> > - there are realtive imports all over the place.  This is a deprecated
> >  unfeature, switch to "from gluon.html import blah" instead of the
> >  current "from html import blah".

This is an aesthetic issue. The current code works well with all
supported python interpreters. Anyway, I do not oppose changing it,
but it is tedious to do.

> > - the email code in tools is vulnerable to header injections if the
> >  input is unfiltered.  If this is intentional it should be documented
> >  in the code.

Can you provide an example? If this is the case, it is not
intentional.

> > - There is no proper undecoding of the path info in main.py but spaces
> >  are transformed into underscores.  However a plus is converted into a
> >  space as well which is against the RFC.  Plus is only an alias for  
> > %20
> >  in the querystring of a URL.

web2py does not allow spaces in incoming urls. It just replaces them
with _. It should also replace + with _ . Are you saying that is not
the case? Perhaps you see a problem that I do not see. can you provide
an example of when this would be a problem?

> > - request.vars is either a dict of lists or a dict of plain strings.
> >  This means the user has to perform instance checks each access, not
> >  very clever.  Other systems fix that by providing multi-value dicts.

This cannot be changed because would break backward compatibility.
I think it is fine as it is. This is how the cgi module parses vars.

> > - time.strftime should not be used for header-date formatting because
> >  it's affected by locales.  That means if a user is using locales it
> >  will break all headers sent that deal with dates.

We do specify the format string. How do you suggest we change it?

> > This list was created in ~15 minutes from quickly checking some files.
> > If one does a check on the code more things will come up.  Some things
> > are also underspecified or unclear in the code.  For example how the
> > code is supposed to deal with unicode (it does that in very different
> > ways) and how headers are quoted/unquoted, how URLs are decoded etc.
> > Also the code architecture is suboptimal in places.  Hardcoded support
> > for Google appengine in places where you wouldn't expect it (like
> > validators etc.)

Unfortunately that is hard to avoid. GAE does things differently.

> > For an "enterprise" framework this is a bit too much smelly code  
> > that is
> > found in no time.
> > What would I fix?  Stop advertising the framework as enterprise ready
> > until it matured more.  And do less advertising in general.  And if  
> > you
> > can, fix some of the things in the code.  Those that are caused by
> > design choices won't be fixed anyway.

As I mentioned we say "advocating" not "advertising". The "enterprise"
feature of web2py is not that it is perfect (and I do not know of any
enterprise application that is perfect, what would perfect mean
anyway).

By "enterprise" we mean:
- our target users are small and medium businesses and non-profits
(the term enterprise includes both)
- we understand the needs of users by guaranteeing backward
compatibility thus protecting their long term investment in web2py
(this is not negotiable)
- we care about usability and thus we try make web2py as easy as
possible.
- we fix bugs regularly, usually within 24 of when they are reported.

"enterprise" does not mean that code cannot be improved. There is no
code that cannot be improved.
I will look into your comments more in details and I encourage other
users to do so too.
I appreciate them and they will help us improve.

web2py was originally copyrighted as "Enterprise Web Framework" so
this stuck with the name. The name changed to Gluon because there is a
company called "EWF". The name changed again to web2py because there
is a company that owns the "Gluon" trademark. It is makes you feel
better feel free to read "Enterprise" as the "Star Trek Starship USS
Enterprise NCC-1701". Perhaps we should change the tagline. web2py: to
boldly go where no web framework has been before". ;-)

Thanks Armin,

Massimo

Armin Ronacher

unread,
Aug 2, 2009, 4:18:04 PM8/2/09
to web2py-users
Hi,

Tired so just the most important parts first.

On Aug 2, 9:21 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> No there is not. Why do you think there is a race condition?
If the execution of the module takes longer than, say a few minutes,
the timestamp would be off of course. Long running HTTP requests are
pretty common these days thanks to stuff like comet.

> Yes. It was a design decision to be closer to Rails than Django in
> this respect. I think this is what is making web2py popular, more than
> anything else.
That's not django/rails that's more python vs ruby. In Ruby the
module/require work a lot different to python modules and the rails
people do have problems with that namespacing there because of the
silent reopening of classes.

> Can you provide an example of what you mean? I am pretty sure there is
> no leak.

_controlled = set()
def controlled(f):
_controlled.add(f)
return f

from lib import controlled

@controlled
def foo():
pass

If that code is reexecuted each time there would be a humongous memory
leak because over the func_globals even the temporary namespace is
preserved with everything in it.

> I disagree with "nobody knows how these idioms scale or behave"?
> We have been doing this for 2 years. We know very well how it works
> and scales.
> It actually works very well because as soon as a request is completed
> the environment is deleted, this is clean, avoids conflicts when
> multiple apps are running, and prevents memory leaks.
Python is not designed for short running code or throwaway modules.
Things can leak incredible fast and it's hard to spot. Here a piece
of code that would leak the hell out of a web2py application if used
inside one those throw-away environment things:

from collections import Container
class Foo(object):
def foo(self):
print "Meh"
Container.register(Foo)

That's maybe not the most obvious example because not many people will
use the ABC registry in those modules, but similar registration
systems are part of ORMs and many other systems. Even modules in the
standard library (csv dialects for instance)

> I do not see the problem. Files are automatically closed when the file
> object goes out of scope.
> Can you provide an example when it does not?
There is no such thing as a file going out of scope. The file is
closed as part of the object's __del__ method and that might or might
not be invoked. In cpython (and only in that python implementation)
there is a reference counter that often closes files when you think it
would, but even that's not always the case. Let me show you an
example where a file is leaked:
(Using a fake file here)

>>> class FakeFile(object):
... encoding = 'utf-8'
... write = lambda x, s: None
... def __del__(self):
... print "file deleted"
...
>>> class SortedLineWriter(object):
... def __init__(self, func, file):
... self.func = func
... self.file = file
... def write_lines(self, lines):
... for line in sorted(lines, key=self.func):
... self.file.write(line)
...
>>> def not_closing_directly():
... f = FakeFile()
... slw = SortedLineWriter(lambda x: x.decode(slw, 'ignore').lower
(), f)
...
KeyboardInterrupt
>>> def not_closing_directly():
... f = FakeFile()
... slw = SortedLineWriter(lambda x: x.decode(slw.file.encoding,
'ignore').lower(), f)
... slw.write_lines([])
...
>>> import gc
>>> not_closing_directly()
>>> gc.collect()
file deleted
5

As you can see, the file is not deleted right now, just after the gc
ran. This is standard behavior in pypy, jython and many others. In
cpython just if refcounting fails.

> I do not believe that is the case but I may be wrong. Can you provide
> an example of what you say?
Seems like I was misreading the code. repr() is properly used.

mdipierro

unread,
Aug 2, 2009, 4:37:39 PM8/2/09
to web2py-users
Thanks Armin,
my command below:

On Aug 2, 3:18 pm, Armin Ronacher <armin.ronac...@active-4.com> wrote:
> Hi,
>
> Tired so just the most important parts first.
>
> On Aug 2, 9:21 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:> No there is not. Why do you think there is a race condition?
>
> If the execution of the module takes longer than, say a few minutes,
> the timestamp would be off of course. Long running HTTP requests are
> pretty common these days thanks to stuff like comet.

True. but I would not call it a race condition. We timestamp
everything with the time when a request arrives, not when it is
processed, unless specified otherwise (datetime.now() instead of
request.now)

> > Yes. It was a design decision to be closer to Rails than Django in
> > this respect. I think this is what is making web2py popular, more than
> > anything else.
>
> That's not django/rails that's more python vs ruby.  In Ruby the
> module/require work a lot different to python modules and the rails
> people do have problems with that namespacing there because of the
> silent reopening of classes.
>
> > Can you provide an example of what you mean? I am pretty sure there is
> > no leak.
>
>   _controlled = set()
>   def controlled(f):
>    _controlled.add(f)
>    return f
>
>   from lib import controlled
>
>   @controlled
>   def foo():
>    pass
>
> If that code is reexecuted each time there would be a humongous memory
> leak because over the func_globals even the temporary namespace is
> preserved with everything in it.

True but I believe we never do that in web2py. It is also true that
nothing prevent the user from doing it but the same would be true with
other frameworks.
Yes but because all relevant application code is executed within a
context and there are no references outside the context to stuff
inside the context, when a request is completed, the context is
deleted and everything should be garbage collected.

Anyway. I agree with you it would be better to always close files
explicitly. I will do so unless omebody sends me a patch first.

Bottiger

unread,
Aug 2, 2009, 8:46:49 PM8/2/09
to web2py-users
I haven't had time to verify the other findings, but there are
definitely file handle leakages. I can never delete an application if
I just visited it once.
> ...
>
> read more »

Pynthon

unread,
Aug 3, 2009, 9:18:27 AM8/3/09
to web2py-users
I'm reading this topic for a time and it really makes me doubt about
web2py =[. Hopefully Massimo can fix this =].
> ...
>
> meer lezen »

Jason Brower

unread,
Aug 3, 2009, 9:26:07 AM8/3/09
to web...@googlegroups.com
I am happy for the efforts being made in web2py. I think we should have
a better way to file a bug or request a feature. But after buying the
book, I am very happy documentation wise.

Pynthon

unread,
Aug 3, 2009, 9:50:25 AM8/3/09
to web2py-users
True, hopefully Massimo can make the inner of W2P nicer.
> ...
>
> meer lezen »

suiato

unread,
Aug 3, 2009, 10:04:25 AM8/3/09
to web2py-users
I found this thread encouraging. It shows Massimo and members here are
open to different views/opinions while maintaining integrity. That's
essential for open-source software to evolve. E.g., who thought in
the early 1990's Linux was going to be as serious a software as we see
today?

Cheers,
Teru

Pynthon Pynthon

unread,
Aug 3, 2009, 10:09:37 AM8/3/09
to web...@googlegroups.com
Indeed, but I still hope that Massimo can fix this stuff :P.

2009/8/3 suiato <Hom...@gmail.com>

Álvaro Justen [Turicas]

unread,
Aug 3, 2009, 11:13:44 AM8/3/09
to web...@googlegroups.com
On Mon, Aug 3, 2009 at 11:09, Pynthon Pynthon<foru...@gmail.com> wrote:
> Indeed, but I still hope that Massimo can fix this stuff :P.

Think in community: WE can change this (including you!).
Massimo have done a lot of work in web2py and I think users need to
help more this project.

> 2009/8/3 suiato <Hom...@gmail.com>
>>
>> I found this thread encouraging. It shows Massimo and members here are
>> open to different views/opinions while maintaining integrity. That's
>> essential for open-source software to evolve. E.g., who thought  in
>> the early 1990's Linux was going to be as serious a software as we see
>> today?
>>
>> Cheers,
>> Teru
>>
>
>
>
--
Álvaro Justen
Peta5 - Telecomunicações e Software Livre
21 3021-6001 / 9898-0141
http://www.peta5.com.br/

Pynthon

unread,
Aug 3, 2009, 11:18:22 AM8/3/09
to web2py-users
If I could I would help...

On 3 aug, 17:13, Álvaro Justen [Turicas] <alvarojus...@gmail.com>
wrote:
> On Mon, Aug 3, 2009 at 11:09, Pynthon Pynthon<forumx...@gmail.com> wrote:
> > Indeed, but I still hope that Massimo can fix this stuff :P.
>
> Think in community: WE can change this (including you!).
> Massimo have done a lot of work in web2py and I think users need to
> help more this project.
>
> > 2009/8/3 suiato <Homm...@gmail.com>

Armin Ronacher

unread,
Aug 3, 2009, 4:22:05 AM8/3/09
to web2py-users
Hi,

> True. but I would not call it a race condition. We timestamp
> everything with the time when a request arrives, not when it is
> processed, unless specified otherwise (datetime.now() instead of
> request.now)
True. But that does not make it a better idea. Also, datetime.now()
should be consistently replaced with datetime.utcnow() because using
anythign else than UTC data internally is problematic for various
reasons. See the discussion on that topic in various i18n/l10n
libraries such as babel / pytz.

> True but I believe we never do that in web2py. It is also true that
> nothing prevent the user from doing it but the same would be true with
> other frameworks.
You're not doing it, a user might be doing that by accident or because
he things it should work. This problem does not exist in other
frameworks because besides web2py I don't know a single one that does
this sort of execfile() + namespace thing or uses any other kind of
throwaway modules. As soon as a single reference leaks from the
execfile()'d namespace you're in big troubles and due to the open
nature of Python this could happen very, very fast.

> Yes but because all relevant application code is executed within a
> context and there are no references outside the context to stuff
> inside the context, when a request is completed, the context is
> deleted and everything should be garbage collected.
That depends on two things. First not having a reference leaked,
which could happen with abstract base classes and other stuff that
uses registries or steals non-weak references. Also and more
importantly, the file descriptor limit is very low and the majority of
Python implementations will only collect that on the GC run (always,
no matter where references are). Say you're opening three files per
request and you have more than 100 requests/sec you could lose all
file descriptors that are available before the GC even thought about
running.


Regards,
Armin

Yarko Tymciurak

unread,
Aug 3, 2009, 12:59:28 PM8/3/09
to web...@googlegroups.com
Pynthon -

Remember that just recently you mentioned you have been 3 months since you looked at Python, and you did not remember lists and dict's...

It is easy to have doubts out of fear, but remember to watch the discussion, and see what develops, rather than assume everything raised "must be a problem".   The point of discourse is to air views, and come to understanding.

Don't worry, rather watch and read background and try to understand the topics being discussed.

Regards,
- Yarko

Pynthon

unread,
Aug 3, 2009, 1:57:23 PM8/3/09
to web2py-users
Thanks Yarko, indeed because I don't know about it and Armin (Massimo
too and a lot of more guys here) looks like an advanced programmer I
thought maybe he is right and thats the reason why I'm kinda scared.

On 3 aug, 18:59, Yarko Tymciurak <yark...@gmail.com> wrote:
> Pynthon -
>
> Remember that just recently you mentioned you have been 3 months since you
> looked at Python, and you did not remember lists and dict's...
>
> It is easy to have doubts out of fear, but remember to watch the discussion,
> and see what develops, rather than assume everything raised "must be a
> problem".   The point of discourse is to air views, and come to
> understanding.
>
> Don't worry, rather watch and read background and try to understand the
> topics being discussed.
>
> Regards,
> - Yarko
>
> On Mon, Aug 3, 2009 at 9:09 AM, Pynthon Pynthon <forumx...@gmail.com> wrote:
> > Indeed, but I still hope that Massimo can fix this stuff :P.
>
> > 2009/8/3 suiato <Homm...@gmail.com>

waTR

unread,
Aug 3, 2009, 4:13:37 PM8/3/09
to web2py-users
Hi all,

What do you guys think of, at this time, filing a few of these issues
as ISSUES on the google code page for this project. Then, start a
debate on each individual issue separately. They have a great platform
for this on google code. Also, it keeps the discussion focused, as
right now there seem to be multiple issues being discussed here, and
it could get messy very quickly. Furthermore, do web2py users need to
see this discussion in web2py-users? It seems more of a web2py-
developer discussion or a google code "issues" page discussion.
Lastly, you guys really need to start using google code, it is simple,
easy to contribute on, fast, and while there may be a lot of issues
that will be killed by the team as a result of the conclusions reached
in the debates for each issue, it will show great project activity for
project lurkers. Nothing against launch-pad, it just feels too bloated
for me.

Just my 1.91 CAD cents for our $2 USD.

P.S. Armin, it's great to have you posting on this mailing list!! You
sound like you have a lot of love for this framework (or at least
hope), which is up to the project team to make sure is not dashed.
Though definitely stick around and continue to poke the src code and
find soft-spots / holes. I really feel it is people like you, working
with/in the web2py team who can take this project to the next level
re: quality, etc.

Pynthon

unread,
Aug 3, 2009, 4:38:29 PM8/3/09
to web2py-users
Indeed, I think its also a good idea or post this thing son the web2py
developers group. Maybe new users see this and think "Hmmm..." and
that is now what we want (well I think that).
And indeed Armin is a good guy!

JohnMc

unread,
Aug 3, 2009, 6:34:12 PM8/3/09
to web2py-users
I take the long view on all of this. Tools/Systems are a process not a
end. I point that out only because from time to time I will get a
notice that a bug has been found in C/C++ and the required patches or
replacement code to use. C is nearly as old as dirt (1972 for me) and
still has fixes and we complain about Web2Py that is 3 years in the
making has bugs? For its lifecycle Web2Py is probably about average in
code issues/fixes from what I see. That does not discourage me from
using the tool.

Armin has done a public service and should be encouraged to point out
more.

JohnMc

On Aug 3, 9:09 am, Pynthon Pynthon <forumx...@gmail.com> wrote:
> Indeed, but I still hope that Massimo can fix this stuff :P.
>
> 2009/8/3 suiato <Homm...@gmail.com>
>

mdipierro

unread,
Aug 4, 2009, 12:56:10 AM8/4/09
to web2py-users
A minor corrections. web2py is not 3 years old. It will be 2 years old
on October 1st, 2009.
We are already working on fixing many of these issues although I
should point out none of these issues so far has manifested itself as
a bug, as far as I know. People should submit a bug report on the
google code if they think there is a bug, and it is not been
addressed,

Massimo

mdipierro

unread,
Aug 4, 2009, 3:04:35 AM8/4/09
to web2py-users
Changing now into utcnow would break backward compatibility.

I do agree with you that othen people may want to use

Field(....,default=datetime.utcnow())

instead of

Field(....,default=request.now)

I will add a comment about this in the book.

Massimo

Pynthon

unread,
Aug 4, 2009, 6:32:08 AM8/4/09
to web2py-users
Nice, Massimo will there be a new book?

mdipierro

unread,
Aug 4, 2009, 8:42:59 AM8/4/09
to web2py-users
Almost done. 1-2 weeks.

Pynthon

unread,
Aug 4, 2009, 8:49:15 AM8/4/09
to web2py-users
Kool, I know it sounds greedy but do you need to buy the PDF again
or..?

Thanks for the fast support!

Alex Fanjul

unread,
Aug 4, 2009, 1:55:05 PM8/4/09
to web...@googlegroups.com
Massimo,
Many times I have seen that, due to backward compatibility, we are
forcing to write "messy" code in web2py applitacations.
Evenmore some issues will never fix in the right way bacause of that...
Won't you consider/planning to do a breakpoint with a major version
web2py 2.0? and solve such things?

Python did it with 3.0, isn't it?

Only out of curiosity, sorry if it's reduplicate question, regards,
Alex F
--
Alejandro Fanjul Fdez.
alex....@gmail.com
www.mhproject.org

Iceberg

unread,
Aug 4, 2009, 2:15:13 PM8/4/09
to web2py-users
Perhaps we will eventually have a web2py 2.0 in the way which Alex
Fanjul suggests.

Meanwhile, we can take closer look into those "many times" of "due to
backward compatibility" issue, and see what can be adjusted. We did
that before at least for IS_STRONG.

This time, for example, the datetime.utcnow issue can be easily
addressed by a request.utcnow, and keep request.now as is but
obsolete. Oops, problem solved without breaking backward
compatibility.

Regards,
Iceberg

Pynthon

unread,
Aug 4, 2009, 2:30:55 PM8/4/09
to web2py-users
Yeah maybe a complete web2py 2 rewrite. With nice and better coding...
Alex Fanjul has a point IMO.

Fran

unread,
Aug 4, 2009, 3:00:46 PM8/4/09
to web2py-users
On Aug 4, 7:30 pm, Pynthon <forumx...@gmail.com> wrote:
> Yeah maybe a complete web2py 2 rewrite. With nice and better coding...

I believe this is a bad idea as we really don't want to break existing
apps - this is a *key* strength of web2py & one many of it's adherents
really value. This is what justifies the term 'Enterprise'.

Many of the 'issues' are easy to solve without doing so & the rest are
pretty much design decisions.
They are seen as key strengths for some & key weaknesses for others.
Let those who see them as strengths enjoy this & if others cannot live
with what they see as weaknesses, let them choose from the many other
frameworks out there.

F

waTR

unread,
Aug 4, 2009, 5:08:34 PM8/4/09
to web2py-users
I don't feel it is necessary to re-write anything yet. There is a lot
of good in it as it is. Don't just give up on it when you run into a
few difficulties. We are not talking about anything that won't be
addressed either. It is more of an issue of how it is addressed, and
for all the "weaknesses" in this framework, there are a hell of a lot
more strengths.

Have patience, and don't forget this is a VERY young project. Also,
you may not agree with all the decisions all the time, but then no one
will in any framework. You can't please all the people all the time.
The key is to remember what brought you to this framework, and if that
still stands. All software has weaknesses, the question is whether it
is being dealt with or monitored. If it is, then one day in the near
future it will be addressed, but don't confuse speed of resolving all
problems brought up with quality solutions. Often the best solution
will require a lot of dialogue and a lot of analysis before
implementing.

Also, not breaking backwards compatibility is a great rule to prevent
software from moving too fast. If the software breaks backwards
compatibility with every minor release, no one will use it because it
is not serious. If it breaks with every major release, no serious
project will use it because it would be too expensive to maintain.
Therefore, if all I have to do is use some added code in order to
solve the problem but preserve backward compatibility, then I am all
for it. If you have a small project that you can re-write in a day,
you won't care, but then I would not care about these minor issues if
that is all I needed it for. I am still following this framework
because I believe I can build something serious with it, not just a 5
minute wiki AND I know I won't have to re-write everything once a
year. Breaking backwards compatibility is a very serious decision, and
I am very happy Massimo takes it VERY seriously. This attitude is what
will set this framework apart, and will make it deserve the title of
"Enterprise Framework". Otherwise, it is just another framework.

If this project has to break backwards compatibility though, I would
hope it is done at the same time that it transitions to Python 3, as
that would take care of everything all at once. However, there is a
LOT of work to be done in assembling a list of issues and solutions
before that happens. So just be patient, and continue to use the
framework, that is how more issues can be discovered, and solutions
proposed. However, for today, there is no reason for even talking
about breaking backwards compatibility.

Kuba Kucharski

unread,
Aug 4, 2009, 5:10:50 PM8/4/09
to web...@googlegroups.com

+1 Really bad idea. I have never seen even a shadow of reason for this. We trust Massimo with his declaration. Please, try to understand. This is about serious production environments, important applications and making money using web2py. We could consider it if there was a real need. But there is none.

On Aug 4, 2009 9:00 PM, "Fran" <franc...@googlemail.com> wrote:

On Aug 4, 7:30 pm, Pynthon <forumx...@gmail.com> wrote: > Yeah maybe a complete web2py 2 rewrite. W...

I believe this is a bad idea as we really don't want to break existing
apps - this is a *key* strength of web2py & one many of it's adherents
really value. This is what justifies the term 'Enterprise'.

Many of the 'issues' are easy to solve without doing so & the rest are
pretty much design decisions.
They are seen as key strengths for some & key weaknesses for others.
Let those who see them as strengths enjoy this & if others cannot live
with what they see as weaknesses, let them choose from the many other
frameworks out there.

F --~--~---------~--~----~------------~-------~--~----~ You received this message because you are s...

mdipierro

unread,
Aug 4, 2009, 5:13:42 PM8/4/09
to web2py-users
This is a good point but I disagree and agree.

I disagree because I do not think we ever wrote messy code because we
needed to keep backward compatibility. So far we never had to
compromise much. Moreover backward compatibility is a strength of
web2py.

I agree because one day all existing database divers will be ported to
Python 3.0 and GAE will support python 3.0. When this happen (in 2-3
years) we may want to release a backward incompatible version of
web2py (while keeping support for the old one). That would be the time
to change a few other things and that will be the time to discuss what
to change.

Massimo
> alex.fan...@gmail.comwww.mhproject.org

JohnMc

unread,
Aug 4, 2009, 5:53:51 PM8/4/09
to web2py-users
Hmmm. My bigger question is, are you still looking for assistance or
do you have the book wrapped up?

Richard

unread,
Aug 4, 2009, 8:25:13 PM8/4/09
to web2py-users
> Yeah maybe a complete web2py 2 rewrite. With nice and better coding...

I really REALLY hope not...
web2py is already an incredibly useful web framework and is getting
better each month with new features and bug fixes.
IMO investing energy in a rewrite could split the community and would
certainly set back the pace of development.

And I also don't feel development is significantly constrained by
backwards compatibility. Maintaining backwards compatibility is one of
the core advantages of web2py that attracted me to it.

Richard

mdipierro

unread,
Aug 5, 2009, 3:02:10 AM8/5/09
to web2py-users
The text is almost done. I had significant help from some users.
Some of the images must be redone.
I have already posted here drafts of the 2 of the 3 new chapters. I
will post the third soon.

Massimo

AchipA

unread,
Aug 5, 2009, 6:15:45 AM8/5/09
to web2py-users
I've got no reply to my mail to your cron chapter question... Are we
on track with that or ?

Pynthon

unread,
Aug 5, 2009, 9:10:17 AM8/5/09
to web2py-users
Indeed its indeed a very young project. Hopefully the "code group" can
fix the most of this stuff. And sorry for my "panic".

- Py

mdipierro

unread,
Aug 5, 2009, 9:52:27 AM8/5/09
to web2py-users
Did you email me something? If so, can you send it again. I do not
recall what this is about.

Pynthon

unread,
Aug 5, 2009, 10:00:58 AM8/5/09
to web2py-users
I don't, I just posted this reply. I don't think so :P.

Alex Fanjul

unread,
Aug 5, 2009, 9:37:28 PM8/5/09
to web...@googlegroups.com
Hi Armin,
I'm thinking that if you could review and noticed so many "potential
issues" in 15 minutes, I'm entirely sure that many people here would
appreciate you could invert 1 day of your life in a deeper and
exhaustive review!! I'm sure that would make stronger framework. :-D

cheers,
alex f

Alex Fanjul

unread,
Aug 5, 2009, 9:38:00 PM8/5/09
to web...@googlegroups.com
Some thoughts and commentaries about major version:

-My question came after thinking in Armin thread
-Most of people seems to be convinced that there is no real reason
(big issue) that couldn't be "avoidable enough" to need to realase a non
backward compatible major version (so I figure out we have a strong
framework in front)
-Only real experience people (not like me) know if there are real
motives to consider a non backward compatible major version
-Many people say that web2py is so young to think in "big changes in
code", but In case it would be real motives to do a "non backward
compatible major version", and due to the youthfulness of the framework,
wouldn't be better to do it as soon as possible and not after some years
with many more users and applications online?
-In case it would be real motives to do a "non backward compatible
major version", wouldn't you prefer to adapt your application in return
to have a better framework (maybe more secure, flexible, better writed, etc)

These are only (and probably wrong) open thoughts and conjetures, so
don't waste time in reply me...
Alex F
>> apps - this is a *key* strength of web2py& one many of it's adherents
>> really value. This is what justifies the term 'Enterprise'.
>>
>> Many of the 'issues' are easy to solve without doing so& the rest are
>> pretty much design decisions.
>> They are seen as key strengths for some& key weaknesses for others.
>> Let those who see them as strengths enjoy this& if others cannot live
>> with what they see as weaknesses, let them choose from the many other
>> frameworks out there.
>>
>> F
>>
> >
>
>

Bottiger

unread,
Aug 6, 2009, 12:55:59 AM8/6/09
to web2py-users
Agree 100%

If you are going to rewrite, best to do it soon instead of later.

A quick look at the available applications:http://www.web2py.com/
appliances

There simply is not that many of them, and most are just proofs of
concept. They are not building blocks for Web2Py except for CAS which
isn't necessary, and could probably be rewritten fairly quickly.
> alex.fan...@gmail.comwww.mhproject.org

mdipierro

unread,
Aug 6, 2009, 1:05:31 AM8/6/09
to web2py-users
To my knowledge there is no issue cannot be solved in a backward
compatible way.

Most of the issues that Armin raised about web2py have to do with its
internal design decisions. These are not bugs to be fixed. These are
distinctive features of web2py that were thought through before being
developed and I see no reason to change. Every design decision has
pros and cons.

Some of the issues like the "open file issue" do not constitute a bug
(we have no knowledge of any problem with it) nevertheless this has
been addressed in trunk already.

The issues with "request.now" is non-existent. request.now is just a
variable that contains the localtime when a request arrives. It is a
variable that you can use in your programs. It is true that by the
time a request arrives until a request a request ends time passes by,
by there is no race condition. We simply provide a variable to the
users to know for sure when the request arrived, the current time is
in datetime.now().

The issue with now vs utfnow, althought I dismissed because it would
break backward compatibility, I should have been more extensive. The
OS timestamps files using the local time, not utc time. The average
developer prefers using localtime vs utc time. If you store things in
database and you have multiple servers over multiple timezones than
you want to use timestamp records using utc time instead of local
time. Except in this last case localtime is to be preferred and that
is why we store in a variable. If you need utf time that is also
available via the datetime module.

I am not sure there is any open issue here. Which issue in your
opinion would need a backward incompatible change? Not that this is
going to happen.

Massimo
> alex.fan...@gmail.comwww.mhproject.org

mdipierro

unread,
Aug 6, 2009, 1:08:38 AM8/6/09
to web2py-users
Not true. There are many private apps built by users and kept private.
We owe to them. Backward compatibility is a feature of web2py. Anyway.
There is no issue so far that cannot be fixed in a backward compatible
way.

Richard

unread,
Aug 6, 2009, 2:56:24 AM8/6/09
to web2py-users
On Aug 6, 2:55 pm, Bottiger <bottig...@gmail.com> wrote:
> Agree 100%
>
> If you are going to rewrite, best to do it soon instead of later.
>
> A quick look at the available applications:http://www.web2py.com/
> appliances
>
> There simply is not that many of them, and most are just proofs of
> concept.

I have a number of private apps at my work place that I don't want to
redebug next time I upgrade. Some of these apps were written a year
ago and still work with the current web2py version.

What is the show stopping feature that requires breaking backward
compatibility?

Richard

Fran

unread,
Aug 6, 2009, 3:26:29 AM8/6/09
to web2py-users
On Aug 6, 6:05 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
> The issue with now vs utfnow, althought I dismissed because it would
> break backward compatibility, I should have been more extensive. The
> OS timestamps files using the local time, not utc time. The average
> developer prefers using localtime vs utc time. If you store things in
> database and you have multiple servers over multiple timezones than
> you want to use timestamp records using utc time instead of local
> time. Except in this last case localtime is to be preferred and that
> is why we store in a variable. If you need utf time that is also
> available via the datetime module.

Would you be up for adding an accessible request.utcnow?
I definitely want this & it seems like a standard thing for the
'enterprise'.
A recipe example would be an OK alternative...a quick play didn't
reveal to me how to convert default=request.now into a utcnow
equivalent...

F

Alex Fanjul

unread,
Aug 6, 2009, 4:06:29 AM8/6/09
to web...@googlegroups.com
Yea Massimo, thats the point, I don't know if there are really big
issues and it seems actually there aren't. It's almost a mind question
to you and big developers, is there any big issue to consider non
backward? It seems is not by your comments, :-D

Timbo

unread,
Aug 6, 2009, 10:29:31 AM8/6/09
to web2py-users
It's been a while since I've been around but this seems to be an
important discussion that I'd like to weigh in on.

Massimo is right. Most of Armin's critiques are thought-out design
decisions. The perceived problems from just looking at the code don't
play out in reality due to being either a non-issue handled by some
other mechanism.

The issue of open file-handles is partly an issue and partly Armin
showing his ignorance of WSGI. Any iterator passed back to a WSGI
server will be "closed" if it has a close methods and will be
appropriately deleted. For all the other instances, yes, we should
close them.

I'm not going to weigh in on the ORM since I currently do not use it
(however, this might change with the new ORM coming out).

Fact is, I've used web2py for production purposes for more than a year
now and the only issues I've had with my setup were either my own code
or issues with python's broken socket library (which the included
wsgiserver uses).

As for the future of Python...I see cPython becoming what the vanilla
Linux kernel is today. That is to say, it's sort of the base-line,
but no distribution uses just the vanilla kernel, but they all add
their own patches and customizations. Pypy will likely bypass both
cPython and Jython in performance and flexibility within the next 2
years and popularity in the following 5. Python developers will refer
to cPython more than end-users will actually use it. Google will keep
using it since they're investing in Unladen-Swallow, but independent
projects will gear more towards Pypy compatibility due to it's
platform flexibility, speed and scalability.

That being said, Python 3.x will not be adopted as quickly as some
would hope. Py3 support is barely on the radar for Pypy and Jython.
Frankly, even at Python3.1 it's still horribly broken for web-
development. The Cherrypy and Werkzeug devs are building their own
versions of the cgi library.

Eventually, web2py will need to have some implementation on Python3,
but I don't see that as needed for at least 2 more years. Python2 is
still going very strong. It will be more than 2 years before Google
migrates. They will likely be the driving force behind moving beyond
Py2.

Just my 2c

-tim

waTR

unread,
Aug 6, 2009, 1:06:53 PM8/6/09
to web2py-users
Re-writing a framework after 2-3 years is suicide. First, and
foremost, we do not have enough developers to take-on such a huge
task. Even if we did, it is still too early because they are not
organized for such a task (look at Perl 6...and how long it is taking
them to release a new version).

Backwards compatibility is EXTREMELY important for a framework to be
taken seriously. If you are looking at this issue from your bedroom
and thinking I want a really fun hot-rod framework to play with, thats
fine. However, for a business, you are looking for something that is:
1. Going to still be around in 5 years
2. Going to still be stable & Secure in 5 years
3. Going to still be compatible in 5 years.

Don't forget that some companies are still using code written back in
the 1970s (think COBOL). While I agree that it would be FUN to play
with the latest web-framework features, I am sure you are free to fork
this project and take it in an experimental direction. However, try
not to push such a promising potential "enterprise" framework in that
direction.

As Massimo says, there are no bugs here, rather design decisions. If
you disagree with a design decision, fork and re-design. If you can
live with it, provide some work-around plug-ins to do it the way you
feel it should be done.

Speaking of which... is there a plug-in function in web2py yet??? It
would be great for web2py to allow people to plug-in these crazy
features so people can choose to try every idea talked about by simply
downloading the plug-in. Then, if it is popular enough, it can be
added to the list of plug-ins that will be included in the main
release once the framework moves to Python 3.

Yarko Tymciurak

unread,
Aug 6, 2009, 2:10:51 PM8/6/09
to web...@googlegroups.com
I think Massimo has already spoken to all these concerns / issues;

Closing files explicitly is stylistic - saying what you mean (and not counting on side-effects of the system); it makes for more readable code in general (although some will complain about an extra line of typing, it is explicit to say "I'm done with this handle now!).   Among other things, not depending on side effects of a particular system means your code is more portable to other systems/languages (you may want to move processing from javascript to a controller or vice-versa; you can think of other situations where language choice of a part is beyond your choice).

Massimo has chosen to do this already.

As for rewrite for Python 3 at some point in the future (as Massimo alluded to being a possible fork from backward compatibility) that will be an opportunity to look at the application writer's idioms, and if something can be changed to more naturally express the work-flow of expressing business logic, this will be an opportunity to explore that.

In the meantime, no worries - openness to feedback is a good thing.

mdipierro

unread,
Aug 6, 2009, 5:12:08 PM8/6/09
to web2py-users
Why should I add when you can do it in your model?

import datetime
request.utcnow=datetime.datetime.utcnow()

Massimo

Fran

unread,
Aug 7, 2009, 12:06:17 PM8/7/09
to web2py-users
On Aug 6, 10:12 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> you can do it in your model?
> import datetime
> request.utcnow=datetime.datetime.utcnow()

Perfect, thanks :)

I'd suggest putting this into scaffolding app myself, but entirely
your call...

F

Fran

unread,
Aug 7, 2009, 12:11:03 PM8/7/09
to web2py-users
On Aug 6, 10:12 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> Why should I add when you can do it in your model?

So that it can be included in Auth without needing to subclass.

Seems like a good enough reason to me...

F

Yarko Tymciurak

unread,
Aug 7, 2009, 1:06:35 PM8/7/09
to web...@googlegroups.com
even a commented out version of this in the scaffold, showing what you can uncomment if you need utc might be useful.
Reply all
Reply to author
Forward
0 new messages