web2py exec in the new again

[Massimo Di Pierro]

From the usual source:

http://lucumr.pocoo.org/2011/2/1/exec-in-python/

My answer:

Armin suggests exec'ing in an environment and not in the outer
environment. That is what web2py does and in fact we already follow
much of the advice he gives us about about how to use exec. We just do
not follow the advice not to use exec. ;-)

Performance is not an issue because 1) the time to bytecompile is
often small compared to the database IO time and because 2) web2py
allows you to bytecode compile apps anyway.

In web2py you cannot store instances of objects into session. While
Armin makes it tool like this is a web2py problem this is more complex
and general than that. In Python if you pickle and object that is
instance of a class defined in /path1/mymodule.py and then you
unpickle on a different installation where the class is defined in /
path2/mymodule.py you may run into problems. Because web2py does not
require installation (it is a feature) and has hot plug and play of
apps (another feature) we cannot pickle instances (the price we pay
for those features). To me it is worth it.

Once again Armin raises the issue of memory leaks. Unfortunately
CPython does not use a proper garbage collection but uses reference
counting. It suffers from memory leaks when circular references are
created. This is not a web2py specific problem. This is a problem with
any Python program. For example the following code

class A: pass
a=A()
a.b=a

causes a memory leak in ANY python program.

I agree with Armin that in the case of exec (and web2py uses exec),
the self references are created whether you want them or not. That
means one should not define classes in Models and Controllers. I have
issued that warning already to our users and, I have never seen this
being a problem in practice.

Nothing in life is perfect and every design decision has a tradeoff.

What would really be useful to us is an example of how to overcome
that problem by showing how to break those implicit self references.

Massimo

[VP]

One question:
If I define a class externally, and use local_import to import it into
controller or model, will it have these potential problems or not?

[Massimo Di Pierro]

I run this test (following Armin's example):

# in file a.py
class Foo(object):
def __del__(self):
print 'Deleted'

#in file b.py
from a import A
foo=Foo()

#in file c.py
execfile('b.py', {})
execfile('b.py', {})
execfile('b.py', {})
import gc
gc.collect()

running c.py printes
Deleted
Deleted
Deleted

therefore there is NO memory leak.

[Anthony]

But what if 'local_import' is used (instead of 'from a import Foo') -- does local_import work the same as regular Python imports?

 

On Tuesday, February 1, 2011 5:09:25 PM UTC-5, Massimo Di Pierro wrote:

I run this test (following Armin's example):

# in file a.py
class Foo(object):
    def __del__(self):
        print 'Deleted'

#in file b.py
from a import A
foo=Foo()

#in file c.py
execfile('b.py', {})
execfile('b.py', {})
execfile('b.py', {})
import gc
gc.collect()

running c.py printes
Deleted
Deleted
Deleted

therefore there is NO memory leak.

[VP]

Maybe unnecessary, but here's another test:

+ In foo.py:

class Foo(object):
def __del__(self):
print 'Deleted'

def bar(self):
print "bar::"
b = Foo()

def f():
print "local_func::"
c = Foo()


+ In test.py:

from foo import *
a = Foo()
a.bar()
f()


+ Then, in interpreter:
>>> execfile('test.py',{})
bar::
Deleted
local_func::
Deleted
Deleted


==============

It looks like all variables a, b, and c are properly destroyed.

Is this the pattern of usage in web2py?
(1) No class in controller.
(2) External classes are imported using local_import (does
local_import work exactly like import?).


If so, I think webp2y is okay with respect to memory leak.

[Anthony]

On Tuesday, February 1, 2011 3:16:12 PM UTC-5, Massimo Di Pierro wrote:

In web2py you cannot store instances of objects into session. While

Armin makes it tool like this is a web2py problem this is more complex

and general than that. In Python if you pickle and object that is

instance of a class defined in /path1/mymodule.py and then you

unpickle on a different installation where the class is defined in /

path2/mymodule.py you may run into problems. Because web2py does not

require installation (it is a feature) and has hot plug and play of

apps (another feature) we cannot pickle instances (the price we pay

for those features). To me it is worth it.

 

Is this limitation documented anywhere?

 

I agree with Armin that in the case of exec (and web2py uses exec),

the self references are created whether you want them or not. That

means one should not define classes in Models and Controllers. I have

issued that warning already to our users and, I have never seen this

being a problem in practice.

 

My understanding is that with the recent fix (using gc), we can include classes in models and controllers, as long as they don't have a __del__ method (https://groups.google.com/forum/#!msg/web2py/zE1cGK2x288/R6fTdEQjNOoJ). Is that right? That should be documented as well.

 

Nothing in life is perfect and every design decision has a tradeoff.

 

A quote from Armin's post:

"Because one of the things we all have to keep in mind: if a Python developer starts his journeys in the twisted world of wrongly executed Python modules they will be very confused when they continue their travels in another Python environment. And having different semantics in different frameworks/modules/libraries is very hurtful for Python as a runtime and language."

He (and Jacob, and others) seems to be very concerned about following conventions. Of course, that's reasonable, as conventions can be very helpful, but there's always going to be some tension be convention and innovation. The above quote implies dire consequences if a few libraries and frameworks happen to buck this particular convention, but is there actual evidence of significant developer confusion or other supposed harm to Python? If we extend this logic, Python itself probably shouldn't exist because it violates conventions established by C and other earlier languages and might confuse developers as they move between environments.

 

Anyway, it's interesting that Armin didn't find arguments about the importance of convention so compelling when it came to creating an alternative to the Standard Library logging module: http://plumberjack.blogspot.com/2010/09/python-logging-functionality-facts-vs.html

 

Anthony 

[VP]

It's interesting to note that even the variable "c" in my example
above is properly destroyed with execfile('test.py', {}).


What does this mean?

I think this means that YES you can define classes inside a web2py
controller without memory leaks, as long as there are no global
(within the controller, i.e. foo.py) instances of Foo.

[VP]

If what I thought above is true, a warning like this sufficient:
"Global objects/variables in controllers might (will?) cause memory
leaks".

If this is what the criticism of web2py amounts to, I think it's a
storm in a tea cup.

[Jonathan Lundell]

On Feb 1, 2011, at 12:16 PM, Massimo Di Pierro wrote:
>
> I agree with Armin that in the case of exec (and web2py uses exec),
> the self references are created whether you want them or not. That
> means one should not define classes in Models and Controllers. I have
> issued that warning already to our users and, I have never seen this
> being a problem in practice.
>
> Nothing in life is perfect and every design decision has a tradeoff.
>
> What would really be useful to us is an example of how to overcome
> that problem by showing how to break those implicit self references.

I agree. This is a subtle problem that isn't all that obvious to most of us, including me.

I've got classes defined in models (it's sort of the obvious place to define custom validators, for example), and I'd like to understand the implications better than I do.

[Massimo Di Pierro]

You are correct. There ar no problems using trunk (and future web2py
versions) if classes have no __del__ method.

> alternative to the Standard Library logging module:http://plumberjack.blogspot.com/2010/09/python-logging-functionality-...
>
> Anthony

[Massimo Di Pierro]

Not quite like that. It is more like (web2py specific):

"Do not declare classes with a __del__ method in web2py models or
controllers or instances of those classes will cause memory leaks"

AND (for any python program)

"Do not declare classes with a __del__ method in any Python program
unless you are sure there are no circular references"
Circular references are very easy to create accidentally in Python.


Massimo

[Massimo Di Pierro]

There is no problem if they have no __del__ method.

[cjrh]

On Feb 1, 10:16 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

> From the usual source:
> http://lucumr.pocoo.org/2011/2/1/exec-in-python/

His post is pretty good and informative.

> What would really be useful to us is an example of how to overcome
> that problem by showing how to break those implicit self references.

"Now now we know the cause, why doesn't it happen if you have a
module? The reason for that is that Python will do a trick when it
shuts down modules. It will override all global values that do not
begin with an underscore with None"

Is quite an interesting point. Is there a way we can do this during
the exec cycle, or is too low a level necessary?

Furthermore, this:

"With that I encourage the web2py developers to reconsider their
decision on the use of the exec statement and using regular Python
modules."

is never going to happen because we're fully backward-compatible.
And besides, he and JKM are entitled to their opinion, but that
doesn't mean they are right. I have used Django before. I intend one
day using Flask. I know what's out there.

I think the bottom line is that one should always understand what
you're doing, and why you're doing it. I could care less about some
python developer being confused with different semantics. I mean,
heck, just moving between a few different languages already have huge
semantic differences. That's really a non-argument. I *like* the
fact that we use exec. In one of my applications, I even call exec
inside a controller on another piece of dynamically-uploaded code.
Implementation details must be managed, obviously (the memory leak
issue comes to mind), but the execution scheme is clearly viable.

I also *like* the fact that lotsa fanboys are going to listen to
negative posts like these about web2py and stay away. We do not need
to grow bigger simply for the sake of growing. The only direction
that matters is quality, not quantity. We need only focus on how to
make awesome web applications.

I could care less about "joining the world of python web frameworks",
or whatever condescending drivel JKM said in his last web2py reddit
post.

[desfrenes]

On 1 fév, 21:16, Massimo Di Pierro <massimo.dipie...@gmail.com> wrote:

> In web2py you cannot store instances of objects into session. While
> Armin makes it tool like this is a web2py problem this is more complex
> and general than that. In Python if you pickle and object that is
> instance of a class defined in /path1/mymodule.py and then you
> unpickle on a different installation where the class is defined in /
> path2/mymodule.py you may run into problems. Because web2py does not
> require installation (it is a feature) and has hot plug and play of
> apps (another feature) we cannot pickle instances (the price we pay
> for those features). To me it is worth it.

I know for experience that storing instances in session is usually a
bad application design idea.

[Massimo Di Pierro]

On Feb 2, 1:26 am, cjrh <caleb.hatti...@gmail.com> wrote:
> "Now now we know the cause, why doesn't it happen if you have a
> module? The reason for that is that Python will do a trick when it
> shuts down modules. It will override all global values that do not
> begin with an underscore with None"

Can you provide a proof of concept example code to do this?

Massimo

[Anthony]

Anyone? Is using local_import to import a class into a model/controller just as safe as using 'from a import A' (i.e., in terms of memory leaks), or do you have to use the Python import statement directly?

[Jonathan Lundell]

On Feb 2, 2011, at 2:08 PM, Anthony wrote:

Anyone? Is using local_import to import a class into a model/controller just as safe as using 'from a import A' (i.e., in terms of memory leaks), or do you have to use the Python import statement directly?

It should be equivalent; both end up calling __import__() to do the actual work.