Reloading modules

[Yarin]

For dev purposes, it would be nice to be able to reload modules without having to restart the server- or to have a setting that automatically reloads on each request- Otherwise it's almost impossible to do active module development from within web2py.

[simon]

from gluon.custom_import import track_changes; track_changes(True)

[Anthony]

On Saturday, April 28, 2012 10:24:21 AM UTC-4, simon wrote:

from gluon.custom_import import track_changes; track_changes(True)

 And note that only reloads when the module changes, not on every request.

Anthony 

[Yarin]

Awesome- thanks!

On Saturday, April 28, 2012 10:24:21 AM UTC-4, simon wrote:

[Yarin]

Anthony- is this a performance hit, or can it be left on?

[Yarin]

I've been trying this and it is not working.

I put this exact line:

from gluon.custom_import import track_changes; track_changes(True)

in my 0.py model file, before any other code. But still any changes I make in my modules do not take effect until I restart server- or if they do it is inconsistently- certainly not on every new page request. Basically, no different than before. 

On Saturday, April 28, 2012 10:24:21 AM UTC-4, simon wrote:

[Anthony]

On Saturday, April 28, 2012 1:50:25 PM UTC-4, Yarin wrote:

Anthony- is this a performance hit, or can it be left on?

It has to check the modification time of each file upon import, so may be a small performance hit.

Anthony

[Anthony]

Hmm, it's working for me. Can you post a module and some code for which it's not working?

Anthony

[Yarin]

@Anthony- There's really nothing specific to post, its a general thing, and not the first time i've dealt with it- just that this code fix doesn't change anything. Moreover, now I'm experiencing some truly odd behavior.

1. I have a module that was throwing an error, and fixing it doesn't seem to update.
2. I quit python, restart web2py
3. The page loads fine, module working.
4. I make a change- ANY change, including change in comments- and i start getting error tickets again, as if the old broken version of the module is still cached somewhere- how this would persist after quiting and restarting web2py makes no sense to me. It's just too erratic to track down.

I'm working locally with the rocket server on a Mac, if that means anything to anybody. 

[Bruno Rocha]

I have a similar problem.

example:

if I change this modules under this package: https://github.com/rochacbruno/Movuca/tree/master/modules/datamodel they never reloads, so I need to restart the server.

But if I change something in modules under this: https://github.com/rochacbruno/Movuca/tree/master/modules/handlers It works ok, and I dont need to restart for reload.

My track changes is in: https://github.com/rochacbruno/Movuca/blob/master/models/0.py

--

Bruno Rocha

[http://rochacbruno.com.br]

[Yarin]

@Bruno- Thanks for confirming this issue

@Anthony - Thoughts?

[Yarin]

Got to say this is scary- we're about to go into production with our first web2py app, and having erratic module behavior persist across server restarts is not something we signed up for. Please let's address this.

[Anthony]

@Bruno- Thanks for confirming this issue

@Anthony - Thoughts?

Not sure what's going on. Perhaps Massimo or Pierre (creator of custom importer) could weigh in.

Anthony 

[Massimo Di Pierro]

Let's clarify something....

web2py always uses the most comment models/controllers/views.

web2py (as Python does) loads modules onces and keeps them in memory, even when modules are shipped with the app.

The fact that 

    from gluon.custom_import import track_changes; track_changes(True)

overrides this behavior if a feature to be used in development, not in production.

Reloading modules is a bad idea. It has performance penalties and can cause undesired effects because of global objects defined in the modules.

A production application should not do this. 

[Richard Galka]

I would also like to comment that I suspect (unconfirmed) that reloading of modules while references are still around appears to potentially cause memory leaks.. during dev while modules are reloaded, our app's memory usage grows significantly. With this disabled, we find a consistent footprint throughout the apps life. 

[Massimo Di Pierro]

Yes. because it depends on what module does. You should not do this in production. Many "other" frameworks treat models and controllers and they potentially suffer from this problem (depending on the module). This is why in web2py models and controllers are not modules, to avoid this problem.

reloading modules is not good.

Massimo

[Yarin]

@Massimo- The initial issue was about reloading modules for development not production.

    from gluon.custom_import import track_changes; track_changes(True)

was not consistently reloading modules on each request while we were developing locally on the Rocket server, though I can't consistently reproduce this.

The second issue, which has me concerned about production, is that right now it's not clear to me what the boundaries are of modules and the execution environment in general. I had been under the assumption that, in wsgi applications, restarting the server would always clear out all modules and spawn a new execution environment. But we saw some behavior after Rocket restarts that looked like things were persisting across server restarts, though I can't say for certain.

Is it feasible that anything in the environment persist across wsgi server restarts? (If so then my understanding is all wrong). With Apache, does the configuration of mod_wsgi, and whether it's in embedded or daemon mode, have any bearing on this?

[Yarin]

@Massimo- you're saying a production application should not reload modules, but then what is the correct process for updating an app in production - i.e. installing an app by overwriting an old one - if it has modules that have changed?

On Sunday, April 29, 2012 12:02:52 PM UTC-4, Massimo Di Pierro wrote:

[Massimo Di Pierro]

Models and controllers are executed at every request therefore it is safe to change them while the app is running. Web2py handles them the way it does to allow you how replacement.

Python modules - in general - should not be replaced without turning the app off and on again.

It really depends on the module and this is not a web2py specific issue. A module may define a global object and if you reload the module you end up with two of those objects. Depending on how it is used this may have unintended consequences. The fact that you technically can force python to reload a module does not mean that is  good practice in general. 

[Yarin]

@Massimo- Thank you, I understand the practice but still not the prescription. By turning the app on/off you mean disable/enable? Don't all apps live in the same site-wide Python process, and wouldn't that mean that the global objects would remain?

[Yarin]

I think I've figured out the issue I was having with the auto reloader- the modules I import into the app do in fact reload, it was their dependancies that weren't- so if made a change on a module that was being imported only by another module, the changed module wasn't reloaded. 

The solution is to import every module, including dependencies, into your app directly. So far it's working..

On Saturday, April 28, 2012 2:40:35 PM UTC-4, Yarin wrote:

[Alex Benfica]

I'm having a seriuos problem having to stop and start Google App Engine SDK when I want edit modules, otherwise, they are not reloaded.
The problem is that, doing this, I lose the entire database content on GAE SDK...

Is there a way to edit web2py modules and get then reloaded each time I run the app inside GAE SDK?
I have these lines on the beginning of my first model file... and it is not working yet...

    from gluon.custom_import import track_changes
    track_changes(True)

Any ideas?

On Saturday, April 28, 2012 10:43:42 AM UTC-3, Yarin wrote:

For dev purposes, it would be nice to be able to reload modules without having to restart the server- or to have a setting that automatically reloads on each request- Otherwise it's almost impossible to do active module development from within web2py.

[Ricardo Pedroso]

On Wed, May 9, 2012 at 8:35 PM, Alex Benfica <alexb...@gmail.com> wrote:
>
> I'm having a seriuos problem having to stop and start Google App Engine SDK
> when I want edit modules, otherwise, they are not reloaded.
> The problem is that, doing this, I lose the entire database content on GAE
> SDK...

You can persist the data on GAE SDK

https://developers.google.com/appengine/docs/python/tools/devserver#Using_the_Datastore

Ricardo

[Alex BENFICA]

Oi Ricardo,

Thanks.

I add the option to use sql lite and databse keeps data when I restart
app engine SDK.
And about the module reloading... did you have any problems with it?

--
--

Atenciosamente,
Alex BENFICA

--
"O que não se mede não se gerencia."

[howesc]

i use the persistent local datastore on GAE SDK as well.  i have not tried to turn on dynamic module re-loading on GAE, but i have a shell script that starts/stops GAE and i just restart that.  it's not as fast as auto-reloading but it has been ok for me (though now that i'm doing more and more in modules it does slow things down)

[Alex BENFICA]

Hi...

When you say "now that i'm doing more and more in modules it does slow
things down"... are you talking about development time, right? is not
about the performace of you web2py app... is that?

The performance increase when using modules instead put code on models
worth all this extra work?
Do you have any measure about this performance gain?

[Christian Foster Howes]

yes, sorry, i meant in development time.

in one project i moved models to modules and shaved about 33% off of the
GAE production request time. another project is in test this week and i
will hopefully have numbers soon. note that when you don't need auth
(not all of my requests use auth) not setting it up saves about 200ms on
GAE.

cfh

>>> "O que n�o se mede n�o se gerencia."
>
>
>

[Andrew Rogers]

This is an old post. But I had this problem too. 'track_changes' works fine in a brand new project. But i can't get it to work in my now quite large project. 

Eventually I found something that works. My module is called 'public.py' and it lives where it should - in the 'web2py/applications/myapp/modules' folder.  I include the following at the top of the db.py file in the 'models' folder.

import importlib

import sys

importlib.reload(sys.modules['public'])

I am  not sure what the proper approach is to develop using modules. I have a lot of code in the models folder currently. However everything i have read says 'dont put too much stuff in the models folder' and 'dont reload modules'.

I know i shouldn't reload modules when in production. But it would be unpleasant to develop code in modules if i had to restart the server everything each time i made a change. If anyone has any thoughts on the right way to do this i would be love to hear them.

Cheers

Andrew