Problem importing modules from other app (having symlinked models and modules)

[Lisandro]

I have two web2py applications; modules are defined in the first one, and symlinked from the second one. .

This is the structure (also notice some symlinked models):

applications/app1/

applications/app1/models/0.py

applications/app1/models/0_template.py  ---> symlink to app2/models/0_template.py

applications/app1/models/db.py

applications/app1/modules/

applications/app1/modules/globales.py

applications/app2/

applications/app2/models/0.py  ---> symlink to app1/models/0.py

applications/app2/models/0_template.py

applications/app2/models/db.py  ---> symlink to app1/models/db.py

applications/app2/modules/ ---> symlink to app1/modules/

This two applications are two separated mercurial repositories.

Every once in a while, I make some change to one (or both) application, then I go to the server and I do a hg pull and hg update. 

After the hg update, the application shows an intermitent error 500 with this traceback:

Traceback (most recent call last):
  File "/home/gonguinguen/medios/gluon/restricted.py", line 227, in restricted
    exec ccode in environment
  File "/home/gonguinguen/medios/applications/fundapres_panel/models/0_template.py", line 3, in <module>
    from globales import Template, VariableConfiguracion, Estilo, Tema
  File "/home/gonguinguen/medios/gluon/custom_import.py", line 95, in custom_importer
    return base_importer(pname, globals, locals, fromlist, level)
ImportError: No module named fundapres_panel.modules.globales

In order to get it to work again, I have to restart uwsgi process (that's what I'm using, nginx + uwsgi). 

Restarting uwsgi process solves the problem, but I don't want to restart uwsgi process every time I do an update.

Notice that the line throwing the error is in models/0_template.py, which is one of the models that is symlinked. 

Also, I don't understand why the error is intermitent (but it's there, the error shows up once every two or three calls).

And one last thing: the error appears even if the update doesn't update anything. Even if nothing has changed in the repository, if I do an update, the error appears and I'm forced to restart uwsgi process.

I'm not using track_changes or anything like that. 

I'm using the simple form of the mercurial update, just "hg update", no parameters provided.

I know it's possible to import modules in other ways (using __import__ or importlib), but I've tried that and the problem remains. Any suggestion will be appreciated. Thanks!

[Niphlod]

beside from the fact that is hardly a web2py problem, what the heck ? you udpate the app and you don't want to reload the process that runs it ?

[Lisandro]

Yes, I understand it's not a web2py problem, I was just wondering if there was a way of achieving that without restarting uwsgi process.

I can see the need of restarting the process that runs the app after updating that app. However, most of the times, the update involves changes to static files (css, some javascript, maybe some images), and I'm forced to restart uwsgi process when the app code hasn't actually changed. Moreover: I'm running a lot of web2py applications (between 40 and 50), and I don't like the idea of restarting the uwsgi process that serves all that apps just because I had to update some css files in just one app.

But I understand this is not web2py related, I was just wondering if it would be possible to do it in another way. 

I will look into Mercurial to see if there is a way of updating just the files that have actually changed, don't know if this would actually work. 

Another obvious thing to do is avoiding frecuent small updates and, instead, updating the app once in a while with all the changes needed. However in my case this is not possible, because each app correspond to a customer, and sometimes the customer needs the change to be applied inmediately.

[Niphlod]

why don't you use the emperor mode with uwsgi pointing each process to a file that, when you modify it, reloads automatically the process ?

[Lisandro]

That's what I have actually :)

I have emperor mode configured as suggested in the book. When I want to restart uwsgi process, I just make a touch to the uwsgi.ini file and the process restart successfully. It's inmediate and transparent to the users (I mean, users browsing the websites). 

The "problem" is when I want to update several of the installed apps. In this case, I have two options:

1) Do a "hg update" in all the apps, and **finally**, restart uwsgi process. But in this case, the updated apps will be throwing that error at least for the time that takes to finish the update in all apps. 

2) For every app, do a "hg update" and **inmediately after** restart uwsgi process. I've tried this but it didn't went too well, because the uwsgi process was being restarted several times in a very short period of time (I mean, it was restarted as much times as apps I had updated). When I tried that, I ended app with nginx not finding the uwsgi process and all apps throwing errors (about socket not being available or something like that).

[Niphlod]

there's a distinct lack of singular/plurals in your statements that may pinpoint the problem.

multiple apps --> multiple uwsgi processes

either you "hg update" all the apps, and restart ALL uwsgi processes (more a reload than a restart, hopefully, touching EACH and EVERYONE of the files uwsgi watches for changes, without them being simlinked) OR for every app, you hg update and then restart THE uwsgi process, touching the single file that reloads THAT app.

IMHO you'd need to do the latter, without EVER simlinking anything. There are tons of deploy facilities lying around that will make symlinking an old-fashioned thingy.

[Lisandro]

I see your point. I'm reconsidering the whole thing. Your answer made me think that maybe my setup isn't quite right.

First of all, I've found a solution for my problem of the error when updating code: I will just copy the modules from one app to another, instead of symlinking. I think now it's the best practical solution, and it can be done quickly and easy. This solves the "problem" and allows me to update my web2py apps without reloading uwsgi (of course there will be cases where reloading will have to be done).


Now, regarding the setup. You say multiple apps --> multiple uwsgi processes, but I think my configuration is different (and maybe wrong). 

What I have is the result of having followed this steps of the book:
http://web2py.com/books/default/chapter/29/13/deployment-recipes#Nginx

So I have:
 - one only web2py instance
 - one only uwsgi configuration file, as in the book: /etc/uwsgi/web2py.xml

Then, I installed several web2py apps under /var/www/web2py/applications, and I've configured routes.py to serve each web2py app through a specific domain. Inside nginx's configuration, I've defined one virtual host (server block) per domain, and all of them make an uwsgi_pass to the same socket, thus, all web2py apps are using the same socket.

I also have uwsgi emperor mode. I know, what's the point of having emperor mode if I have only one uwsgi configuration file? It's because I'm planning to run two more web2py instances, but not right now. Right now, emperor mode is only watching the file /etc/uwsgi/web2py.xml. That file is pretty much like the one showed by the book. So, when I say "reload or restart uwsgi" I mean touching that file, in order to emperor reloading the configuration.

Is this setup correct (or at least acceptable) for deploying multiple web2py apps?

[Niphlod]

then I really don't know why are you complaining about uwsgi restarted several times. if you have a single process handling all the things, said process should be reloaded every time you update any of your apps.

[Lisandro Rostagno]

I wasn't complaining about restarting uwsgi, I was wondering about
what was causing the error after an update, even when the update only
involved just static files like css. First I thought it could be the
modules folder being symlinked from one app to another, so I removed
the symlink and copy the modules, but it was the same: when I updated
from repository, the error was there again, until I reloaded uwsgi
process.

I think the problem could be the models being symlinked (the line that
throws the error is in one of those symlinked models).

Nevermind anyway. For the moment I will continue reloading the uwsgi
process everytime an app is updated. I guess I'll have to organize my
work in a way that doesn't require frequent updates.

As always, thank you very much for the help.

> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "web2py-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/web2py/MVRBJaEYcyA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> web2py+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.