How to keep project separate from web2py itself?

[Brendan Barnwell]

I've recently begun looking at web2py.  It looks very nice.  However, one thing that seems strange to me is that it is apparently not set up to be installed.  Rather, to "install" it you just copy the web2py files (or clone the repo) into a directory.  That install then "knows" about all the applications stored under its directory tree.

This seems like a rather odd setup.  With this setup, my application is stored under the web2py directory tree.  But I want to put my application code in its own directory --- in particular, in its own repository.  My application's code should be handled separately from the code of web2py itself.

Also, it's not clear to me how I would then deploy my application without also "installing" web2py separately for each app.  So, if I want to copy some files to my web server to run an app, I would expect to install web2py ONCE to the server, and then set up my app to use the existing web2py installation.  But with the way it's actually set up, it appears I would have to install separate copies of web2py for each app, which seems rather wasteful.

I found an old post (https://groups.google.com/forum/#!topic/web2py/3FWZCmLzhzc) where someone talks about handling this by symlinking from the web2py directory to the "real" app directory.  Is this really the way you have to do it?  Isn't there a way to take JUST my own application code, and tell it where to find web2py (or tell web2py where to find my app), and then get web2py to run my app regardless of where they each are on the drive?  I want a clean separation between the code that is part of the web2py framework itself and the code that is specific to my app.

Thanks,
Brendan

[Massimo Di Pierro]

Hello Brendan, there is not one solution to this problem. What is important is that there is one one web2py folder and one applications (plural) folder. Normally the latter is under the former but it does not have to be. Each application you run must be a subfolder of that one applications folder.

This is because unlike Flask and other frameworks where the application is the entry point and it imports the modules you need and therefore you run one server per each application, in web2py, web2py is the entry point. You run one server which serves all your applications. It also runs the web based IDE (unless you remove it) and that needs to know where to put applications uploaded via the web interface.

So the two options are (not exclusive)

1) move the applications folder outside the web2py folder (this is cone by copying web2py/examples/options_std.py into web2py/options.py, edit it, and set folder = to the location where to find applications)

2) move an individual app outside the applications folder (this require that you symlink it under the applications folder)

Massimo

[Anthony]

This seems like a rather odd setup.  With this setup, my application is stored under the web2py directory tree.  But I want to put my application code in its own directory --- in particular, in its own repository.  My application's code should be handled separately from the code of web2py itself.

In addition to Massimo's explanation, note that just because an application folder exists inside the web2py folder does not preclude you from making an individual application its own repository -- you can locate a repository anywhere in the filesystem.

Anthony

[Brendan Barnwell]

Thanks.  The "folder" option in options.py seems like a decent solution for the moment.  However, if I understand right, it would still require me to move the existing applications (like admin) along with my own apps.

Have you considered adding a feature to web2py to allow the directory for each application to be specified individually, rather than having them all under a single global applications directory?  For instance, why couldn't the "folder" option you mentioned instead be a list of folders?

[Brendan Barnwell]

Yes, I realize I can do that.  The problem is that (if I understand right) unless I put it in the web2py folder, web2py won't know about it.  I can symlink it (or directory-junction it, in windows), but that still seems quite awkward to me.  I feel like it would make more sense if the information about where web2py looks for applications was part of web2py's own configuration, not so closely tied to the directory structure on disk.

[Dave S]

On Wednesday, January 27, 2016 at 11:37:47 PM UTC-8, Brendan Barnwell wrote:

Yes, I realize I can do that.  The problem is that (if I understand right) unless I put it in the web2py folder, web2py won't know about it.  I can symlink it (or directory-junction it, in windows), but that still seems quite awkward to me.  I feel like it would make more sense if the information about where web2py looks for applications was part of web2py's own configuration, not so closely tied to the directory structure on disk.

As a Mercurial user, I am quite comfortable having my repository in the applications/myapp directory.  Because Mercurial is a DVCS, I can keep my code elsewhere as well (even on another system) and pull or push to keep them synchronized.  Git is also a DVCS, and I would not be surprised to find that it can hide nicely in the application folder as well.  I'd even expect web2py to play nice with an SVN workspace in the application folder (but the repo would be on a real SVN server only).

Because I'm comfortable with doing my source control in situ, I don't feel any need to not have my app as a subdirectory in the tree.  And, independent of source control, fab files make it easy to deploy an updated app from development to production.  (Or if you're a chef or salt guru, I'm sure knocking off a recipe is pretty straightforward, but I've only read the tutorials for those guys, and fab files may be all I need.)

My 2 cents, which isn't going to buy a cup of coffee anywhere, and I've only done smallish projects so far.  I'm sure there are folks in the group who have done actual large scale deployments and have tried different things.  I can tell from their posts that some of the regulars here (like Anthony) have seen many an app over the course of time.

/dps

[Dave S]

On Thursday, January 28, 2016 at 12:19:41 AM UTC-8, Dave S wrote:

On Wednesday, January 27, 2016 at 11:37:47 PM UTC-8, Brendan Barnwell wrote:

Yes, I realize I can do that.  The problem is that (if I understand right) unless I put it in the web2py folder, web2py won't know about it.  I can symlink it (or directory-junction it, in windows), but that still seems quite awkward to me.  I feel like it would make more sense if the information about where web2py looks for applications was part of web2py's own configuration, not so closely tied to the directory structure on disk.

As a Mercurial user, I am quite comfortable having my repository in the applications/myapp directory.

I should point out that I don't do web2py core development, so I don't need to maintain a repository for web2py itself.  I typically download the current stable zip (or windows exe), unpack in a new directory, and use Beyond Compare to see what changed in the welcome and adminapp applications, and then to copy my application files into their new home.

Even if I did clone a repo for the core, it would be easy enough to tell the DVCS to ignore my app and to manage the app from its repo in the subdirectory.  Mercurial also has the notion of subrepos, if I felt had to be able to synchronize web2py and myapp closely.  Although sometimes it's better to use a build products manager to handle that type of synchronization (I think that's what gradle does, but my Continuous Integration reading has all run together in a big blur).

[Anthony]

On Thursday, January 28, 2016 at 2:37:47 AM UTC-5, Brendan Barnwell wrote:

Yes, I realize I can do that.  The problem is that (if I understand right) unless I put it in the web2py folder, web2py won't know about it. 

I'm not sure I follow. If you realize you can leave your application folder inside /web2py/applications and simply create a version control repository right there, then why is it a "problem" that it must be inside the web2p folder?
 

I can symlink it (or directory-junction it, in windows), but that still seems quite awkward to me.  I feel like it would make more sense if the information about where web2py looks for applications was part of web2py's own configuration, not so closely tied to the directory structure on disk.

Well, you can already move the entire /applications folder somewhere else. And you can also create a separate version control repository for each application within the /applications folder. What additional benefit do you get by moving each application folder to some arbitrary place in the filesystem? In the rare cases where that is necessary, you can always use existing OS facilities, such as symlinking.

I suppose we could allow further configuration, but that would add complexity to the routing system and the functioning of the admin app. We tend to avoid additional complexity unless there is a compelling benefit. It's hard to see the compelling benefit here.

Anthony

[Brendan Barnwell]

On Thursday, January 28, 2016 at 10:21:54 AM UTC-8, Anthony wrote:

On Thursday, January 28, 2016 at 2:37:47 AM UTC-5, Brendan Barnwell wrote:

Yes, I realize I can do that.  The problem is that (if I understand right) unless I put it in the web2py folder, web2py won't know about it. 

I'm not sure I follow. If you realize you can leave your application folder inside /web2py/applications and simply create a version control repository right there, then why is it a "problem" that it must be inside the web2p folder?

 
Basically because my web2py application is only one part of a larger project, other parts of which have nothing to do with web2py.  (I write web apps for use as online experiments/surveys.  The app collects the data, but the analysis is done offline, and then other things are done with that data/analysis that are unconnected with the web2py app that was used to collect it.)  What I want is to have a directory for that project, and a subdirectory within that for the web2py app.  But I can't do that now without symlinking my subdirectory into the web2py folder.

I can symlink it (or directory-junction it, in windows), but that still seems quite awkward to me.  I feel like it would make more sense if the information about where web2py looks for applications was part of web2py's own configuration, not so closely tied to the directory structure on disk.

Well, you can already move the entire /applications folder somewhere else. And you can also create a separate version control repository for each application within the /applications folder. What additional benefit do you get by moving each application folder to some arbitrary place in the filesystem? In the rare cases where that is necessary, you can always use existing OS facilities, such as symlinking.

I suppose we could allow further configuration, but that would add complexity to the routing system and the functioning of the admin app. We tend to avoid additional complexity unless there is a compelling benefit. It's hard to see the compelling benefit here.

The compelling benefit I see is that it enables a web2py app to be one part of a larger project, rather than forcing that entire project to be stuffed under the web2py directory.

Symlinking is a sort-of solution, but on Windows symlinking is not as simple as it might sound.  (As indicated in the other thread I linked to, you have to create a "directory junction", which is different than an ordinary file link.)  Also, symlinking raises the possibility that other filesystem operations I might do in the future will unexpectedly recurse into the linked directory.  I don't want my directory to act as if it were in the web2py directory in general, just for web2py's own internal purposes, which is why I say I think it makes more sense for this to be part of web2py config rather than filesystem-level things like symlinks.

I see this situation as analogous to how Python libraries work.  Normally, when you install a library, it gets put under site-packages.  But if you want, you can put the library somewhere else, and use a .PTH file (which you put in site-packages) to tell Python where to find the library.  This is for instance how pip editable installs work.  Of course the "normal" situation is to install the library directly in site-packages, but there are many situations where for whatever reason (e.g., development) you don't want to do that.  Python doesn't handle this by relying on OS-level symlinking; it has its own system (.PTH files) that allows you to tell Python where your libraries are independent of where they are in the filesystem.  I think that flexibility would be valuable in web2py as well.

Anyway, thanks for your continued dialogue with me on this.  Web2py does seem really cool and I think I can make it work for me for the time being.  I'm mainly raising this question with an eye down the road to future projects I might do with it, where I foresee it might become unwieldy to handle as I have more and more of these projects of which a web2py app is only one piece.

[Anthony]

On Thursday, January 28, 2016 at 3:40:17 PM UTC-5, Brendan Barnwell wrote:

On Thursday, January 28, 2016 at 10:21:54 AM UTC-8, Anthony wrote:

On Thursday, January 28, 2016 at 2:37:47 AM UTC-5, Brendan Barnwell wrote:

Yes, I realize I can do that.  The problem is that (if I understand right) unless I put it in the web2py folder, web2py won't know about it. 

I'm not sure I follow. If you realize you can leave your application folder inside /web2py/applications and simply create a version control repository right there, then why is it a "problem" that it must be inside the web2p folder?

 
Basically because my web2py application is only one part of a larger project, other parts of which have nothing to do with web2py.  (I write web apps for use as online experiments/surveys.  The app collects the data, but the analysis is done offline, and then other things are done with that data/analysis that are unconnected with the web2py app that was used to collect it.)  What I want is to have a directory for that project, and a subdirectory within that for the web2py app.  But I can't do that now without symlinking my subdirectory into the web2py folder.

Got it. It's helpful to understand the use case.
 

I suppose we could allow further configuration, but that would add complexity to the routing system and the functioning of the admin app. We tend to avoid additional complexity unless there is a compelling benefit. It's hard to see the compelling benefit here.

The compelling benefit I see is that it enables a web2py app to be one part of a larger project, rather than forcing that entire project to be stuffed under the web2py directory.

By compelling benefit, I meant for the larger community -- I'm not sure this is a common scenario. If there's demand, though, I'm sure something can be done.

Anthony

[Niphlod]

PS: didn't have time to read through all, but did you consider using web2py.py -f /path/to/applications_directory ? I just fixed a couple of issues in trunk but it works fine.

[Anthony]

On Friday, January 29, 2016 at 3:00:47 AM UTC-5, Niphlod wrote:

PS: didn't have time to read through all, but did you consider using web2py.py -f /path/to/applications_directory ? I just fixed a couple of issues in trunk but it works fine.

I think he wants to be able to put each individual application in a separate folder somewhere on the filesystem (as each is part of a larger project and ideally would be nested in the folder structure of its respective project).

Anthony

[Dave S]

With Git and Mercurial, he can have a repo in his project tree and sync it to the repo in the applications folder.  Slightly more complicated than a symbolic link, but still pretty easy-peasy.  (I've done that sort of thing.)

/dps

 

[Brendan Barnwell]

On Friday, January 29, 2016 at 6:27:36 AM UTC-8, Anthony wrote:

Right, that's what I'm looking for.  A separate, independently-located folder per application, not just a still-global applications folder in a different location.

[Niphlod]

some things in web2py are not app-dependant so I guess it'll be quite not possible to have a per-app folder.

[Anthony]

On Tuesday, February 2, 2016 at 2:38:59 AM UTC-5, Niphlod wrote:

some things in web2py are not app-dependant so I guess it'll be quite not possible to have a per-app folder.

Presumably if there were some configuration file mapping applications to folders (perhaps even in routes.py), it could be made to work (not saying we should do it).

Anthony