Re: Subfolders in models

[Anthony]

You can now do:

response.models_to_run = ['list', 'of', 'regexes']

You can set that (and change it) in any model file. The regexes are relative to the /models folder, so any model file with a relative file path that matches one of the regexes will be executed. By default, the response.models_to_run regexes are defined to enable the standard conditional models (i.e., execute models in subfolders whose names match the current controller and function). However, you can change it to whatever you like. To run all models in all subfolders, somewhere in the first model file, just do:

response.models_to_run = ['.*']

It is checked after each model is executed, so it can be changed dynamically from one model file to another. Also, note that when specifying file paths, always use '/' as the path separator, even on Windows.

Anthony

On Thursday, November 22, 2012 9:04:24 AM UTC-5, Joel Samuelsson wrote:

Hi,

I would like to have a subfolder structure in my models folder. I am aware that subfolders are only executed upon request from the user. I saw this thread however:
https://groups.google.com/forum/?fromgroups=#!searchin/web2py/models$20subfolders/web2py/o8Ya4ZqNhLk/eoeLK7_45Z4J
But after that, I haven't been able to find any mention if the fix (mentioned in last message in above thread) was ever implemented.

Can I somehow get subfolders in my models catalogue to be executed without moding compileapp myself?

Best regards,

Joel

[Joel Samuelsson]

I've been testing the way you described above and it works, to a point. The models are not imported in alphabetical order. This seems to be controlled by this line (521):

models = listdir(path, '^\w+\.py$', 0, sort=False) in compileapp.py
listdir is essentially os.walk with a regexp for filtering and os.walk doesn't give any guarantees for ordering at all from what I understand.

Later in compileapp, the models found with listdir are looped through, once (not dependant on the regexp) which gives changing the regexp between models limited use.

To explain my issue, I'll try to use an example. Please bear with me!

Let's say I have the following models:

dbInterface/someDbFile.py

anotherInterface/someAnotherInterface.py

__init__.py
logic.py

Now let's assume that listdir lists the models root dir first, then anotherInterface dir and then dbInterface dir.

I want the dbInterface to load before anotherInterface since anotherInterface is dependant on dbInterface.

So in __init__.py I do:

response.models_to_run =['dbInterface/']

And in the someDbFile.py model, I change the regexp to:

response.models_to_run = ['anotherInterface/']

This will however cause stuff in anotherInterface to never be loaded since the loop in compileapp has already passed (and filtered out) all anotherInterface files once I get to the dbInterface files.

Is there any way I can control the order things are imported from subfolders without changing web2py? I am aware that just setting sort to true in the listdir-call will fix my issue but I would like to be able to update web2py without having to make that change each time, if possible.

Best regards,

Joel

[Anthony]

I've been testing the way you described above and it works, to a point. The models are not imported in alphabetical order. This seems to be controlled by this line (521):

models = listdir(path, '^\w+\.py$', 0, sort=False) in compileapp.py
listdir is essentially os.walk with a regexp for filtering and os.walk doesn't give any guarantees for ordering at all from what I understand.

Note, it is not os.listdir -- it is the listdir function from gluon/fileutils.py, which does sort the files alphabetically: http://code.google.com/p/web2py/source/browse/gluon/fileutils.py#88.

 

Later in compileapp, the models found with listdir are looped through, once (not dependant on the regexp) which gives changing the regexp between models limited use.

It loops through the files alphabetically -- within any given file, response.models_to_run can be changed, which can affect whether models that come later in alphabetical order get run (obviously it cannot cause models that come earlier in alphabetical order to be run). Using response.models_to_run does not change the order in which models are run, nor the ability of later models to affect earlier ones. It merely controls which models get run, with some ability for earlier models to make dynamic changes that affect later ones.

 

To explain my issue, I'll try to use an example. Please bear with me!

Let's say I have the following models:

dbInterface/someDbFile.py

anotherInterface/someAnotherInterface.py

__init__.py
logic.py

Now let's assume that listdir lists the models root dir first, then anotherInterface dir and then dbInterface dir.

That should be exactly how listdir lists the models.

 

I want the dbInterface to load before anotherInterface since anotherInterface is dependant on dbInterface.

Very simple -- just change the names so dbInterface comes before anotherInterface.

 

Is there any way I can control the order things are imported from subfolders without changing web2py? I am aware that just setting sort to true in the listdir-call will fix my issue but I would like to be able to update web2py without having to make that change each time, if possible.

As noted, sort is already True, so you should be OK. There is no way to change the order in which web2py executes the models, though (other than to generate your folder and file names to produce the order you desire).

 

Anthony

[Joel Samuelsson]

I'm not sure if we're using the same web2py version. I am using the current stable release (2.2.1). The call to listdir looks like this (just downloaded a fresh copy and double-checked):

models = listdir(path, '^\w+\.py$', 0, sort=False)

I.e. sort=False. Otherwise, what you describe is true. With sort=False though, sort order is not alphabetical and what you describe does not work since you can't know in what order models are run.

/Joel

[Massimo Di Pierro]

How did that get in there? Fixed in trunk. Thanks Joel for reporting this.

[Anthony]

Looks like sort=False was initially introduced here -- not clear exactly why. Note, fileutils.listdir automatically sorts the files within folders, so this appears to affect only the sorting of the folders. It wasn't really an issue until the recent introduction of response.models_to_run because prior to that, with conditional models, only one folder was run.

Anthony

[Massimo Di Pierro]

sort=False was probably introduce to avoid un-necessary sorting... until it become necessary again and I did not set it back to True.

[Joel Samuelsson]

Glad to help! You do excellent work. Thanks!

[Joel Samuelsson]

In version 2.6.3, sort=False is still there. Unsure if the way it's supposed to be used has changed. Got the same bug as before when I updated.