Call functions from other applications without network requests

[Brendan Barnwell]

I have a situation where I have (or am going to have) multiple applications running on a single web2py instance.  Each application will do its own thing, but they are all using the same databases.  Some of the applications may serve as backend API endpoints where certain kinds of data can be retrieved.  Other applications may sometimes want to make calls to these endpoints to get the data and use it in their own operations.

So what I'm wondering is: is there any way for one application to call a controller in another application and get the raw data that WOULD be used to build an actual HTTP response, but without actually going through the network?  That is, suppose I have and endpoint /app1/controller1/get_data and another /app2/controller2/do_something .  Inside do_something, I want to get the data that get_data gets.

Currently I have get_data set up as a JSON service.  I could certainly make the request to the other endpoint, but that involves a number of extra steps: generating an outgoing HTTP request, serializing any data to passed with it, generating the incoming HTTP request object, deserializing the request data, then serializing the response and deserializing it again when it's received.  What I really want to do is to just call the function get_data AS IF a request had been submitted, but have everything happen on the server, without any network requests actually happening.  As far as I can tell, I can't in any way "import" the get_data function from the other app, because if I call it, it won't have the magic environment set up (e.g., "db" won't exist for it to do its database queries).  I would even be okay with "pretending" like I was doing a request, as long as web2py could somehow realize that the app I want to call is running on the same web2py instance, and call it "directly" instead of submitting a network request to itself.

I realize that this architecture involves apps that are more interdependent than they would be in a "by the book" web2py project.  But basically the idea is that the functionality of the apps is independent, and in theory they should work if running on different servers, by making real requests.  It's just that, IF they are running on the same server, I'd like to communicate between them without going through the network, to improve performance.  Also, I can't really change the architecture since it's for a project I'm working on but don't ultimately control.

[Niphlod]

never though about just scheduling the functions via the scheduler ?

[Anthony]

Check out http://web2py.com/books/default/chapter/29/04/the-core#Execution-environment and http://web2py.com/books/default/chapter/29/04/the-core#Cooperation.

You might also consider whether the shared functionality can be abstracted into Python modules that can be imported by multiple applications.

Anthony

[Brendan Barnwell]

The shared functionality involves DB access, so as far as I can tell it cannot be abstracted into modules that would then be imported with a normal import.  But I will take a look at the exec_environment thing, which may be what I'm after.

[Anthony]

On Saturday, January 7, 2017 at 1:57:36 PM UTC-5, Brendan Barnwell wrote:

The shared functionality involves DB access, so as far as I can tell it cannot be abstracted into modules that would then be imported with a normal import.

You can use the DAL from modules as well. As Niphlod suggested, you can also use the scheduler to schedule and run a task in the context of another app, though that might not be as fast as you'd like, as even setting immediate=True, it could take up to "heartbeat" seconds for the worker to pick up and execute the task, and then you have to check for the results.

Anthony

[Brendan Barnwell]

I can use the DAL, but can I use the "db" object defined in my models file?  Or do you mean I could use the DAL if I rewrote my other app to use a "model-less" approach where the models were defined in some other manner?

[Anthony]

You can use the DAL from modules as well. As Niphlod suggested, you can also use the scheduler to schedule and run a task in the context of another app, though that might not be as fast as you'd like, as even setting immediate=True, it could take up to "heartbeat" seconds for the worker to pick up and execute the task, and then you have to check for the results.

I can use the DAL, but can I use the "db" object defined in my models file?  Or do you mean I could use the DAL if I rewrote my other app to use a "model-less" approach where the models were defined in some other manner?

Yes, the point was that you can put whatever you want into modules, which means the code can be shared across apps. If you want to share the code to create a database connection object, then put that code in a module. Here is the basic idea:

/app1/modules/db.py:

import os
from gluon import *

def dal(**kwargs):
    web2py_folder = current.request.env.applications_parent
    db_folder = os.path.abspath(os.path.join(web2py_folder, 'applications',
                                             'app1', 'databases'))
    return DAL('sqlite://storage.sqlite', folder=db_folder, **kwargs)

def define_tables(db):
    db.define_table('table1', Field('field1'))

/app2/models/db.py:

from applications.app1.modules.db import dal, define_tables

app1_db = dal()
define_tables(app1_db)

# Now use app1_db as usual.

Of course, the module defining the database connection and tables doesn't have to be in the /modules folder of any particular application, nor does the folder specified as the "folder" argument -- that can all be centralized somewhere if desired.

Also, note there is a way to share models defined in the model files of another app, though with limitations. See http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#Using-DAL-without-define-tables. That approach can be used to access model definitions in app1 model files from within app2 model files. However, the model definitions within app2 will only include the table and field attributes that can be gleaned from the *.table migration files in app1's /databases folder, which only includes attributes needed for the database schema definition (i.e., it doesn't include any web2py-specific attributes, such as validators, labels, represent functions, etc.).

Anthony

[Brendan Barnwell]

On Friday, January 13, 2017 at 8:21:48 AM UTC-8, Anthony wrote:

You can use the DAL from modules as well. As Niphlod suggested, you can also use the scheduler to schedule and run a task in the context of another app, though that might not be as fast as you'd like, as even setting immediate=True, it could take up to "heartbeat" seconds for the worker to pick up and execute the task, and then you have to check for the results.

I can use the DAL, but can I use the "db" object defined in my models file?  Or do you mean I could use the DAL if I rewrote my other app to use a "model-less" approach where the models were defined in some other manner?

Yes, the point was that you can put whatever you want into modules, which means the code can be shared across apps. If you want to share the code to create a database connection object, then put that code in a module. Here is the basic idea:

/app1/modules/db.py:

import os
from gluon import *

def dal(**kwargs):
    web2py_folder = current.request.env.applications_parent
    db_folder = os.path.abspath(os.path.join(web2py_folder, 'applications',
                                             'app1', 'databases'))
    return DAL('sqlite://storage.sqlite', folder=db_folder, **kwargs)

def define_tables(db):
    db.define_table('table1', Field('field1'))

/app2/models/db.py:

from applications.app1.modules.db import dal, define_tables

app1_db = dal()
define_tables(app1_db)

# Now use app1_db as usual.

Of course, the module defining the database connection and tables doesn't have to be in the /modules folder of any particular application, nor does the folder specified as the "folder" argument -- that can all be centralized somewhere if desired.

Okay.  So I've been experimenting with different approaches here.

In your example above, it's one model defining tables from another model.  But in my case, what I have is a controller function that needs access to the db, which is defined in the model file.  But I found the hard way (and later also from an old thread: https://groups.google.com/forum/#!topic/web2py-developers/5TPI9oYOZHM ) that exec_environment does not load models.  So this puts me kind of back to where I was.

The simplest solution I found was to put, in the controller file, code that looks like this:

from applications.otherapp.modules.db_definer
db = db_definer.define_db()

Then every time my controller is run, it will redefine the DB, using the code set up in the other app's module to do this in a consistent way (along the lines you suggested).

What I don't like about this is that it means the DB-setup code is run multiple times on a "normal" request.  That is, if I have code to set up the DB in my model file, and I visit my db-accessing controller, it runs the code in the model file to set up the DB, then runs the code again when it runs the controller file.  I imagine this is not a huge penalty but it is annoying.

I tried the suggestion from that old thread to use gluon.shell.env, but that did not work for me.  gluon.shell.env seems to have two main problems.  First, it stomps on the "current" object, causing the newly-created environment to have a new request/response/etc. and losing the old ones from the original request.  Second, the new request apparently isn't considered as an HTTPS request, which caused problems for me because my models file has request.requires_https().

I do have a couple questions about this: First, am I right that the performance impact of re-running the DAL object creation is negligible?  Is it something I should be concerned about?  Second, is there anything wrong with using request.requires_https() in the top-level code of a model file?  This part of the codebase was done by someone else, so if that is a misuse of web2py I could potentially try to get it changed.  I think the intent was to ensure that every single request must be HTTPS.  The problem is that I don't want these internal cross-app calls to be considered "requests"; I just want them to be calls to controller functions, so anything like requires_https should be ignored (or just proxied to the "real" request which is making this sub-call to another app).

The upshot of all this appears to be that there is no real way to get web2py to give me access to the model/controller combination without an HTTP request.  In web2py the models and controllers are tightly coupled to the HTTP request.  I find this somewhat irritating, as to my mind in an MVC framework the models and controllers shouldn't be so closely tied to the transport mechanism.  That is, I should be able to say "use this model and this controller and give me the data", without having to involve HTTP at all.  It seems that, in web2py this coupling is encouraged because controller functions directly access objects like request and response to get their arguments, rather than having a separate "routing" mechanism that maps HTTP request parameters to Python functions.  (This is what the "Service" function does, but it requires a level of indirection with a "call" entry point.)

In essence, the way I think of it, when a request comes in, it is routed to a controller function.  That controller then runs along with its associated models. and the controller function is called.  It returns some data, which is passed to a view.  What I would like is the ability to do just the middle part of that: take a controller, load its models, and run a function.  No HTTP.  No request.  No response.  No network.  No view.  No nothing.  Just the controller and the model.  For now I have a way to do it, but it involves duplicating the model-controller linkage that web2py already does (by "manually" recreating the DAL object from within the controller).

[Anthony]

In your example above, it's one model defining tables from another model.  But in my case, what I have is a controller function that needs access to the db, which is defined in the model file.  But I found the hard way (and later also from an old thread: https://groups.google.com/forum/#!topic/web2py-developers/5TPI9oYOZHM ) that exec_environment does not load models.  So this puts me kind of back to where I was.

The simplest solution I found was to put, in the controller file, code that looks like this:

from applications.otherapp.modules.db_definer
db = db_definer.define_db()

Then every time my controller is run, it will redefine the DB, using the code set up in the other app's module to do this in a consistent way (along the lines you suggested).

What I don't like about this is that it means the DB-setup code is run multiple times on a "normal" request.  That is, if I have code to set up the DB in my model file, and I visit my db-accessing controller, it runs the code in the model file to set up the DB, then runs the code again when it runs the controller file.  I imagine this is not a huge penalty but it is annoying.

So, then don't do it that way. You have two options. First, if you have a controller that doesn't need any of the model definitions from your application's model files, then you should probably re-architect your application so only the models that are needed get defined when they are needed (either using the conditional models functionality or by defining all models in modules that are imported where needed).

The other option is simply to avoid re-running the DB setup code and instead use the already defined db object, passing it to the module code from the other app:

app1/models/db.py:

db = DAL(...)

app2/modules/db_definer.py:

def define_tables(db):
    db.define_table('my_app2_table', ...)

app1/controllers/default.py:

from app2.modules.db_definer import define_tables

def myfunc():
    define_tables(db)
    ...

The above assumes both apps are using the same database, but if they are not, then there is no "re-running the DB setup code" anyway, as you would be dealing with two different databases, which would each need their own setup code.
 

I do have a couple questions about this: First, am I right that the performance impact of re-running the DAL object creation is negligible?  Is it something I should be concerned about?

The impact should be minimal, but see above for alternatives.
 

  Second, is there anything wrong with using request.requires_https() in the top-level code of a model file?

That's typically where it would be -- usually one of the first lines in the first model file to avoid unnecessary processing (as it results in a redirect). This is assuming you want the whole app over HTTPS.

The problem is that I don't want these internal cross-app calls to be considered "requests"; I just want them to be calls to controller functions, so anything like requires_https should be ignored (or just proxied to the "real" request which is making this sub-call to another app).

It should only take effect for HTTP requests, not executions via the shell, scheduler, or cron.
 

The upshot of all this appears to be that there is no real way to get web2py to give me access to the model/controller combination without an HTTP request.  In web2py the models and controllers are tightly coupled to the HTTP request.  I find this somewhat irritating, as to my mind in an MVC framework the models and controllers shouldn't be so closely tied to the transport mechanism.  That is, I should be able to say "use this model and this controller and give me the data", without having to involve HTTP at all.  It seems that, in web2py this coupling is encouraged because controller functions directly access objects like request and response to get their arguments, rather than having a separate "routing" mechanism that maps HTTP request parameters to Python functions.  (This is what the "Service" function does, but it requires a level of indirection with a "call" entry point.)

 

In essence, the way I think of it, when a request comes in, it is routed to a controller function.  That controller then runs along with its associated models. and the controller function is called.  It returns some data, which is passed to a view.  What I would like is the ability to do just the middle part of that: take a controller, load its models, and run a function.  No HTTP.  No request.  No response.  No network.  No view.  No nothing.  Just the controller and the model.  For now I have a way to do it, but it involves duplicating the model-controller linkage that web2py already does (by "manually" recreating the DAL object from within the controller).

If you need to share code across applications, I suggest you put that code into modules and then import in whatever applications need the functionality. Of course, you can also make internal HTTP requests from one application to another (or use an RPC mechanism).

Anthony

[Brendan Barnwell]

On Thursday, February 16, 2017 at 6:39:37 AM UTC-8, Anthony wrote:

In your example above, it's one model defining tables from another model.  But in my case, what I have is a controller function that needs access to the db, which is defined in the model file.  But I found the hard way (and later also from an old thread: https://groups.google.com/forum/#!topic/web2py-developers/5TPI9oYOZHM ) that exec_environment does not load models.  So this puts me kind of back to where I was.

The simplest solution I found was to put, in the controller file, code that looks like this:

from applications.otherapp.modules.db_definer
db = db_definer.define_db()

Then every time my controller is run, it will redefine the DB, using the code set up in the other app's module to do this in a consistent way (along the lines you suggested).

What I don't like about this is that it means the DB-setup code is run multiple times on a "normal" request.  That is, if I have code to set up the DB in my model file, and I visit my db-accessing controller, it runs the code in the model file to set up the DB, then runs the code again when it runs the controller file.  I imagine this is not a huge penalty but it is annoying.

So, then don't do it that way. You have two options. First, if you have a controller that doesn't need any of the model definitions from your application's model files, then you should probably re-architect your application so only the models that are needed get defined when they are needed (either using the conditional models functionality or by defining all models in modules that are imported where needed).

Thanks for your quick reply.

Not sure what you mean there about the models.  What you describe is the opposite of my situation.  My controller does need the models.  That is exactly the problem, because I want to call that controller from another controller, but to do so I need the model files to be run.
 

The other option is simply to avoid re-running the DB setup code and instead use the already defined db object, passing it to the module code from the other app:

app1/models/db.py:

db = DAL(...)

app2/modules/db_definer.py:

def define_tables(db):
    db.define_table('my_app2_table', ...)

app1/controllers/default.py:

from app2.modules.db_definer import define_tables

def myfunc():
    define_tables(db)
    ...

The above assumes both apps are using the same database, but if they are not, then there is no "re-running the DB setup code" anyway, as you would be dealing with two different databases, which would each need their own setup code.
 

What already-defined DB object?  If I use exec_environment, model files aren't run, so there is no db object.  In fact both controllers are using the same models, but if I use exec_environment, the object I get doesn't know about any models.  I could put code to create the models in the controller function (or in the controller top-level), which seems to be what your example is doing), but then I am, again, re-running that model-creation code twice (once in the original controller and once in the exec-ed controller).

Incidentally, I tried to just inject the db object into the object I get from exec_environment, but even this is difficult.  The reason is that exec_environment executes the file in a dict namespace, but then what it returns is not that dict, but a Storage object that is a (shallow) copy of it.  So if I do "c = exec_environment('some/controller.py')", the object c is not in fact the global namespace of the execed file, but a copy of it.  Normally I would be able to do "c.db = db" to inject db as a global variable, but instead I had to do tricky things like "c.some_function.func_globals['db'] = db".

Anyway, your response makes me think maybe I haven't explained my situation correctly.  What I have this: controllerA.py and controllerB both needs the model in modelA.py  controllerB wants to call a function in controllerA.  What I want is a way, from within controller B, to say "run controllerA, making sure all the models it needs are available to it, and then give me controllerA as a module object (or something equivalent), so that I can call controllerA.some_function() and get the results".  All without any additional network requests.  (In fact, some_function is a service, so I will pass arguments to it.).

The problem is that I don't want these internal cross-app calls to be considered "requests"; I just want them to be calls to controller functions, so anything like requires_https should be ignored (or just proxied to the "real" request which is making this sub-call to another app).

It should only take effect for HTTP requests, not executions via the shell, scheduler, or cron.
 

As I mentioned, the problem is that exec_environment does not run models, and gluon.shell.env creates a new "request" object which apparently does not preserve the HTTPS-ness of the original request.  (And it also stomps on the "current" object.)  So what I want is neither an HTTP request, nor an execution via the shell, scheduler or cron, but a direct execution of an "extra" controller, in a way that ensures the controller has access to all the models that it needs.
 

The upshot of all this appears to be that there is no real way to get web2py to give me access to the model/controller combination without an HTTP request.  In web2py the models and controllers are tightly coupled to the HTTP request.  I find this somewhat irritating, as to my mind in an MVC framework the models and controllers shouldn't be so closely tied to the transport mechanism.  That is, I should be able to say "use this model and this controller and give me the data", without having to involve HTTP at all.  It seems that, in web2py this coupling is encouraged because controller functions directly access objects like request and response to get their arguments, rather than having a separate "routing" mechanism that maps HTTP request parameters to Python functions.  (This is what the "Service" function does, but it requires a level of indirection with a "call" entry point.)

 

In essence, the way I think of it, when a request comes in, it is routed to a controller function.  That controller then runs along with its associated models. and the controller function is called.  It returns some data, which is passed to a view.  What I would like is the ability to do just the middle part of that: take a controller, load its models, and run a function.  No HTTP.  No request.  No response.  No network.  No view.  No nothing.  Just the controller and the model.  For now I have a way to do it, but it involves duplicating the model-controller linkage that web2py already does (by "manually" recreating the DAL object from within the controller).

If you need to share code across applications, I suggest you put that code into modules and then import in whatever applications need the functionality. Of course, you can also make internal HTTP requests from one application to another (or use an RPC mechanism).

How would such code access the DB?  By putting the define_tables-type code at the module top-level?  Or by putting it in a function and having every db-accessing function call "db = get_DB()" at the beginning?  That seems similar to what I'm doing, except that the code is in a controller instead of a module.  It seems the bottom line is that there is no way for a controller to internally specify a "dependency" on particular models.  The normal web2py mechanism will run the right model files on a regular HTTP request (based on the naming conventions), but if you want to call the controller file some other way, it will have to run the model-creation code itself, even if that code has already been run in another controller.  There isn't a way to say "if such-and-such tables have already been created, give me the existing db object; otherwise, create them now".  Is that right?

Thanks for continuing to help me suss this out.  I recognize that what I'm doing is somewhat going against the grain of web2py, but I'd still like to find the smoothest way to get the inter-controller cooperation I'm looking fir.
 

[Anthony]

What I don't like about this is that it means the DB-setup code is run multiple times on a "normal" request.  That is, if I have code to set up the DB in my model file, and I visit my db-accessing controller, it runs the code in the model file to set up the DB, then runs the code again when it runs the controller file.  I imagine this is not a huge penalty but it is annoying.

So, then don't do it that way. You have two options. First, if you have a controller that doesn't need any of the model definitions from your application's model files, then you should probably re-architect your application so only the models that are needed get defined when they are needed (either using the conditional models functionality or by defining all models in modules that are imported where needed).

Not sure what you mean there about the models.  What you describe is the opposite of my situation.  My controller does need the models.  That is exactly the problem, because I want to call that controller from another controller, but to do so I need the model files to be run.

You said you were running the db setup code twice. Since the second run overwrites the first, by definition you must not need the first, so just avoid running it the first time.
 

The other option is simply to avoid re-running the DB setup code and instead use the already defined db object, passing it to the module code from the other app:

app1/models/db.py:

db = DAL(...)

app2/modules/db_definer.py:

def define_tables(db):
    db.define_table('my_app2_table', ...)

app1/controllers/default.py:

from app2.modules.db_definer import define_tables

def myfunc():
    define_tables(db)
    ...

The above assumes both apps are using the same database, but if they are not, then there is no "re-running the DB setup code" anyway, as you would be dealing with two different databases, which would each need their own setup code.
 

What already-defined DB object?

You weren't quite clear about what you were doing. You showed the following code:

from applications.otherapp.modules.db_definer
db = db_definer.define_db()

and complained that the code to create the "db" object was being run twice (presumably once in this app's model files, and once in the db_definer module from otherapp). My suggestion is that if both "db" objects are for the same database, then just create the "db" object once (i.e., in this app's model files) and pass it to any module code that needs to define specific tables (i.e., don't create the "db" object in the module).

 

  If I use exec_environment, model files aren't run, so there is no db object.

My assumption is we are not using exec_environment here. I don't think that will be a fruitful approach.
 

Anyway, your response makes me think maybe I haven't explained my situation correctly.  What I have this: controllerA.py and controllerB both needs the model in modelA.py  controllerB wants to call a function in controllerA.  What I want is a way, from within controller B, to say "run controllerA, making sure all the models it needs are available to it, and then give me controllerA as a module object (or something equivalent), so that I can call controllerA.some_function() and get the results".  All without any additional network requests.  (In fact, some_function is a service, so I will pass arguments to it.).

 

If you need to share code across applications, I suggest you put that code into modules and then import in whatever applications need the functionality. Of course, you can also make internal HTTP requests from one application to another (or use an RPC mechanism).

How would such code access the DB?  By putting the define_tables-type code at the module top-level?  Or by putting it in a function and having every db-accessing function call "db = get_DB()" at the beginning?

Those are not the only options. First, note that you can separate the creation of the DAL object and the defining of its tables. You could have one function in a module define the DAL object for a given database and call that function from the model file of any given app. You can then have different functions for defining various models, and those functions can be called from anywhere -- a model, the top level of a controller, within a controller function, or even another module.
 

  That seems similar to what I'm doing, except that the code is in a controller instead of a module.  It seems the bottom line is that there is no way for a controller to internally specify a "dependency" on particular models.

If you create functions in modules to define the models, you can specify the dependency by importing the module from the controller.
 

  The normal web2py mechanism will run the right model files on a regular HTTP request (based on the naming conventions), but if you want to call the controller file some other way, it will have to run the model-creation code itself, even if that code has already been run in another controller.  There isn't a way to say "if such-and-such tables have already been created, give me the existing db object; otherwise, create them now".  Is that right?

The simplest approach is to put shared functionality into modules. Of course, if you don't know in what context a given bit of code will run, you can easily check whether particular tables have been defined and then call the table defining code if necessary.

Maybe it would help to show some actual code from two apps that you would like to share.

Anthony

[Brendan Barnwell]

The simplest approach is to put shared functionality into modules. Of course, if you don't know in what context a given bit of code will run, you can easily check whether particular tables have been defined and then call the table defining code if necessary.

Maybe it would help to show some actual code from two apps that you would like to share.

Yeah, you're right, it is getting a bit abstract.  I made a simple example.  This is with only one app (with two controllers) but it seems to illustrate the same kind of problem I'm having with multiple apps, so maybe we can start with this.  I created a sample app called test_env.  Everything in db.py is left as the default, except I uncommented request.requires_https().  I then made two controller files:

# db_worker.py
def call():
    return service()
   
@service.json
def work_service(a, b=2):
    return a, b, str(db)
   
def work_plain():
    return str(db)

# boss.py
import gluon.shell

def order_service():
    othermod = gluon.shell.exec_environment('applications/test_env/controllers/db_worker.py')
    return "Hello", othermod.work(a=1, b=2)
   
def order_plain():
    othermod = gluon.shell.exec_environment('applications/test_env/controllers/db_worker.py')
    return "Hello", othermod.work_plain()

I just used "print the db object" as a dummy "do something with the db" usage.

If I visit /test_env/db_worker/call/json/work_service?a=1&b=2, I get the right result.  What I want is to be able to visit /test_env/boss/order_service and get the exact same result.  Right now, I get "name 'service' is not defined" (because db.py is not run, and "service" is defined there).  Likewise, work_plain works, but order_plain gives the same error.  If I comment out work_service (to bypass the "name service not found" error) and try order_plain, I now get "name 'db' is not defined" (because "db" is also defined in db.py, which is still not run).

Now, as near as I can tell, everything I've done above in db_worker.py is straight out of the book as basic web2py stuff.  But what you seem to be saying is that, if I want to make "order" work, I have to significantly change the way I've written db.py and worker.py.  Namely, I have to abandon the idea of the normal execution framework whereby model files are automatically executed and dump their contents into the namespace of controllers, and instead I have to factor out the db-creation code into a separate module, and then, instead of having it automatically executed and provided to the db_worker controller, I have to manually import and run that db-creation code from within db_worker.py.  Is that correct?

If so, I understand what you're saying, but I guess it seems a bit perverse.  Web2py has all this nice magic for automatically deciding which model files to run, running them, and automatically providing the names they define to the controller files.  It's great and makes things very convenient and reduces boilerplate.  But that all has to go out the window for one controller to access another (even within the same app)?  Is there no way to say "do everything just as you would if a request came in for it -- that is, run the model files and the controller file -- but just don't actually create a request or call any of the controller functions yet, just return me an object representing the controller file"?

[Anthony]

If I visit /test_env/db_worker/call/json/work_service?a=1&b=2, I get the right result.  What I want is to be able to visit /test_env/boss/order_service and get the exact same result.  Right now, I get "name 'service' is not defined" (because db.py is not run, and "service" is defined there).

Because work_service is defined as a JSON-RPC service, the proper way to call it would be via an RPC call, as explained here: http://web2py.com/books/default/chapter/29/10/services#Accessing-JSONRPC-services-from-web2py. Otherwise, there is not much point in making it an RPC service.
 

Now, as near as I can tell, everything I've done above in db_worker.py is straight out of the book as basic web2py stuff.  But what you seem to be saying is that, if I want to make "order" work, I have to significantly change the way I've written db.py and worker.py.  Namely, I have to abandon the idea of the normal execution framework whereby model files are automatically executed and dump their contents into the namespace of controllers, and instead I have to factor out the db-creation code into a separate module, and then, instead of having it automatically executed and provided to the db_worker controller, I have to manually import and run that db-creation code from within db_worker.py.  Is that correct?

You make it sound more complicated than it is. If you need to share both a database and model definitions across applications, you can simply take the code you would otherwise have put in a model file and instead put it in a function in a module. Then in each app that needs it, import the function and run it. This involves maybe a couple extra lines of code in each app.

Note, web2py already employs this approach to provide some of its functionality. Consider Auth and Scheduler -- both of those classes take a db object and define a number of models. Yet they are defined in modules and must be imported and instantiated in any given application that wants to use them. This is the standard mechanism for sharing code in Python.
 

If so, I understand what you're saying, but I guess it seems a bit perverse.  Web2py has all this nice magic for automatically deciding which model files to run, running them, and automatically providing the names they define to the controller files.  It's great and makes things very convenient and reduces boilerplate.  But that all has to go out the window for one controller to access another (even within the same app)?

The workflow of an HTTP request involves things like setting up the session, and some of the code (particularly controller code) may depend on attributes of the request object. If you want to make what amounts to an internal request to another app or controller, it will not be clear what elements of the current request to retain (e.g., should the current session, request headers, query string, etc. be replicated in the second call?). Note, you can achieve this to some extent by calling gluon.shell.env and passing in some or all of the current request attributes via the extra_request parameter, but things can get messy very quickly. Should everything happen in the same database transaction? What if there is a redirect or HTTP exception in the second app? You will generally be better off isolating full HTTP requests to a single app.
 

Is there no way to say "do everything just as you would if a request came in for it -- that is, run the model files and the controller file -- but just don't actually create a request or call any of the controller functions yet, just return me an object representing the controller file"?

I wonder how often you really need to do something like this. If there is some common set of functionality that needs to be shared across apps, it probably does not have to be in the form of a controller function, but could instead be a more general function that can be used as part of a controller. Again, consider how the Auth functionality works -- it does not provide a set of ready-made controller functions but instead provides a mechanism to (a) define a common set of database tables and (b) expose methods that can be called from within controllers where needed. Do you have a use case that cannot easily be accommodated by that pattern? If so, what do you imagine would be the API for doing what you want to do, and would it look much different from simply making an HTTP call (or possibly an RPC call) to the other app (which you can already do)?

Anthony

[Anthony]

Do you have a use case that cannot easily be accommodated by that pattern? If so, what do you imagine would be the API for doing what you want to do, and would it look much different from simply making an HTTP call (or possibly an RPC call) to the other app (which you can already do)?

And just to be clear, if you are OK with this:

def order_plain():
    othermod = gluon.shell.exec_environment('applications/test_env/controllers/db_worker.py')
    return "Hello", othermod.work_plain()

then why not this:

def order_plain():
    work_plain_response = gluon.tools.fetch(URL('db_worker', 'work_plain', host='localhost'))
    return "Hello %s" % work_plain_response

Anthony

[Anthony]

It may also be worth noting that you can get the output of another controller function within the same application via LOAD(..., ajax=False) (note, ajax=False is actually the default). This, however, will also execute the called function's view and return the full rendered output. In this case, the execution environment of the current request is re-used (i.e., it does not re-execute the models but simply runs the requested function in the current environment), which is why this does not work across applications. So, taking your example, you could do:

def order_plain():
    work_plain_response = LOAD('db_worker', 'work_plain')

    return "Hello %s" % work_plain_response

Anthony

[Brendan Barnwell]

If I understand right, the second one generates an HTTP request.  That means the data that is fetched will be serialized and deserialized.  In some cases this data is sizable, so I want to avoid that overhead.  There will likely also be some overhead associated with actually sending/receiving the network request; this is probably smaller but I'd still rather avoid it.  As I understand it, these operations will also happen with an RPC call.  That's why I keep saying that I don't want an HTTP request.  I want everything to happen totally internally, within the same request, same process, same everything, nothing going over the wire.

In your other message you said:

The workflow of an HTTP request involves things like setting up the session, and some of the code (particularly controller code) may depend on attributes of the request object. If you want to make what amounts to an internal request to another app or controller, it will not be clear what elements of the current request to retain (e.g., should the current session, request headers, query string, etc. be replicated in the second call?).

Yeah, that is true.  To my mind, though, the point of using something like Service is to constrain the function's interface so that the situation you describe won't arise.  Instead, the function is supposed to depend only on the arguments you pass it.  However, I can see that it is difficult for web2py to support this use case, because there is no way to enforce the restriction that nothing in the code path (i.e., nothing in the model or controller code) directly accesses the request object.  In some sense, even my function does not really just depend on its arguments, because it still depends on the global "db" object.  What I was hoping is that there is a way to get stuff like "db" (i.e., global objects I created in my model code) without the need to have objects like "request" (i.e., global objects built into web2py).  But I see your point: the normal web2py workflow places no constraints on how code may access request/response/session/etc.  Since these objects are "allowed" to be directly accessed (and even mutated) from anywhere in user code, trying to call code without them would probably fail more often than not.

(I mostly included work_plain there just to be able to verify that the db object wasn't being created.  My real use case is more like work_service.)

Because work_service is defined as a JSON-RPC service, the proper way to call it would be via an RPC call, as explained here: http://web2py.com/books/default/chapter/29/10/services#Accessing-JSONRPC-services-from-web2py. Otherwise, there is not much point in making it an RPC service.

 
It's not an RPC service per se.  The decorator is @service.json, not @service.jsonrpc.  The goal of using Service here is what I mentioned above: to allow the function's API to be its argument signature, rather than allowing it to grab arbitrary stuff from the request object.  The idea is that sometimes I do want to actually visit the URL in a browser (or retrieve it programmatically), and in that case of course a real request must be generated; but just in case the "request" is actually an internal call, I would rather have a "shortcut" that doesn't do anything that has to do with going over the wire.

That is also why it's not purely a matter of "common functionality that is shared across apps".  Rather, there is functionality that I want to be accessible BOTH as a "real" URL endpoint AND internally between apps --- but without the request overhead in the latter case.  So I do want a controller that exposes the function.  (I could of course write a "dummy" controller function that just calls a "worker" function in a module, but that is essentially what Service already gives me.)

Anyway, thanks for your help on this matter, I'll look at modularizing my db-creation code a bit so that I can more easily create the db in each module that needs it.

[Anthony]

If I understand right, the second one generates an HTTP request.  That means the data that is fetched will be serialized and deserialized.  In some cases this data is sizable, so I want to avoid that overhead.  There will likely also be some overhead associated with actually sending/receiving the network request; this is probably smaller but I'd still rather avoid it.  As I understand it, these operations will also happen with an RPC call.  That's why I keep saying that I don't want an HTTP request.  I want everything to happen totally internally, within the same request, same process, same everything, nothing going over the wire.

Correct, there would be some overhead due to the HTTP request.
 

Because work_service is defined as a JSON-RPC service, the proper way to call it would be via an RPC call, as explained here: http://web2py.com/books/default/chapter/29/10/services#Accessing-JSONRPC-services-from-web2py. Otherwise, there is not much point in making it an RPC service.

 
It's not an RPC service per se.  The decorator is @service.json, not @service.jsonrpc.

Oh, right, sorry about that. The same reasoning applies, though -- you would simply make an HTTP request rather than use an RPC client.
 

That is also why it's not purely a matter of "common functionality that is shared across apps".  Rather, there is functionality that I want to be accessible BOTH as a "real" URL endpoint AND internally between apps --- but without the request overhead in the latter case.

I suppose it would help to understand your real world use case. It's not clear what the problem is with using modules. It might also help to know how you would handle this in other frameworks.
 

  So I do want a controller that exposes the function.  (I could of course write a "dummy" controller function that just calls a "worker" function in a module, but that is essentially what Service already gives me.)

Not exactly. The functions exposed by Service are not accessible as HTTP request endpoints on their own.

Note, if you have a lot of functions in a module that all need to be exposed as controller endpoints, you could route to them via request.args. For example, in /modules/mycontroller.py:

from gluon import current

def myfunc():
    return dict(message=current.request.vars.message)

And in /controllers/default.py:

import mycontroller

def index():
    return getattr(mycontroller, request.args(0))()

Then access /myapp/default/index/myfunc to get the output of myfunc (you can use the router to hide "default" and "index", so the URL can be re-written to /myapp/myfunc). You can put as many functions as you want in mycontroller.py, and they can be accessed from any app.

Here is an example of this approach: https://github.com/TechEmpower/FrameworkBenchmarks/tree/2623eae7135205c8ca4da27d44ea1ccda6c19a56/frameworks/Python/web2py/app/standard.

Anthony