Arbitrary args/kwargs with service?

[Brendan Barnwell]

Hi, I've been experimenting the web2py services functionality.  Very neat!  But what I'm wondering is whether it is possible to expose a service that accepts arbitrary URL parameters and receives them as *args and/or **kwargs.  That is, I'd like to have code like this:

@service.json
def myfinc(this, that, *args, **kwargs):
    # ...

And if I visit myapp/default/call/json/this/that/something/else?key=foo&otherkey=bar , then I would get ("something", "else") in args, and {"key": "foo", "otherkey": "bar"} in kwargs.  Is such a thing possible.  Looking at the code in gluon.tools.py (specifically the "universal_caller" function), I'm guessing no, because it doesn't seem to just silently swallow any extra arguments that don't match with the function signature, and it doesn't check to see whether the function accepts varargs.  Has there been any discussion about adding such a feature?

[Brendan Barnwell]

Just bumping this up to see if anyone has any ideas, as I have again run up against this limitation.

The situation I'm in is that I have a number of controller functions which all accept some arguments in common (for instance, an API key that has to be checked to allow access to the resource).  My natural inclination was to handle that with a decorator that strips off the arguments in question and passes the rest along, something like

def check_secret(func):
    @functools.wraps(func)
    def wrapper(secret, *args, **kwargs):
        secret = request.vars.pop('secret')
        if not valid_secret(secret):
            error_response()
        else:
             return func(*args, **kwargs)
    return wrapper

@service.json
@check_secret
def some_service(real_arg_1, real_arg2):
    # whatever

However, this doesn't work.  The decorator returns a wrapped function that accepts only one "real" argument.  The service handler reads the argspec and only passes the "secret" argument.

I could of course write my own system for doing this, but I'd have to reimplement all of the function-registering that "service" does.  It would be nice if I could just get "service" to pass ALL of the arguments it receives, not just the ones named in the function argspec.  I think this could actually be done by making service use inspect.getargspec rather than manually reading co_argcount and so on, which seems much more fragile.

What is the right way to handle this situation, where I want to "factor out" handling of arguments that are common in different controller functions?

[Leonel Câmara]

I'm using @request.restful for this instead of @service.json. Then just return json using response.json.

[Dave S]

On Sunday, December 4, 2016 at 3:24:35 AM UTC-8, Leonel Câmara wrote:

I'm using @request.restful for this instead of @service.json. Then just return json using response.json.

@request.restful is dependent on your field definitions, no?  It's not clear that that is what Brendan is after; he might not be thinking of his data as a tree.  Perhaps the smart_query feature might help (next section in Chapter 10).

/dps

[Anthony]

On Monday, December 5, 2016 at 2:20:59 PM UTC-5, Dave S wrote:

On Sunday, December 4, 2016 at 3:24:35 AM UTC-8, Leonel Câmara wrote:

I'm using @request.restful for this instead of @service.json. Then just return json using response.json.

@request.restful is dependent on your field definitions, no?

No, the @request.restful decorator simply maps the incoming request type to the matching nested function and passes *request.args and **request.vars to that function. It has nothing to do with the DAL. The parse_as_rest and smart_query methods are part of the DAL and can be used in conjunction with @request.restful, though they are not required.

Anthony

[Dave S]

Okay, time for me to re-read.

/dps

' 

[Brendan Barnwell]

It looks like I could use request.restful for that, but to be honest I find that mechanism somewhat awkward.  Creating functions as local variables and then returning locals() just seems gross to me.  :-)

But from the other side of things, I guess the question is, if we can do that, is there ever any reason to use the "service" mechanism at all?  Or can request.restful do everything service can do, and more?

On Sunday, December 4, 2016 at 3:24:35 AM UTC-8, Leonel Câmara wrote:

[Anthony]

On Thursday, January 5, 2017 at 12:39:13 AM UTC-5, Brendan Barnwell wrote:

It looks like I could use request.restful for that, but to be honest I find that mechanism somewhat awkward.  Creating functions as local variables and then returning locals() just seems gross to me.  :-)

Not sure why that is "gross," but you don't have to code it like that -- the decorated function simply has to return a dictionary whose keys are one or more of the HTTP verbs (GET, POST, etc.) and whose keys are the functions that should be called in response to those verbs. You can create that dictionary however you like and define the functions wherever you like.
 

But from the other side of things, I guess the question is, if we can do that, is there ever any reason to use the "service" mechanism at all?  Or can request.restful do everything service can do, and more?

I suppose their use cases are overlapping. If @request.restful works for you in this case, then use it. The @service mechanism lets you easily expose functions for use in remote procedure calls, including via standardized RPC protocols. The @service decorators also automatically return the correct response format, whereas @request.restful requires the use of the generic views (disabled by default) to automatically generate different response formats (though it will automatically generate JSON if the request specifies "application/json").

Anthony

[Brendan Barnwell]

On Thursday, January 5, 2017 at 7:04:56 AM UTC-8, Anthony wrote:

But from the other side of things, I guess the question is, if we can do that, is there ever any reason to use the "service" mechanism at all?  Or can request.restful do everything service can do, and more?

I suppose their use cases are overlapping. If @request.restful works for you in this case, then use it. The @service mechanism lets you easily expose functions for use in remote procedure calls, including via standardized RPC protocols. The @service decorators also automatically return the correct response format, whereas @request.restful requires the use of the generic views (disabled by default) to automatically generate different response formats (though it will automatically generate JSON if the request specifies "application/json").

Anthony

I guess the request.restful approach works better for me at this point.  Is there a way to tell web2py to enable generic views only for restful controller functions, and disable them otherwise?

In continuing to explore both approaches, I gained a better understanding of my problem with the service approach, and I want to share it here because I think it is worth thinking about in terms of the design of web2py.  The problem I have with the service approach is that it seems to make impossible to use a decorator to wrap a service controller function in order to add or remove arguments, because the service function will "block" web2py from seeing what arguments the real service function accepts, causing web2py to pass the wrong arguments.

There is a certain Python idiom that goes something like this:

def accept_extra_arg(func):
    def wrapper(extra_arg, *args, **kwargs):
        # maybe do something with extra_arg here
        result = func(*args, **kwargs)
        # maybe do something with extra_arg here
        return result
    return wrapper

@accept_extra_arg
    def foo(x, y):
        return x+y

The idea is that the accept_extra_arg decorator allows you to wrap your function with a handler that accepts an extra argument which the function does not see, but which is used to preprocess the function's arguments and/or postprocess its result.  In the case of web APIs, for instance, this is a natural way to do something like write a series of API functions that require an API key to be used: you can have a @requires_key decorator that wraps functions, allowing the function itself to be "pure" and just concentrate on returning the data, without having to handle the checking of the API key.  There is a related idiom which is the reverse, involving writing a decorator that accepts an argument and returns a decorated function accepting *fewer* arguments that the original, allowing a behavior akin to functools.partial, by which some arguments are specified in advance via the decorator, rather than being passed on each call to the decorated function.

The web2py service mechanism breaks this idiom.  The problem is that because the wrapped function is supposed to be agnostic as to the arguments of the function it wraps, it just accepts *args and **kwargs (in addition to its extra argument).  This means that web2py won't pass it anything.  This kind of breaks the RPC idea that calling the function internally should be the same as calling it via the network API: if my function wants to use *args and/or **kwargs, web2py apparently doesn't provide a way for me to use it in a seamless manner for internal vs external calls.

I *think* I have been able to get around this with request.restful, because it passes everything along without trying to match the arguments.  But it is somewhat annoying for the case of read-only APIs because of the extra layer of indirection required (returning a function that returns the data, rather than just returning the data).

What I really want is the ability to write an arbitrary function, accepting arbitrary arguments.  I want the ability to wrap that function with decorators that absorb or add arguments as I please.  And then I want to be able to expose that arbitrarily wrapped function as a service endpoint (presumably by wrapping the function with a web2py decorator as the last one).  I don't want any of the decorators nor the original function ever to have to worry about *how* the arguments are being passed (e.g., positionally or by keyword), and I don't want any function in the chain to ever have to worry about the number or names of any arguments other than the ones it explicitly processes from the arguments passed to it.  So, basically, I want to be able to use decorators the same way I would use them in ordinary Python code, and not have web2py step in at some point and disrupt the process by which decorated functions pass their arguments to the functions they wrap.

Now, that's not to say that I don't appreciate the tools web2py does offer.  I've found them quite powerful.  And for now I think I can make things work with the restful decorator.  But just mention above to maybe be considered for future web2py development.  The ability to manipulate function signatures with decorators in a way that remains transparent from the perspective of the wrapped functions is a really nice feature of Python, and the ability to use varargs is powerful too, and I think it would be good for web2py to allow such functions to be exposed without sacrificing those features of Python.

[Anthony]

Feel free to open a Github issue, or better yet, send a pull request.

Depending on your use case, another option is to instead decorate the call() action in order to pass in extra args (you can strip them out of request.args and request.vars before the service method gets called).

Anthony

[Brendan Barnwell]

On Thursday, January 12, 2017 at 1:17:30 AM UTC-8, Anthony wrote:

Feel free to open a Github issue, or better yet, send a pull request.

I created an issue here: https://github.com/web2py/web2py/issues/1561

I can probably do a pull request, but wanted to see if anyone knows of a reason why the existing service functionality restricts the passed arguments, instead of just passing them all to the service function, as request.restful does.

[Anthony]

I created an issue here: https://github.com/web2py/web2py/issues/1561

I can probably do a pull request, but wanted to see if anyone knows of a reason why the existing service functionality restricts the passed arguments, instead of just passing them all to the service function, as request.restful does.

I'm not aware of any reason for the current restriction.

This brings to mind another possible workaround -- you can monkey patch universal_caller in gluon.tools.

Anthony 

[Brendan Barnwell]

Yeah, that is a possibility.  That is obviously more risky though as it would globally affect all code in all apps.

I made a pull request (https://github.com/web2py/web2py/pull/1566).  My modification adds a new FlexibleService class that just passes all arguments to the service function; this way it doesn't interfere with anything the existing Service does.  It also uses a method instead of a top-level function for the equivalent of universal_caller, so that subclasses can override it to add more complex argument-matching if they want.

[Anthony]

I made a pull request (https://github.com/web2py/web2py/pull/1566).  My modification adds a new FlexibleService class that just passes all arguments to the service function; this way it doesn't interfere with anything the existing Service does.  It also uses a method instead of a top-level function for the equivalent of universal_caller, so that subclasses can override it to add more complex argument-matching if they want.

Thanks. I made a suggestion to simplify by just altering the current Service class to take a new argument adding the new desired behavior as an option. This would keep the code DRY and the API less cluttered. I would lean toward just replacing the old behavior with the new (or at least making the new behavior the default), though, unless there is a good reason to retain the old.

Anthony