Meta decorator with parameters, defined in explicit functions

[Ben Finney]

Howdy all,

I want an explicit replacement for a common decorator idiom.

There is a clever one-line decorator that has been copy-pasted without
explanation in many code bases for many years::

decorator_with_args = lambda decorator: lambda *args, **kwargs: lambda func: decorator(func, *args, **kwargs)

My problem with this is precisely that it is clever: it explains nothing
about what it does, has many moving parts that are not named, it is
non-obvious and lacks expressiveness.

Even the widely-cited ActiveState recipe by Peter Hunt from 2005
<URL:http://code.activestate.com/recipes/465427-simple-decorator-with-arguments/>
gives no clue as to what this is doing internally nor what the names of
its parts should be.

I would like to see a more Pythonic, more explicit and expressive
replacement with its component parts easily understood.

--
\ “[H]ow deep can a truth be — indeed, how true can it be — if it |
`\ is not built from facts?” —Kathryn Schulz, 2015-10-19 |
_o__) |
Ben Finney

[Zachary Ware]

On Tue, Jun 28, 2016 at 12:02 AM, Ben Finney <ben+p...@benfinney.id.au> wrote:
> Howdy all,
>
> I want an explicit replacement for a common decorator idiom.
>
> There is a clever one-line decorator that has been copy-pasted without
> explanation in many code bases for many years::
>
> decorator_with_args = lambda decorator: lambda *args, **kwargs: lambda func: decorator(func, *args, **kwargs)
>
> My problem with this is precisely that it is clever: it explains nothing
> about what it does, has many moving parts that are not named, it is
> non-obvious and lacks expressiveness.
>
> Even the widely-cited ActiveState recipe by Peter Hunt from 2005
> <URL:http://code.activestate.com/recipes/465427-simple-decorator-with-arguments/>
> gives no clue as to what this is doing internally nor what the names of
> its parts should be.
>
> I would like to see a more Pythonic, more explicit and expressive
> replacement with its component parts easily understood.

Try this on for size:

'''
import functools

def decorator_with_args(decorator):
"""
Meta-decorator for decorators that take arguments.

Usage:

@decorator_with_args
def some_decorator(func, foo, bar=None)
if foo:
func.bar = bar
return func

@some_decorator(True, bar='quux')
def some_decorated_function(some_arg)
return some_arg

assert some_decorated_function.bar == 'quux'

Returns a function that takes arguments which are to be
passed to the decorator along with the function to be
decorated.

This allows you to just write your decorator as taking
the arguments you want, without having to write a decorator
factory that creates and returns a decorator.
"""

@functools.wraps(decorator)
def factory(*args, **kwargs):
"""Generic decorator factory.

Returns a decorator that calls the original decorator
with the function to be decorated and all arguments
passed to this factory.
"""

def decorator_wrapper(func):
"""Thin wrapper around the real decorator."""
return decorator(func, *args, **kwargs)
return decorator_wrapper
return factory
'''

I make no guarantees about this; this is completely untested and is
based solely upon how the original translated in my mind. If I
misunderstood how the original works, this is worse than useless :).
Also, I'm not sure how close I got on the "easily understood" part.
It's certainly more explanatory than the original, but it's not the
simplest of concepts anyway: a function that takes a function and
returns a function that takes arbitrary arguments and returns a
function that takes a function and returns the result of calling the
originally passed function with arguments consisting of the passed
function and the aforementioned arbitrary arguments has too many
layers of indirection to keep in mind at once!

--
Zach

[Paul Rubin]

Ben Finney <ben+p...@benfinney.id.au> writes:
> decorator_with_args = lambda decorator: lambda *args, **kwargs:
> lambda func: decorator(func, *args, **kwargs)

> I would like to see a more Pythonic, more explicit and expressive
> replacement with its component parts easily understood.

How's this:

from functools import partial
def dwa(decorator):
def wrap(*args,**kwargs):
return partial(decorator, *args, **kwargs)
return wrap

[dieter]

Ben Finney <ben+p...@benfinney.id.au> writes:
> I want an explicit replacement for a common decorator idiom.
>
> There is a clever one-line decorator that has been copy-pasted without
> explanation in many code bases for many years::
>

> decorator_with_args = lambda decorator: lambda *args, **kwargs: lambda func: decorator(func, *args, **kwargs)
>

> My problem with this is precisely that it is clever: it explains nothing
> about what it does, has many moving parts that are not named, it is
> non-obvious and lacks expressiveness.

I have been able to understand the purpose of the definition above -
based solely on background knowledge about "decorator" and the definition.

A decorator is a function which takes a function as argument and
returns another function, the decorated function.

In its simplest form, it is usually is used like:

@decorator
def f....


However, there is a more complex use: a decorator with arguments.
It takes the form

@decorator(...args...)
def f...

In this use, the decorator is not "decorator" itself but
"decorator(...args...)". "decorator" itself is in fact a "meta" decorator
taking arguments and returning the real decorator.
The standard way to define such a meta decorator would be to have
a local function definition in its body and return that.


Locally defining functions and returning them looks a bit nasty in
Python. You might want to avoid it. That's what the
"decorator_with_args" above tries to facilitate.

It allows you to define a function "decorator"
with arguments "func, *args, **kw", decorate it with
"decorator_with_args" and use this as a decorator with arguments
("*args, **kw).

"decorator_with_args" essentially is a signature transform.
It transforms a function "func, *args, **kw --> ..." into
a function "*args, **kw --> func --> ..." *AND*
it does this in a completely natural and understandable way.

[Steven D'Aprano]

On Tuesday 28 June 2016 15:02, Ben Finney wrote:

> Howdy all,
>
> I want an explicit replacement for a common decorator idiom.
>
> There is a clever one-line decorator that has been copy-pasted without
> explanation in many code bases for many years::
>
> decorator_with_args = lambda decorator: lambda *args, **kwargs: lambda
> func: decorator(func, *args, **kwargs)

I've never seen it before, and I'll admit I really had to twist my brain to
understand it, but I came up with an example showing the traditional style of
decorator-factory versus this meta-decorator.

# Standard idiom.

def chatty(name, age, verbose=True): # the factory
def decorator(func): # the decorator returned by the factory
if verbose:
print("decorating function...")
@functools.wraps(func)
def inner(*args, **kwargs):
print("Hi, my name is %s and I am %d years old!" % (name, age))
return func(*args, **kwargs)
return inner
return decorator

@chatty("Bob", 99)
def calculate(x, y, z=1):
return x+y-z


# Meta-decorator variant.

decorator_with_args = (lambda decorator: lambda *args, **kwargs: lambda func:
decorator(func, *args, **kwargs))

@decorator_with_args
def chatty(func, name, age, verbose=True):
if verbose:
print("decorating function...")
@functools.wraps(func)
def inner(*args, **kwargs):
print("Hi, my name is %s and I am %d years old!" % (name, age))
return func(*args, **kwargs)
return inner

@chatty("Bob", 99)
def calculate(x, y, z=1):
return x+y-1



I agree -- it's very clever, and completely opaque in how it works. The easy
part is expanding the lambdas:

def decorator_with_args(decorator):
def inner(*args, **kwargs):
def innermost(func):
return decorator(func, *args, **kwargs)
return innermost
return inner


but I'm not closer to having good names for inner and innermost than you.


--
Steve

[Lawrence D’Oliveiro]

On Tuesday, June 28, 2016 at 5:03:08 PM UTC+12, Ben Finney wrote:
> There is a clever one-line decorator that has been copy-pasted without
> explanation in many code bases for many years::
>
> decorator_with_args = lambda decorator: lambda *args, **kwargs: lambda func: decorator(func, *args, **kwargs)
>
> My problem with this is precisely that it is clever: it explains nothing
> about what it does, has many moving parts that are not named, it is
> non-obvious and lacks expressiveness.

It is written in a somewhat roundabout way: why not just

decorator_with_args = lambda decorator, *args, **kwargs : lambda func : decorator(func, *args, **kwargs)

? Could it be this was not valid in earlier versions of Python?

Anyway, it’s a function of a function returning a function. Example use (off the top of my head):

my_decorator = decorator_with_args(actual_func_to_call)(... remaining args for actual_func_to_call)

(for the original version) or

my_decorator = decorator_with_args(actual_func_to_call,... remaining args for actual_func_to_call)

(for the simpler version). Then an example use would be

@my_decorator
def some_func...

which would call “actual_func_to_call” with some_func plus all those extra args, and assign the result back to some_func.

> I would like to see a more Pythonic, more explicit and expressive
> replacement with its component parts easily understood.

I don’t know why this fear and suspicion of lambdas is so widespread among Python users ... former Java/C# programmers, perhaps?

[Lawrence D’Oliveiro]

On Tuesday, June 28, 2016 at 5:03:08 PM UTC+12, Ben Finney wrote:

> decorator_with_args = lambda decorator: lambda *args, **kwargs: lambda func: decorator(func, *args, **kwargs)

Ah, I see why there are 3 lambdas, instead of 2. It’s so that you can write

decorator_func = decorator_with_args(actual_func)

then you can use it like this:

@decorator_func(...additional args for actual_func...)
def some_func...

to invoke actual_func(some_func, ... additional args for actual_func...) on some_func, and assign the result back to some_func.

[Steven D'Aprano]

On Thu, 30 Jun 2016 12:43 pm, Lawrence D’Oliveiro wrote:

> On Tuesday, June 28, 2016 at 5:03:08 PM UTC+12, Ben Finney wrote:
>> There is a clever one-line decorator that has been copy-pasted without
>> explanation in many code bases for many years::
>>
>> decorator_with_args = lambda decorator: lambda *args, **kwargs:
>> lambda func: decorator(func, *args, **kwargs)
>>
>> My problem with this is precisely that it is clever: it explains nothing
>> about what it does, has many moving parts that are not named, it is
>> non-obvious and lacks expressiveness.
>
> It is written in a somewhat roundabout way: why not just
>
> decorator_with_args = lambda decorator, *args, **kwargs : lambda func
> : decorator(func, *args, **kwargs)
>
> ? Could it be this was not valid in earlier versions of Python?

Your version has a much inferior API than the original. You can't wrap your
decorators ahead of time, you have to keep calling decorator_with_args each
time you want to use them. Contrast your version:


# LDO's version
import functools
decorator_with_args = (

lambda decorator, *args, **kwargs :
lambda func : decorator(func, *args, **kwargs)

)

def chatty(func, name, age, verbose=True):
if verbose:
print("decorating function...")
@functools.wraps(func)
def inner(*args, **kwargs):
print("Hi, my name is %s and I am %d years old!" % (name, age))
return func(*args, **kwargs)
return inner

@decorator_with_args(chatty, "Bob", 99)

def calculate(x, y, z=1):
return x+y-z

@decorator_with_args(chatty, "Sue", 35, False)
def spam(n):
return ' '.join(['spam']*n)



Here's the original. It's a much better API, as the decorator "chatty" only
needs to be wrapped once, rather than repeatedly. For a library, it means
that now you can expose "chatty" as a public function, and keep
decorator_with_args as an internal detail, instead of needing to make them
both public:


# original meta decorator version

import functools
decorator_with_args = (

lambda decorator:
lambda *args, **kwargs:
lambda func: decorator(func, *args, **kwargs)

)

@decorator_with_args
def chatty(func, name, age, verbose=True):
if verbose:
print("decorating function...")
@functools.wraps(func)
def inner(*args, **kwargs):
print("Hi, my name is %s and I am %d years old!" % (name, age))
return func(*args, **kwargs)
return inner

@chatty("Bob", 99)
def calculate(x, y, z=1):

return x+y-z

@chatty("Sue", 35, False)
def spam(n):
return ' '.join(['spam']*n)



Think of the use-case where you are the author of a library that provides
various decorators. `decorator_with_args` is an implementation detail of
your library: *you* say:

@decorator_with_args
def chatty(func, name, age, verbose=True): ...

@decorator_with_args
def logged(func, where_to_log): ...

@decorator_with_args
def memoise(func, cache_size): ...


and then offer chatty, logged and memoise as public functions to the users
of your library, who just write:

@memoise(1000)
def myfunc(arg): ...

etc. as needed. But with your version, you have to make decorator_with_args
a public part of the API, and require the user to write:

@decorator_with_args(memoise, 1000)
def myfunc(arg): ...


which I maintain is a much inferior API for the users of your library.

> I don’t know why this fear and suspicion of lambdas is so widespread among
> Python users ... former Java/C# programmers, perhaps?

Not so much fear as a little healthy respect for them, I think.




--
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[Peter Otten]

If there is "fear" I don't share it. However, for

foo = lambda <args>: <expr>

there is syntactic sugar in Python that allows you to write it as

def foo(<args>):
return <expr>

with the nice side effects that it improves the readability of tracebacks
and allows you to provide a docstring.

So yes, assigning a lambda to a name raises my "suspicion".

If a lambda is provided as an argument or as part of an expression I wonder
how it is tested, and if even if it is trivial like

def reduce(items, func=lambda x, y: x + y): ...

the alternative

def reduce(items, func=add): ...

looks more readable in my eyes even though somewhere -- under the rug or in
the operator module -- you need

def add(x, y):
"""
>>> add(3, 4)
7
"""
return x + y

>> decorator_with_args = lambda decorator: lambda *args, **kwargs:
lambda func: decorator(func, *args, **kwargs)
>

> Ah, I see why there are 3 lambdas, instead of 2. It’s so that you can
write

I have a suspicion that there would not have been a correction "I see why
there are three functions instead of two" ;)

[Lawrence D’Oliveiro]

On Thursday, June 30, 2016 at 7:26:01 PM UTC+12, Peter Otten wrote:
> foo = lambda <args>: <expr>
>
> there is syntactic sugar in Python that allows you to write it as
>
> def foo(<args>):
> return <expr>
>
> with the nice side effects that it improves the readability of tracebacks
> and allows you to provide a docstring.

True, but then again the original had three lambdas, so one line would have to become at least 3×2 = 6 lines, more if you want docstrings.

> def reduce(items, func=lambda x, y: x + y): ...

There was a reason why “reduce” was removed from being a builtin function in Python 2.x, to being banished to functools in Python 3.

> the alternative
>
> def reduce(items, func=add): ...
>

> looks more readable in my eyes even though somewhere ...

Just use “sum” in this case.

[Steven D'Aprano]

On Thursday 30 June 2016 17:43, Lawrence D’Oliveiro wrote:

> On Thursday, June 30, 2016 at 7:26:01 PM UTC+12, Peter Otten wrote:
>> foo = lambda <args>: <expr>
>>
>> there is syntactic sugar in Python that allows you to write it as
>>
>> def foo(<args>):
>> return <expr>
>>
>> with the nice side effects that it improves the readability of tracebacks
>> and allows you to provide a docstring.
>
> True, but then again the original had three lambdas, so one line would have
> to become at least 3×2 = 6 lines, more if you want docstrings.

I hear that we've passed "Peak Newlines" now, so adding extra lines will get
more and more expensive. I guess languages like C and Java will soon have to be
abandoned, and everyone will move to writing minified Javascript and Perl one-
liners.

>> def reduce(items, func=lambda x, y: x + y): ...
>
> There was a reason why “reduce” was removed from being a builtin function in
> Python 2.x, to being banished to functools in Python 3.

Yes, and that reason is that Guido personally doesn't like reduce.



--
Steve

[Peter Otten]

Lawrence D’Oliveiro wrote:

> On Thursday, June 30, 2016 at 7:26:01 PM UTC+12, Peter Otten wrote:
>> foo = lambda <args>: <expr>
>>
>> there is syntactic sugar in Python that allows you to write it as
>>
>> def foo(<args>):
>> return <expr>
>>
>> with the nice side effects that it improves the readability of tracebacks
>> and allows you to provide a docstring.
>
> True, but then again the original had three lambdas, so one line would
> have to become at least 3×2 = 6 lines, more if you want docstrings.
>
>> def reduce(items, func=lambda x, y: x + y): ...
>
> There was a reason why “reduce” was removed from being a builtin function
> in Python 2.x, to being banished to functools in Python 3.

You do understand what an example is?

>> the alternative
>>
>> def reduce(items, func=add): ...
>>
>> looks more readable in my eyes even though somewhere ...
>
> Just use “sum” in this case.

Nah, the implementation is of course

def reduce(items, func=lambda x, y: x + y):
"""

>>> list(reduce([1,2,3]))
[1, 3, 6]

>>> list(reduce([42]))
[42]

>>> reduce([])
Traceback (most recent call last):
...
TypeError: reduce() with empty sequence
"""
items = iter(items)
try:
first = next(items)
except StopIteration:
raise TypeError("reduce() with empty sequence")

def _reduce(accu=first):
for item in items:
yield accu
accu = func(accu, item)
yield accu

return _reduce()

Yes, I'm kidding...

[dieter]

Peter Otten <__pet...@web.de> writes:

Maybe, it's the name ("lambda").

In Javascript, anonymous functions are widespread (and extensively
used for e.g. callback definitions) - but it uses the much more familiar
"function" (rather than "lambda") for this purpose.

[Ben Bacarisse]

dieter <die...@handshake.de> writes:
<snip>
>> Lawrence D’Oliveiro wrote:
<snip>

>>> I don’t know why this fear and suspicion of lambdas is so widespread among
>>> Python users ... former Java/C# programmers, perhaps?

By replying I'm not accepting the premise -- I have no idea if there is
widespread fear and suspicion of lambdas among Python users but it seems
unlikely.

> Maybe, it's the name ("lambda").
>
> In Javascript, anonymous functions are widespread (and extensively
> used for e.g. callback definitions)

Yes, but in Python they are restricted to a single expression. It's
therefore quite likely that programmers who want the semantics of an
anonymous function just define a named function and use that instead.
That saves you from having to decide, up front, if an expression is
going to be enough or from having to change it later if you find that it
isn't.

> - but it uses the much more familiar
> "function" (rather than "lambda") for this purpose.

... or the new arrow notation.

--
Ben.

[Lawrence D’Oliveiro]

On Tuesday, June 28, 2016 at 5:03:08 PM UTC+12, Ben Finney wrote:

> There is a clever one-line decorator that has been copy-pasted without
> explanation in many code bases for many years::
>
> decorator_with_args = lambda decorator: lambda *args, **kwargs: lambda func: decorator(func, *args, **kwargs)
>

For those who want docstrings, I’ll give you docstrings:

def decorator_with_args(decorator) :
"given function decorator(func, *args, **kwargs), returns a decorator which," \
" given func, returns the result of decorator(func, *args, **kwargs)."

def decorate(*args, **kwargs) :

def generated_decorator(func) :
return \
decorator(func, *args, **kwargs)
#end generated_decorator

#begin decorate
generated_decorator.__name__ = "decorator_{}".format(decorator.__name__)
generated_decorator.__doc__ = "decorator which applies {} to the previously-specified arguments".format(decorator.__name__)
return \
generated_decorator
#end decorate

#begin decorator_with_args
decorate.__name__ = "decorate_with_{}".format(decorator.__name__)
decorate.__doc__ = "generates a decorator which applies {} to the given arguments".format(decorator.__name__)
return \
decorate
#end decorator_with_args

[Ben Finney]

Ben Bacarisse <ben.u...@bsb.me.uk> writes:

> By replying I'm not accepting the premise -- I have no idea if there
> is widespread fear and suspicion of lambdas among Python users but it
> seems unlikely.

I can testify, as the person who started this thread, that there is no
fear or suspicion of lambda here. I use it quite frequently without
qualm for creating self-explanatory functions that need no name.

Rather, the motivation was that a complex thing, with many moving parts,
has an unexplained implementation: a nested set of functions without
names to explain their part in the pattern.

That these are then immediately bound to a name, to me defeats the
purpose of using lambda in the first place. If you want a named
function, lambda is not the tool to reach for; we have the ‘def’
statement for that.

But by using ‘lambda’ the author avoided one of the more important parts
of publishing the code: making it readable and self-explanatory. If
they'd chosen a name for each function, that would at least have
prompted them to explain what they're doing.

So ‘lambda’ is great, and I use it without worry for creating simple
self-explanatory nameless functions. But it's not the right tool for
this job: This is not self-explanatory, and the component functions
should not be nameless.

--
\ “It is forbidden to steal hotel towels. Please if you are not |
`\ person to do such is please not to read notice.” —hotel, |
_o__) Kowloon, Hong Kong |
Ben Finney

[Ian Kelly]

You should use functools.wraps instead of clobbering the decorated
function's name and docstring:

@functools.wraps(decorator)
def decorate(*args, **kwargs):
...

> return \
> decorate

Just to satisfy my own curiosity, do you have something against
putting the return keyword and the returned expression on the same
line?

[Ben Finney]

Ian Kelly <ian.g...@gmail.com> writes:

> You should use functools.wraps instead of clobbering the decorated
> function's name and docstring:
>
> @functools.wraps(decorator)
> def decorate(*args, **kwargs):
> ...

Thanks. Can you write the full implementation with that, so I can be
sure of where you expect that to go?

--
\ “I knew things were changing when my Fraternity Brothers threw |
`\ a guy out of the house for mocking me because I'm gay.” |
_o__) —postsecret.com, 2010-01-19 |
Ben Finney

[Ian Kelly]

On Fri, Jul 1, 2016 at 11:32 PM, Ben Finney <ben+p...@benfinney.id.au> wrote:
> Ian Kelly <ian.g...@gmail.com> writes:
>
>> You should use functools.wraps instead of clobbering the decorated
>> function's name and docstring:
>>
>> @functools.wraps(decorator)
>> def decorate(*args, **kwargs):
>> ...
>
> Thanks. Can you write the full implementation with that, so I can be
> sure of where you expect that to go?

def decorator_with_args(decorator):
"""Given function decorator(func, *args, **kwargs), returns a
wrapper that accepts args and kwargs separately from func.
"""
@functools.wraps(decorator)
def apply(*args, **kwargs):
# inner_decorator is transient, so its name and docstring
# are unimportant.
def inner_decorator(func):
return decorator(func, *args, **kwargs)
return inner_decorator
return apply

[Lawrence D’Oliveiro]

On Saturday, July 2, 2016 at 5:10:06 PM UTC+12, Ian wrote:
> You should use functools.wraps instead of clobbering the decorated
> function's name and docstring:

Where am I doing that?

> Just to satisfy my own curiosity, do you have something against
> putting the return keyword and the returned expression on the same
> line?

Yes.

[dieter]

Ben Finney <ben+p...@benfinney.id.au> writes:
> ...

> Rather, the motivation was that a complex thing, with many moving parts,
> has an unexplained implementation: a nested set of functions without
> names to explain their part in the pattern.

In a previous reply, I have tried to explain (apparently without success)
that the "thing" is not "complex" at all but a simple signature
transform for a decorator definitition and that the nested function
implementation is natural for this purpose.

[Ben Finney]

dieter <die...@handshake.de> writes:

> Ben Finney <ben+p...@benfinney.id.au> writes:
> > ... Rather, the motivation was that a complex thing, with many
> > moving parts, has an unexplained implementation: a nested set of
> > functions without names to explain their part in the pattern.
>
> In a previous reply, I have tried to explain (apparently without
> success) that the "thing" is not "complex" at all

Your explanation was clear. I disagree with it; the code is not at all
obvious in its intention or operation.

Naming the parts descriptively, and writing a brief synopsis docstring
for each function, is a way to address that. Which is my motivation for
this thread.

> but a simple signature transform for a decorator definitition

You're making my case for me: To anyone who doesn't already know exactly
what's going on, that is not at all obvious from the code as presented
in the first message.

> and that the nested function implementation is natural for this
> purpose.

The nested function implementation is not the problem, as I hope I've
made clear elsewhere in this thread.

--
\ “People's Front To Reunite Gondwanaland: Stop the Laurasian |
`\ Separatist Movement!” —wiredog, http://kuro5hin.org/ |
_o__) |
Ben Finney

[Ian Kelly]

On Sat, Jul 2, 2016 at 12:40 AM, Lawrence D’Oliveiro
<lawren...@gmail.com> wrote:
> On Saturday, July 2, 2016 at 5:10:06 PM UTC+12, Ian wrote:
>> You should use functools.wraps instead of clobbering the decorated
>> function's name and docstring:
>
> Where am I doing that?

Using the implementation you posted:

>>> @decorator_with_args
... def my_decorator(func, *args, **kwargs):
... """Returns func unmodified."""
... return func
...
>>> my_decorator.__doc__
'generates a decorator which applies my_decorator to the given arguments'

[Lawrence D’Oliveiro]

On Sunday, July 3, 2016 at 4:49:15 PM UTC+12, Ian wrote:

>
> On Sat, Jul 2, 2016 at 12:40 AM, Lawrence D’Oliveiro wrote:
>>
>> On Saturday, July 2, 2016 at 5:10:06 PM UTC+12, Ian wrote:
>>>
>>> You should use functools.wraps instead of clobbering the decorated
>>> function's name and docstring:
>>
>> Where am I doing that?
>
> Using the implementation you posted:
>
>>>> @decorator_with_args
> ... def my_decorator(func, *args, **kwargs):
> ... """Returns func unmodified."""
> ... return func
> ...
>>>> my_decorator.__doc__
> 'generates a decorator which applies my_decorator to the given arguments'

That is a function that I am generating, so naturally I want to give it a useful docstring. Am I “clobbering” any objects passed in by the caller, or returned by the caller? No, I am not.

[Ian Kelly]

It isn't useful, though. If somebody writes a docstring for a
function, it's because they want that to be the docstring for the
function, not some arbitrary and vague docstring assigned by a
decorator. If I type "help(my_decorator)" in the interactive
interpreter, I should see what my_decorator actually does, not
"generates a decorator which applies my_decorator", which just
confuses with its recursiveness. The only way to find out what
my_decorator actually does from the interactive interpreter would be
to realize that it's been opaquely decorated and type out something
like this:

>>> my_decorator.__closure__[0].cell_contents.__doc__
'Returns func unmodified.'

That's far too obscure. Alternatively, one could go read the source.
But if you have to go read the source to understand what a function
does, then you've lost the purpose of having a docstring in the first
place.

[Lawrence D’Oliveiro]

On Sunday, July 3, 2016 at 11:53:46 PM UTC+12, Ian wrote:

> On Sat, Jul 2, 2016 at 11:37 PM, Lawrence D’Oliveiro wrote:
>> That is a function that I am generating, so naturally I want to give it a
>> useful docstring. Am I “clobbering” any objects passed in by the caller,
>> or returned by the caller? No, I am not.
>
> It isn't useful, though. If somebody writes a docstring for a
> function, it's because they want that to be the docstring for the

> function ...

Which I am not “clobbering” at all. Do you get that yet? My docstrings apply to *my* functions, not those supplied or generated by the caller.

[Ian Kelly]

Sorry, but you're the one who doesn't seem to get it. Because it's a
decorator, "your" function is replacing the caller's function in the
caller's namespace. If the docstring written in the code for function
m.f is x, but the value of m.f.__doc__ is actually y, then the
docstring x is, for all practical purposes, clobbered. That you're not actually
modifying the original function is irrelevant.

For the purpose of the built-in help, it's the value of m.f.__doc__
that counts. For the purpose of documentation built with pydoc, it's
the value of m.f.__doc__ that counts. If the documentation for every
decorator in a module just reads "generates a decorator which applies
<function name> to the given arguments", that's useless documentation.

Which brings me to the other half of the point I was trying to make:
your decorator_with_args doesn't even *need* a docstring. It's not
part of the public interface. As a nested function that doesn't have
its own name within the module, help() or pydoc won't even include it
in their output. If you want to put something in the code to explain
what it does, that's fine, but use a comment for that. The docstring
doesn't add any value.

[Lawrence D’Oliveiro]

On Monday, July 4, 2016 at 10:39:30 AM UTC+12, Ian wrote:
> Sorry, but you're the one who doesn't seem to get it. Because it's a
> decorator, "your" function is replacing the caller's function in the
> caller's namespace.

No it is not. The replacement happens here:

def generated_decorator(func) :
return \
decorator(func, *args, **kwargs)
#end generated_decorator

As you can see, the function being called, “decorator”, is supplied by the caller. The function being decorated, “func”, is supplied by the caller (along with the additional *args and **kwargs). And the returned function is returned *as-is*, so whatever docstring was assigned by the caller is *NOT* “clobbered”.

Once again, the only docstrings I am setting are the ones for functions that *I* generate.

Do you understand that yet?

[Chris Angelico]

Yes, in a technical sense, you are correct. It's also technically
correct to say that str.upper() doesn't convert its argument to
uppercase - it constructs a completely new string of all uppercase
letters. But in normal usage, people expect "x = x.upper()" to return
*the same string, uppercased*. In the same way, people expect "f =
deco(f)" to return *the same function, decorated*. Technically it's
generally going to be a new function, because functions are immutable
(more-or-less), but it should broadly be the same function.

def require_login(f):
def modified(*a, **kw):
if not logged_in:
return "You must be logged in to do this."
return f(*a, **kw)
return modified

@require_login
def fire_ze_missiles():
raise Exhausted

@require_login
def reboot_server(server):
"""Reboot the specified server"""
server.send_command("sudo reboot")

Tell me, should the docstring for reboot_server be "Reboot the
specified server", or not? If not, why not?

ChrisA

[Lawrence D’Oliveiro]

On Monday, July 4, 2016 at 11:39:56 AM UTC+12, Chris Angelico wrote:
> In the same way, people expect "f = deco(f)" to return *the same function,
> decorated*.

I am not changing that in anyway. The decorated function generated by the caller is returned *unchanged*.

Once again, the only docstrings I am setting are for the intermediate functions *I* generate.

Suppose we have the following:

def try_func(func, arg) :
"sample function to be turned into a decorator."

def result() :
"returns func unchanged."
return \
func()
#end result

#begin try_func
result.__name__ = func.__name__
result.__doc__ = func.__doc__
return \
result
#end try_func

then:

>>> f = decorator_with_args(try_func)
>>> help(f)
Help on function decorate_with_try_func in module decorator_try:

decorate_with_try_func(*args, **kwargs)
generates a decorator which applies try_func to the given arguments

>>> @f(None)
... def g() :
... "my docstring--should be unchanged."
... return \
... 1
... #end g
...
>>> help(g)
Help on function g in module decorator_try:

g()
my docstring--should be unchanged.

Do you understand now?

[Ian Kelly]

On Sun, Jul 3, 2016 at 5:25 PM, Lawrence D’Oliveiro
<lawren...@gmail.com> wrote:
> On Monday, July 4, 2016 at 10:39:30 AM UTC+12, Ian wrote:
>> Sorry, but you're the one who doesn't seem to get it. Because it's a
>> decorator, "your" function is replacing the caller's function in the
>> caller's namespace.
>
> No it is not. The replacement happens here:
>
> def generated_decorator(func) :
> return \
> decorator(func, *args, **kwargs)
> #end generated_decorator
>
> As you can see, the function being called, “decorator”, is supplied by the caller. The function being decorated, “func”, is supplied by the caller (along with the additional *args and **kwargs). And the returned function is returned *as-is*, so whatever docstring was assigned by the caller is *NOT* “clobbered”.

I'm talking about the docstring of the *decorator*, not func. I agree
that whatever docstring func has is entirely up to func and the
decorator, not your code. Here again is the example I posted earlier:

>>> @decorator_with_args
... def my_decorator(func, *args, **kwargs):
... """Returns func unmodified."""
... return func
...
>>> my_decorator.__doc__
'generates a decorator which applies my_decorator to the given arguments'

I never made any claim about the docstring of the decorated func.

[Chris Angelico]

On Mon, Jul 4, 2016 at 9:59 AM, Lawrence D’Oliveiro
<lawren...@gmail.com> wrote:
> #begin try_func
> result.__name__ = func.__name__
> result.__doc__ = func.__doc__
> return \
> result
> #end try_func

This is the bit we're talking about. You're doing manually what
functools.wraps does for you (and it does other stuff too, to make
help() more useful), and you weren't doing this before.

Your original docstring on result() is completely ignored.

ChrisA

[Lawrence D’Oliveiro]

On Monday, July 4, 2016 at 12:02:23 PM UTC+12, Ian wrote:
> I'm talking about the docstring of the *decorator*, not func.

*Sigh*. Originally somebody was complaining that the lambda version didn’t allow for good (or any) docstrings. So I posted my version, and I went to all the effort of ensuring that every generated function had a good docstring. And now you are trying to knock me down for that?

What the hell is wrong with you people?!?

[Ian Kelly]

Sorry. I don't know who that was or what they wrote, because your post
is the first mention of docstrings that I see in this thread. Maybe
you're thinking of some other thread. I don't read more than a
fraction of what gets posted to this list.

But I read your post and I saw a way to improve the code in it, so I
suggested it. Next time I won't bother.

[Lawrence D’Oliveiro]

On Monday, July 4, 2016 at 12:06:21 PM UTC+12, Chris Angelico wrote:

> On Mon, Jul 4, 2016 at 9:59 AM, Lawrence D’Oliveiro wrote:
>> #begin try_func
>> result.__name__ = func.__name__
>> result.__doc__ = func.__doc__
>> return \
>> result
>> #end try_func
>
> This is the bit we're talking about.

This isn’t part of my decoration mechanism, this is just an example of it in use.