[Python-ideas] Conventions for function annotations

[Thomas Kluyver]

Function annotations (PEP 3107) are a very interesting new feature, but so far have gone largely unused. The only project I've seen using them is plac, a command-line option parser. One reason for this is that because function annotations can be used to mean anything, we're wary of doing anything in case we interfere with some other use case. A recent thread on ipython-dev touched on this [1], and we'd like to suggest some conventions to make annotations useful for everyone.

1. Code inspecting annotations should be prepared to ignore annotations it can't understand.

2. Code creating annotations should use wrapper classes to indicate what the annotation means. For instance, we are contemplating a way to specify options for a parameter, to be used in tab completion, so we would do something like this:

from IPython.core.completer import options
def my_io(filename, mode: options('read','write') ='read'):
    ...

3. There are a couple of important exceptions to 2:
- Annotations that are simply a string can be used like a docstring, to be displayed to the user. Inspecting code should not expect to be able to parse any machine-readable information out of these strings.
- Annotations that are a built-in type (int, str, etc.) indicate that the value should always be an instance of that type. Inspecting code may use these for type checking, introspection, optimisation, or other such purposes. Note that for now, I have limited this to built-in types, so other types can be used for other purposes, but this could be extended. For instance, the ABCs from collections (collections.Mapping et al.) could well be added to this category.

4. There should be a convention for attaching multiple annotations to one value. I propose that all code using annotations expects to handle tuples/lists of annotations. (We also considered dictionaries, but the result is long and ugly). So in this definition:

def my_io(filename, mode: (options('read','write'), str, 'The mode in which to open the file') ='read'):
    ...

the mode parameter has a set of options (ignored by frameworks that don't recognise it), should always be a string, and has a description.

Any thoughts and suggestions are welcome.

As an aside, we may also create a couple of decorators to fill in __annotations__ on Python 2, something like:

@return_annotation('A file obect')
@annotations(mode=(options('read','write'), str, 'The mode in which to open the file'))
def my_io(filename, mode='read'):
    ...

[1] http://mail.scipy.org/pipermail/ipython-dev/2012-November/010697.html


Thanks,
Thomas

[Andrew Svetlov]

I think code related to annotations is tightly coupled with annotated
function usage context (decorator, metaclass, function caller).
So annotation really can mean anything and it depends from context.
I don't see use case when context need to ignore unexpected
annotation. In my practice annotation is always expected if specified,
absence of annotation for parameter is mark to do nothing with it (it
can be allowed or disabled depending of context requirements).
The same for multiple annotations. If your context allow it — that's
up to you. Exact kind of composition to use depends from context — it
can be tuple, dict, user-defined composition object.

My point is: we dont need to restrict annotations in any way. If some
libraries want to share annotations that means they are tightly enough
coupled and can make rules for itself. All other code can go in the
wild.

> _______________________________________________
> Python-ideas mailing list
> Python...@python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>

--
Thanks,
Andrew Svetlov
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas

[Thomas Kluyver]

I think annotations are potentially very useful for things like introspection and static analysis. For instance, your IDE could warn you if you pass a parameter that doesn't match the type specified in an annotation. In these cases, the code reading the annotations isn't coupled with the function definitions.

I'm not aiming to restrict annotations, just to establish some conventions to make them useful. We have a convention, for instance, that attributes with a leading underscore are private. That's a useful basis that everyone understands, so when you do obj.<tab> in IPython, it doesn't show those attributes by default. I'd like to have some conventions of that nature around annotations.

Thomas

[Nick Coghlan]

On Sun, Dec 2, 2012 at 4:26 PM, Robert McGibbon <rmcg...@gmail.com> wrote:

By being tolerant and well behaved when confronted with annotations that our library doesn't understand I, I

think we can use function annotations without a short-range decorator that translates their information in some other

structure. If other annotation-using libraries are also willing to ignore our tabbing annotations if/when they encounter them,

then can't we all get along smoothly?

(For reference, the feature will look/work something like this)

In[1]: def foo(filename : tab_glob('*.txt')):  # tab completion that recommends files/directories that match a glob pattern

...          pass

...

In[2]: foo(<TAB>

'a.txt'        'b.txt'

'c.txt'         'dir/'

You're missing the other key reason for requiring decorators that interpret function annotations: they're there for the benefit of *readers*, not just other software. Given your definition above, I don't know what the annotations are for, except by recognising the "tab_glob" call. However, that then breaks as soon as the expression is put into a named variable earlier in the file:

    def foo(filename : text_files): # What does this mean?

        pass

But the reader can be told *explicitly* what the annotations are related to via a decorator:

    @tab_expansion

    def foo(filename : text_files): # Oh, it's just a tab expansion specifier

        pass

Readers no longer have to guess from context, and if the tab_expansion decorator creates IPython-specific metadata, then the interpreter doesn't need to guess either.

(Note that you *can* use ordinary mechanisms like class decorators, metaclasses, post-creation modification of classes and IDE snippet inclusion to avoid the need to type out the "this is what these annotations mean" decorator explicitly. However, that's just an application of Python's standard abstraction tools, rather than a further special case convention)

Mixing annotations intended for different consumers is a fundamentally bad idea, as it encourages unreadable code and complex dances to avoid stepping on each other's toes. It's better to design a *separate* API that supports composition by passing the per-parameter details directly to a decorator factory (which then adds appropriate named attributes to the function), with annotations used just as syntactic sugar for simple cases where no composition is involved.

The important first question to ask is "How would we solve this if annotations didn't exist?" and only *then* look at the shorthand case for function-annotations. For cases where function annotations make code more complex or less robust, *don't use them*.

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Nick Coghlan]

On Sun, Dec 2, 2012 at 2:30 AM, Thomas Kluyver <tho...@kluyver.me.uk> wrote:

I think annotations are potentially very useful for things like introspection and static analysis. For instance, your IDE could warn you if you pass a parameter that doesn't match the type specified in an annotation. In these cases, the code reading the annotations isn't coupled with the function definitions.

I'm not aiming to restrict annotations, just to establish some conventions to make them useful. We have a convention, for instance, that attributes with a leading underscore are private. That's a useful basis that everyone understands, so when you do obj.<tab> in IPython, it doesn't 

show those attributes by default. I'd like to have some conventions of that nature around annotations.

Indeed, composability is a problem with annotations. I suspect the only way to resolve this systematically is to adopt a convention where annotations are used *strictly* for short-range communication with an associated decorator that transfers the annotation details to a *different* purpose-specific location for long-term introspection.

Furthermore, if composability is going to be possible in general, annotations can really *only* be used as a convenience API, with an underlying API where the necessary details are supplied directly to the decorator. For example, here's an example using the main decorator API for a cffi callback declaration [1]:

    @cffi.callback("int (char *, int)"):
    def my_cb(arg1, arg2):
        ...

The problem with this is that it can get complicated to map C-level types to parameter names as the function signature gets more complicated. So, what you may want to do is write a decorator that builds the CFFI signature from annotations on the individual parameters:

    @annotated_cffi_callback
    def my_cb(arg1: "char *", arg2: "int") -> "int":
        ...

The decorator would turn that into an ordinary call to cffi.callback, so future introspection wouldn't look at the annotations mapping at all, it would look directly at the CFFI metadata.

Annotations should probably only ever be introspected by their associated decorator, and if you really want to apply multiple decorators with annotation support to a single function, you're going to have to fall back to the non-annotation based API for at least some of them. Once you start trying to overload the annotation field with multiple annotations, the readability gain for closer association with the individual parameters is counterbalanced by the loss of association between the subannotations and their corresponding decorators.

[1] http://cffi.readthedocs.org/en/latest/#callbacks

[Ramchandra Apte]

There can be a class which contains annotations not following this convention. e.g.

def f(a:other("positive integer") ,b:other("negative integer")):pass

[Ramchandra Apte]

+1 even I was thinking of this.

It would be useful for Python to C/C++ translators too.

[Ramchandra Apte]

Oops, here is it, corrected.

There can be a class which contains annotations not following this convention. e.g.

def f(a:other(something) ,b:other(something)):pass

[Robert McGibbon]

Nick,

Thanks! You make a very convincing argument.

Especially if this represents the collective recommendation of the python core development team on the proper conventions surrounding the use of function annotations, I would encourage you guys to perhaps make it more widely known (blogs, etc). As python 3.x adoption continues to move forward, this type of thing could become an issue if shmucks like me start using the annotation feature more widely.

-Robert

[Nick Coghlan]

On Sun, Dec 2, 2012 at 8:12 PM, Robert McGibbon <rmcg...@gmail.com> wrote:

Nick,

Thanks! You make a very convincing argument.

Especially if this represents the collective recommendation of the python core development team on the proper conventions surrounding the use of function annotations, I would encourage you guys to perhaps make it more widely known (blogs, etc). As python 3.x adoption continues to move forward, this type of thing could become an issue if shmucks like me start using the annotation feature more widely.

Last time it came up, the collective opinion on python-dev was still to leave PEP 8 officially neutral on the topic so that people could experiment more freely with annotations and the community could help figure out what worked well and what didn't. Admittedly this was long enough ago that I don't remember the details, just the obvious consequence that PEP 8 remains largely silent on the matter, aside from declaring that function annotations are off-limits for standard library modules: "The Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style. Instead, the annotations are left for users to discover and experiment with useful annotation styles."

Obviously, I'm personally rather less open-minded on the topic of *composition* in particular, as that's a feature I'm firmly convinced should be left in the hands of ordinary decorator usage. I believe trying to contort annotations to handle that cause is almost certain to result in something less readable than the already possible decorator equivalent.

However, the flip-side of the argument is that if we assume my opinion is correct and document it as an official recommendation in PEP 8, then many people won't even *try* to come up with good approaches to composition for function annotations. Maybe there *is* an elegant, natural solution out there that's superior to using explicit calls to decorator factories for the cases that involve composition. If PEP 8 declares "just use decorator factories for cases involving composition, and always design your APIs with a non-annotation based fallback for such cases", would we be inadvertently shutting down at least some of the very experimentation we intended to allow?

After all, while I don't think the composition proposal in this thread reached the bar of being more readable than just composing decorator factories to handle more complex cases, I *do* think it is quite a decent attempt.

[Thomas Kluyver]

On 2 December 2012 11:43, Nick Coghlan <ncog...@gmail.com> wrote:

However, the flip-side of the argument is that if we assume my opinion is correct and document it as an official recommendation in PEP 8, then many people won't even *try* to come up with good approaches to composition for function annotations. Maybe there *is* an elegant, natural solution out there that's superior to using explicit calls to decorator factories for the cases that involve composition. If PEP 8 declares "just use decorator factories for cases involving composition, and always design your APIs with a non-annotation based fallback for such cases", would we be inadvertently shutting down at least some of the very experimentation we intended to allow?

My concern with this is that it's tricky to experiment with composition. If you want to simultaneously use annotations for, say, one framework that checks argument types, and one that documents individual arguments based on annotations, they need to be using the same mechanism to compose annotation values. Alternatively, the first one to access the annotations could decompose the values, leaving them in a form the second can understand - but that sounds brittle and opaque.

Another proposed mechanism (Robert's idea) which I didn't mention above is to override __add__, so that multiple annotations can be composed like this:

def my_io(filename, mode: tab('read','write') + typed(str) ='read'):
    ...

As a possible workaround, here's a decorator for decorators that makes the following two definitions equivalent:
https://gist.github.com/4189289

@check_argtypes
def checked1(a:int, b:str):
    pass

@check_argtypes(a=int, b=str)
def checked2(a, b):
    pass

With this, it's easy to use annotations where possible, and you benefit from the extra clarity, but it's equally simple to pass the values as arguments to the decorator, for instance if the annotations are already in use for something else. It should also work under Python 2, using the non-annotated version.

Thomas

[Steven D'Aprano]

On 02/12/12 22:43, Nick Coghlan wrote:

> Last time it came up, the collective opinion on python-dev was still to
> leave PEP 8 officially neutral on the topic so that people could experiment
> more freely with annotations and the community could help figure out what
> worked well and what didn't. Admittedly this was long enough ago that I
> don't remember the details, just the obvious consequence that PEP 8 remains
> largely silent on the matter, aside from declaring that function
> annotations are off-limits for standard library modules: "The Python
> standard library will not use function annotations as that would result in
> a premature commitment to a particular annotation style. Instead, the
> annotations are left for users to discover and experiment with useful
> annotation styles."

I fear that this was a strategic mistake. The result, it seems to me, is that
annotations have been badly neglected.

I can't speak for others, but I heavily use the standard library as a guide
to what counts as good practice in Python. I'm not a big user of third party
libraries, and most of those are for 2.x, so with the lack of annotations in
the std lib I've had no guidance as to what sort of things annotations could
be used for apart from "type checking".

I'm sure that I'm not the only one.



--
Steven

[David Townshend]

> I fear that this was a strategic mistake. The result, it seems to me, is that
> annotations have been badly neglected.
>
> I can't speak for others, but I heavily use the standard library as a guide
> to what counts as good practice in Python. I'm not a big user of third party
> libraries, and most of those are for 2.x, so with the lack of annotations in
> the std lib I've had no guidance as to what sort of things annotations could
> be used for apart from "type checking".
>
> I'm sure that I'm not the only one.
>
>
>
> --
> Steven
>

+1

[Raymond Hettinger]

On Dec 2, 2012, at 3:43 AM, Nick Coghlan <ncog...@gmail.com> wrote:

 Admittedly this was long enough ago that I don't remember the details, just the obvious consequence that PEP 8 remains largely silent on the matter, aside from declaring that function annotations are off-limits for standard library modules: 

PEP 8 is not "largely silent" on the subject:

"''

The Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style. Instead, the annotations are left for users to discover and experiment with useful annotation styles.

Early core developer attempts to use function annotations revealed inconsistent, ad-hoc annotation styles. For example:

• [str] was ambiguous as to whether it represented a list of strings or a value that could be either str or None.
• The notation open(file:(str,bytes)) was used for a value that could be either bytes or str rather than a 2-tuple containing a str value followed by a bytesvalue.
• The annotation seek(whence:int) exhibited an mix of over-specification and under-specification: int is too restrictive (anything with __index__ would be allowed) and it is not restrictive enough (only the values 0, 1, and 2 are allowed). Likewise, the annotation write(b: bytes) was also too restrictive (anything supporting the buffer protocol would be allowed).
• Annotations such as read1(n: int=None) were self-contradictory since None is not an int. Annotations such as source_path(self, fullname:str) -> objectwere confusing about what the return type should be.
• In addition to the above, annotations were inconsistent in the use of concrete types versus abstract types: int versus Integral and set/frozenset versus MutableSet/Set.
• Some annotations in the abstract base classes were incorrect specifications. For example, set-to-set operations require other to be another instance of Setrather than just an Iterable.
• A further issue was that annotations become part of the specification but weren't being tested.
• In most cases, the docstrings already included the type specifications and did so with greater clarity than the function annotations. In the remaining cases, the docstrings were improved once the annotations were removed.
• The observed function annotations were too ad-hoc and inconsistent to work with a coherent system of automatic type checking or argument validation. Leaving these annotations in the code would have made it more difficult to make changes later so that automated utilities could be supported.
• '''

Raymond

[Antoine Pitrou]

Le Mon, 3 Dec 2012 00:09:17 -0800,
Raymond Hettinger
<raymond....@gmail.com> a écrit :

>
> Early core developer attempts to use function annotations revealed
> inconsistent, ad-hoc annotation styles. For example:
>
> [str] was ambiguous as to whether it represented a list of strings or
> a value that could be either str or None. The notation
> open(file:(str,bytes)) was used for a value that could be either
> bytes or str rather than a 2-tuple containing a str value followed by
> a bytesvalue. The annotation seek(whence:int) exhibited an mix of
> over-specification and under-specification: int is too restrictive
> (anything with __index__ would be allowed) and it is not restrictive
> enough (only the values 0, 1, and 2 are allowed). Likewise, the
> annotation write(b: bytes) was also too restrictive (anything
> supporting the buffer protocol would be allowed). Annotations such as
> read1(n: int=None) were self-contradictory since None is not an int.
> Annotations such as source_path(self, fullname:str) -> objectwere
> confusing about what the return type should be. In addition to the
> above, annotations were inconsistent in the use of concrete types
> versus abstract types: int versus Integral and set/frozenset versus
> MutableSet/Set. Some annotations in the abstract base classes were
> incorrect specifications. For example, set-to-set operations require
> other to be another instance of Setrather than just an Iterable.

In short, we have discovered that declarative typing isn't very
useful :-)

Regards

Antoine.

[Paul Moore]

Sorry, should have gone to the list

On 3 December 2012 09:30, Paul Moore <p.f....@gmail.com> wrote:

> On 3 December 2012 09:21, Antoine Pitrou <soli...@pitrou.net> wrote:
>> In short, we have discovered that declarative typing isn't very
>> useful :-)
>

> .. but haven't thought of any other useful applications of
> annotations, and nor has the collective community on PyPI.
>
> Annotations seem like a solution looking for a problem, to me. (Which
> is a shame, as they look like a pretty cool solution)
> Paul

[Robert McGibbon]

The IPython community has thought of using annotations to do argument specific

tab completion in the interactive interpreter.

For example, a load function whose first argument is supposed to be files matching 

a certain glob pattern might use a function annotation on that argument to specify

the glob pattern.

A sympy maintainer, Aaron Meurer,  has also expressed interest in using this feature

-- as implemented in ipython -- to annotate sympy functions' return values by type

to facilitate tab completion for chained calls like f(x).<TAB>

I'm working on this feature for IPython (PR: Function annotation based hooks into the tab completion system).

I've already benefited a lot from the discussion on this thread in terms of the design

of the API. Specifically Nick Coghlan's arguments have been very enlightening.

Comments, suggestions, contributions, etc are welcome!

-Robert

[Thomas Kluyver]

On 3 December 2012 09:30, Paul Moore <p.f....@gmail.com> wrote:

> .. but haven't thought of any other useful applications of
> annotations, and nor has the collective community on PyPI.

I suspect that the lack of applications is partly due to people not knowing about them, code having to still support Python 2, and an absence of guidelines about how to use them safely.

For our part, I think we'll push forwards following Nick's suggestions - annotations to be accessed by closely coupled decorators only.

Thanks all,
Thomas

[Nick Coghlan]

On Mon, Dec 3, 2012 at 6:09 PM, Raymond Hettinger <raymond....@gmail.com> wrote:

On Dec 2, 2012, at 3:43 AM, Nick Coghlan <ncog...@gmail.com> wrote:

 Admittedly this was long enough ago that I don't remember the details, just the obvious consequence that PEP 8 remains largely silent on the matter, aside from declaring that function annotations are off-limits for standard library modules: 

PEP 8 is not "largely silent" on the subject:

It's effectively silent on the matters at hand, which are:

* the advisability of using annotations without an associated decorator that makes the interpretation currently in play explicit (while the examples given do illustrate why *not* doing this is a bad idea, it doesn't explicitly state that conclusion, merely "we're not going to use them in the standard library at this point")
* the advisability of providing a pure annotations API, without any fallback to an explicit decorator factory
* the advisability of handling composition within the annotations themselves, rather than by falling back to explicit decorator factories
* the advisability of using the __annotations__ dictionary for long-term introspection, rather than using the decorator to move the information to a purpose-specific location in a separate function attribute

I would be *quite delighted* if people are open to the idea of making a much stronger recommendation along the following lines explicit in PEP 8:

==================

* If function annotations are used, it is recommended that:
    * the annotation details should be designed with a specific practical use case in mind
    * the annotations are used solely as a form of syntactic sugar for passing arguments to a decorator factory that would otherwise accept explicit per-parameter arguments
    * the decorator factory name should provide the reader of the code with a strong hint as to the intended meaning of the parameter annotations (or at least a convenient reference point to look up in the documentation)
    * in simple cases, using parameter and return type annotations will then allow the per-parameter details to be mapped easily by both the code author and later readers without requiring repetition of parameter names or careful alignment of factory arguments with parameter positions.
    * the explicit form remains available to handle more complex situations (such as applying multiple decorators to the same function) without requiring complicated conventions for composing independent annotations on a single function

==================

In relation to the last point, I consider composing annotations to be analogous to composing function arguments. Writing:

    @g
    @f
    def annotated(arg1: (a, x), arg2: (b, y), arg3: (c, z)):
        ...

instead of the much simpler:

    @g(x, y, z)
    @f(a, b, c)
    def annotated(arg1, arg2, arg3):
        ...

is analagous to writing:

    args = [(a, x), (b, y), (c, z)]
    f(*(x[0] for x in args))
    g(*(x[1] for x in args))

instead of the more obvious:

    f(a, b, c)
    g(x, y, z)

[Andrew Svetlov]

On Mon, Dec 3, 2012 at 1:08 PM, Nick Coghlan <ncog...@gmail.com> wrote:
> * the advisability of using the __annotations__ dictionary for long-term
> introspection, rather than using the decorator to move the information to a
> purpose-specific location in a separate function attribute

My 5 cents: perhaps you don't need to use __annotations__ at all,
Signature object (PEP 362) gives more convenient way for gathering
information about function spec.

[Nick Coghlan]

On Mon, Dec 3, 2012 at 9:45 PM, Andrew Svetlov <andrew....@gmail.com> wrote:

On Mon, Dec 3, 2012 at 1:08 PM, Nick Coghlan <ncog...@gmail.com> wrote:
> * the advisability of using the __annotations__ dictionary for long-term
> introspection, rather than using the decorator to move the information to a
> purpose-specific location in a separate function attribute

My 5 cents: perhaps you don't need to use __annotations__ at all,
Signature object (PEP 362) gives more convenient way for gathering
information about function spec.

I don't quite understand that comment - PEP 362 is purely an access mechanism. The underlying storage is still in __annotations__ (at least as far any annotations are concerned).

However, using separate storage is a natural consequence of also providing an explicit decorator factory API, so I didn't bring it up.

[Andrew Svetlov]

Ok, you right. I told about access mechanism only.

--
Thanks,
Andrew Svetlov

[Matt Williams]

On Saturday, 1 December 2012 16:30:49 UTC, Thomas Kluyver wrote:
> I think annotations are potentially very useful for things like introspection and static analysis.
> For instance, your IDE could warn you if you pass a parameter that doesn't match the type
> specified in an annotation. In these cases, the code reading the annotations isn't coupled
> with the function definitions.

Indeed. The Python plugin for KDevelop is using function annotations for help with type deduction
http://scummos.blogspot.co.uk/2012/10/vienna-kdevkate-sprint-kdevelop.html

Matt

[Barry Warsaw]

On Dec 03, 2012, at 09:08 PM, Nick Coghlan wrote:

>I would be *quite delighted* if people are open to the idea of making a
>much stronger recommendation along the following lines explicit in PEP 8:

I am -1 for putting any of what followed in PEP 8, and in fact, I think the
existing examples at the bottom of PEP 8 are inappropriate.

PEP 8 should be prescriptive of explicit Python coding styles. Think "do
this, not that". It should be as minimal as possible, and in general provide
rules that can be easily referenced and perhaps automated (e.g. pep8.py).

Some of the existing text in PEP 8 already doesn't fall under that rubric, but
it's close enough (e.g. designing for inheritance).

I don't think annotations reach the level of consensus or practical experience
needed to be added to PEP 8.

OTOH, I wouldn't oppose a new informational PEP labeled "Annotations Best
Practices", where some of these principles can be laid out and explored.

Cheers,
-Barry

[Guido van Rossum]

Hm. I agree PEP 8 seems an odd place for Nick's recommendation. Even
if I were to agree with hos proposal I would think it belongs in a
different PEP than PEP 8.

But personally I haven't given up on using annotations to give type
hints -- I think it can at some times be a useful augmentation to
static analysis (whose use I see mostly as an aid to human readers
and/or tools like linters, IDEs, and refactoring tools, not for
guiding compiler optimizations). I know of several projects (both
public and private) for improving the state of the art of Python
static analysis with this goal in mind. With the advent of e.g.
TypeScript and Dart in the JavaScript world, optional type annotations
for dynamic languages appear to be becoming more fashionable, and
maybe we can get some use out of them.

FWIW, as far as e.g. 'int' being both overspecified and
underspecified: I don't care about the underspecification so much,
that's always going to happen; and for the overspecification, we can
either use some abstract class instead, or simply state that the
occurrence of certain concrete types must be taken as a shorthand for
a specific abstract type. This could be part of the registration call
of the concrete type, or something.

Obviously this would require inventing and standardizing notations for
things like "list of X", "tuple with items X, Y, Z", "either X or Y",
and so on, as well as a standard way of combining annotations intended
for different tools.

*This* would be a useful discussion. What to do in the interim... I
think the current language in PEP 8 is just fine until we have a
better story.

--Guido

--
--Guido van Rossum (python.org/~guido)

[Nick Coghlan]

So long as any type hinting semantics are associated with a "@type_hints" decorator, none of those ideas conflict with my suggestions for good annotation usage practices.

The explicit decorators effectively end up serving as dialect specifiers for the annotations, for the benefit of other software (by moving the metadata out to purpose specific attributes) and for readers (simply by being present).

Anyway, the reactions here confirmed my recollection of a lack of consensus amongst the core team. I'll just put something up on my own site, instead.

Cheers,
Nick.

--
Sent from my phone, thus the relative brevity :)

[David Townshend]

Just thought of a couple of usages which don't fit into the decorator model.  The first is using the return annotation for early binding:

    def func(seq) -> dict(sorted=sorted):
        return func.__annotations__['return']['sorted'](seq)

Stangely enough, this seems to run slightly faster than

    def func(seq, sorted=sorted):
        return sorted(seq)

My test shows the first running in about 0.376s and the second in about 0.382s (python 3.3, 64bit).


The second is passing information to base classes.  This is a rather contrived example which could easily be solved (better) in plenty of other ways, but it does illustrate a pattern which someone else may be able to turn into a genuine use case.

class NumberBase:

    def adjust(self, value):
        return self.adjust.__annotations__['return'](value)


class NegativeInteger(NumberBase):

    def adjust(self, value) -> int:
        return super().adjust(-value)


>>> ni = NegativeInteger()
>>> ni.adjust(4.3)
-4


Cheers

David

[Jasper St. Pierre]

Indeed. I've looked at annotations before, but I never understood the purpose. It seemed like a feature that was designed and implemented without some goal in mind, and where the community was supposed to discover the goal themselves.

So, if I may ask, what was the original goal of annotations? The PEP gives some suggestions, but doesn't leave anything concrete. Was it designed to be an aid to IDEs, or static analysis tools that inspect source code? Something for applications themselves to munge through to provide special behaviors, like a command line parser, or runtime static checker?

The local decorator influence might work, but that has the problem of only being able to be used once before we fall back to the old method. Would you rather:

    @tab_expand(filename=glob('*.txt'))
    @types
    def read_from_filename(filename:str, num_bytes:int) -> bytes:
        pass

or

    @tab_expand(filename=glob('*.txt'))
    @types(filename=str, num_bytes=int, return_=bytes)
    def read_from_filename(filename, num_bytes):
        pass

For consistency's sake, I'd prefer the latter.

Note that we could take a convention, like Thomas suggests, and adopt both:

    @tab_expand
    @types
    def read_from_filename(filename:(str, glob('*.txt')), num_bytes:int) -> bytes:
        pass

But that's a "worst of both worlds" approach: we lose the locality of which argument applies to which decorator (unless we make up rules about positioning in the tuple or something), and we gunk up the function signature, all to use a fancy new Python 3 feature.

With a restricted and narrow focus, I could see them gaining adoption, but for now, it seems like extra syntax was introduced simply for the point of having extra syntax.

--
  Jasper

[Thomas Kluyver]

On 4 December 2012 16:43, Jasper St. Pierre <jstp...@mecheye.net> wrote:

The local decorator influence might work, but that has the problem of only being able to be used once before we fall back to the old method. Would you rather:

    @tab_expand(filename=glob('*.txt'))
    @types
    def read_from_filename(filename:str, num_bytes:int) -> bytes:
        pass

or

    @tab_expand(filename=glob('*.txt'))
    @types(filename=str, num_bytes=int, return_=bytes)
    def read_from_filename(filename, num_bytes):
        pass

For consistency's sake, I'd prefer the latter.

Using the decorator decorator I posted (https://gist.github.com/4189289 ), you could use these interchangeably, so the annotations are just a convenient alternative syntax for when you think they'd make the code more readable.

Thomas

[Ned Batchelder]

On 12/4/2012 11:43 AM, Jasper St. Pierre wrote:
> Indeed. I've looked at annotations before, but I never understood the
> purpose. It seemed like a feature that was designed and implemented
> without some goal in mind, and where the community was supposed to
> discover the goal themselves.
>
> So, if I may ask, what was the original goal of annotations? The PEP
> gives some suggestions, but doesn't leave anything concrete. Was it
> designed to be an aid to IDEs, or static analysis tools that inspect
> source code? Something for applications themselves to munge through to
> provide special behaviors, like a command line parser, or runtime
> static checker?

A telling moment for me was during an early Py3k keynote at PyCon
(perhaps it was in Dallas or Chicago?), Guido couldn't remember the word
"annotation," and said, "you know, those things that aren't type
declarations?" :-)

--Ned.

[Guido van Rossum]

On Tue, Dec 4, 2012 at 9:12 AM, Ned Batchelder <n...@nedbatchelder.com> wrote:
> On 12/4/2012 11:43 AM, Jasper St. Pierre wrote:
>>
>> Indeed. I've looked at annotations before, but I never understood the
>> purpose. It seemed like a feature that was designed and implemented without
>> some goal in mind, and where the community was supposed to discover the goal
>> themselves.

To the contrary. There were too many use cases that immediately looked
important, and we couldn't figure out which ones would be the most
important or how to combine them, so we decided to take a two-step
approach: in step 1, we designed the syntax, whereas in step 2, we
would design the semantics. The idea was very clear that once the
syntax was settled people would be free to experiment with different
semantics -- just not in the stdlib. The idea was also that
eventually, from all those experiments, one would emerge that would be
fit for the stdlib.

The process was somewhat similar to the way decorators were
introduced. In Python 2.3, we introduced things like staticmethod,
classmethod and property. But we *didn't* introduce the @ syntax,
because we couldn't agree about it at that point. Then, for 2.4, we
sorted out the proper syntax, having by then conclusively discovered
that the original way of using e.g. classmethod (an assignment after
the end of the method definition) was hard on the human reader.

(Of course, you may note that for decorators, we decided on semantics
first, syntax second. But no two situations are quite the same, and in
the case of annotations, without syntax it would be nearly impossible
to experiment with semantics.)

>> So, if I may ask, what was the original goal of annotations? The PEP gives
>> some suggestions, but doesn't leave anything concrete. Was it designed to be
>> an aid to IDEs, or static analysis tools that inspect source code? Something
>> for applications themselves to munge through to provide special behaviors,
>> like a command line parser, or runtime static checker?

Pretty much all of the above to some extent. But for me personally,
the main goal was always to arrive at a notation to specify type
constraints (and maybe other constraints) for arguments and return
values. I've toyed at various times with specific ways of combining
types. E.g. list[int] might mean a list of integers, and dict[str,
tuple[float, float, float, bool]] might mean a dict mapping strings to
tuples of three floats and a bool. But I felt it was much harder to
get consensus about such a notation than about the syntax for argument
annotations (think about how many objections you can bring in to these
two examples :-) -- I've always had a strong desire to use "var: type
= default" and to make the type a runtime expression to be evaluated
at the same time as the default.

> A telling moment for me was during an early Py3k keynote at PyCon (perhaps
> it was in Dallas or Chicago?), Guido couldn't remember the word
> "annotation," and said, "you know, those things that aren't type
> declarations?" :-)

Heh. :-)

--
--Guido van Rossum (python.org/~guido)

[Masklinn]

On 2012-12-03, at 18:27 , Guido van Rossum wrote:
>
> Obviously this would require inventing and standardizing notations for
> things like "list of X", "tuple with items X, Y, Z", "either X or Y",
> and so on, as well as a standard way of combining annotations intended
> for different tools.

I've always felt that __getitem__ and __or__/__ror__ on type 1. looked
rather good and 2. looked similar to informal type specs and type specs
of other languages. Although that's the issue with annotations being
Python syntax: it requires changing stuff fairly deep into Python to
be able to experiment.

The most bothersome part is that I "feel" "either X or Y" (aka `X | Y`)
should be a set of type (and thus the same as {X, Y}[0]) but that doesn't
work with `isinstance` or `issubclass`. Likewise, `(a, b, c)` in an
annotation feels like it should mean the same as `tuple[a, b, c]` ("a
tuple with 3 items of types resp. a, b and c") but that's at odds with
the same type-checking functions.

The first could be fixable by relaxing slightly the constraints of
isinstance and issubclass, but not so for the second.

[0] which works rather neatly for anonymous unions as `|` is the union
of two sets, so the arithmetic would be `type | type -> typeset`,
`type | typeset -> typeset` and `typeset | typeset -> typeset`,
libraries could offer opaque types/typesets which would be composable
without their users having to know whether they're type atoms or
typesets

[Eric Snow]

Check out http://www.artima.com/weblogs/viewpost.jsp?thread=89161

-eric

[Guido van Rossum]

On Tue, Dec 4, 2012 at 1:37 AM, David Townshend <aquav...@gmail.com> wrote:
> Just thought of a couple of usages which don't fit into the decorator model.
> The first is using the return annotation for early binding:
>
> def func(seq) -> dict(sorted=sorted):
> return func.__annotations__['return']['sorted'](seq)

You've got to be kidding...

> Stangely enough, this seems to run slightly faster than
>
> def func(seq, sorted=sorted):
> return sorted(seq)
>
> My test shows the first running in about 0.376s and the second in about
> 0.382s (python 3.3, 64bit).

Surely that's some kind of random variation. It's only a 2% difference.

> The second is passing information to base classes. This is a rather
> contrived example which could easily be solved (better) in plenty of other
> ways, but it does illustrate a pattern which someone else may be able to
> turn into a genuine use case.
>
> class NumberBase:
>
> def adjust(self, value):
> return self.adjust.__annotations__['return'](value)
>
>
> class NegativeInteger(NumberBase):
>
> def adjust(self, value) -> int:
> return super().adjust(-value)
>
>
>>>> ni = NegativeInteger()
>>>> ni.adjust(4.3)
> -4

This looks like a contrived way to use what is semantically equivalent
to function attributes. The base class could write

def adjust(self, value):
return self.adjust.adjuster(value)

and the subclass could write

def adjust(self, value):
return super().adjust(-value)
adjust.adjuster = int

Or invent a decorator to set the attribute:

@set(adjuster=int)
def adjust(self, value):
return super().adjust(-value)

But both of these feel quite awkward compared to just using a class attribute.

class NumberBase:

def adjust(self, value):

return self.adjuster(value)

class NegativeInteger(NumberBase):

adjuster = int
# No need to override adjust()

IOW, this is not a line of thought to pursue.

[Guido van Rossum]

On Tue, Dec 4, 2012 at 10:12 AM, Masklinn <mask...@masklinn.net> wrote:
> On 2012-12-03, at 18:27 , Guido van Rossum wrote:
>>
>> Obviously this would require inventing and standardizing notations for
>> things like "list of X", "tuple with items X, Y, Z", "either X or Y",
>> and so on, as well as a standard way of combining annotations intended
>> for different tools.
>
> I've always felt that __getitem__ and __or__/__ror__ on type 1. looked
> rather good and 2. looked similar to informal type specs and type specs
> of other languages. Although that's the issue with annotations being
> Python syntax: it requires changing stuff fairly deep into Python to
> be able to experiment.

So, instead of using

def foo(a: int, b: str) -> float:
<blah>

you use

from experimental_type_annotations import Int, Str, Float

def foo(a: Int, b: Str) -> Float:
<blah>

And now we're ready for experimentation.

[Warning: none of this is particularly new; I've had these things in
my brain for years, as the referenced Artima blog post made clear.]

> The most bothersome part is that I "feel" "either X or Y" (aka `X | Y`)
> should be a set of type (and thus the same as {X, Y}[0]) but that doesn't
> work with `isinstance` or `issubclass`. Likewise, `(a, b, c)` in an
> annotation feels like it should mean the same as `tuple[a, b, c]` ("a
> tuple with 3 items of types resp. a, b and c") but that's at odds with
> the same type-checking functions.

Note that in Python 3 you can override isinstance, by defining
__instancecheck__ in the class:
http://docs.python.org/3/reference/datamodel.html?highlight=__instancecheck__#class.__instancecheck__

So it shouldn't be a problem to make isinstance(42, Int) work.

We can also make things like List[Int] and Dict[Str, Float] work, and
even rig it so that

isinstance([1, 2, 3], List[Int]) == True

while

isinstance([1, 2, 'booh'], List[Int]) == False

Of course there are many bikeshedding topics like whether we should
ever write List -- maybe we should write Iterable or Sequence instead,
and maybe we have to be able to express mutability, and so on. The
numeric tower (PEP 3141) is also good to keep in mind. I think that's
all solvable once we start experimenting a bit.

Some important issues to bikeshed over:

- Tuples. Sometimes you want to say e.g. "a tuple of integers, don't
mind the length"; other times you want to say e.g. "a tuple of fixed
length containing an int and two strs". Perhaps the former should be
expressed using ImmutableSequence[Int] and the second as Tuple[Int,
Str, Str].

- Unions. We need a way to say "either X or Y". Given that we're
defining our own objects we may actually be able to get away with
writing e.g. "Int | Str" or "Str | List[Str]", and isinstance() would
still work. It would also be useful to have a shorthand for "either T
or None", written as Optional[T] or Optional(T).

- Whether to design notations to express other constraints. E.g.
"integer in range(10, 100)", or "one of the strings 'r', 'w' or 'a'",
etc. You can go crazy on this.

- Composability (Nick's pet peeve, in that he is against it). I
propose that we reserve plain tuples for this. If an annotation has
the form "x: (P, Q)" then that ought to mean that x must conform to
both P and Q. Even though Nick doesn't like this, I don't think we
should do everything with decorators. Surly, the decorators approach
is good for certain use cases, and should take precedence if it is
used. But e.g. IDEs that use annotations for suggestions and
refactoring should not require everything to be decorated -- that
would just make the code too busy.

- Runtime enforcement. What should we use type annotations for? IDEs,
static checkers (linters) and refactoring tools only need the
annotations when they are parsing the code. While it is tempting to
invent some kind of runtime checking that automatically checks the
actual types against the annotations whenever a function is called, I
think this is rarely useful, and often prohibitively slow. So I'd say
don't focus on this. Instead, explicit type assertions like "assert
isinstance(x, List[Int])" might be used, sparingly, for those cases
where we'd otherwise write a manual assertion with the same meaning
(which is also sparingly!). A decorator to do this might be useful
(especially if there's a separate mechanism for turning actual
checking on or off through some configuration mechanism).

> The first could be fixable by relaxing slightly the constraints of
> isinstance and issubclass, but not so for the second.
>
> [0] which works rather neatly for anonymous unions as `|` is the union
> of two sets, so the arithmetic would be `type | type -> typeset`,
> `type | typeset -> typeset` and `typeset | typeset -> typeset`,
> libraries could offer opaque types/typesets which would be composable
> without their users having to know whether they're type atoms or
> typesets

I like this for declaring union types. I don't like it for composing
constraints that are intended for different tools.

--
--Guido van Rossum (python.org/~guido)

[Ben Hoyt]

> - Tuples. Sometimes you want to say e.g. "a tuple of integers, don't
> mind the length"; other times you want to say e.g. "a tuple of fixed
> length containing an int and two strs". Perhaps the former should be
> expressed using ImmutableSequence[Int] and the second as Tuple[Int,
> Str, Str].

Nice, that seems very explicit. ImmutableSequence is long, but clear.
In this specific case, should it be just Sequence, and a mutable one
would be MutableSequence (to be consistent with collections.abc
names?).

> - Unions. We need a way to say "either X or Y". Given that we're
> defining our own objects we may actually be able to get away with
> writing e.g. "Int | Str" or "Str | List[Str]", and isinstance() would
> still work. It would also be useful to have a shorthand for "either T
> or None", written as Optional[T] or Optional(T).

Definitely useful to have a notation for "either T or None", as it's a
pretty heavily-used pattern. But what about using the same approach,
something like "T | None" or "T | NoneType". Though if you use the
real None rather than experimental_type_annotations.None, is that
confusing? In any case, it seems unnecessary to have a special
Optional(T) notation when you've already got the simple "T1 | T2"
notation.

> - Whether to design notations to express other constraints. E.g.
> "integer in range(10, 100)", or "one of the strings 'r', 'w' or 'a'",
> etc. You can go crazy on this.

Yes, I think this is dangerous territory -- it could get crazy very
fast. Statically typed languages don't have this. Then again, I guess
type annotations have the potential to be *more* powerful in this
regard. Still, it'd have to be an awfully nice and general notation
for it to be useful. Even then, your "def" line complete with
type/constraint annotations may get far too long to be readable...

-Ben

[Bruce Leban]

On Wed, Dec 5, 2012 at 11:22 AM, Guido van Rossum <gu...@python.org> wrote:

- Unions. We need a way to say "either X or Y". Given that we're
defining our own objects we may actually be able to get away with
writing e.g. "Int | Str" or "Str | List[Str]", and isinstance() would
still work. It would also be useful to have a shorthand for "either T
or None", written as Optional[T] or Optional(T).

Optional is not the same as "or None" to me:

Dict(a=Int, b=Int | None, c=Optional(Int))

suggests that b is required but might be None while c is not required, i.e., {'a': 3, b: None} is allowed while {'a': 3, c: None} is not. 

Ditto for Tuples:

Tuple[Int, Str | None, Optional(Int)]

where (3, None) matches as does (3, 'a', 4) but not (3, None, None).

Optionals might be restricted to the end as matching in the middle would be complicated and possibly error-prone:

Tuple[Int, Optional(Int | None), Int | Str, Int | None]

--- Bruce

Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com

[Aaron Meurer]

> - Runtime enforcement. What should we use type annotations for?

There's the original example that started this whole thread, which is
tab completion annotations for IPython. The example that started that
thread was to annotate a function argument as taking files ending in
.txt, so that func(<TAB> would tab complete (in IPython) all .txt
files and all directories in the current directory. This is a simple
example, but even it is sufficiently complicated to suggest something
that should work at run time, not parsing time (or at least, the
annotation objects must be actually run, supposing e.g, that you want
to hook this into an IDE as well). In other words, something like

def loadfile(file: File(extension=".txt"):
...

Could probably be handled by a parser, but if the constriction gets
more complicated, or dynamic at all, the general case will just be

def func(arg: TabCompletionHintsObject()):
...

Meaning that TabCompletionHintsObject() has to be run dynamically.

This is similar to argument checking, but a little different. For
argument checking, you are trying to see if the arguments are valid.
For tab completion, you are trying to gather a list of all possible
(or reasonably possible) arguments, in the hopes that that list will
be small enough to help you type the code faster. Just like true type
checking is impossible due to the dynamic nature of the language, so
is true tab completion in just a parsing stage, because e.g.,

def func(arg: int):
...

might mean, "tab complete against all currently defined variables that
have the type int".

Aaron Meurer

[Masklinn]

On 2012-12-05, at 20:22 , Guido van Rossum wrote:
>
>> The most bothersome part is that I "feel" "either X or Y" (aka `X | Y`)
>> should be a set of type (and thus the same as {X, Y}[0]) but that doesn't
>> work with `isinstance` or `issubclass`. Likewise, `(a, b, c)` in an
>> annotation feels like it should mean the same as `tuple[a, b, c]` ("a
>> tuple with 3 items of types resp. a, b and c") but that's at odds with
>> the same type-checking functions.
>
> Note that in Python 3 you can override isinstance, by defining
> __instancecheck__ in the class:
> http://docs.python.org/3/reference/datamodel.html?highlight=__instancecheck__#class.__instancecheck__
>
> So it shouldn't be a problem to make isinstance(42, Int) work.

My problem there was more about having e.g. Int | Float return a set,
but isinstance not working with a set. But indeed it could return a
TypeSet which would implement __instancecheck__.

> - Tuples. Sometimes you want to say e.g. "a tuple of integers, don't
> mind the length"; other times you want to say e.g. "a tuple of fixed
> length containing an int and two strs". Perhaps the former should be
> expressed using ImmutableSequence[Int] and the second as Tuple[Int,
> Str, Str].



> - Unions. We need a way to say "either X or Y". Given that we're
> defining our own objects we may actually be able to get away with
> writing e.g. "Int | Str" or "Str | List[Str]", and isinstance() would
> still work. It would also be useful to have a shorthand for "either T
> or None", written as Optional[T] or Optional(T).

Well if `|` is the "union operator", as Ben notes `T | None` works well,
is clear and is sufficient. Though that's if and only if "Optional[T]"
is equivalent to "T or None" which Bruce seems to disagree with. There's
some history with this pattern:
http://journal.stuffwithstuff.com/2010/08/23/void-null-maybe-and-nothing/
(bottom section, from "Or Some Other Solution")

> - Whether to design notations to express other constraints. E.g.
> "integer in range(10, 100)", or "one of the strings 'r', 'w' or 'a'",
> etc. You can go crazy on this.

Yes this is going in Oleg territory, a sound core is probably a
good starting idea. Although basic enumerations ("one of the strings
'r', 'w' or 'a'") could be rather neat.

> - Composability (Nick's pet peeve, in that he is against it). I
> propose that we reserve plain tuples for this. If an annotation has
> the form "x: (P, Q)" then that ought to mean that x must conform to
> both P and Q. Even though Nick doesn't like this, I don't think we
> should do everything with decorators. Surly, the decorators approach
> is good for certain use cases, and should take precedence if it is
> used. But e.g. IDEs that use annotations for suggestions and
> refactoring should not require everything to be decorated -- that
> would just make the code too busy.
>
> - Runtime enforcement. What should we use type annotations for? IDEs,
> static checkers (linters) and refactoring tools only need the
> annotations when they are parsing the code.

For IDEs, that's pretty much all the time though, either they're parsing
the code or they're trying to perform static analysis on it, which uses
the annotations.

> While it is tempting to
> invent some kind of runtime checking that automatically checks the
> actual types against the annotations whenever a function is called, I
> think this is rarely useful, and often prohibitively slow.

Could be useful for debug or testing runs though, in the same way
event-based profilers are prohibitively slow and can't be enabled all
the time but are still useful. Plus it might be possible to
enable/disable this mechanism with little to no source modification via
sys.setprofile (I'm not sure what hooks it provides exactly and the
documentation is rather sparse, so I'm not sure if the function object
itself is available to the setprofile callback, looking at
Lib/profiler.py it might only get the code object).

[Guido van Rossum]

Those are not the semantics I had in mind for Optional.

[Guido van Rossum]

On Wed, Dec 5, 2012 at 12:34 PM, Masklinn <mask...@masklinn.net> wrote:
> On 2012-12-05, at 20:22 , Guido van Rossum wrote:
>>
>>> The most bothersome part is that I "feel" "either X or Y" (aka `X | Y`)
>>> should be a set of type (and thus the same as {X, Y}[0]) but that doesn't
>>> work with `isinstance` or `issubclass`. Likewise, `(a, b, c)` in an
>>> annotation feels like it should mean the same as `tuple[a, b, c]` ("a
>>> tuple with 3 items of types resp. a, b and c") but that's at odds with
>>> the same type-checking functions.
>>
>> Note that in Python 3 you can override isinstance, by defining
>> __instancecheck__ in the class:
>> http://docs.python.org/3/reference/datamodel.html?highlight=__instancecheck__#class.__instancecheck__
>>
>> So it shouldn't be a problem to make isinstance(42, Int) work.
>
> My problem there was more about having e.g. Int | Float return a set,
> but isinstance not working with a set. But indeed it could return a
> TypeSet which would implement __instancecheck__.

Right, that's what I meant.

>> - Tuples. Sometimes you want to say e.g. "a tuple of integers, don't
>> mind the length"; other times you want to say e.g. "a tuple of fixed
>> length containing an int and two strs". Perhaps the former should be
>> expressed using ImmutableSequence[Int] and the second as Tuple[Int,
>> Str, Str].
>
>
>
>> - Unions. We need a way to say "either X or Y". Given that we're
>> defining our own objects we may actually be able to get away with
>> writing e.g. "Int | Str" or "Str | List[Str]", and isinstance() would
>> still work. It would also be useful to have a shorthand for "either T
>> or None", written as Optional[T] or Optional(T).
>
> Well if `|` is the "union operator", as Ben notes `T | None` works well,
> is clear and is sufficient. Though that's if and only if "Optional[T]"
> is equivalent to "T or None" which Bruce seems to disagree with. There's
> some history with this pattern:
> http://journal.stuffwithstuff.com/2010/08/23/void-null-maybe-and-nothing/
> (bottom section, from "Or Some Other Solution")

Actually, I find "T|None" somewhat impure, since None is not a type
but a value. If you were allow this, what about "T|False"? And then
what about "True|None"? (There's no way to make the latter work!) And
I think "T|NoneType" is obscure; hence my proposal of Optional(T).
(Not Optional[T], since Optional is not a type.)

>> - Whether to design notations to express other constraints. E.g.
>> "integer in range(10, 100)", or "one of the strings 'r', 'w' or 'a'",
>> etc. You can go crazy on this.
>
> Yes this is going in Oleg territory, a sound core is probably a
> good starting idea. Although basic enumerations ("one of the strings
> 'r', 'w' or 'a'") could be rather neat.
>
>> - Composability (Nick's pet peeve, in that he is against it). I
>> propose that we reserve plain tuples for this. If an annotation has
>> the form "x: (P, Q)" then that ought to mean that x must conform to
>> both P and Q. Even though Nick doesn't like this, I don't think we
>> should do everything with decorators. Surly, the decorators approach
>> is good for certain use cases, and should take precedence if it is
>> used. But e.g. IDEs that use annotations for suggestions and
>> refactoring should not require everything to be decorated -- that
>> would just make the code too busy.
>>
>> - Runtime enforcement. What should we use type annotations for? IDEs,
>> static checkers (linters) and refactoring tools only need the
>> annotations when they are parsing the code.
>
> For IDEs, that's pretty much all the time though, either they're parsing
> the code or they're trying to perform static analysis on it, which uses
> the annotations.

Yeah, they're parsing it, but they're not executing it.

>> While it is tempting to
>> invent some kind of runtime checking that automatically checks the
>> actual types against the annotations whenever a function is called, I
>> think this is rarely useful, and often prohibitively slow.
>
> Could be useful for debug or testing runs though, in the same way
> event-based profilers are prohibitively slow and can't be enabled all
> the time but are still useful. Plus it might be possible to
> enable/disable this mechanism with little to no source modification via
> sys.setprofile (I'm not sure what hooks it provides exactly and the
> documentation is rather sparse, so I'm not sure if the function object
> itself is available to the setprofile callback, looking at
> Lib/profiler.py it might only get the code object).

Hence my idea of using a decorator to enable this on specific functions.

--
--Guido van Rossum (python.org/~guido)

[Bruce Leban]

On Wed, Dec 5, 2012 at 3:01 PM, Guido van Rossum <gu...@python.org> wrote:

Those are not the semantics I had in mind for Optional.

I know that. My point was that the standard meaning of the word optional is that something may or may not be given (or whatever the applicable verb is). That's quite different from saying it must be provided but may be None. Since you invited a bit of bikeshedding, I felt it was appropriate to point that out and then I got distracted by discussing the alternative that you weren't talking about. Sorry that was confusing.

In C#, this is called Nullable and you can write Nullable<String> to indicate the type (String or null type). The shorthand for that is String?.

If you want a shorthand to specify that None is allowed, I'd suggest ~Str.

--- Bruce

P.S. Optional[T] is not literally a shorthand for T | None as the former is 11 characters and the latter is 10 characters even if we include and count the spaces. :-)

P.P.S. I don't think Str | None rather than Str | NoneType is confusing.

[Nick Coghlan]

On Thu, Dec 6, 2012 at 5:22 AM, Guido van Rossum <gu...@python.org> wrote:

- Composability (Nick's pet peeve, in that he is against it). I
propose that we reserve plain tuples for this. If an annotation has
the form "x: (P, Q)" then that ought to mean that x must conform to
both P and Q. Even though Nick doesn't like this, I don't think we
should do everything with decorators. Surly, the decorators approach
is good for certain use cases, and should take precedence if it is
used. But e.g. IDEs that use annotations for suggestions and
refactoring should not require everything to be decorated -- that
would just make the code too busy.

I'm not against using composition within a particular set of annotation semantics, I'm against developing a convention for arbitrary composition of annotations with *different* semantics.

Instead, I'm advocating for the following guidelines to avoid treading on each others toes when experimenting with annotations and to leave scope for us to define standard annotation semantics at a future date:

1. Always use a decorator that expresses the annotation semantics in use (e.g. tab completion, type descriptions, parameter documentation)
2. Always *move* the annotations out to purpose-specific storage as part of the decorator (don't leave them in the annotations storage)
3. When analysing a function later, use only the purpose-specific attribute(s), not the raw annotations storage
4. To support composition with other sets of annotation semantics, always provide an alternate API that accepts the per-parameter details directly (e.g. by name or index) rather than relying solely on the annotations

The reason for this is so that if, at some future point in the time, python-dev agrees to bless some particular set of semantics as *the* meaning of function annotations (such as the type hinting system being discussed), then that won't break anything. Otherwise, if people believe that it's OK for them to simply assume that the contents of the annotations mean whatever they mean for their particular project, then it *will* cause problems further down the road as annotations written for one set of semantics (e.g. tab completion, parameter documentation) get interpreted by a processor expecting different semantics (e.g. type hinting).

Here's how your example experiment would look under such a scheme:

    from experimental_type_annotations import type_hints, Int, Str, Float

    # After type_hints runs, foo.__annotations__ would be empty, and the type
    # hinting data would instead be stored in (e.g.) a foo._type_hints attribute.
    @type_hints

    def foo(a: Int, b: Str) -> Float:
        <blah>

This is then completely clear and unambigious:
- readers can see clearly that these annotations are intended as type hints
- the type hinting processor can see that there *is* type hinting information available, due to the presence of a _type_hints attribute
- other automated processors see that there are no "default" annotations (which is good, since there is currently no such thing as "default" annotation semantics)

Furthermore, (as noted elsewhere in the thread) an alternate API can then easily be provided that supports composition with other annotations:

    @type_hints(Int, Str, _return=Float)
    def foo(a, b):
        <blah>

[Guido van Rossum]

Hi Nick,

I understand your position completely (and I did before). I just disagree. :-)

I think that requiring the experiment I am proposing to use a
decorator on each function that uses it (rather than just an import at
the top of the module) will cause too much friction, and the
experiment won't get off the ground. That's why I am proposing a
universal composition convention:

When an annotation for a particular argument is a tuple, then any
framework or decorator that tries to assign meanings to annotations
must search the items of the tuple for one that it can understand. For
the experimental type annotation system I am proposing this should be
simple enough -- the type annotation system can require that the
things it cares about must all be subclasses of a specific base class
(let's call it TypeConstraint). If the annotation is not a tuple, it
should be interpreted as a singleton tuple.

Yes, it is possible that a mistake leaves an annotation unclaimed. But
that's no worse than currently, where all annotations are ignored. And
for TypeConstraint there is no runtime behavior anyway (unless you
*do* add a decorator) -- its annotations are there for other tools to
parse and interpret. It's like pylint directives -- if you
accidentally misspell it 'pylnt' you get no error (but you may still
notice that something's fishy, because when pylint runs it doesn't
suppress the thing you tried to suppress :-).

--Guido

[David Townshend]

On Wed, Dec 5, 2012 at 8:17 PM, Guido van Rossum <gu...@python.org> wrote:

On Tue, Dec 4, 2012 at 1:37 AM, David Townshend <aquav...@gmail.com> wrote:
> Just thought of a couple of usages which don't fit into the decorator model.
> The first is using the return annotation for early binding:
>
>     def func(seq) -> dict(sorted=sorted):
>         return func.__annotations__['return']['sorted'](seq)

You've got to be kidding... 

> Stangely enough, this seems to run slightly faster than
>
>     def func(seq, sorted=sorted):
>         return sorted(seq)
>
> My test shows the first running in about 0.376s and the second in about
> 0.382s (python 3.3, 64bit).

Surely that's some kind of random variation. It's only a 2% difference.

It's consistent.  I ran several tests and came out with the same 2% difference every time.

IOW, this is not a line of thought to pursue.

I wasn't suggesting that this is a good idea, I was merely trying to point out that there are currently ways of using annotations beyond type declarations with decorators, and that there may be other use cases out there which will work well.  Documenting recommendations that annotations only be used with decorators, or only be used for type declarations will limit the possibilities because nobody will bother to look further, and if they do, the ideas will no doubt be shut down as being bad style because they go against the recommended usage.  I thought that limiting annotations like this was what you wanted to avoid?

Having said that, I've never found a good use for annotations in my own code, so I'm not emotionally invested one way or the other.  I do think that the best usage I've seen is exactly what is being discussed here and it would be great if there was some prescribed use for annotations.  Perhaps people would actually use them then.

David

[Masklinn]

On 2012-12-06, at 00:06 , Guido van Rossum wrote:
>
>>> - Unions. We need a way to say "either X or Y". Given that we're
>>> defining our own objects we may actually be able to get away with
>>> writing e.g. "Int | Str" or "Str | List[Str]", and isinstance() would
>>> still work. It would also be useful to have a shorthand for "either T
>>> or None", written as Optional[T] or Optional(T).
>>
>> Well if `|` is the "union operator", as Ben notes `T | None` works well,
>> is clear and is sufficient. Though that's if and only if "Optional[T]"
>> is equivalent to "T or None" which Bruce seems to disagree with. There's
>> some history with this pattern:
>> http://journal.stuffwithstuff.com/2010/08/23/void-null-maybe-and-nothing/
>> (bottom section, from "Or Some Other Solution")
>
> Actually, I find "T|None" somewhat impure, since None is not a type
> but a value. If you were allow this, what about "T|False"? And then
> what about "True|None"? (There's no way to make the latter work!) And
> I think "T|NoneType" is obscure; hence my proposal of Optional(T).
> (Not Optional[T], since Optional is not a type.)

Why would Optional not be a type? It's coherent with Option or Maybe types
in languages with such features, or C#'s Nullable.

[Andrew Svetlov]

On Wed, Dec 5, 2012 at 9:22 PM, Guido van Rossum <gu...@python.org> wrote:
> - Unions. We need a way to say "either X or Y". Given that we're
> defining our own objects we may actually be able to get away with
> writing e.g. "Int | Str" or "Str | List[Str]", and isinstance() would
> still work. It would also be useful to have a shorthand for "either T
> or None", written as Optional[T] or Optional(T).

Just to note: there are https://github.com/Deepwalker/trafaret library
intended for checking on complex enough structures.

[rand...@fastmail.us]

On Thu, Dec 6, 2012, at 3:43, Masklinn wrote:
> Why would Optional not be a type? It's coherent with Option or Maybe
> types in languages with such features, or C#'s Nullable.

C#'s Nullable doesn't really work outside a static typing system - when
you assign a Nullable to an 'object' or a 'dynamic', you get either the
original type (e.g. Int32) or a null reference (which has no type). It's
a real type only as far as the static typing system goes: it can be the
type of a field or a local variable, it _cannot_ be the type of an
object on the heap.

And since python doesn't have static typing...

[Daniel Wong]

This proposal looks great. The only thing is that I don't understand the point of annotations in the first place, since Python has decorators. As the last part of your post describes, decorators can be used to do the same thing. With decorators, it is even possible to use annotation-like syntax:

def defaults_as_parameter_metadata(f):

  names, args_name, kwargs_name, defaults = inspect.getargspec(f) 

  assert len(names) == len(defaults)  # To keep this example simple...

  f.parameter_metadata = {}
  for name, meta in zip(names, defaults):

    f.parameter_metadata[name] = meta

  f.__defaults__ = ()  # Again, for simplicity.

  return f

@defaults_as_parameter_metadata
def make_ice_cream(flavor=(options('vanilla', 'chocolate', ...),
                           str, "What kind of delicious do you want?"),

                   quantity=(positive, double,

                             "How much (in pounds) do you want?")):

  ...

I know this addresses a different issue, but I was directed to this thread from an answer that I got on StackOverflow, and this thread seems related enough. Sorry if I'm going off the rails here.

On Saturday, December 1, 2012 4:28:50 AM UTC-8, Thomas Kluyver wrote:

Function annotations (PEP 3107) are a very interesting new feature, but so far have gone largely unused. The only project I've seen using them is plac, a command-line option parser. One reason for this is that because function annotations can be used to mean anything, we're wary of doing anything in case we interfere with some other use case. A recent thread on ipython-dev touched on this [1], and we'd like to suggest some conventions to make annotations useful for everyone.

1. Code inspecting annotations should be prepared to ignore annotations it can't understand.

2. Code creating annotations should use wrapper classes to indicate what the annotation means. For instance, we are contemplating a way to specify options for a parameter, to be used in tab completion, so we would do something like this:

from IPython.core.completer import options
def my_io(filename, mode: options('read','write') ='read'):
    ...

3. There are a couple of important exceptions to 2:
- Annotations that are simply a string can be used like a docstring, to be displayed to the user. Inspecting code should not expect to be able to parse any machine-readable information out of these strings.
- Annotations that are a built-in type (int, str, etc.) indicate that the value should always be an instance of that type. Inspecting code may use these for type checking, introspection, optimisation, or other such purposes. Note that for now, I have limited this to built-in types, so other types can be used for other purposes, but this could be extended. For instance, the ABCs from collections (collections.Mapping et al.) could well be added to this category.

4. There should be a convention for attaching multiple annotations to one value. I propose that all code using annotations expects to handle tuples/lists of annotations. (We also considered dictionaries, but the result is long and ugly). So in this definition:

def my_io(filename, mode: (options('read','write'), str, 'The mode in which to open the file') ='read'):
    ...

the mode parameter has a set of options (ignored by frameworks that don't recognise it), should always be a string, and has a description.

Any thoughts and suggestions are welcome.

As an aside, we may also create a couple of decorators to fill in __annotations__ on Python 2, something like:

@return_annotation('A file obect')
@annotations(mode=(options('read','write'), str, 'The mode in which to open the file'))
def my_io(filename, mode='read'):
    ...

[1] http://mail.scipy.org/pipermail/ipython-dev/2012-November/010697.html


Thanks,
Thomas