[Python-Dev] Type hints -- a mediocre programmer's reaction

[Harry Percival]

Hi all,

tldr; type hints in python source are scary. Would reserving them for stub files be better?

For people that don't know me (most of you I think), I don't have a long experience of programming (perhaps 5 years, barring a bit of messing about with BASIC in the 80s), I've never made any commits on cPython, and so I don't speak from a great height of experience.  I am very much a "mediocre programmer" (TM).  I wouldn't have felt it appropriate to email python-dev about this, except that one of the members encouraged me to do so -- perhaps they felt there hadn't been enough "balance" in discussions to date.  Also, I've had enough positive reactions in person and on the twitters from people well-known in the Python community, to make me feel like maybe the issue is at least worth discussing...  And I do have some experience with teaching Python to beginners, enough to worry about anything I think might make their lives more difficult.

So, in outline:

I think:

- type hints are ugly

- they make the language harder to understand

- particularly for beginners

- the defense that "they're optional" doesn't really work, for several reasons.

- but maybe there's a way to keep them and their benefits, without incurring the above costs

My first reaction to type hints was "yuck", and I'm sure I'm not the only one to think that.  viz (from some pycon slides):

    def zipmap(f: Callable[[int, int], int], xx: List[int],
               yy: List[int]) -> List[Tuple[int, int, int]]:

arg.  and imagine it with default arguments.

Of course, part of this reaction is just a knee-jerk reaction to the new and unfamiliar, and should be dismissed, entirely justifiably, as mere irrationality.  But I'm sure sensible people agree that they do make our function definitions longer, more complex, and harder to read.

No doubt this has occurred to everyone that's been working on them.  There is a cost. But the benefits make it worthwhile.

I don't want to spend too long debating the benefits -- Guido gave an outline of them at Pycon, and I know y'all wouldn't be doing all this work for no reason.  All I will say is -- it sounds like the people that will benefit are Google and other "Enterprise" users, IDE vendors, and the people that will pay for it in sweat and confusion are beginners and John Q. Mediocre Programmer.

But what I really want to dwell on are the costs. 

I've heard this argument that, because the type hints are optional, they're not something that beginners or the average user needs to worry about. Beginners won't need to learn them, and they'll probably never see them.  And average users won't need them either, so they don't need to worry about them either.  So the costs are minimal!  I should relax.

I'm not so sure.  My worry is that once type hinting gets standardised, then they will become a "best practice", and there's a particular personality type out there that's going to start wanting to add type hints to every function they write.  Similarly to mindlessly obeying PEP8 while ignoring its intentions, hobgoblin-of-little-minds style, I think we're very likely to see type hints appearing in a lot of python source, or a lot of pre-commit-hook checkers.  Pretty soon it will be hard to find any open source library code that doesn't have type hints, or any project style guide that doesn't require them.

It may not even be an irrational, cargo-cult thing -- they really may be paying dividends. But it does mean we will all have to wade through a bunch of type hints before we can understand any function.  Not the end of the world.  The extra effort may even help us understand our functions better.  But it's just that little bit uglier, just that little extra mental effort, just that little extra barrier that is going to mean some people, at the margin, just give up on learning programming, or switch to javascript, or whatever it'll be.

Now I'm aware that throwing out type hints altogether is unlikely to be a popular proposal at this stage.  So I'm casting around for an alternative.  And it seems to me that stub files might be the answer.

From what I understand, type hint files are files with the extension .pyi that provide stub versions of the functions in the matching .py files, but that contain the type hints, while the .py file doesn't.  In fact they're likely to be a very popular approach anyway, since they allow type hints for codebases that need to work under both 2 and 3.

That sounds like the best of both worlds to me.

- .py files stay beautiful, concise, and easy to read.

- beginners don't have to worry about wading through type definitions when they find themselves browsing someone else's source

- type information is available to the linters and static file checkers, so we get all the benefits.

Sounds great right?  Everybody will be happy!  So let's nail it down! If I was in charge, here's what I'd do:

* standardise the syntax for type hints in 3.5, as per PEP484

* but: recommend the use of stub files as the preferred place to store hints

* and: deprecate function annotations in the core language

* remove them from the core language altogether in 3.6

How about that?

HP

[Barry Warsaw]

On Apr 20, 2015, at 07:30 PM, Harry Percival wrote:

>tldr; type hints in python source are scary. Would reserving them for stub
>files be better?

I think so. I think PEP 8 should require stub files for stdlib modules and
strongly encourage them for 3rd party code.

Cheers,
-Barry
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Chris Kaynor]

On Mon, Apr 20, 2015 at 11:30 AM, Harry Percival <hj...@cantab.net> wrote:
> My first reaction to type hints was "yuck", and I'm sure I'm not the only
> one to think that. viz (from some pycon slides):
>
> def zipmap(f: Callable[[int, int], int], xx: List[int],
> yy: List[int]) -> List[Tuple[int, int, int]]:

My opinion of type hints is: they are great inline, when they are
simple. Cases like (I know its not a real-world case, but there are
plenty of similarly simple functions):
def add(a: int, b: int) -> int:
are fine. When you get more complicated examples, especially when they
involve complex types (list, tuple, callable, etc). In such cases, it
may make sense to predefine a type (I believe the typehints PEP
supports some form of typedef - I think just a simple assignment) and
use that instead of the full type name. This will generally make the
function definition MUCH easier to read.

> That sounds like the best of both worlds to me.
> - .py files stay beautiful, concise, and easy to read.
> - beginners don't have to worry about wading through type definitions when
> they find themselves browsing someone else's source
> - type information is available to the linters and static file checkers, so
> we get all the benefits.
>
> Sounds great right? Everybody will be happy! So let's nail it down! If I
> was in charge, here's what I'd do:
>
> * standardise the syntax for type hints in 3.5, as per PEP484
> * but: recommend the use of stub files as the preferred place to store hints
> * and: deprecate function annotations in the core language
> * remove them from the core language altogether in 3.6

The main drawback of using a stub file is that it is much more likely
to end up out-of-date, and thus useless at best, than if the types are
defined inline. The same issue applies to various alternative
proposals such as having the typehints in decorators, but to a lesser
degree. The same issue also applies to general comments as well.

I would say there are a few major reasons to use stub files:
1) You cannot put the annotations in the source, probably either
because the module is a C library, or you must support versions which
do not support annotations.
2) You must use annotations for other purposes.
3) The annotations are particularly complicated, and you cannot
provide nice names for the various types (likely, because the types
are just very difficult to name, such as in the zipmap example, though
that one might be easier in a more specific context).

Outside of those specific reasons, I would vastly prefer any
annotations to be included in the source, where they are much more
likely to be updated, rather than in a separate file, where they will
likely be forgotten.

[Tymoteusz Jankowski]

<voice-from-audience>

burn the witch..

</voice-from-audience>

More seriously.. +1 to Harry voice. Adding type hints to function code is so ugly that that i'm breaking silence and i'm expressing it here before you, so:

It's ugly

Perhaps this question was asked a million times, but why not docstrings, which seems to be more elegant and natural place for that?

I will not use it.

Damn, if i had reputation this could be a threat. ;)

Unsubscribe: https://mail.python.org/mailman/options/python-dev/tymoteusz.jankowski%40gmail.com

[Paul Moore]

On 20 April 2015 at 19:41, Barry Warsaw <ba...@python.org> wrote:
>>tldr; type hints in python source are scary. Would reserving them for stub
>>files be better?
>
> I think so. I think PEP 8 should require stub files for stdlib modules and
> strongly encourage them for 3rd party code.

Agreed. I have many of the same concerns as Harry, but I wouldn't have
expressed them quite as well. I'm not too worried about actually
removing annotations from the core language, but I agree that we
should create a strong culture of "type hints go in stub files" to
keep source files readable and clean.

On that note, I'm not sure "stub" files is a particularly good name.
Maybe "type files" would be better? Something that emphasises that
they are the correct place to put type hints, not a workaround.

Paul

[Ethan Furman]

On 04/20, Barry Warsaw wrote:
> On Apr 20, 2015, at 07:30 PM, Harry Percival wrote:
>
>>tldr; type hints in python source are scary. Would reserving them for stub
>>files be better?
>
> I think so. I think PEP 8 should require stub files for stdlib modules and
> strongly encourage them for 3rd party code.

+1

--
~Ethan~

[Łukasz Langa]

On Apr 20, 2015, at 11:30 AM, Harry Percival <hj...@cantab.net> wrote:

I think:

- type hints are ugly

Making them work with the current Python syntax was a challenge. Granted, the end result is not perfect. It can be improved *if* type hints prove to be generally useful and popular. This might not happen.

- they make the language harder to understand

- particularly for beginners

A counter-point to that would be that good APIs already expose this information but using informal languages in docstrings, pure comments or straight API documentation. This only standardizes the notation in a way that:

- let’s those annotations be accessed at runtime

- let’s the entire Python ecosystem learn to read them statically and use them for REPL, IDE completion and type checking

- moves them to a place where it’s less likely that they become outdated

    def zipmap(f: Callable[[int, int], int], xx: List[int],

               yy: List[int]) -> List[Tuple[int, int, int]]:

Yeah, so agreed, this is pretty busy. For such cases, reformatting makes it less confusing (see: Screenshot 1).

As already stated, you can also name the types, in which case they’re less busy. That’s a compomise, too, since they require indirection when reading.

No doubt this has occurred to everyone that's been working on them.  There is a cost. But the benefits make it worthwhile.

Yes, they start to read more transparently after a while.

It sounds like the people that will benefit are Google and other "Enterprise" users, IDE vendors, and the people that will pay for it in sweat and confusion are beginners and John Q. Mediocre Programmer.

Any startup that at the beginning bangs out code as fast as possible - and eventually *becomes successful* - will be happy to be able to introduce gradual typing as a measure of improving quality.

My worry is that once type hinting gets standardised, then they will become a "best practice”

For library authors and APIs, they might. There’s nothing alarming with that, as I said, for those use cases you already expect “type hints” in the form of docstrings, comments, API docs, tests, etc.

there's a particular personality type out there that's going to start wanting to add type hints to every function they write. Similarly to mindlessly obeying PEP8 while ignoring its intentions, hobgoblin-of-little-minds style, I think we're very likely to see type hints appearing in a lot of python source, or a lot of pre-commit-hook checkers.

Yes, this is going to happen sometimes. If you can’t fight a hobgoblin in your community/organization, you’ve got bigger problems than ugly hints. Ultimately, we can’t optimize for the lowest common denominator.

Pretty soon it will be hard to find any open source library code that doesn't have type hints, or any project style guide that doesn't require them.

Harry, you must be the biggest Python 3 supporter ever <3 More seriously, no, this is not going to happen fast.

Sounds great right?  Everybody will be happy!  So let's nail it down!

Stub files have many downsides, too, unfortunately:

- we don’t *want* to have them, but we *need* to have them (C extensions, third-party modules, Python 2, …)

- they bring cognitive overhead of having to switch between two files

- they require the author to repeat himself quite a lot

- they might go out of date much easier than annotations in the function signature

- they can’t help with local variable inference

Since it was mentioned in a different e-mail in this thread: yes, the standard library is not getting any type annotations. When we decide to ship type hints with Python 3.6, they will be added as stubs.

As for the actual proposal:

* standardise the syntax for type hints in 3.5, as per PEP484

* but: recommend the use of stub files as the preferred place to store hints

* and: deprecate function annotations in the core language

* remove them from the core language altogether in 3.6

That is still pretty radical. I’ll leave the final words to the other authors of PEP 484 but from my perspective it’s not type annotations that are ugly, it’s the stubs. The annotation syntax might be improved for future releases. Stubs - by design - will always be inferior because of the reasons stated above.

-- 

Best regards,
Łukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

[Paul Moore]

On 20 April 2015 at 20:35, Łukasz Langa <luk...@langa.pl> wrote:

Since it was mentioned in a different e-mail in this thread: yes, the standard library is not getting any type annotations. When we decide to ship type hints with Python 3.6, they will be added as stubs.

Why is this? Surely this is something where the stdlib should "eat its own dogfood" and include inline hints rather than stub files?

Paul

[Mark Young]

Just another peanut from the gallery: I pretty much agree with everything that harry said. My current response to type annotations is "Yuck, that kills readability. I hope no code I ever have to read uses this.".

[Guido van Rossum]

Actually, "eat your own dogfood" is not one of the goals of the stdlib -- nor is it supposed to be an example of how to code. This is often misunderstood.  The stdlib contains a lot of Python code, and you can learn a lot from it, but good coding habits aren't generally something you learn there -- the code is crusty (some of the oldest Python code in existence lives in the stdlib!), often has to bend over backwards to support backward compatibility, and is riddled with performance hacks.

Based on some events in the distant past, there's actually an active ban against sweeping changes to the stdlib that attempt to "modernize" it or use new features -- because there is so much code in the stdlib, review of such sweeping (often near-mechanical) changes is inevitably less thorough than when a new feature is implemented, and even the best tests don't catch everything, so regressions in dark corners are often the result.

The only place in the stdlib where I expect inline type hints to be used is in brand new modules introduced in 3.6 or later, and then only when the author believes inline type hints to be clearer than wordy docstrings.

The situation is possibly even bleaker (or happier, depending on your position :-) for inline type hints in 3rd party packages -- few package authors will be satisfied with supporting only Python 3.5 and later. True, you can support Python 3.2 and up by declaring the 3rd party typing package as a dependency (unless Python 3.5+ is detected), but I don't expect this to become a popular approach overnight.

So I think the rumors of Python's death are greatly exaggerated.

--

--Guido van Rossum (python.org/~guido)

[Eric Snow]

On Mon, Apr 20, 2015 at 1:35 PM, Łukasz Langa <luk...@langa.pl> wrote:

Yeah, so agreed, this is pretty busy. For such cases, reformatting makes it less confusing (see: Screenshot 1).

While it helps, this sort of best-practice is still unsettled (and apparently not obvious).  In the short term it would make more sense to recommend using stub files for all the reason Harry enumerated.  Once the best practices are nailed down through experience with stub files, then we can make recommendations regarding inline type hints.

-eric

[Robert Collins]

On 21 April 2015 at 08:07, Guido van Rossum <gu...@python.org> wrote:

> The situation is possibly even bleaker (or happier, depending on your
> position :-) for inline type hints in 3rd party packages -- few package
> authors will be satisfied with supporting only Python 3.5 and later. True,
> you can support Python 3.2 and up by declaring the 3rd party typing package
> as a dependency (unless Python 3.5+ is detected), but I don't expect this to
> become a popular approach overnight.

mypy has a codec for 2.x which strips type annotations -
https://github.com/JukkaL/mypy/tree/master/mypy/codec - while you
can't run mypy under 2.x, you can run it under 3.x to perform the
analysis, and ones code still runs under 2.x.

Another route - the one I've been experimenting with as I get familiar
with mypy - is to just use type comments exclusively. Function type
comments currently break, but that seems like a fairly shallow bug to
me, rather than something that shouldn't work. The advantage of that
route is that editors which make comments appear in subtle colours,
makes the type hints be unobtrusive without specific syntax colouring
support.

-Rob


--
Robert Collins <rbtco...@hp.com>
Distinguished Technologist
HP Converged Cloud

[Robert Collins]

On 21 April 2015 at 08:10, Eric Snow <ericsnow...@gmail.com> wrote:
>
>
>
> While it helps, this sort of best-practice is still unsettled (and apparently not obvious). In the short term it would make more sense to recommend using stub files for all the reason Harry enumerated. Once the best practices are nailed down through experience with stub files, then we can make recommendations regarding inline type hints.
>
> -eric

Forgive my ignorance, but can stub files can't annotate variables
within functions? E.g. AIUI if there is a stub file, it is used in the
static analysis instead of the actual source. Likely I've got it
modelled wrong in my head :)

[Nikolaus Rath]

On Apr 20 2015, Harry Percival <hj...@cantab.net> wrote:
> My first reaction to type hints was "yuck", and I'm sure I'm not the only
> one to think that. viz (from some pycon slides):
>
> def zipmap(f: Callable[[int, int], int], xx: List[int],
> yy: List[int]) -> List[Tuple[int, int, int]]:
>
> arg. and imagine it with default arguments.

This is indeed ugly as hell and I certainly would not want to see this
in any Python file I'm working with.

However, I always assumed that whatever tools consume these annotations
are expected to be good enough at automatic type inference that
something like this would never be necessary?

In practice, I would hope that the above simplifies to

def zipmap(f, xx: List[int], yy: List[int]):

because (just picking a probably buggy random implementation)

zz = [] # --> z must be List[A]
for i in range(min(len(xx), len(yy))):
x = xx[i] # --> x must be int
y = xx[i] # --> y must be int
z = f(x,y) # --> f must be Callable[(int,int], B]
zz[i] = (x,y,z) # --> A must be Tuple[int,int,B]

return zz # --> return value must be List[Tuple[int,int,B]]

it doesn't catch that B = int, but I think that's acceptable.

Is this not the case? Are we really expecting people to write stuff like
the above?

Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Guido van Rossum]

On Mon, Apr 20, 2015 at 1:15 PM, Robert Collins <rob...@robertcollins.net> wrote:

On 21 April 2015 at 08:07, Guido van Rossum <gu...@python.org> wrote:

> The situation is possibly even bleaker (or happier, depending on your
> position :-) for inline type hints in 3rd party packages -- few package
> authors will be satisfied with supporting only Python 3.5 and later. True,
> you can support Python 3.2 and up by declaring the 3rd party typing package
> as a dependency (unless Python 3.5+ is detected), but I don't expect this to
> become a popular approach overnight.

mypy has a codec for 2.x which strips type annotations -
https://github.com/JukkaL/mypy/tree/master/mypy/codec - while you
can't run mypy under 2.x, you can run it under 3.x to perform the
analysis, and ones code still runs under 2.x.

I know, it was my idea. :-) But I think stubs are usually better of you want to support PY2. The experience when the codec is not installed is really poor.

 

Another route - the one I've been experimenting with as I get familiar
with mypy - is to just use type comments exclusively. Function type
comments currently break, but that seems like a fairly shallow bug to
me, rather than something that shouldn't work. The advantage of that
route is that editors which make comments appear in subtle colours,
makes the type hints be unobtrusive without specific syntax colouring
support.

There are definitely some tooling issues around annotations (e.g. I've had to file at least one bug with "pep8"), but in cases like this the language must lead, and tooling will follow. The idea of using comments exclusively is interesting, although I think there are also some real downsides. (As an example: Jython is thinking of making some use of type hints for code generation, and they strongly prefer to have the hints in the AST, whereas comments don't show up in the AST at all.)

[Stephen Hansen]

Sounds great right?  Everybody will be happy!  So let's nail it down! If I was in charge, here's what I'd do:

* standardise the syntax for type hints in 3.5, as per PEP484

* but: recommend the use of stub files as the preferred place to store hints

* and: deprecate function annotations in the core language

* remove them from the core language altogether in 3.6

Personally, I'm not all that keen on the verbosity of the syntax; I'm sad that List[int] has to be how you spell things instead of just [int], but I'm sure there's reasons that can't work.

That said, I hate stub files. I understand why they may be needed sometimes so accept them as a necessary evil, but that doesn't make them a goal to me. Stub files become "optional headers" which I'll have to keep in sync with the actual code -- which in my opinion, just about guarantees a maintenance burden that will fall by the side of the road. If I have to look at another file know or change the function arguments in the code I'm working on, that hurts readability, too.

Will it take some getting used to, this syntax? Yes. At one point I thought comprehensions and ternary expressions were unreadable. I use them all the time now and find them very elegant. I'm doubtful I'll ever find type hints /elegant/, but I'm pretty sure they won't be "ugly" forever. Ugly has a lot to do with familiarity. It's also deeply subjective.

But its an objective reality, imho, that having to maintain and sync up function definitions in *two different files* is a burden. And that is a burden I really don't want to deal with.

--Stephen

[Guido van Rossum]

On Mon, Apr 20, 2015 at 1:17 PM, Robert Collins <rob...@robertcollins.net> wrote:

On 21 April 2015 at 08:10, Eric Snow <ericsnow...@gmail.com> wrote:
>
>
>
> While it helps, this sort of best-practice is still unsettled (and apparently not obvious).  In the short term it would make more sense to recommend using stub files for all the reason Harry enumerated.  Once the best practices are nailed down through experience with stub files, then we can make recommendations regarding inline type hints.
>
> -eric

Forgive my ignorance, but can stub files can't annotate variables
within functions? E.g. AIUI if there is a stub file, it is used in the
static analysis instead of the actual source. Likely I've got it
modelled wrong in my head :)

-Rob

Correct, stub files are only used to type-check *users* of a module. If you want a module itself to be type-checked you have to use inline type hints. (Though it has been suggested to combine the hints from the stub with the implementation and use this to type-check the implementation, and some tool chains may actually implement this.)

[R. David Murray]

I wrote a longer response and then realized it didn't really add much to
the discussion. So let me be short: type annotations do *not* appeal to
me, and I am not looking forward to the cognitive overhead of dealing
with them. Perhaps I will eventually grow to like them if the tools
that use them really add value. You'll have to sell me on it, though.

On Mon, 20 Apr 2015 12:35:33 -0700, <luk...@langa.pl> wrote:
> Stub files have many downsides, too, unfortunately:
> - we don’t *want* to have them, but we *need* to have them (C extensions, third-party modules, Python 2, …)
> - they bring cognitive overhead of having to switch between two files
> - they require the author to repeat himself quite a lot
> - they might go out of date much easier than annotations in the function signature

The whole point of type hints is for the linters/IDEs, so IMO it is
perfectly reasonable to put the burden of making them useful onto the
linters/IDEs. The UI for it can unify the two files into a single
view...I know because way back in the dark ages I wrote a small
editor-based IDE that did something very analogous on an IBM Mainframe,
and it worked really well as a development environment.

--David

[Guido van Rossum]

Jukka (mypy's author) believes that specifying full annotations will catch more bugs, and makes the error messages easier to understand -- in practice when you let the type inferencer run free it will often infer bizarre union types or other complex forms, and the errors may well appear in the wrong spot. This is a common issue with type inferencing, and anyone who has tried to learn Haskell (or C++ :-) has plenty of experience with such errors.

The good news is that I don't actually expect people to have to write or even read stuff like the above; the example was quoted out of context. In code that the typical (mediocre :-) programmer writes, argument types won't be more complex than some built-in primitive type (e.g. int or str) or a List of primitive types, or perhaps a user-defined class or List of such. And you can always leave the annotation out if you find yourself fighting the type checker -- it will default to Any and the type checker will just shut up. That's the beauty of gradual typing (and it differs greatly from strict typing as seen in Haskell or C++).

[Isaac Morland]

On Mon, 20 Apr 2015, Paul Moore wrote:

> On 20 April 2015 at 19:41, Barry Warsaw <ba...@python.org> wrote:
>>> tldr; type hints in python source are scary. Would reserving them for stub
>>> files be better?
>>
>> I think so. I think PEP 8 should require stub files for stdlib modules and
>> strongly encourage them for 3rd party code.
>
> Agreed. I have many of the same concerns as Harry, but I wouldn't have
> expressed them quite as well. I'm not too worried about actually
> removing annotations from the core language, but I agree that we
> should create a strong culture of "type hints go in stub files" to
> keep source files readable and clean.
>
> On that note, I'm not sure "stub" files is a particularly good name.
> Maybe "type files" would be better? Something that emphasises that
> they are the correct place to put type hints, not a workaround.

How about "header" files?

(ducks...)

Isaac Morland CSCF Web Guru
DC 2619, x36650 WWW Software Specialist

[Harry Percival]

> stub files are only used to type-check *users* of a module. If you want a module itself to be type-checked you have to use inline type hints

is this a fundamental limitation, or just the current state of tooling?

On 20 April 2015 at 21:48, Harry Percival <harry.p...@gmail.com> wrote:

> "I hate stub files. [...] in my opinion, [it] just about guarantees a maintenance burden that will fall by the side of the road.

I'm not so pessimistic.  It's not like documentation or docstrings or comments -- the whole point is that it should be very easy to have an automated check for whether your stubs are in sync with your source, because both are in code.  Unlike docs or comments which can easily become out of date, because there's no automated process to tell you they need updating...  I'm thinking of it as a thing your editor will warn you of.  Like pyflakes warnings about unused variables & co, I'm never happy until I've silenced them all in a file, and similarly, your editor will keep bugging you until you've got your stubs inline with your code...

Unsubscribe: https://mail.python.org/mailman/options/python-dev/hjwp2%40cantab.net

--

------------------------------
Harry J.W. Percival
------------------------------
Twitter: @hjwp
Mobile:  +44 (0) 78877 02511
Skype:         harry.percival

--

------------------------------
Harry J.W. Percival
------------------------------
Twitter: @hjwp
Mobile:  +44 (0) 78877 02511
Skype:         harry.percival

[Harry Percival]

> "I hate stub files. [...] in my opinion, [it] just about guarantees a maintenance burden that will fall by the side of the road.

I'm not so pessimistic.  It's not like documentation or docstrings or comments -- the whole point is that it should be very easy to have an automated check for whether your stubs are in sync with your source, because both are in code.  Unlike docs or comments which can easily become out of date, because there's no automated process to tell you they need updating...  I'm thinking of it as a thing your editor will warn you of.  Like pyflakes warnings about unused variables & co, I'm never happy until I've silenced them all in a file, and similarly, your editor will keep bugging you until you've got your stubs inline with your code...

On 20 April 2015 at 20:37, Isaac Morland <ijmo...@uwaterloo.ca> wrote:

Unsubscribe: https://mail.python.org/mailman/options/python-dev/hjwp2%40cantab.net

[Robert Collins]

On 21 April 2015 at 08:50, Harry Percival <harry.p...@gmail.com> wrote:
>> stub files are only used to type-check *users* of a module. If you want a
>> module itself to be type-checked you have to use inline type hints
>
> is this a fundamental limitation, or just the current state of tooling?

AIUI its the fundamental design. Stubs don't annotate python code,
they *are* annotated code themselves. They aren't merged with the
observed code at all.

Could they be? Possibly. I don't know how much work that would be.

-Rob

--
Robert Collins <rbtco...@hp.com>
Distinguished Technologist
HP Converged Cloud

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Guido van Rossum]

On Mon, Apr 20, 2015 at 1:50 PM, Harry Percival <harry.p...@gmail.com> wrote:

> stub files are only used to type-check *users* of a module. If you want a module itself to be type-checked you have to use inline type hints

is this a fundamental limitation, or just the current state of tooling?

It's not fundamental, it's just more in line with the original purpose of stubs (to describe C extensions).

[Ryan Gonzalez]

Only if you want Java users burning all written copies of the PEP...

Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

--

Ryan

[ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong.

http://kirbyfan64.github.io/

 

[Harry Percival]

@Lukasz:

Of course you're right, ugly is a matter of perspective, and I'm sure I could grow to love them, and they might evolve into a more polished direction

> "they start to read more transparently after a while."

But I'm still worried about beginners, and I'm even worried about me.  I like to be able to scan through some code and see the essence of it.   I learned Java at school, and I got it figured out, but i'm glad I left it behind. Every so often I read a TDD book and the examples are all in java and it just feels like obfuscation -- public void static private String[] class blabla... so many keywords and types getting in the way of *what the code is actually doing*.  That's what's so appealing about Python, it strips it down to just the basics.  And, to me, type hints are always going to be some unnecessary chaff that gets in the way of my understanding -- not useless, not that they don't have a purpose or that we should remove them completely.  But if there was a way of just hiding them, so that I don't have to think about them, as a beginner, or as a normal programmer, most of the time, in the 90% of cases where I don't need to see them, I shouldn't have to...  that's why i'm so keen on this stub files idea.

One thing I don't understand is this "local variable inference" thing -- can that not be made to work in stub files?

[Ian Cordasco]

So I've generally stayed out of this but I feel there is some context that people are missing in general.

First, allow me to provide some context: I maintain a /lot/ of Python code[1] and nearly all of it is designed to be compatible with Pythons 2.6, 2.7, 3.2, 3.3, 3.4 (and eventually 3.5) and sometimes 2.5 (depending on the project). If I want to improve a developer's experience with some of that code using Type Hints I will essentially have no way to do that unless I write the code with the annotations and ship versions with annotations stripped and other versions with annotations? That's a lot more overhead. If I could provide the annotations in stubs that means that only the people who care about using them will have to use them.

Is it more overhead to manage twice the number of files? Yes. Do I feel it would be worth it to not overly complicate how these packages are released? Yes.

Further, there are far more reasons to make stubs the baseline (in my opinion) the biggest reason of all is that people want to provide stubs for popular yet unmaintained libraries as third party packages. Should everyone using PIL be using Pillow? Of course. Does that mean they'll migrate or be allowed to migrate? No. Should they be able to benefit from this? Yes the should. The only way for PIL users to be able to do that is if stub files can be packaged separately for PIL and distributed by someone else.

I think while the authors are currently seeing stubs as a necessary *evil* they're missing points where they're a better backwards compatible solution for people who want to give users with capable IDEs the ability to use stub (or hint) files.

Cheers,

Ian

[1]: I am a maintainer of flake8, requests, chardet, github3.py uritemplate.py, rfc3986, sqlobject, the requests toolbelt, a bunch of the requests auth plugins, and I'm a core developer on a small number of openstack projects.

[Łukasz Langa]

On Apr 20, 2015, at 3:02 PM, Ian Cordasco <graffatc...@gmail.com> wrote:

I think while the authors are currently seeing stubs as a necessary *evil* they're missing points where they're a better backwards compatible solution for people who want to give users with capable IDEs the ability to use stub (or hint) files.

We might have not chosen the wording that makes this obvious to you but we definitely did consider all of the points you’re referring to:

https://www.python.org/dev/peps/pep-0484/#stub-files

[Robert Collins]

On 21 April 2015 at 10:02, Ian Cordasco <graffatc...@gmail.com> wrote:
>
>

>
> So I've generally stayed out of this but I feel there is some context that
> people are missing in general.
>
> First, allow me to provide some context: I maintain a /lot/ of Python
> code[1] and nearly all of it is designed to be compatible with Pythons 2.6,
> 2.7, 3.2, 3.3, 3.4 (and eventually 3.5) and sometimes 2.5 (depending on the
> project). If I want to improve a developer's experience with some of that
> code using Type Hints I will essentially have no way to do that unless I
> write the code with the annotations and ship versions with annotations
> stripped and other versions with annotations? That's a lot more overhead. If
> I could provide the annotations in stubs that means that only the people who
> care about using them will have to use them.

2.5? I'm so sorry :).

Being in approximately the same boat, I definitely want to be able to
improve the developer experience.

That said, with one key exception (str/bytes/unicode) Python code
generally has the same type on all versions. Sure it might be imported
from somewhere else, and you're restricted to the common subset of
APIs, but the types in use don't vary per-python.

So - as long as your *developers* can run mypy on 3.2+, they can
benefit from type checking. mypy itself requires 3.2+ to run, but
programs with type annotations should be able to run on all those
python versions you mention.

Now, what is the minimum barrier for entry?

Nothing :) - at the moment every file can be analysed, and mypy ships
with a bunch of stubs that describe the stdlib. So - you'll get some
benefit immediately, where bad use of stdlib routines is happening.

Constraining the types of functions gets you better errors (because
you are expressing intent rather than what-might-happen which the
inference has to work from otherwise. In particular, constraining the
type of *inputs* can let bad callers be detected, rather than the
engine assuming they are valid-until-a-contradiction-occurs. You can
do that with stub files: put them in repo A, and add them to the
MYPYPATH when working on repo B which calls the code in repo A. You
can also add those stubs to repo B, but I wouldn't do that because
then they will skew vs repo A.

A further step up would be to annotate A in its code, rather than
using stubs. That way, developers of repo A will be warned about bad
uses of other repo A code.

But - if you have stubs *or* annotations in-line in repo A, *everyone*
changing repo A needs to care. Because if the code mismatches the
stub, the folk that do care will now be unable to use repo A correctly
- their type checker will complain about valid uses, and fail to
complain about more invalid uses.

I'm particularly interested in mypy for OpenStack because for some
repos > 10% of reported bugs are type mismatch errors which mypy may
well have avoided.

> Is it more overhead to manage twice the number of files? Yes. Do I feel it
> would be worth it to not overly complicate how these packages are released?
> Yes.

> Further, there are far more reasons to make stubs the baseline (in my
> opinion) the biggest reason of all is that people want to provide stubs for
> popular yet unmaintained libraries as third party packages. Should everyone
> using PIL be using Pillow? Of course. Does that mean they'll migrate or be
> allowed to migrate? No. Should they be able to benefit from this? Yes the
> should. The only way for PIL users to be able to do that is if stub files
> can be packaged separately for PIL and distributed by someone else.

stubs can certainly be packaged and distributed separately. That
doesn't make the case that we should use stubs for projects that are
opting in.

> I think while the authors are currently seeing stubs as a necessary *evil*
> they're missing points where they're a better backwards compatible solution
> for people who want to give users with capable IDEs the ability to use stub
> (or hint) files.

-Rob


--
Robert Collins <rbtco...@hp.com>
Distinguished Technologist
HP Converged Cloud

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Guido van Rossum]

On Mon, Apr 20, 2015 at 2:01 PM, Robert Collins <rob...@robertcollins.net> wrote:

On 21 April 2015 at 08:50, Harry Percival <harry.p...@gmail.com> wrote:
>> stub files are only used to type-check *users* of a module. If you want a
>> module itself to be type-checked you have to use inline type hints
>
> is this a fundamental limitation, or just the current state of tooling?

AIUI its the fundamental design. Stubs don't annotate python code,
they *are* annotated code themselves. They aren't merged with the
observed code at all.

Could they be? Possibly. I don't know how much work that would be.

It's fundamental in the implementation of mypy. It doesn't have to be in the implementation of other type checkers (and IIRC the Google folks are planning to merge the two streams). However if you are using typing.get_type_hints(func) this is not intended to give you access to hints defined in stubs (it would require a huge amount of machinery to implement that right).

[Harry Percival]

So I guess the main difference is that type annotations in stub files wouldn't be available at runtime?  Ie, they wouldn't magically appear in __annotations__ (unless the python interpreter itself started to evaluate stub files too)

[Harry Percival]

exactly.  yay stub files!  we all agree! everyone loves them!  boo type annotations inline in python source.  only some people love them.  and even then, only after a while, and only tentatively... and some people fear them, mightily...

[Jack Diederich]

Twelve years ago a wise man said to me "I suggest that you also propose a new name for the resulting language"

I talked with many of you at PyCon about the costs of PEP 484. There are plenty of people who have done a fine job promoting the benefits.

* It is not optional. Please stop saying that. The people promoting it would prefer that everyone use it. If it is approved it will be optional in the way that PEP8 is optional. If I'm reading your annotated code it is certainly /not/ optional that I understand the annotations.

* Uploading stubs for other people's code is a terrible idea. Who do I contact when I update the interface to my library? The random Joe who "helped" by uploading annotations three months ago and then quit the internet? I don't even want to think about people maliciously adding stubs to PyPI.

* The cognitive load is very high. The average function signature will double in length. This is not a small cost and telling me it is "optional" to pretend that every other word on the line doesn't exist is a farce.

* Every company's style guide is about to get much longer. That in itself is an indicator that this is a MAJOR language change and not just some "optional" add-on.

* People will screw it up. The same people who can't be trusted to program without type annotations are also going to be *writing* those type annotations.

* Teaching python is about to get much less attractive. It will not be optional for teachers to say "just pretend all this stuff over here doesn't exist"

* "No new syntax" is a lie. Or rather a red herring. There are lots of new things it will be required to know and just because the compiler doesn't have to change doesn't mean the language isn't undergoing a major change.

If this wasn't in a PEP and it wasn't going to ship in the stdlib very few people would use it. If you told everyone they had to install a different python implementation they wouldn't. This is much worse than that - it is Python4 hidden away inside a PEP.

There are many fine languages that have sophisticated type systems. And many bondage & discipline languages that make you type things three times to make really really sure you meant to type that. If you find those other languages appealing I invite you to go use them instead.

-Jack

https://mail.python.org/pipermail/python-dev/2003-February/033291.html

[Ryan Gonzalez]

Although I like the concept of type annotations and the PEP, I have to agree with this. If I saw these type annotations when learning Python (I'm self-taught), there's a 99% chance I would've freaked.

It's the same issue as with teaching C++: it's wrong to say, "Hey, I taught you the basics, but there's other stuff that's going to confuse you to a ridiculous extent when you read it." People can't ignore it. It'll become a normal part of Python programs.

At least now you can say, "I'm using the mypy type checker."

Don't get me wrong; I like mypy. I helped with their documentation and am watching the GitHub repo. But this is dead-on.

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

[R. David Murray]

+1 to this from me too. I'm afraid that means I'm -1 on the PEP.

I didn't write this in my earlier email because I wasn't sure about it,
but my gut reaction after reading Harry's email was "if type annotations
are used in the stdlib, I'll probably stop contributing". That doesn't
mean that's *true*, but that's the first time I've ever had that
thought, so it is probably worth sharing.

Basically, it makes Python less fun to program in. That may be be an
emotional reaction and irrational, but I think it matters. And yes, I
write production Python code for a living, though granted not at Google
or Facebook or Dropbox scale.

> Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com

[Chris Angelico]

On Tue, Apr 21, 2015 at 9:41 AM, Jack Diederich <jack...@gmail.com> wrote:
> * It is not optional. Please stop saying that. The people promoting it would
> prefer that everyone use it. If it is approved it will be optional in the
> way that PEP8 is optional. If I'm reading your annotated code it is
> certainly /not/ optional that I understand the annotations.
>

> * The cognitive load is very high. The average function signature will
> double in length. This is not a small cost and telling me it is "optional"
> to pretend that every other word on the line doesn't exist is a farce.
>
> * Every company's style guide is about to get much longer. That in itself is
> an indicator that this is a MAJOR language change and not just some
> "optional" add-on.

Maybe I'm completely misreading everything here, but I would have
thought that there are two completely different use-cases here:

1) Library authors
2) Application authors

When you're writing a library, it can be a great help to provide type
annotations, because every application that uses your library can
benefit. When you're writing an application, you can completely ignore
them, but still get the benefit of everyone else's.

Most company style guides are going to spend most of their time
dealing with application code. Maybe you'd put a few type annotations
on some of your internal library-like routines, but you certainly
don't need to adorn every single function parameter and return value
in code that's called only from elsewhere. Say you have an application
framework like Flask - you're going to be writing a bunch of functions
that you never call from your own code, but which get called by the
framework. Annotating their return values is almost completely useless
- you're not going to be checking Flask itself for type errors! And
the benefit of annotating their parameters is constrained to the
functions themselves, so you can take your pick whether or not you
annotate any particular function.

Type hints are on par with all those other structured function
signature tidbits - formatted docstrings, autodoc comments, etc, etc,
etc. Has any one of those become mandatory for all code? Nope. And
possibly the best answer to anyone who tries to demand type hints for
all code is TheDailyWTF, where you can find JavaDoc like this:

http://thedailywtf.com/articles/If_At_First_You_Don_0x27_t_Succeed
http://thedailywtf.com/articles/SelfDocumenting

I have no fears for my own code. Are you afraid for yours?

ChrisA

[Ben Finney]

Chris Angelico <ros...@gmail.com> writes:

> On Tue, Apr 21, 2015 at 9:41 AM, Jack Diederich <jack...@gmail.com> wrote:
> > * It is not optional. Please stop saying that. The people promoting
> > it would prefer that everyone use it. If it is approved it will be
> > optional in the way that PEP8 is optional. If I'm reading your
> > annotated code it is certainly /not/ optional that I understand the
> > annotations.

[…]
>
> Maybe I'm completely misreading everything here […]

I think you've misunderstood the complaint.

> When you're writing a library, it can be a great help to provide type
> annotations, because every application that uses your library can
> benefit. When you're writing an application, you can completely ignore
> them, but still get the benefit of everyone else's.

Jack is not complaining only about *writing* code. He's complaining
about the effect this will have on code that we all are expected to
*read*.

Programmers spend a great deal of time reading code written by other
people. The costs of this proposal are only partly on the writers of the
code; they are significantly borne by the people *reading* that code.
For them, it is not optional.

> I have no fears for my own code. Are you afraid for yours?

Jack, if I understand correctly, fears for the code that will be written
by others in conformance with this proposal, that he will then have to
read and understand.

--
\ “The opposite of a correct statement is a false statement. But |
`\ the opposite of a profound truth may well be another profound |
_o__) truth.” —Niels Bohr |
Ben Finney

[Chris Angelico]

On Tue, Apr 21, 2015 at 10:52 AM, Ben Finney <ben+p...@benfinney.id.au> wrote:
> Chris Angelico <ros...@gmail.com> writes:
>
>> On Tue, Apr 21, 2015 at 9:41 AM, Jack Diederich <jack...@gmail.com> wrote:
>> > * It is not optional. Please stop saying that. The people promoting
>> > it would prefer that everyone use it. If it is approved it will be
>> > optional in the way that PEP8 is optional. If I'm reading your
>> > annotated code it is certainly /not/ optional that I understand the
>> > annotations.
> […]
>>
>> Maybe I'm completely misreading everything here […]
>
> I think you've misunderstood the complaint.
>
>> When you're writing a library, it can be a great help to provide type
>> annotations, because every application that uses your library can
>> benefit. When you're writing an application, you can completely ignore
>> them, but still get the benefit of everyone else's.
>
> Jack is not complaining only about *writing* code. He's complaining
> about the effect this will have on code that we all are expected to
> *read*.

Ahh. Yes, that's a concern. When you go digging into that library to
find out how it works, yes, you'd be face-to-face with their type
annotations.

I doubt the worst-case complex ones are going to come up all that
often, though. Sure, you might declare that something returns a list
of dictionaries mapping tuples of integers and strings to list of sets
of atoms, but that's hardly common. (And chances are you can just
declare that it returns List[Dict] and have done with it.)

Maybe it'd be of value to have a quick "code stripper" that takes away
all the annotations, plus any other junk/framing that you're not
interested in, and gives you something you can browse in a text
editor? It could take away all the Sphinx adornments from docstrings,
any source control versioning markers, all that kind of thing. Then
you could read through the code in a simpler form, while still having
type annotations there for you if you need them.

ChrisA

[Nick Coghlan]

On 20 Apr 2015 14:44, "Barry Warsaw" <ba...@python.org> wrote:

>
> On Apr 20, 2015, at 07:30 PM, Harry Percival wrote:
>
> >tldr; type hints in python source are scary. Would reserving them for stub
> >files be better?
>
> I think so.  I think PEP 8 should require stub files for stdlib modules and
> strongly encourage them for 3rd party code.

+1

Having stub files take precedence over inline annotations would also neatly deal with the annotation compatibility problem (you can use a stub file to annotate code using annotations for another purpose).

Another point in favour of that approach: stub files could be evaluated holistically rather than statement at a time, permitting implicit forward references.

Cheers,
Nick.

>
> Cheers,
> -Barry

> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> https://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

[Steven D'Aprano]

On Mon, Apr 20, 2015 at 11:34:51PM +0100, Harry Percival wrote:
> exactly. yay stub files! we all agree! everyone loves them!

Not even close.


--
Steve

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Steven D'Aprano]

On Mon, Apr 20, 2015 at 07:30:39PM +0100, Harry Percival wrote:
> Hi all,

>
> tldr; type hints in python source are scary. Would reserving them for stub
> files be better?

No no no, a thousand times no it would not!

Please excuse my extreme reaction, but over on the python-list mailing
list (comp.lang.python if you prefer Usenet) we already had this
discussion back in January.

Anyone wishing to read those conversations should start here:

https://mail.python.org/pipermail/python-list/2015-January/697202.html

https://mail.python.org/pipermail/python-list/2015-January/697315.html

Be prepared for a long, long, long read. Nothing in your post hasn't
already been discussed (except for your proposal to deprecate
annotations altogether). So if you feel that I'm giving any of your
ideas or concerns short-shrift, I'm not, it's just that I've already
given them more time than I can afford. And now I get to do it all
again, yay.

(When reading those threads, please excuse my occasional snark towards
Rick, he is a notorious troll on the list and sometimes I let myself be
goaded into somewhat less than professional responses.)

While I sympathise with your thought that "it's scary", I think it is
misguided and wrong. As a thought-experiment, let us say that we roll
back the clock to 1993 or thereabouts, just as Python 1.0 (or so) was
about to be released, and Guido proposed adding *default values* to
function declarations [assuming they weren't there from the start]. If
we were used to Python's clean syntax:

def zipmap(f, xx, yy):

the thought of having to deal with default values:

def zipmap(f=None, xx=(), yy=()):

might be scary. Especially since those defaults could be arbitrarily
complex expressions. Twenty years on, what should we think about such
fears?

Type hinting or declarations are extremely common in programming
languages, and I'm not just talking about older languages like C and
Java. New languages, both dynamic and static, like Cobra, Julia,
Haskell, Go, Boo, D, F#, Fantom, Kotlin, Rust and many more include
optional or mandatory type declarations. You cannot be a programmer
without expecting to deal with type hints/declarations somewhere. As
soon as you read code written in other languages (and surely you do
that, don't you? you practically cannot escape Java and C code on the
internet) and in my opinion Python cannot be a modern language without
them.

I'm going to respond to your recommendation to use stub files in
another post (replying to Barry Warsaw), here I will discuss your
concerns first.

> My first reaction to type hints was "yuck", and I'm sure I'm not the only
> one to think that. viz (from some pycon slides):
>
> def zipmap(f: Callable[[int, int], int], xx: List[int],
> yy: List[int]) -> List[Tuple[int, int, int]]:
>
> arg. and imagine it with default arguments.

You've picked a complex example and written it poorly. I'd say yuck too,
but let's use something closer to PEP-8 formatting:

def zipmap(f: Callable[[int, int], int],
xx: List[int],
yy: List[int]
) -> List[Tuple[int, int, int]]:

Not quite so bad with each parameter on its own line. It's actually
quite readable, once you learn what the annotations mean. Like all new
syntax, of course you need to learn it. But the type hints are just
regular Python expressions.


> Of course, part of this reaction is just a knee-jerk reaction to the new
> and unfamiliar, and should be dismissed, entirely justifiably, as mere
> irrationality. But I'm sure sensible people agree that they do make our
> function definitions longer, more complex, and harder to read.

Everything has a cost and all features, or lack of features, are a
trade-off. Function definitions could be even shorter and simpler and
easier to read if we didn't have default values.


[...]
> I'm not so sure. My worry is that once type hinting gets standardised,
> then they will become a "best practice", and there's a particular
> personality type out there that's going to start wanting to add type hints
> to every function they write. Similarly to mindlessly obeying PEP8 while
> ignoring its intentions, hobgoblin-of-little-minds style, I think we're
> very likely to see type hints appearing in a lot of python source, or a lot
> of pre-commit-hook checkers. Pretty soon it will be hard to find any open
> source library code that doesn't have type hints, or any project style
> guide that doesn't require them.

I doubt that very much. I'm not a betting man, but if I were, I would
put money on it.

Firstly: libraries tend to be multi-version, and these days they are
often hybrid Python 2 & 3 code. Since annotations are 3 only, libraries
cannot use these type hints until they drop support for Python 2.7,
which will surely be *no less* than five years away. Probably more like
ten. So "annotations everywhere" are, at best, many years away.

Secondly, more importantly, there are a lot of Python programmers who
(like you) dislike type hinting. They are not going to suddenly jump on
the band wagon and use them. Even if the libraries that they use have
annotations, most people don't read the source and will never know.

But the biggest reason that I know that there will not be a sudden rush
to horrible enterprisey code in Python due to type annotations is that
Python has allowed enterprisey code for 20+ years and it hasn't become
common practice. We can overuse design patterns, and turn everything
into a property (or worse, getter/setter methods) just like Java, and we
simple don't.

Well, perhaps some libraries do. Since we don't tend to read their
source code, how would we know if they are full of properties and
decorators and metaclasses doing the most horrid things imaginable? All
we care about is that the code we write is ordinary readable Python
code, and it works. (There may be exceptions, but they are
*exceptions*.)

There is no epidemic of style guides making properties mandatory, or git
check-ins that reject any function that fails to have doctests, or any
of the bad things that have been possible in Python for two decades.
Type hints can be abused too, and just like all the other things that
can be, they won't be abused to the degree you fear.


My comments on stub files to follow.

[Steven D'Aprano]

On Mon, Apr 20, 2015 at 02:41:06PM -0400, Barry Warsaw wrote:
> On Apr 20, 2015, at 07:30 PM, Harry Percival wrote:
>

> >tldr; type hints in python source are scary. Would reserving them for stub
> >files be better?
>

> I think so. I think PEP 8 should require stub files for stdlib modules and
> strongly encourage them for 3rd party code.

A very, very strong -1 to that.

Stub files are a necessary evil. Except where absolutely necessary,
they should be strongly discouraged. A quote from the Go FAQs:

Dependency management is a big part of software development
today but the “header files” of languages in the C tradition
are antithetical to clean dependency analysis—and fast
compilation.

http://golang.org/doc/faq#What_is_the_purpose_of_the_project

Things that go together should be together. A function parameter and
its type information (if any) go together: the type is as much a part
of the parameter declaration as the name and the default. Putting them
together is the best situation:

def func(n: Integer): ...

and should strongly be prefered as best practice for when you choose to
use type hinting at all. Alternatives are not as good. Second best is to
put them close by, as in a decorator:

@typing(n=Integer) # Don't Repeat Yourself violation
def func(n): ...

A distant third best is a docstring. Not only does it also violate DRY,
but it also increases the likelyhood of errors:

def func(n):
"""Blah blah blah

blah blah blah

Arguments:

m: Integer

"""

Keeping documentation and code in synch is hard, and such mistakes are
not uncommon.

Putting the type information in a stub file is an exponentially more
distant fourth best, or to put it another way, *the worst* solution for
where to put type hints. Not only do you Repeat Yourself with the name
of the parameter, but also the name of the function (or method and
class) AND module. The type information *isn't even in the same file*,
which increases the chance of it being lost, forgotten, deleted, out of
date, unmaintained, etc.

[Guido van Rossum]

On Mon, Apr 20, 2015 at 4:41 PM, Jack Diederich <jack...@gmail.com> wrote:

Twelve years ago a wise man said to me "I suggest that you also propose a new name for the resulting language"

The barrage of FUD makes me feel like the woman who asked her doctor for a second opinion and was told "you're ugly too."

 

I talked with many of you at PyCon about the costs of PEP 484. There are plenty of people who have done a fine job promoting the benefits.

So excuse me if I also defend the PEP against your attack.

 

* It is not optional. Please stop saying that. The people promoting it would prefer that everyone use it. If it is approved it will be optional in the way that PEP8 is optional. If I'm reading your annotated code it is certainly /not/ optional that I understand the annotations.

The PEP 8 analogy is actually great -- that PEP's style advice *is* optional and plenty of people ignore it happily. The benefits come out when you're sharing code but violations are no big deal.

And as to requiring what the annotations mean: the CPython interpreter itself doesn't even use them when executing your code, so you can certainly pretend that the annotations aren't there and read the body of the function or the docstring instead.

 

* Uploading stubs for other people's code is a terrible idea. Who do I contact when I update the interface to my library? The random Joe who "helped" by uploading annotations three months ago and then quit the internet? I don't even want to think about people maliciously adding stubs to PyPI.

We're certainly not planning to let arbitrary people upload stubs for arbitrary code to PyPI that will automatically be used by the type checkers. (We can't stop people from publishing their stubs, just as you can't stop people from writing blog posts or stackoverflow answers with examples for your library.)

The actual plan is to have a common repository of stubs (a prototype exists at https://github.com/JukkaL/typeshed) but we also plan some kind of submission review. I've proposed that when submitting stubs for a package you don't own, the typeshed owners ask the package owner what their position is, and we expect the answers to fall on the following spectrum:

- I don't want stubs uploaded for my package
- I'll write the stubs myself

- I want to review all stubs that are uploaded for my package before they are accepted

- Please go ahead and add stubs for my package and let me know when they're ready

- Go ahead, I trust you

This seems a reasonable due diligence policy that avoids the scenarios you're worried about. (Of course if you refuse stubs a black market for stubs might spring into existence. That sounds kind of exciting... :-)

 

* The cognitive load is very high. The average function signature will double in length. This is not a small cost and telling me it is "optional" to pretend that every other word on the line doesn't exist is a farce.

Have you never found yourself trying to read someone else's code and being stopped in your tracks because you couldn't find out what kind of argument was expected somewhere? I'm personally quite tired of seeing that an argument is a "file" and not knowing whether it's meant to be an IO stream or a filename (and was that bytes or unicode? :-). Or seeing something documented as "creation time" without knowing whether that's a datetime or a a POSIX timestamp (or maybe milliseconds or a string encoding a date). Etc., etc.

And yes, you can put the info in docstrings or comments (though it won't be less verbose, and it will be more likely wrong). One way to see type hints is as a way to save on docstrings and comments by using a standard notation that ties type info to the argument without the need to repeat the argument name.

Yes, you need to use your brain. But don't tell my that all the Python code you read or write is self-explanatory, because it isn't.

 

* Every company's style guide is about to get much longer. That in itself is an indicator that this is a MAJOR language change and not just some "optional" add-on.

Actually, standardization *reduces* the length of a company style guide, like PEP 8 did before. Would you rather have instructions for the specific linter your company uses, instead of the single phrase "use type hints (PEP 484)"?

 

* People will screw it up. The same people who can't be trusted to program without type annotations are also going to be *writing* those type annotations.

I don't follow this argument. So what? People will screw up anything you give them. introspection, metaclasses, import hooks, reduce(), you name it.

 

* Teaching python is about to get much less attractive. It will not be optional for teachers to say "just pretend all this stuff over here doesn't exist"

This argument is particularly weak. Teachers are actually *very* effective teaching only the parts of Python they need -- some start with just functions and variables. Others with just classes and methods. Few ever even teach operator overloading.

 

* "No new syntax" is a lie. Or rather a red herring. There are lots of new things it will be required to know and just because the compiler doesn't have to change doesn't mean the language isn't undergoing a major change.

The language undergoes major changes all the time. There was a time before context managers. You've survived print() becoming a function. We're getting a matrix multiply operator. The sky isn't falling.

 

If this wasn't in a PEP and it wasn't going to ship in the stdlib very few people would use it. If you told everyone they had to install a different python implementation they wouldn't. This is much worse than that - it is Python4 hidden away inside a PEP.

But the nice thing is that this time you really *can* mix Python 3 and Python 4 together in one app. :-P

 

There are many fine languages that have sophisticated type systems. And many bondage & discipline languages that make you type things three times to make really really sure you meant to type that. If you find those other languages appealing I invite you to go use them instead.

-Jack

https://mail.python.org/pipermail/python-dev/2003-February/033291.html

So finally it's payback time? Well played sir. :-)

[Terry Reedy]

On 4/20/2015 9:07 PM, Chris Angelico wrote:
> On Tue, Apr 21, 2015 at 10:52 AM, Ben Finney <ben+p...@benfinney.id.au> wrote:

>> Jack is not complaining only about *writing* code. He's complaining
>> about the effect this will have on code that we all are expected to
>> *read*.

For reading, good function and parameter names and a good summary as the
first line of the docstring are probably more useful *to humans* than
formal type annotations.

> Ahh. Yes, that's a concern. When you go digging into that library to
> find out how it works, yes, you'd be face-to-face with their type
> annotations.

> Maybe it'd be of value to have a quick "code stripper" that takes away
> all the annotations,

I was already thinking about that for Idle. The file should then be
either semi read-only or renamed. Also options to merge annotations in
from stub files for editing and to split back out when writing.

> plus any other junk/framing that you're not
> interested in, and gives you something you can browse in a text
> editor? It could take away all the Sphinx adornments from docstrings,
> any source control versioning markers, all that kind of thing.

Interesting idea.

--
Terry Jan Reedy

[M.-A. Lemburg]

Hmm, that's the first time I've heard about this. I agree with
Jack that it's a terrible idea to allow this for 3rd party
packages.

If people want to contribute stubs, they should contribute them
to the package in the usual ways, not in a side channel. The important
part missing in the above setup is maintenance and to some extent
an external change of the API definitions.

Both require active participation in the package project,
not the separated setup proposed above, to be effective and
actually work out in the long run.

For the stdlib, typesched looks like a nice idea to spread the
workload.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Apr 21 2015)
>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Antoine Pitrou]

On Mon, 20 Apr 2015 20:43:38 -0400
"R. David Murray" <rdmu...@bitdance.com> wrote:
> +1 to this from me too. I'm afraid that means I'm -1 on the PEP.
>
> I didn't write this in my earlier email because I wasn't sure about it,
> but my gut reaction after reading Harry's email was "if type annotations
> are used in the stdlib, I'll probably stop contributing". That doesn't
> mean that's *true*, but that's the first time I've ever had that
> thought, so it is probably worth sharing.

I think it would be nice to know what the PEP means for daily stdlib
development. If patches have to carry typing information each time they
add/enhance an API that's an addition burden. If typing is done
separately by interested people then it sounds like it wouldn't have
much of an impact on everyone else's workflow.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Cory Benfield]

On 21 April 2015 at 01:45, Chris Angelico <ros...@gmail.com> wrote:
> When you're writing a library, it can be a great help to provide type
> annotations, because every application that uses your library can
> benefit.

It can be a great help to whom? Not to me (the library author),
because I can't use them in my library code, because I have to support
2.7. That's by no means a bad thing (after all, most libraries are
written to help others), but I found it really unclear who was being
advantaged here.

To show the downside, I decided to annotate requests' API a little
bit. Then I stopped, because I really couldn't work out how to do it.

For example, requests.get() takes a URL parameter. This can be an
instance of basestring, or anything that provides a __str__ or
__unicode__ method that provides us a string that looks like a URL
(such as a URL abstraction class). We don't know ahead of time what
those objects may be, so there's no type signature I can provide here
that is of any use beyond 'Any'.

We also provide headers/params options, whose type signature would be
(as far as I can tell) Union[Iterable[Tuple[basestring, basestring]],
Mapping[basestring, basestring]], which is a fairly unpleasant type
(and also limiting, as we totally accept two-element lists here as
well as two-tuples. That's got *nothing* on the type of the `files`
argument, which is the most incredibly polymorphic argument I've ever
seen: the best I can work out it would be:

Optional[
Union[
Mapping[
basestring,
Union[
Tuple[basestring, Optional[Union[basestring, file]]],
Tuple[basestring, Optional[Union[basestring, file]],
Optional[basestring]],
Tuple[basestring, Optional[Union[basestring, file]],
Optional[basestring], Optional[Headers]]
]
],
Iterable[
Tuple[
basestring,
Union[
Tuple[basestring, Optional[Union[basestring, file]]],
Tuple[basestring, Optional[Union[basestring,
file]], Optional[basestring]],
Tuple[basestring, Optional[Union[basestring,
file]], Optional[basestring], Optional[Headers]]
]
]
]
]

Headers = Union[
Mapping[basestring, basestring],
Iterable[Tuple[basestring, basestring]],
]

This includes me factoring out the Headers type for my own sanity, and
does not include the fact that in practice almost all instances of
'basestring' above actually mean 'anything that can be safely cast to
basestring', and 'file' means 'any file-like object including BytesIO
and StringIO' (speaking of which, the PEP doesn't even begin to go
into how to handle that kind of requirement: do we have some kind of
Readable interface that can be implemented? Or do I have to manually
union together all types someone might want to use?).

Now, this can be somewhat improved: the three types of tuple (again,
not just tuples, often lists!) can be factored out, as can the
Union[basestring, file]. However, there's still really no way around
the fact that this is a seriously complex type signature!

Python has had years of people writing APIs that feel natural, even if
that means weird contortions around 'types'. Writing an argument as
'clever' as the 'files' argument in requests in a statically typed
language is an absolute nightmare, but Python makes it an easy and
natural thing to do.

Further, Python's type system is not sufficiently flexible to allow
library authors to adequately specify the types their code actually
works on. I need to be able to talk about interfaces, because
interfaces are the contract around which APIs are build in Python, but
I cannot do that with this system in a way that makes any sense at
all. To even begin to be useful for library authors this PEP would
need to allow some kind of type hierarchy that is *not* defined by
inheritance, but instead by interfaces. We've been discouraging use of
'type' and 'isinstance' for years because they break duck typing, but
that has *nothing* on what this PEP appears to do to duck typing.

I suppose the TLDR of this email is that I think that libraries with
'pythonic' APIs are the least likely to take up this typing system
because it will provide the least value to them. Maintaining
signatures will be a pain (stub files are necessary), writing clear
signatures will be a pain (see above), and writing signatures that are
both sufficiently open to be back-compatible and sufficiently
restrictive to be able to catch bugs seems like it might be very
nearly impossible.

I'd love for someone to tell me I'm wrong, though. I want to like this
PEP, I really do.

[Chris Angelico]

On Tue, Apr 21, 2015 at 6:58 PM, Cory Benfield <co...@lukasa.co.uk> wrote:
> On 21 April 2015 at 01:45, Chris Angelico <ros...@gmail.com> wrote:
>> When you're writing a library, it can be a great help to provide type
>> annotations, because every application that uses your library can
>> benefit.
>
> It can be a great help to whom? Not to me (the library author),
> because I can't use them in my library code, because I have to support
> 2.7. That's by no means a bad thing (after all, most libraries are
> written to help others), but I found it really unclear who was being
> advantaged here.

Mainly application users get the benefit, I expect. But I may be wrong.

> ... That's got *nothing* on the type of the `files`

> argument, which is the most incredibly polymorphic argument I've ever
> seen: the best I can work out it would be:
>
> Optional[
> Union[
> Mapping[
> basestring,
> Union[
> Tuple[basestring, Optional[Union[basestring, file]]],
> Tuple[basestring, Optional[Union[basestring, file]],
> Optional[basestring]],
> Tuple[basestring, Optional[Union[basestring, file]],
> Optional[basestring], Optional[Headers]]
> ]
> ],
> Iterable[
> Tuple[
> basestring,
> Union[
> Tuple[basestring, Optional[Union[basestring, file]]],
> Tuple[basestring, Optional[Union[basestring,
> file]], Optional[basestring]],
> Tuple[basestring, Optional[Union[basestring,
> file]], Optional[basestring], Optional[Headers]]
> ]
> ]
> ]
> ]

At this point, you may want to just stop caring about the exact type.
Part of the point of gradual typing is that you can short-cut a lot of
this. And quite frankly, this isn't really helping anything. Just skip
it and say that it's Union[Mapping, Iterable, None].

ChrisA

[Cory Benfield]

On 21 April 2015 at 10:10, Chris Angelico <ros...@gmail.com> wrote:
> At this point, you may want to just stop caring about the exact type.
> Part of the point of gradual typing is that you can short-cut a lot of
> this. And quite frankly, this isn't really helping anything. Just skip
> it and say that it's Union[Mapping, Iterable, None].

I think this is my problem with the proposal. This change doesn't
catch any of the bugs anyone was going to hit (no-one is passing a
callable to this parameter), and it doesn't catch any of the bugs
someone might actually hit (getting the tuple elements in the wrong
order, for example). At this point the type signature is worse than
useless.

It seems like the only place the type annotations will get used is in
relatively trivial cases where the types are obvious anyway. I don't
deny that *some* bugs will be caught, but I suspect they'll
overwhelmingly be crass ones that would have been caught quickly
anyway. The minute a type signature might actually help prevent a
subtle bug we can't use them anymore because the toolchain isn't
powerful enough to do it so we just put 'Any' and call it a day. And
even then we haven't helped the trivial case much because anyone who
wants to provide a duck-typed object that should be perfectly suitable
in this case no longer can.

What this proposal *needs* to be generally useful, IMO, is the ability
to specify types that actually match the way we have encouraged people
to write code: duck typed, favouring composition over inheritance,
etc. At the very least that means 'type constraints' (see Haskell,
Rust, really any language with a powerful and robust type system:
hell, even see Go's interfaces) need to be something we can specify.

[Steven D'Aprano]

On Mon, Apr 20, 2015 at 08:37:28PM -0700, Guido van Rossum wrote:
> On Mon, Apr 20, 2015 at 4:41 PM, Jack Diederich <jack...@gmail.com> wrote:
>
> > Twelve years ago a wise man said to me "I suggest that you also propose a
> > new name for the resulting language"
> >
>
> The barrage of FUD makes me feel like the woman who asked her doctor for a
> second opinion and was told "you're ugly too."

Don't worry Guido, some of us are very excited to see this coming to
fruition :-)

It's been over ten years since your first blog post on optional typing
for Python. At least nobody can accuse you of rushing into this.

http://www.artima.com/weblogs/viewpost.jsp?thread=85551


--
Steve

[Rob Cliffe]

On 21/04/2015 10:33, Cory Benfield wrote:

On 21 April 2015 at 10:10, Chris Angelico <ros...@gmail.com> wrote:

At this point, you may want to just stop caring about the exact type.
Part of the point of gradual typing is that you can short-cut a lot of
this. And quite frankly, this isn't really helping anything. Just skip
it and say that it's Union[Mapping, Iterable, None].

I think this is my problem with the proposal. This change doesn't
catch any of the bugs anyone was going to hit (no-one is passing a
callable to this parameter), and it doesn't catch any of the bugs
someone might actually hit (getting the tuple elements in the wrong
order, for example). At this point the type signature is worse than
useless.

Exactly.  At this point putting the type signatures in seems to be a pointless exercise in masochism (for the author) and sadism (for the reader).

The refreshing thing about Python is that it is a fantastic, concise, dynamically typed language, at the opposite end of the spectrum from C++ (ugh!).
We have got used to the consequences (good and bad) of this:
    Duck typing, e.g. a function to sort numbers (sorry Tim Peters bad example) turns out to support any kind of object (e.g. strings) that supports comparison.
        Not to mention objects of some class that will be written in 5 years time.
        (Adding a type hint that restricted the argument to say a sequence of numbers turns out to be a mistake.  And what is a number?
         Is Fraction?  What about complex numbers, which can't be sorted?  What if the function were written before the Decimal class?)
    Errors are often not caught until run time that would be caught at compile time in other languages (though static code checkers help).
        (Not much of a disadvantage because of Python's superb error diagnostics.)
     Python code typically says what it is doing, with the minimum of syntactic guff.  (Well, apart from colons after if/while/try etc. :-) )
    Which makes it easy to read.
Now it seems as if this proposal wants to start turning Python in the C++ direction, encouraging adding ugly boilerplate code.  (This may only be tangentially relevant, but I want to scream when I see some combination of public/private/protected/static/extern etc., most of which I don't understand.)

Chris A makes the valid point (if I understand correctly) that
    Authors of libraries should make it as easy as possible to
          (i) know what object types can be passed to functions
          (ii) diagnose when the wrong type of object is passed
    Authors of apps are not under such obligation, they can basically do what they want.

Well,
    (i) can be done with good documentation (docstrings etc.).
    (ii) can be done with appropriate runtime checks and good error messages.
E.g. (Cory's example) I'm sure it is possible to test somehow if an object is file-like (if only by trying to access it like a file).
Is thorough argument checking and provision of good diagnostics going to be easy for the library author?  No.  Is it going to be a lot of work to do thoroughly?  Almost certainly yes.
But what the hell, writing a production-quality library is not an exercise for newbies.

It seems to me that type hints are attempting to be a silver bullet and to capture in a simple formula what is often, in practice, not simple at all, viz. "Is this passed object suitable?".  Attempting - and failing, except in the simplest cases.

Apologies, Guido, but:
There was once a Chinese student who was a marvellous painter.  He painted a perfect life-like picture of a snake.
But he was unable to stop and leave it alone.  In his zeal he went on to paint feet on the snake.  Which of course completely spoiled the picture, as snakes don't have feet.
Hence "to paint feet on the snake": to be unable to resist tinkering with something that is already good.  (I suppose "If it ain't broke, don't fix it" is an approximate Western equivalent.)
You see where I'm going with this - adding type hints to Python feels a bit like painting feet on the snake.  Or at least turning it into a different language.

Best wishes
Rob Cliffe

[Gustavo Carneiro]

Documentation is not checked.  It often loses sync with the actual code.  Docs say one thing, code does another.

Documenting types in docstrings forces you to repeat yourself.  You have to write the parameter names twice, once in the function definition line, another in the docstring.  And you need to understand the syntax of whatever docstring format will be used to check the code.

I'm sure I'm not the only one that thinks that type hints actually *improve readability*.  I will not argue whether it makes the code uglier or prettier, but I am certain that most of the time the type hints make the code easier to read because you don't have to spend so much mental effort trying to figure out "gee, I wonder if this parameter is supposed to be bool, string, or int?"  The type (or types) that a parameter is expected to have becomes explicit, otherwise you have to read the entire function body to understand.

 

    (ii) can be done with appropriate runtime checks and good error messages.
E.g. (Cory's example) I'm sure it is possible to test somehow if an object is file-like (if only by trying to access it like a file).
Is thorough argument checking and provision of good diagnostics going to be easy for the library author?  No.  Is it going to be a lot of work to do thoroughly?  Almost certainly yes.
But what the hell, writing a production-quality library is not an exercise for newbies.

It seems to me that type hints are attempting to be a silver bullet and to capture in a simple formula what is often, in practice, not simple at all, viz. "Is this passed object suitable?".  Attempting - and failing, except in the simplest cases.

Even if it only helps in the simplest cases, it saves the reader of the code a lot of effort if you add up all those little cases together.

My suggestion, wherever an annotated type is too complex, don't annotate it.  If you do /only/ the simple cases, the end result is easier to read.

 

Apologies, Guido, but:
There was once a Chinese student who was a marvellous painter.  He painted a perfect life-like picture of a snake.
But he was unable to stop and leave it alone.  In his zeal he went on to paint feet on the snake.  Which of course completely spoiled the picture, as snakes don't have feet.
Hence "to paint feet on the snake": to be unable to resist tinkering with something that is already good.  (I suppose "If it ain't broke, don't fix it" is an approximate Western equivalent.)
You see where I'm going with this - adding type hints to Python feels a bit like painting feet on the snake.  Or at least turning it into a different language.

Oh, $DEITY, I really hope this is not going to be another excuse lazy programmers will use to avoid porting to Python 3 :-(

 

Best wishes
Rob Cliffe

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com

--

Gustavo J. A. M. Carneiro

Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert

[Chris Withers]

On 20/04/2015 20:09, Paul Moore wrote:

> On 20 April 2015 at 19:41, Barry Warsaw <ba...@python.org> wrote:
>>> tldr; type hints in python source are scary. Would reserving them for stub
>>> files be better?
>> I think so. I think PEP 8 should require stub files for stdlib modules and
>> strongly encourage them for 3rd party code.

> Agreed. I have many of the same concerns as Harry, but I wouldn't have
> expressed them quite as well. I'm not too worried about actually
> removing annotations from the core language, but I agree that we
> should create a strong culture of "type hints go in stub files" to
> keep source files readable and clean.
>
> On that note, I'm not sure "stub" files is a particularly good name.
> Maybe "type files" would be better? Something that emphasises that
> they are the correct place to put type hints, not a workaround.
Or we could just be honest and admit that we're choosing to add header
files to Python.

It's a shame, as it's more complexity, and it's being inflicted on those
who might be writing a library for the first time, or those becoming
core committers for the first time, or those just trying to "do the
right thing".

Currently, the burden is a heavier one (type inference, rather than
reading it from a file) but borne by people best placed to handle it
(authors of IDEs, type checking software, etc).

Chris

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Chris Withers]

On 20/04/2015 19:30, Harry Percival wrote:
> Hi all,
>

> tldr; type hints in python source are scary. Would reserving them for
> stub files be better?

I think Jack's summary of this is excellent and aligns well with where I
think I'm coming from on this:

https://mail.python.org/pipermail/python-dev/2015-April/139253.html

Harry Also makes just as many good points so I'll reply here, with a
note that while switching to (type/header/stub - my ordered preference
for how to describe .pyi files) is preferable to type hints inside
files, it's still a massive change to the language, not one I can say
I've missed over the past 15 years, and one if asked to vote on (I know
that's not the case ;-)) that I would choose to vote against.

Anyway, I've not posted much to python-dev in quite a while, but this is
a topic that I would be kicking myself in 5-10 years time when I've had
to move to Javascript or <insert new language here> because everyone
else has drifted away from Python as it had become ugly...

[Chris Withers]

On 21/04/2015 12:23, Gustavo Carneiro wrote:

Well,

    (i) can be done with good documentation (docstrings etc.).

Documentation is not checked.  It often loses sync with the actual code.  Docs say one thing, code does another.

That certainly something that could be fixed by formalising the documentation specification and having a checker check that rather than changing the syntax of the language...

Chris

[Cory Benfield]

On 21 April 2015 at 12:23, Gustavo Carneiro <gjcar...@gmail.com> wrote:
> Documentation is not checked. It often loses sync with the actual code.
> Docs say one thing, code does another.

Agreed. I don't think anyone would disagree here. I'm talking from the
position of being a library author, where supporting versions of
Python lower than 3.5 will be a reality for at least 5 more years. I
will not be able to inline my type hints, so they'll have to go in
stub files, and now we've got the same problem: type hints can go out
of step just as easily as documentation can.

> Documenting types in docstrings forces you to repeat yourself. You have to
> write the parameter names twice, once in the function definition line,
> another in the docstring. And you need to understand the syntax of whatever
> docstring format will be used to check the code.

Yes, but I'm repeating myself O(small integer) lines away from where I
first wrote it. The maintenance burden of keeping those things in step
is not very high. As for your last point, my code is not checked
against my docstrings, primarily for the reasons I mentioned in my
original email. My docstrings are merely rendered into documentation.

> I'm sure I'm not the only one that thinks that type hints actually *improve
> readability*.

In some cases, sure. No contest.

> but I am certain that most of the time the type hints make the
> code easier to read because you don't have to spend so much mental effort
> trying to figure out "gee, I wonder if this parameter is supposed to be
> bool, string, or int?"

Yes. If you spend a lot of your time doing this, then type hints will
help you, *assuming* the programmer who originally wrote the ambiguous
code diligently maintains the type hints. I have no reason to believe
that a programmer whose code is that ambiguous is going to do that, or
even write type hints at all.

The bigger problem is that, previously, Python functions never took
bool, string, or int. They took 'object that can be evaluated in a
boolean context', 'object that behaves like strings in some undefined
way', and 'objects that behave like ints in some undefined way'. The
type hint you provide is *not helpful*. What I *need* is a type hint
that says 'object that has the .endswith() method' or 'object that can
be safely cast to a string', so I can tell whether I can pass you a
custom URL class that happens to render into a textual URL or whether
it definitely has to be a string object. But you can't express that
relationship in this model, so you will just say 'string', and now I
will turn off type checking because the type hinter keeps shouting at
me for using my URLBuilder object.

If you only work with primitive types and only want your users to use
primitive types, this proposal almost certainly works brilliantly. If
you *don't*, this proposal begins to become extremely uncomfortable
extremely fast.

> Even if it only helps in the simplest cases, it saves the reader of the code
> a lot of effort if you add up all those little cases together.
>
> My suggestion, wherever an annotated type is too complex, don't annotate it.
> If you do /only/ the simple cases, the end result is easier to read.

If that's genuinely the bar for success for this PEP, then fine, I
wish it the best. I'd *like* a PEP that can deal with the complex
cases too, but I freely admit to having no idea how such a thing may
be implemented. All I can say is that I think this PEP will have
limited uptake amongst many third-party libraries, particularly the
complex ones.

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Steven D'Aprano]

On Tue, Apr 21, 2015 at 11:56:15AM +0100, Rob Cliffe wrote:

> (Adding a type hint that restricted the argument to say a
> sequence of numbers turns out to be a mistake.

Let's find out how big a mistake it is with an test run.

py> def sorter(alist: List[int]) -> List[int]:
... return sorted(alist)
...
py> data = (chr(i) + 'ay' for i in range(97, 107))
py> type(data)
<class 'generator'>
py> sorter(data)
['aay', 'bay', 'cay', 'day', 'eay', 'fay', 'gay', 'hay', 'iay', 'jay']


When we say that type checking is optional, we mean it.

[Disclaimer: I had to fake the List object, since I don't have the
typing module, but everything else is exactly as you see it.]

Annotations will be available for type checking. If you don't want to
type check, don't type check. If you want to go against the type hints,
you can go against the type hints, and get exactly the same runtime
errors as you have now:

py> sorter(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in sorter
TypeError: 'NoneType' object is not iterable

There is no compile time checking unless you choose to run a type
checker or linter -- and even if the type checker flags errors, you can
ignore it and run the program regardless. Just like today with linters
like PyFlakes, PyLint and similar.

For those who choose not to run a type checker, the annotations will be
nothing more than introspectable documentation.

> And what is a number?
> Is Fraction? What about complex numbers, which can't be
> sorted? What if the function were written before the Decimal class?)

I know that you are intending these as rhetorical questions, but Python
has had a proper numeric tower since version 2.5 or 2.6. So:

py> from numbers import Number
py> from decimal import Decimal
py> isinstance(Decimal("1.25"), Number)
True
py> isinstance(2+3j, Number)
True

> Errors are often not caught until run time that would be caught at
> compile time in other languages (though static code checkers help).

Yes they do help, which is exactly the point.

> (Not much of a disadvantage because of Python's superb error
> diagnostics.)

That's certainly very optimistic of you.

If I had to pick just one out of compile time type checking versus run
time unit tests, I'd pick run time tests. But it is naive to deny the
benefits of compile time checks in catching errors that you otherwise
might not have found even with extensive unit tests (and lets face it,
we never have enough unit tests).

Ironically, type hinting will *reduce* the need for intrusive,
anti-duck-testing explicit calls to isinstance() at runtime:

def func(x:float):
if isinstance(x, float): ...
else: raise TypeError


Why bother making that expensive isinstance call every single time the
function is called, if the type checker can prove that x is always a
float?

> Python code typically says what it is doing, with the minimum of
> syntactic guff. (Well, apart from colons after if/while/try etc. :-) )
> Which makes it easy to read.
> Now it seems as if this proposal wants to start turning Python in the
> C++ direction, encouraging adding ugly boilerplate code. (This may only
> be tangentially relevant, but I want to scream when I see some
> combination of public/private/protected/static/extern etc., most of
> which I don't understand.)

Perhaps if you understood it you would be less inclined to scream.

> Chris A makes the valid point (if I understand correctly) that
> Authors of libraries should make it as easy as possible to
> (i) know what object types can be passed to functions
> (ii) diagnose when the wrong type of object is passed
> Authors of apps are not under such obligation, they can basically
> do what they want.
>
> Well,
> (i) can be done with good documentation (docstrings etc.).
> (ii) can be done with appropriate runtime checks and good error
> messages.

How ironic. After singing the praises of duck-typing, now you are
recommending runtime type checks.

As far as good error messages go, they don't help you one bit when the
application suddenly falls over in a totally unexpected place due to a
bug in your code.

I can't go into too many details due to commercial confidentiality, but
we experienced something similar recently. A situation nobody foresaw,
that wasn't guarded against, and wasn't tested for, came up after
deployment. There was a traceback, of course, but a failure in the field
200km away with a stressed customer and hundreds of angry users is not
as useful as a compile-time failure during development.

> You see where I'm going with this - adding type hints to Python feels a
> bit like painting feet on the snake.

Pythons are one of the few snakes which have vestigal legs:

http://en.wikipedia.org/wiki/Pelvic_spur


--
Steve

[Antoine Pitrou]

On Tue, 21 Apr 2015 22:47:23 +1000
Steven D'Aprano <st...@pearwood.info> wrote:
>
> Ironically, type hinting will *reduce* the need for intrusive,
> anti-duck-testing explicit calls to isinstance() at runtime:

It won't, since as you pointed out yourself, type checks are purely
optional and entirely separate from compilation and runtime evaluation.

> Why bother making that expensive isinstance call every single time the
> function is called, if the type checker can prove that x is always a
> float?

Because the user might not run the type checker, obviously. To quote
you: """When we say that type checking is optional, we mean it."""

You can't at the same time point out that type checking has no
power or control over runtime behaviour, and then claim that type
checking makes runtime behaviour (for example, ability to accept or
reject certain types) saner. It is a trivial contradiction.

(and because of the former property, type hints can be entirely wrong -
for example incomplete, or too strict, or too lax - without anyone
noticing as long as they are self-consistent, since runtime behaviour is
unaffected by them)

Regards

Antoine.

[Steven D'Aprano]

On Tue, Apr 21, 2015 at 01:25:34PM +0100, Chris Withers wrote:

> Anyway, I've not posted much to python-dev in quite a while, but this is
> a topic that I would be kicking myself in 5-10 years time when I've had
> to move to Javascript or <insert new language here> because everyone
> else has drifted away from Python as it had become ugly...

Facebook released Flow, a static typechecker for Javascript, to a very
positive reaction. From their announcement:

Flow’s type checking is opt-in — you do not need to type check all
your code at once. However, underlying the design of Flow is the
assumption that most JavaScript code is implicitly statically typed;
even though types may not appear anywhere in the code, they are in
the developer’s mind as a way to reason about the correctness of the
code. Flow infers those types automatically wherever possible, which
means that it can find type errors without needing any changes to
the code at all. On the other hand, some JavaScript code, especially
frameworks, make heavy use of reflection that is often hard to
reason about statically. For such inherently dynamic code, type
checking would be too imprecise, so Flow provides a simple way to
explicitly trust such code and move on. This design is validated by
our huge JavaScript codebase at Facebook: Most of our code falls in
the implicitly statically typed category, where developers can check
their code for type errors without having to explicitly annotate
that code with types.

Quoted here:

http://blog.jooq.org/2014/12/11/the-inconvenient-truth-about-dynamic-vs-static-typing/

More about flow:

http://flowtype.org/

Matz is interested in the same sort of gradual type checking for Ruby as
Guido wants to add to Python:

https://www.omniref.com/blog/blog/2014/11/17/matz-at-rubyconf-2014-will-ruby-3-dot-0-be-statically-typed/

Julia already includes this sort of hybrid dynamic+static type checking:

http://julia.readthedocs.org/en/latest/manual/types/

I could keep going, but I hope I've made my point. Whatever language you
are using in 5-10 years time, it will almost certainly be either mostly
static with some dynamic features like Java, or dynamic with optional
and gradual typing.

--
Steven

[Paul Moore]

On 21 April 2015 at 13:47, Steven D'Aprano <st...@pearwood.info> wrote:
> On Tue, Apr 21, 2015 at 11:56:15AM +0100, Rob Cliffe wrote:
>
>> (Adding a type hint that restricted the argument to say a
>> sequence of numbers turns out to be a mistake.
>
> Let's find out how big a mistake it is with an test run.
>
> py> def sorter(alist: List[int]) -> List[int]:
> ... return sorted(alist)
> ...
> py> data = (chr(i) + 'ay' for i in range(97, 107))
> py> type(data)
> <class 'generator'>
> py> sorter(data)
> ['aay', 'bay', 'cay', 'day', 'eay', 'fay', 'gay', 'hay', 'iay', 'jay']
>
>
> When we say that type checking is optional, we mean it.

That's a very good point - although I have to say that I sort of feel
like I'm cheating doing that. It doesn't matter how many times you say
"optional", it does still feel like it's wrong, in basically the same
way as using undocumented methods on a class, etc.

Conceded, type checking is optional. But that message seems to be
failing to get through. It's not a technical thing, it's a PR issue.
Which is *very* close to meaning that the problem is FUD, but less in
the sense of "people deliberately spreading FUD" as "people being
worried, not understanding the proposal, and not being sure how type
hints will work in practice".

I'm reasonably willing to just accept that the PEP will be getting
implemented, and to wait and see how things turn out. But my fear,
uncertainty and doubt remain. I'll try not to spread them, though.

>> You see where I'm going with this - adding type hints to Python feels a
>> bit like painting feet on the snake.
>
> Pythons are one of the few snakes which have vestigal legs:
>
> http://en.wikipedia.org/wiki/Pelvic_spur

LOL, instant win :-)

Paul

[Antoine Pitrou]

On Tue, 21 Apr 2015 23:16:19 +1000

Steven D'Aprano <st...@pearwood.info> wrote:
>

> I could keep going, but I hope I've made my point.

I don't think so. Just because other languages are looking at it
doesn't mean it will end up successful. It means it's an interesting
idea, that's all.

A litmus test for this PEP would be to apply it to a sizable code base
and evaluate the results. Looking at isolated function examples doesn't
tell us how it will scale (and I'm obviously not talking about
performance, but the ability to still give useful diagnostics in the
presence of typing "holes" - i.e. untyped functions - or imprecisions,
when factored in a complex graph of call dependencies).

> Whatever language you
> are using in 5-10 years time, it will almost certainly be either mostly
> static with some dynamic features like Java, or dynamic with optional
> and gradual typing.

Anybody can make predictions. As a data point, 6 years ago people were
predicting that the average desktop CPU would have 16 cores nowadays...
(we don't hear much from them anymore :-))

Regards

Antoine.

[Chris Withers]

On 20/04/2015 20:09, Paul Moore wrote:
> On 20 April 2015 at 19:41, Barry Warsaw <ba...@python.org> wrote:

>>> tldr; type hints in python source are scary. Would reserving them for stub
>>> files be better?

>> I think so. I think PEP 8 should require stub files for stdlib modules and
>> strongly encourage them for 3rd party code.
> Agreed. I have many of the same concerns as Harry, but I wouldn't have
> expressed them quite as well. I'm not too worried about actually
> removing annotations from the core language, but I agree that we
> should create a strong culture of "type hints go in stub files" to
> keep source files readable and clean.
>
> On that note, I'm not sure "stub" files is a particularly good name.
> Maybe "type files" would be better? Something that emphasises that
> they are the correct place to put type hints, not a workaround.
Or we could just be honest and admit that we're choosing to add header
files to Python.

It's a shame, as it's more complexity, and it's being inflicted on those
who might be writing a library for the first time, or those becoming
core committers for the first time, or those just trying to "do the
right thing".

Currently, the burden is a heavier one (type inference, rather than
reading it from a file) but borne by people best placed to handle it
(authors of IDEs, type checking software, etc).

Chris

[Harry Percival]

Hey, I just wanted to say to everyone, thanks for being so patient and willing to engage with this discussion, despite my not having done my research and read the (substantial) prior discussion on the topic.  Here it is (or at least, some of it!) for any other newcomers:


https://mail.python.org/pipermail/python-list/2015-January/697202.html
https://mail.python.org/pipermail/python-list/2015-January/697315.html
https://github.com/ambv/typehinting/issues/55

I'm not sure any of it will radically change anyone's mind.  Everyone (I think) agrees that type hints carry some cost in terms of legibility and simplicity of the language.  Everyone agrees they have some benefits.  The disagreement remains over whether it will be worth it, and whether stub files vs inline vs something-else would be the best place to hide them.

Is "let's wait and see" an acceptable position?  Probably the only real way to resolve the disagreement is to start seeing the type hints in use, and form opinions based on real-world practice.  Some people will like them inline.  Lots of people will use stub files, since they're the only way to maintain 2+3 compatibility, so it's my home that the community will standardise around that.  Still other people will want to have another crack at using docstrings, but the standardisation on notation of types will still help them.  And still other others will want no part in them, and will argue against them in their communities.   Hopefully a best practice will evolve.  And hopefully it'll be the stub file one, and then we really can deprecate function annotations in 3.6 ;-)

On 21 April 2015 at 11:56, Rob Cliffe <rob.c...@btinternet.com> wrote:

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/hjwp2%40cantab.net

--

------------------------------
Harry J.W. Percival
------------------------------
Twitter: @hjwp
Mobile:  +44 (0) 78877 02511
Skype:         harry.percival

[Chris Withers]

On 20/04/2015 19:30, Harry Percival wrote:
> Hi all,
>

> tldr; type hints in python source are scary. Would reserving them for
> stub files be better?

I was trying to find Jack's original post as I think his summary is

excellent and aligns well with where I think I'm coming from on this:

https://mail.python.org/pipermail/python-dev/2015-April/139253.html

However, Harry makes just as many good points so I'll reply here, with a

note that while switching to (type/header/stub - my ordered preference
for how to describe .pyi files) is preferable to type hints inside
files, it's still a massive change to the language, not one I can say
I've missed over the past 15 years, and one if asked to vote on (I know
that's not the case ;-)) that I would choose to vote against.

Anyway, I've not posted much to python-dev in quite a while, but this is
a topic that I would be kicking myself in 5-10 years time when I've had
to move to Javascript or <insert new language here> because everyone
else has drifted away from Python as it had become ugly...

Chris

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Arnaud Delobelle]

On Tue, 21 Apr 2015 at 09:59 Cory Benfield <co...@lukasa.co.uk> wrote:

[...] 

Further, Python's type system is not sufficiently flexible to allow
library authors to adequately specify the types their code actually
works on. I need to be able to talk about interfaces, because
interfaces are the contract around which APIs are build in Python, but
I cannot do that with this system in a way that makes any sense at
all. To even begin to be useful for library authors this PEP would
need to allow some kind of type hierarchy that is *not* defined by
inheritance, but instead by interfaces. We've been discouraging use of
'type' and 'isinstance' for years because they break duck typing, but
that has *nothing* on what this PEP appears to do to duck typing.

This is why I feel like this PEP may be a real threat to duck typing.  If people constantly get told by their editor / IDE that they are calling function with the wrong argument types, what are they going to do?  They may start adopting the same approach as in Java / C++ etc... where interfaces must be explicitly defined and the practice of duck typing may become forgotten because discouraged by the tools programmers use.

I guess what I'm saying is that this could encourage a very significant cultural change in the way Python code is written towards a less flexible mindset.

-- 

Arnaud

 

[Paul Sokolovsky]

Hello,

On Tue, 21 Apr 2015 11:56:15 +0100
Rob Cliffe <rob.c...@btinternet.com> wrote:

> On 21/04/2015 10:33, Cory Benfield wrote:
> > On 21 April 2015 at 10:10, Chris Angelico <ros...@gmail.com> wrote:
> >> At this point, you may want to just stop caring about the exact
> >> type. Part of the point of gradual typing is that you can
> >> short-cut a lot of this. And quite frankly, this isn't really
> >> helping anything. Just skip it and say that it's Union[Mapping,
> >> Iterable, None].
> > I think this is my problem with the proposal. This change doesn't
> > catch any of the bugs anyone was going to hit (no-one is passing a
> > callable to this parameter), and it doesn't catch any of the bugs
> > someone might actually hit (getting the tuple elements in the wrong
> > order, for example). At this point the type signature is worse than
> > useless.

> Exactly. At this point putting the type signatures in seems to be a
> pointless exercise in masochism (for the author) and sadism (for the
> reader).

Or interesting and productive exercise, which will lead to further
improvements and benefits - for other kinds of authors and readers.
Those kinds left your kind alone by letting to keep not writing
annotations. Why you feel so embarrassed to let the other kind
progress? Python becomes multi-paradigm language, what's so bad with
that?

You're afraid that you will have discomfort reading other's code? My
god, I love Python, and hate 50% of code written in it, most of it
needs to be rewritten to be satisfactory! Why it bothers you that there
will be direction on how other people may (re)write it?

>
> The refreshing thing about Python is that it is a fantastic,

> *concise*, dynamically typed language, at the opposite end of the
> spectrum from C++ (ugh!).

C++ is fantastic, concise, statically typed language. You've just used
to see 90% of code in it which needs to be rewritten.

[]

> Apologies, Guido, but:
[]

> You see where I'm going with this - adding type hints to Python feels
> a bit like painting feet on the snake. Or at least turning it into a
> different language.

I'm sure type annotations never could be discussed for inclusion if
Guido didn't feel that they fit with Python's way. Sometimes it makes
sense to trust BDFL. It's much less obtrusive and qustionable change
than forcing Unicode by default, making print a function, or keeping
3-headed sys.exc_info() ugliness instead throwing it out of Python3.

Because well, type annotation *are optional*. And it's by now clear
that there will be --strip-type-annotations flag to pip and stickers
"This system runs type annotations free".

>
> Best wishes
> Rob Cliffe



--
Best regards,
Paul mailto:pmi...@gmail.com

[Barry Warsaw]

On Apr 21, 2015, at 01:34 PM, Steven D'Aprano wrote:

>Putting the type information in a stub file is an exponentially more distant
>fourth best, or to put it another way, *the worst* solution for where to put
>type hints. Not only do you Repeat Yourself with the name of the parameter,
>but also the name of the function (or method and class) AND module. The type
>information *isn't even in the same file*, which increases the chance of it
>being lost, forgotten, deleted, out of date, unmaintained, etc.

All true, but the trade-off is the agility and ease of working on, reading,
and understanding the stdlib, all of which IMHO will suffer if type hints are
inlined there.

What I don't want to have happen is for type hints to slowly infiltrate the
stdlib to the point where no patch will be accepted unless it also has hints.
I have the same gut reaction to this as RDM expressed a few posts back.

One of the thing I love most about Python is its dynamic typing. I'm all for
giving linter developers a hook for experimenting with their tools, I just
don't care and I don't want to *have* to care. Maybe some day they will make
it so compelling that I will care, but I want to be convinced first.

So I think stub files in the stdlib are the right compromise today.

Cheers,
-Barry

[Steven D'Aprano]

On Tue, Apr 21, 2015 at 03:08:27PM +0200, Antoine Pitrou wrote:
> On Tue, 21 Apr 2015 22:47:23 +1000
> Steven D'Aprano <st...@pearwood.info> wrote:
> >
> > Ironically, type hinting will *reduce* the need for intrusive,
> > anti-duck-testing explicit calls to isinstance() at runtime:
>
> It won't, since as you pointed out yourself, type checks are purely
> optional and entirely separate from compilation and runtime evaluation.

Perhaps you are thinking of libraries, where the library function has to
deal with whatever junk people throw at it. To such libraries, I believe
that the major benefit of type hints is not so much in proving the
library's correctness in the face of random arguments, but as
documentation. In any case, of course you are correct that public
library functions and methods will continue to need to check their
arguments. (Private functions, perhaps not.)

But for applications, the situation is different. If my application
talks to a database and extracts a string which it passes on to its own
function spam(), then it will be a string. Not a string-like object. Not
something that quacks like a string. A string. Once the type checker is
satisfied that spam() always receives a string, then further isinstance
checks inside spam() is a waste of time. If spam()'s caller changes and
might return something which is not a string, then the type checker
will flag that.

Obviously to get this benefit you need to actually use a type checker. I
didn't think I needed to mention that.


--
Steve

[Antoine Pitrou]

Le 21/04/2015 15:50, Paul Sokolovsky a écrit :
> Hello,
>
> On Tue, 21 Apr 2015 15:08:27 +0200
> Antoine Pitrou <soli...@pitrou.net> wrote:
>
> []

>
>> Because the user might not run the type checker, obviously. To quote
>> you: """When we say that type checking is optional, we mean it."""
>>
>> You can't at the same time point out that type checking has no
>> power or control over runtime behaviour, and then claim that type
>> checking makes runtime behaviour (for example, ability to accept or
>> reject certain types) saner. It is a trivial contradiction.
>

> I suspected there's either short-sightedness, or just a word play for
> for a purpose of word play.
>
> Type annotation are NOT introduced for the purpose of static type
> checking.

I was replying to Steven's message. Did you read it?

> Runtime type checking (run as via "make test", not
> as "production") is much more interesting to catch errors.

Obviously you're giving the word "runtime" a different meaning than I
do. The type checker isn't supposed to actually execute the user's
functions (it's not that it's forbidden, simply that it's not how it
will work in all likelihood): therefore, it doesn't have any handle on
what *actually* happens at runtime, vs. what is declared in the typing
declarations.

But if by "runtime" you mean any action that happens *out-of-band*
during development (such as running pip or a syntax-checking linter),
then sure, fine :-)

> Even more interesting usage is to allow ahead-of-time, and thus
> unbloated, optimization. There're bunch of JITters and AOTters for
> Python language, each of which uses own syntax (via decorators, etc.)
> to annotate functions.

As a developer of one of those tools, I've already said that I find it
unlikely for the PEP to be useful for that purpose. The issue is that
the vocabulary created in the PEP is not extensible enough.

Note I'm not saying it's impossible. I'm just skeptical that in its
current form it will help us. And apparently none of our "competitors"
seems very enthusiastic either (feel free to prove me wrong: I might
have missed something :-)).

> Having language-specified type annotations
> allows for portable syntax for such optimized code.

Only if the annotations allow expressing the subtleties required by the
specific optimizer. For example, "Float" is too vague for Numba: we
would like to know if that is meant to be a single- or double-precision
float.

> Don't block the language if you're stuck
> with an unimaginative implementation, there's much more to Python than
> that.

The Python language doesn't really have anything to do with that. It's
just an additional library with a set of conventions. Which is also why
a PEP wouldn't be required to make it alive, it's just there to make it
an official standard.

[Chris Angelico]

On Tue, Apr 21, 2015 at 7:56 PM, Arnaud Delobelle <arn...@gmail.com> wrote:
> If people constantly get told by their editor / IDE that they are calling
> function with the wrong argument types, what are they going to do? They may
> start adopting the same approach as in Java / C++ etc... where interfaces
> must be explicitly defined and the practice of duck typing may become
> forgotten because discouraged by the tools programmers use.

Style guides should generally be recommending the use of concrete
types for return values, but abstract types for parameters - you might
well have a function that returns a list, but most functions that
accept lists are really accepting sequences. Granted, there are some
vague areas - how many functions take a "file-like object", and are
they all the same? - but between MyPy types and the abstract base
types that already exist, there are plenty of ways to formalize duck
typing. And frankly, even with the uncertainties, I'd still rather
have a function declared as taking a "file-like object" than "an
object with a .read() method that takes an integer and returns up to
that many bytes of data, and a .seek() method that blah blah blah
blah". Sometimes, the precision is completely useless. What an editor
would do with the hint that a function takes a file-like object I'm
not sure, but if it issues a complaint because I'm passing it
something that doesn't have a writelines() method, it's the fault of
the editor, not the type hint.

ChrisA

[Paul Sokolovsky]

Hello,

On Tue, 21 Apr 2015 15:08:27 +0200
Antoine Pitrou <soli...@pitrou.net> wrote:

[]

> Because the user might not run the type checker, obviously. To quote
> you: """When we say that type checking is optional, we mean it."""
>
> You can't at the same time point out that type checking has no
> power or control over runtime behaviour, and then claim that type
> checking makes runtime behaviour (for example, ability to accept or
> reject certain types) saner. It is a trivial contradiction.

I suspected there's either short-sightedness, or just a word play for
for a purpose of word play.

Type annotation are NOT introduced for the purpose of static type

checking. The current PEP just gives an example of their usage for
static type checking as an example backed by the immediate availability
of a tool which does that, MyPy.

But granted, static type checking is the most boring of possible usages
for type annotation. Runtime type checking (run as via "make test", not
as "production") is much more interesting to catch errors. The tooling
for that is yet to be written though.

Even more interesting usage is to allow ahead-of-time, and thus
unbloated, optimization. There're bunch of JITters and AOTters for
Python language, each of which uses own syntax (via decorators, etc.)

to annotate functions. Having language-specified type annotations

allows for portable syntax for such optimized code.

Granted, the most juicy usages described above won't be available in
CPython out of the box. But there should be that motto: "CPython is the
most boring of all Pythons". Don't block the language if you're stuck

with an unimaginative implementation, there's much more to Python than
that.

--
Best regards,
Paul mailto:pmi...@gmail.com

[Alexander Walters]

So. This is how you try and get me to care about Python 3. Can't speak
for others, but this does the opposite for me. This makes me ecstatic
that Python 2 has a nearly-frozen api.

[Cory Benfield]

On 21 April 2015 at 15:31, Chris Angelico <ros...@gmail.com> wrote:
> Granted, there are some
> vague areas - how many functions take a "file-like object", and are
> they all the same? - but between MyPy types and the abstract base
> types that already exist, there are plenty of ways to formalize duck
> typing.

Are there? Can I have a link or an example, please? I feel like I
don't know how I'm supposed to do this, and I'd like to see how that
works. I'll even give a concrete use-case: I want to be able to take a
file-like object that has a .read() method and a .seek() method.

> And frankly, even with the uncertainties, I'd still rather
> have a function declared as taking a "file-like object" than "an
> object with a .read() method that takes an integer and returns up to
> that many bytes of data, and a .seek() method that blah blah blah
> blah". Sometimes, the precision is completely useless.

It is completely useless. Happily, this is a strawman, and no-one was
asking for it, so we can all live happily ever after!

The correct specification is "read method with this type signature"
and "seek method with this type signature". I would even be prepared
to waive the type signatures on read and seek, given that enforcing
the type hinting on others is not a good idea.

I suspect I have a mismatch with several others in this discussion. My
position is that if I'm going to have a type system, I'd like to have
a powerful one: Haskell, not Java. Otherwise I'll get by with the duck
typing that has worked just fine for us over the last 20 years. I
suspect, however, that many others in this conversation want any type
system at all, so long as they can have one.

Is that an accurate characterisation of your position, Chris?

[Nikolaus Rath]

On Apr 20 2015, Chris Angelico <ros...@gmail.com> wrote:
> Maybe it'd be of value to have a quick "code stripper" that takes away
> all the annotations, plus any other junk/framing that you're not
> interested in, and gives you something you can browse in a text
> editor?

If you need to preprocess your source code to make it suitable for human
consumption something is very wrong with your language design. I can't
believe you're seriously suggesting this.

Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Chris Angelico]

On Wed, Apr 22, 2015 at 12:51 AM, Cory Benfield <co...@lukasa.co.uk> wrote:
> On 21 April 2015 at 15:31, Chris Angelico <ros...@gmail.com> wrote:
>> Granted, there are some
>> vague areas - how many functions take a "file-like object", and are
>> they all the same? - but between MyPy types and the abstract base
>> types that already exist, there are plenty of ways to formalize duck
>> typing.
>
> Are there? Can I have a link or an example, please? I feel like I
> don't know how I'm supposed to do this, and I'd like to see how that
> works. I'll even give a concrete use-case: I want to be able to take a
> file-like object that has a .read() method and a .seek() method.

Someone who's been more involved in the details of MyPy can probably
say more specifically what ought to be done about file-like objects.

>> And frankly, even with the uncertainties, I'd still rather
>> have a function declared as taking a "file-like object" than "an
>> object with a .read() method that takes an integer and returns up to
>> that many bytes of data, and a .seek() method that blah blah blah
>> blah". Sometimes, the precision is completely useless.
>
> It is completely useless. Happily, this is a strawman, and no-one was
> asking for it, so we can all live happily ever after!
>
> The correct specification is "read method with this type signature"
> and "seek method with this type signature". I would even be prepared
> to waive the type signatures on read and seek, given that enforcing
> the type hinting on others is not a good idea.
>
> I suspect I have a mismatch with several others in this discussion. My
> position is that if I'm going to have a type system, I'd like to have
> a powerful one: Haskell, not Java. Otherwise I'll get by with the duck
> typing that has worked just fine for us over the last 20 years. I
> suspect, however, that many others in this conversation want any type
> system at all, so long as they can have one.
>
> Is that an accurate characterisation of your position, Chris?

Pretty accurate, yeah. Here's how I see it:

def incremental_parser(input: FileLike) -> List[Token]:
tokens = []
data = ""
while True:
if not data:
data = input.read(64)
token = Token(data[0]); data = data[1:]
while token.wants_more():
token.give_more(data[0]); data = data[1:]
tokens.append(token)
if token.is_end_of_stream(): break
input.seek(-len(data), 1)
return tokens

If you were to exhaustively stipulate the requirements on the
file-like object, you'd have to say:

* Provides a read() method which takes an integer and returns up to
that many bytes
* Provides a seek() method which takes two integers
* Is capable of relative seeking by at least 63 bytes backward
* Is open for reading
* Etcetera

That's not the type system's job. Not in Python. Maybe in Haskell, but
not in Python. So how much _should_ go into the type hint? I'm happy
with "FileLike" or however it's to be spelled; maybe separate readable
files from writable ones, as those are two fairly clear variants, but
that's about all you really need. If you provide incremental_parser()
with an input file that's non-seekable, it's going to have problems -
and your editor may or may not even be able to detect that (some files
are seekable but only in the forward direction, but they'll have the
exact same seek() method).

I might be wrong about where MyPy is trying to go with this, but
doubtless someone will correct me on any inaccuracies above.

ChrisA

[Paul Sokolovsky]

Hello,

On Tue, 21 Apr 2015 16:11:51 +0200
Antoine Pitrou <ant...@python.org> wrote:

[]

> >> You can't at the same time point out that type checking has no
> >> power or control over runtime behaviour, and then claim that type
> >> checking makes runtime behaviour (for example, ability to accept or
> >> reject certain types) saner. It is a trivial contradiction.
> >
> > I suspected there's either short-sightedness, or just a word play
> > for for a purpose of word play.
> >
> > Type annotation are NOT introduced for the purpose of static type
> > checking.
>
> I was replying to Steven's message. Did you read it?

Yes. And I try to follow general course of discussion, as its hard to
follow individual sub-threads. And for example yesterday's big theme
was people blackmailing that they stop contributing to stdlib if
annotations are in, and today's theme appear to be people telling that
static type checking won't be useful. And just your reply to Steven
was a final straw which prompted me to point out that static type
checking is not a crux of it, but just one (not the biggest IMHO) usage.

> > Runtime type checking (run as via "make test", not
> > as "production") is much more interesting to catch errors.
>
> Obviously you're giving the word "runtime" a different meaning than I
> do. The type checker isn't supposed to actually execute the user's
> functions (it's not that it's forbidden, simply that it's not how it
> will work in all likelihood): therefore, it doesn't have any handle on
> what *actually* happens at runtime, vs. what is declared in the typing
> declarations.

Well, maybe "typechecker" is the wrong word then. I'm talking about
instrumented VM which actually interprets type annotation while running
bytecode - for example, on entry to a function, it takes argument
annotations, and executes sequence of equivalent isinstance() checks,
etc., etc. One won't use such instrumented VM at "production" runtime,
but it will be useful to enable while running testsuite (or integration
tests), to catch more issues.

> > Even more interesting usage is to allow ahead-of-time, and thus
> > unbloated, optimization. There're bunch of JITters and AOTters for
> > Python language, each of which uses own syntax (via decorators,
> > etc.) to annotate functions.
>
> As a developer of one of those tools, I've already said that I find it
> unlikely for the PEP to be useful for that purpose. The issue is that
> the vocabulary created in the PEP is not extensible enough.

How so, if it essentially allows you make typedefs? Syntax of such
typedef follow basic conventions (so *any* tool should understand
its *basic* semantics), but you're free to assign (implicit, particular
for your tool, but of course documented for it) semantics.

> Note I'm not saying it's impossible. I'm just skeptical that in its
> current form it will help us. And apparently none of our "competitors"

That's my biggest fear - that "JIT" Python community is not yet ready to
receive and value this feature.

> seems very enthusiastic either (feel free to prove me wrong: I might
> have missed something :-)).

Let me try: MicroPython already uses type annotations for statically
typed functions. E.g.

def add(x:int, y:int):
return x + y

will translate the function to just 2 machine instructions. And we'd
prefer to use standard language syntax, instead of having our own
conventions (e.g. above you see that return type "is inferred").

> > Having language-specified type annotations
> > allows for portable syntax for such optimized code.
>
> Only if the annotations allow expressing the subtleties required by
> the specific optimizer. For example, "Float" is too vague for Numba:
> we would like to know if that is meant to be a single- or
> double-precision float.

Oh really, you care to support single-precisions in Numba? We have a lot
of problems with sfloats in MicroPython, because whole Python stdlib API
was written with implicit assumption that float is (at least) double.
E.g., we cannot have time.time() which both returns fractional seconds
and is based of Jan 1, 1970 - there's simply not enough bits in
single-precision floats! It's on my backlog to bring up this and
related issues before wider Python development community.

Anyway, back to your example, it would be done like:

SFloat = float
DFloat = float

For a random tool out there, "SFloat" and "DFloat" would be just
aliases to floats, but Numba will know they have additional semantics
behind them. (That assumes that typedefs like SFloat can be accessed in
symbolic form - that's certainly possible if you have your own
parser/VM, but might worth to think how to do it on "CPython" level).

> > Don't block the language if you're stuck
> > with an unimaginative implementation, there's much more to Python
> > than that.
>
> The Python language doesn't really have anything to do with that. It's
> just an additional library with a set of conventions. Which is also
> why a PEP wouldn't be required to make it alive, it's just there to
> make it an official standard.

Yes, and IMHO "JIT/compiler" Python community is too fragmented and to
unaware (ignorant?) of other projects' efforts, and having "official
standard" would help both "JIT" community and to establish Python as a
language with support for efficient compile-time optimizations.

>
> Regards
>
> Antoine.

--
Best regards,
Paul mailto:pmi...@gmail.com

[Paul Sokolovsky]

Hello,

On Tue, 21 Apr 2015 08:05:59 -0700
Nikolaus Rath <Niko...@rath.org> wrote:

> On Apr 20 2015, Chris Angelico <ros...@gmail.com> wrote:
> > Maybe it'd be of value to have a quick "code stripper" that takes
> > away all the annotations, plus any other junk/framing that you're
> > not interested in, and gives you something you can browse in a text
> > editor?
>
> If you need to preprocess your source code to make it suitable for
> human consumption something is very wrong with your language design.
> I can't believe you're seriously suggesting this.

I'm sure that was irony, d'oh. Just as my suggestion to have stickers
"This system runs free of type annotations" for the most zealous
anti-annotations folks.

The proposed type annotations are very readable. Not as readable as C's
type syntax, but ok. Sorry, irony again. I remember myself as a
10-years old kid, trying to wrap my head about C's types syntax, and
not getting anything of it. So, please contrast it to C - PEP484 gives
very readable syntax, and even both human- and machine-readable. If you
feel like writing 3-story type annotation, just resist the temptation,
"Any" is your best friend.

--
Best regards,
Paul mailto:pmi...@gmail.com

[Cory Benfield]

Ok, that makes sense to me. =) Looking at mypy's source code, I see
shutil.copyfileobj has the following signature:

def copyfileobj(fsrc: IO[AnyStr], fdst: IO[AnyStr], length: int =
16*1024) -> None:

so it seems that mypy defines a general IO datatype here.

I continue to be worried about the lack of a *general* solution to
this problem, but I'm glad that some specific solutions exist. I still
think library authors will not do much to take up this proposal. =)

[Jim Baker]

On Tue, Apr 21, 2015 at 9:09 AM, Chris Angelico <ros...@gmail.com> wrote:

...

Pretty accurate, yeah. Here's how I see it:

def incremental_parser(input: FileLike) -> List[Token]:
    tokens = []
    data = ""
    while True:
        if not data:
            data = input.read(64)
        token = Token(data[0]); data = data[1:]
        while token.wants_more():
            token.give_more(data[0]); data = data[1:]
        tokens.append(token)
        if token.is_end_of_stream(): break
    input.seek(-len(data), 1)
    return tokens

If you were to exhaustively stipulate the requirements on the
file-like object, you'd have to say:

* Provides a read() method which takes an integer and returns up to
that many bytes
* Provides a seek() method which takes two integers
* Is capable of relative seeking by at least 63 bytes backward
* Is open for reading
* Etcetera

Potentially you could use io.RawIOBase as the ABC for the type you need for FileLike, including read and seek. See the mixins in https://docs.python.org/3/library/io.html#class-hierarchy 

That's not the type system's job. Not in Python. Maybe in Haskell,

Not in Haskell either FWIW in terms of what its type system can prove

 

but
not in Python. So how much _should_ go into the type hint? I'm happy
with "FileLike" or however it's to be spelled; maybe separate readable
files from writable ones, as those are two fairly clear variants, but
that's about all you really need.

RawIOBase is also potential overkill... maybe you just wanted something that duck typed for a few methods you implemented. Any and dynamic typing is a good choice then.

 

If you provide incremental_parser()
with an input file that's non-seekable, it's going to have problems -
and your editor may or may not even be able to detect that (some files
are seekable but only in the forward direction, but they'll have the
exact same seek() method).

With what has been proposed, there are no static guarantees about how many bytes can be read, or that input is even seekable (does seekable() return True?) or it is open for reading. Instead we can only prove a limited amount in static type systems about the runtime dynamic behavior of code, and PEP 484 is weaker than other approaches (for very good reasons IMHO).

Still useful however :)

- Jim

[R. David Murray]

On Wed, 22 Apr 2015 01:09:52 +1000, Chris Angelico <ros...@gmail.com> wrote:
> def incremental_parser(input: FileLike) -> List[Token]:
> tokens = []
> data = ""
> while True:
> if not data:
> data = input.read(64)
> token = Token(data[0]); data = data[1:]
> while token.wants_more():
> token.give_more(data[0]); data = data[1:]
> tokens.append(token)
> if token.is_end_of_stream(): break
> input.seek(-len(data), 1)
> return tokens
>
> If you were to exhaustively stipulate the requirements on the
> file-like object, you'd have to say:
>
> * Provides a read() method which takes an integer and returns up to
> that many bytes
> * Provides a seek() method which takes two integers
> * Is capable of relative seeking by at least 63 bytes backward
> * Is open for reading
> * Etcetera
>
> That's not the type system's job. Not in Python. Maybe in Haskell, but

Just a note that if I'm reading the high level description right,
this kind of analysis is exactly the kind of thing that Flow does
for javascript.

--David

[Guido van Rossum]

On Tue, Apr 21, 2015 at 12:33 AM, M.-A. Lemburg <m...@egenix.com> wrote:

On 21.04.2015 05:37, Guido van Rossum wrote:
> On Mon, Apr 20, 2015 at 4:41 PM, Jack Diederich <jack...@gmail.com> wrote:
>> * Uploading stubs for other people's code is a terrible idea. Who do I
>> contact when I update the interface to my library? The random Joe who
>> "helped" by uploading annotations three months ago and then quit the
>> internet? I don't even want to think about people maliciously adding stubs
>> to PyPI.
>>
>
> We're certainly not planning to let arbitrary people upload stubs for
> arbitrary code to PyPI that will automatically be used by the type
> checkers. (We can't stop people from publishing their stubs, just as you
> can't stop people from writing blog posts or stackoverflow answers with
> examples for your library.)
>
> The actual plan is to have a common repository of stubs (a prototype exists
> at https://github.com/JukkaL/typeshed) but we also plan some kind of
> submission review. I've proposed that when submitting stubs for a package
> you don't own, the typeshed owners ask the package owner what their
> position is, and we expect the answers to fall on the following spectrum:
>
> - I don't want stubs uploaded for my package
> - I'll write the stubs myself
> - I want to review all stubs that are uploaded for my package before they
> are accepted
> - Please go ahead and add stubs for my package and let me know when they're
> ready
> - Go ahead, I trust you
>
> This seems a reasonable due diligence policy that avoids the scenarios
> you're worried about. (Of course if you refuse stubs a black market for
> stubs might spring into existence. That sounds kind of exciting... :-)

Hmm, that's the first time I've heard about this. I agree with
Jack that it's a terrible idea to allow this for 3rd party
packages.

If people want to contribute stubs, they should contribute them
to the package in the usual ways, not in a side channel. The important
part missing in the above setup is maintenance and to some extent
an external change of the API definitions.

Both require active participation in the package project,
not the separated setup proposed above, to be effective and
actually work out in the long run.

For the stdlib, typeshed looks like a nice idea to spread the
workload.

I hesitate to speak for others, but IIUC the reason why typeshed was started is that companies like PyCharm and Google (and maybe others) are *already* creating their own stubs for 3rd party packages, because they have a need to type-check code that *uses* 3rd party packages. Their use cases are otherwise quite different (the user code type-checked by PyCharm is that of PyCharm users, and the code type-checked by Google is their own proprietary code) but they both find themselves needing stubs for commonly used 3rd party packages. mypy finds itself in a similar position.

Think of it this way. Suppose you wrote an app that downloaded some files from the web using the popular Requests package. Now suppose you wanted to run mypy over your app. You're willing to do the work of adding signatures to your own app, and presumably there are stubs for those parts of the stdlib that you're using, but without stubs for Requests, mypy won't do a very good job type-checking your calls into Requests. It's not rocket science to come up with stubs for Requests (there aren't that many classes and methods) but the Requests package is in maintenance mode, and while they respond quickly to security issues they might take their time to release a new version that includes your stub files, and until there are a lot of people clamoring for stubs for Requests, they might not care at all.

So what does Requests have to lose if, instead of including the stubs in Requests, they let the typeshed people distribute stubs for Requests? Presumably having the stubs in typeshed means that PyCharm and mypy (and the 50 other type-checkers that are being written right now :-) can give better diagnostics for code using Requests, and once in a while this may save a user of Requests from doing something dumb and blaming Requests. The only downside would be if something was missing in the stubs and users would get incorrect error messages from their favorite type checker. But it's a long stretch to see this rain down on Requests' reputation -- more likely the type checker will be blamed, so type checker authors/distributors will be vigilant before distributing stubs.

OTOH if you prefer to make and distribute your own stubs, type checkers will use those, and there won't be a need to include stubs in typeshed when a package already provides stubs.

And if you really don't want anything to do with stubs for your package, just tell the typeshed owners and your wish will be respected.

--

--Guido van Rossum (python.org/~guido)

[R. David Murray]

On Tue, 21 Apr 2015 18:27:50 +0300, Paul Sokolovsky <pmi...@gmail.com> wrote:
> > I was replying to Steven's message. Did you read it?
>
> Yes. And I try to follow general course of discussion, as its hard to
> follow individual sub-threads. And for example yesterday's big theme
> was people blackmailing that they stop contributing to stdlib if
> annotations are in, and today's theme appear to be people telling that
> static type checking won't be useful. And just your reply to Steven
> was a final straw which prompted me to point out that static type
> checking is not a crux of it, but just one (not the biggest IMHO) usage.

Please be respectful rather than inflammatory. If you read what I
wrote, I did not say that I was going to stop contributing, I
specifically talked about that gut reaction being both emotional and
illogical. That doesn't make the reaction any less real, and the fact
that such reactions exist is a data point you should consider in
conducting your PR campaign for this issue. (I don't mean that last as
a negative: this issue *requires* an honest PR campaign.)

--David

[Guido van Rossum]

On Tue, Apr 21, 2015 at 12:49 AM, Antoine Pitrou <soli...@pitrou.net> wrote:

On Mon, 20 Apr 2015 20:43:38 -0400
"R. David Murray" <rdmu...@bitdance.com> wrote:
> +1 to this from me too. I'm afraid that means I'm -1 on the PEP.
>
> I didn't write this in my earlier email because I wasn't sure about it,
> but my gut reaction after reading Harry's email was "if type annotations
> are used in the stdlib, I'll probably stop contributing".  That doesn't
> mean that's *true*, but that's the first time I've ever had that
> thought, so it is probably worth sharing.

I think it would be nice to know what the PEP means for daily stdlib
development. If patches have to carry typing information each time they
add/enhance an API that's an addition burden. If typing is done
separately by interested people then it sounds like it wouldn't have
much of an impact on everyone else's workflow.

This point will be moot until new code appears in the stdlib whose author likes type hints. As I said, we won't be converting existing code to add type hints (I'm actively against that for the stdlib, for reasons I've explained already).

*If* type hints prove useful, I expect that adding type hints **to code that deserves them** is treated no different in the workflow than adding tests or docs. I.e. something that is the right thing to do because it has obvious benefits for users and/or future maintainers. If at some point running a type checker over the stdlib as part of continuous integration become routine, type hints can also replace certain silly tests.

Until some point in a possible but distant future when we're all thinking back fondly about the argument we're currently having, it will be the choice of the author of new (and *only* new) stdlib modules whether and how to use type hints. Such a hypothetical author would also be reviewing updates to "their" module and point out lack of type hints just like you might point out an incomplete docstring, an outdated comment, or a missing test. (The type checker would be responsible for pointing out bugs. :-P )

[Antoine Pitrou]

On Tue, 21 Apr 2015 09:28:45 -0700
Guido van Rossum <gu...@python.org> wrote:
> On Tue, Apr 21, 2015 at 12:49 AM, Antoine Pitrou <soli...@pitrou.net>
> wrote:
>
> > On Mon, 20 Apr 2015 20:43:38 -0400
> > "R. David Murray" <rdmu...@bitdance.com> wrote:
> > > +1 to this from me too. I'm afraid that means I'm -1 on the PEP.
> > >
> > > I didn't write this in my earlier email because I wasn't sure about it,
> > > but my gut reaction after reading Harry's email was "if type annotations
> > > are used in the stdlib, I'll probably stop contributing". That doesn't
> > > mean that's *true*, but that's the first time I've ever had that
> > > thought, so it is probably worth sharing.
> >
> > I think it would be nice to know what the PEP means for daily stdlib
> > development. If patches have to carry typing information each time they
> > add/enhance an API that's an addition burden. If typing is done
> > separately by interested people then it sounds like it wouldn't have
> > much of an impact on everyone else's workflow.
> >
>
> This point will be moot until new code appears in the stdlib whose author
> likes type hints. As I said, we won't be converting existing code to add
> type hints (I'm actively against that for the stdlib, for reasons I've
> explained already).

I was thinking of potential stub files. Or are you also putting a ban
on those for existing stdlib code? Sorry if that has already been
answered...

Regards

Antoine.

[Ethan Furman]

On 04/21, Paul Sokolovsky wrote:
>
> And for example yesterday's big theme was people blackmailing that they

> stop contributing to stdlib if annotations are in [...]

A volunteer's honest reaction is not blackmail, and categorizing it as such
is not helpful to the discussion.

--
~Ethan~

[Gregory P. Smith]

On Tue, Apr 21, 2015 at 12:50 AM Antoine Pitrou <soli...@pitrou.net> wrote:

On Mon, 20 Apr 2015 20:43:38 -0400
"R. David Murray" <rdmu...@bitdance.com> wrote:
> +1 to this from me too. I'm afraid that means I'm -1 on the PEP.
>
> I didn't write this in my earlier email because I wasn't sure about it,
> but my gut reaction after reading Harry's email was "if type annotations
> are used in the stdlib, I'll probably stop contributing".  That doesn't
> mean that's *true*, but that's the first time I've ever had that
> thought, so it is probably worth sharing.

I think it would be nice to know what the PEP means for daily stdlib
development. If patches have to carry typing information each time they
add/enhance an API that's an addition burden. If typing is done
separately by interested people then it sounds like it wouldn't have
much of an impact on everyone else's workflow.

Separately by interested people.  That won't change until tools appear and mature that help maintain the types for us.  (if ever)

Nobody wants unreadable code.  Nobody is proposing to make unreadable code happen or encourage its creation.

One thing I feel is often overlooked in the discussion on this PEP: It is about creating a unified type expression syntax for everyone working on Python typing to centralize around. Regardless of if the PEPs version falls short for some purposes. It allows for sharing work. There are multiple ongoing projects that are trying to make use of type information with Python, this allows them to all speak the same language.  (MyPy, MicroPython, Cython, the static analyzer we are trying to create at Google, several others not on the top of my head I'm sure, etc.)

We will not be putting type annotations anywhere in the stdlib or expecting anyone else to maintain them there. That would never happen until tools that are convincing enough in their utility for developers to _want_ to use are available and accepted.  That'll be a developer workflow thing we could address with a later PEP. IF it happens at all.

I view most of this thread as FUD. The fear is understandable, I'm trying to tell people to stop panicing. This PEP does not mean that Python is suddenly going to become unreadable. Just that a set of people working on a common goal have a way to communicate with one another. If that work bears fruit, great, it'll be shared and provided as tools that people want to use. If not, it won't matter in the slightest and the typing module and this PEP will be relegated to history. This is a 100% non-invasive PEP. No new keywords!

Motivation behind static analysis and type checkers comes directly from the success of the type annotations and checking done to Javascript in Google's javascript Closure compiler that has been available for years. Steven mentioned Facebook's Flow which does a similar thing. These are both opt-in and by and large we've found that developers love using them in any decent sized code base. That model is the goal for any of our Python typing related projects to get to. If developers don't _want_ to use it in the end, we have failed and they can happily continue not using it because it was never required.

The reason this PEP exists is for tool developers to be able to do their thing and prove to everyone that it is (a) possible and (b) genuinely useful. IF that proves successful, we can consider if we need a saner syntax for anyone to want to use it. For now we've got this PEP which is a bit of a hack using the Python 3 annotations and a typing module but at the same time doesn't involve any language changes we might regret. I call that a win!

-gps

[Guido van Rossum]

On Tue, Apr 21, 2015 at 7:51 AM, Cory Benfield <co...@lukasa.co.uk> wrote:

The correct specification is "read method with this type signature"
and "seek method with this type signature". I would even be prepared
to waive the type signatures on read and seek, given that enforcing
the type hinting on others is not a good idea.

I suspect I have a mismatch with several others in this discussion. My
position is that if I'm going to have a type system, I'd like to have
a powerful one: Haskell, not Java. Otherwise I'll get by with the duck
typing that has worked just fine for us over the last 20 years. I
suspect, however, that many others in this conversation want any type
system at all, so long as they can have one.

For me, PEP 484 is a stepping stone. Among the authors of PEP 484 there was much discussion about duck typing, and mypy even has some limited support for duck typing (I think you can still find it by searching the mypy code for "protocol"). But we ran out of time getting all the details written up and agreed upon, so we decided to punt -- for now. But duck typing still needs to have a way to talk about things like "seek method with this type signature" (something like `def seek(self, offset: int, whence: int=SEEK_SET) -> int`) so the current proposal gets us part of the way there.

The hope is that once 3.5 is out (with PEP 484's typing.py included *provisional* mode) we can start working on the duck typing specification. The alternative would have been to wait until 3.6, but we didn't think that there would be much of an advantage to postponing the more basic type hinting syntax (it would be like refusing to include "import" until you've sorted out packages). During the run of 3.5 we'll hopefully get feedback on where duck typing is most needed and how to specify it -- valuable input that would be much harder to obtain of *no* part of the type hints notation were standardized.

[Nikolaus Rath]

On Apr 21 2015, Paul Sokolovsky <pmi...@gmail.com> wrote:
> Hello,
>
> On Tue, 21 Apr 2015 08:05:59 -0700
> Nikolaus Rath <Niko...@rath.org> wrote:
>
>> On Apr 20 2015, Chris Angelico <ros...@gmail.com> wrote:
>> > Maybe it'd be of value to have a quick "code stripper" that takes
>> > away all the annotations, plus any other junk/framing that you're
>> > not interested in, and gives you something you can browse in a text
>> > editor?
>>
>> If you need to preprocess your source code to make it suitable for
>> human consumption something is very wrong with your language design.
>> I can't believe you're seriously suggesting this.
>
> I'm sure that was irony, d'oh.

That'd be a relief. It didn't sound ironic to me.

> The proposed type annotations are very readable.

[..]

I don't have an informed opinion about that yet. I was just commenting
on the general idea of stripping them away if they're not readable.

Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Guido van Rossum]

On Tue, Apr 21, 2015 at 9:17 AM, R. David Murray <rdmu...@bitdance.com> wrote:

Please be respectful rather than inflammatory.  If you read what I
wrote, I did not say that I was going to stop contributing, I
specifically talked about that gut reaction being both emotional and
illogical.  That doesn't make the reaction any less real, and the fact
that such reactions exist is a data point you should consider in
conducting your PR campaign for this issue.  (I don't mean that last as
a negative:  this issue *requires* an honest PR campaign.)

Well, my own reactions at this point in the flame war are also quite emotional. :-(

I have done my best in being honest in my PR campaign. But I feel like the opposition (not you, but definitely some others -- have you seen Twitter?) are spreading FUD based on an irrational conviction that this will destroy Python. It will not. It may not prove the solution to all Python's problems -- there's always 3.6. (Oh wait, Python 2.7 is perfect. I've heard that before -- Paul Everitt famously said the same of Python 1.5.2. Aren't you glad I didn't take him literally? :-P )

[Steven D'Aprano]

On Tue, Apr 21, 2015 at 03:51:05PM +0100, Cory Benfield wrote:
> On 21 April 2015 at 15:31, Chris Angelico <ros...@gmail.com> wrote:
> > Granted, there are some
> > vague areas - how many functions take a "file-like object", and are
> > they all the same? - but between MyPy types and the abstract base
> > types that already exist, there are plenty of ways to formalize duck
> > typing.
>
> Are there? Can I have a link or an example, please? I feel like I
> don't know how I'm supposed to do this, and I'd like to see how that
> works. I'll even give a concrete use-case: I want to be able to take a
> file-like object that has a .read() method and a .seek() method.

I've never done this before, so I might not quite have done it
correctly, but this appears to work just fine:

py> import abc
py> class SeekableReadable(metaclass=abc.ABCMeta):
... @classmethod
... def __subclasshook__(cls, C):
... if hasattr(C, 'seek') and hasattr(C, 'read'):
... return True
... return NotImplemented
...
py> f = open('/tmp/foo')
py> isinstance(f, SeekableReadable)
True
py> from io import StringIO
py> issubclass(StringIO, SeekableReadable)
True
py> issubclass(int, SeekableReadable)
False

That gives you your runtime check for an object with seek() and read()
methods. For compile-time checking, I expect you would define
SeekableReadable as above, then make the declaration:

def read_from_start(f:SeekableReadable, size:int):
f.seek(0)
return f.read(size)

So now you have runtime interface checking via an ABC, plus
documentation for the function parameter type via annotation.

But will the static checker understand that annotation? My guess is,
probably not as it stands. According to the docs, MyPy currently
doesn't support this sort of duck typing, but will:

[quote]
There are also plans to support more Python-style “duck typing”
in the type system. The details are still open.
[end quote]

http://mypy.readthedocs.org/en/latest/class_basics.html#abstract-base-classes-and-multiple-inheritance

I expect that dealing with duck typing will be very high on the list
of priorities for the future. In the meantime, for this specific use-case,
you're probably not going to be able to statically check this type hint.
Your choices would be:

- don't type check anything;

- don't type check the read_from_start() function, but type check
everything else;

- don't type check the f parameter (remove the SeekableReadable
annotation, or replace it with Any, but leave the size:int
annotation);

- possibly some type checkers will infer from the function body that f
must have seek() and read() methods, and you don't have to declare
anything (structural typing instead of nominal?);

- (a bad idea, but just for the sake of completeness) leave the
annotation in, and ignore false negatives.

Remember that there is no built-in Python type checker. If you have no
checker, the annotations are just documentation and nothing else will
have changed. If you don't like the checker you have, you'll be able to
replace it with another.

--
Steve

[Antoine Pitrou]

On Tue, 21 Apr 2015 18:27:50 +0300
Paul Sokolovsky <pmi...@gmail.com> wrote:
> Let me try: MicroPython already uses type annotations for statically
> typed functions. E.g.
>
> def add(x:int, y:int):
> return x + y
>
> will translate the function to just 2 machine instructions.

That's quite nice.

> Oh really, you care to support single-precisions in Numba?

Numba is quite specialized. In our area, single-precision arithmetic
can sometimes give double the speed on modern CPUs (especially when
LLVM is able to vectorize the code). The speedup can be even larger on
a GPU (where double-precision execution resources are scarce). I agree
it doesn't make sense for a general-purpose Python compiler.

> Anyway, back to your example, it would be done like:
>
> SFloat = float
> DFloat = float
>
> For a random tool out there, "SFloat" and "DFloat" would be just
> aliases to floats, but Numba will know they have additional semantics
> behind them. (That assumes that typedefs like SFloat can be accessed in
> symbolic form - that's certainly possible if you have your own
> parser/VM, but might worth to think how to do it on "CPython" level).

Fortunately, we don't have our own parser :-) We work from the CPython
bytecode.

Regards

Antoine.

[Guido van Rossum]

No ban, but for now the plan is to collect and manage those separately as part of typeshed. I imagine the typical failure case is where the stubs are more restrictive than the implementation -- after all if the implementation becomes *more* restrictive it's breaking backward compatibility. I think it's okay for the stubs to lag behind like that, so I don't expect pressure on stdlib contributors. If anything the pressure will be on typeshed.

[Daniel Holth]

I am looking forward to using type annotations. I am not very good at
remembering the operations that are available on the objects passed
around in my code, but I am very good at typing CTRL-space. To get
that I am happy to modify my code with weird docstrings or any other
notation. Good support for completion, aided by standard annotations,
eliminates a huge amount of cross-referencing while coding.

I'm also hopeful that static type checking, aided with annotations,
will help with unicode porting. Duck typing does not work very well
when you are trying to differentiate between bytes and str.

Also, Python 1.5.2 was pretty good :-)

[R. David Murray]

On Tue, 21 Apr 2015 16:55:49 -0000, "Gregory P. Smith" <gr...@krypto.org> wrote:
> We will not be putting type annotations anywhere in the stdlib or expecting
> anyone else to maintain them there. That would never happen until tools
> that are convincing enough in their utility for developers to _want_ to use
> are available and accepted. That'll be a developer workflow thing we could
> address with a later PEP. IF it happens at all.

This is the most reassuring statement I've heard so far ;)

--David

[R. David Murray]

On Tue, 21 Apr 2015 10:10:06 -0700, Guido van Rossum <gu...@python.org> wrote:
> On Tue, Apr 21, 2015 at 9:17 AM, R. David Murray <rdmu...@bitdance.com>
> wrote:
>
> > Please be respectful rather than inflammatory. If you read what I
> > wrote, I did not say that I was going to stop contributing, I
> > specifically talked about that gut reaction being both emotional and
> > illogical. That doesn't make the reaction any less real, and the fact
> > that such reactions exist is a data point you should consider in
> > conducting your PR campaign for this issue. (I don't mean that last as
> > a negative: this issue *requires* an honest PR campaign.)
> >
>
> Well, my own reactions at this point in the flame war are also quite
> emotional. :-(
>
> I have done my best in being honest in my PR campaign. But I feel like the

I believe you have. My inclusion of the word 'honest' was meant to
contrast the kind of PR we need with the kind of PR people typically
think about, which is often not particularly honest.

> opposition (not you, but definitely some others -- have you seen Twitter?)
> are spreading FUD based on an irrational conviction that this will destroy

No, I tend only to peek in there occasionally.

> Python. It will not. It may not prove the solution to all Python's problems
> -- there's always 3.6. (Oh wait, Python 2.7 is perfect. I've heard that
> before -- Paul Everitt famously said the same of Python 1.5.2. Aren't you
> glad I didn't take him literally? :-P )

Yes. But somewhere there or not long before was my introduction to
Python, so I remember it fondly :)

[Carol Willing]

On 4/21/15 9:17 AM, R. David Murray wrote:

Please be respectful rather than inflammatory.  

Thank you David.

If you read what I
wrote, I did not say that I was going to stop contributing, I
specifically talked about that gut reaction being both emotional and
illogical.  That doesn't make the reaction any less real, and the fact
that such reactions exist is a data point you should consider in
conducting your PR campaign for this issue.  (I don't mean that last as
a negative:  this issue *requires* an honest PR campaign.)

As David stated, gut reactions are real. These reactions have the potential, if listened to and respected, to lead toward an optimal (not ideal) solution.

Likely, the implementation of optional static type checking will evolve from reasoned, respectful debate of the issues not inflammatory quotes. Quite frankly, belittling someone's understanding or knowledge does not serve the PEP or technical issues at hand.

I like the option of static type checking for critical high availability and high reliability applications (i.e. air traffic control, financial transactions). I'm less interested in static type checking of inputs when prototyping or developing less critical applications.

There have been good technical points made by many on this thread especially given the different use cases. These use cases, and likely a few more, are important to an honest, continued technical refinement of the PEP.

Two areas of clarification would be helpful for me:

1. Optional: What does this really mean in practice? Am I opting in to static type checking and type hints? Or must I opt out of type hints? Having to opt out would probably put more burden on the educational use cases than opting in would for a large corporate project.

2. Clearly, great thought has been put into this PEP. If anyone has a good analysis of the potential impact on Python 3 adoption, please do pass along. I would be interested in reading the information.

Warmly,
Carol

P.S. I do member a time when tools to easily check for memory leaks in C were new and magical. Looking at the Coverity scans, I'm glad that the old magic is reaping real benefits.

--
Carol Willing
Developer | Willing Consulting
https://willingconsulting.com

[Guido van Rossum]

On Tue, Apr 21, 2015 at 10:03 AM, Carol Willing <will...@willingconsulting.com> wrote:

Two areas of clarification would be helpful for me:

1. Optional: What does this really mean in practice? Am I opting in to static type checking and type hints? Or must I opt out of type hints? Having to opt out would probably put more burden on the educational use cases than opting in would for a large corporate project.

You're definitely opting in. You must do two things to opt in: (a) add type hints to some of your signatures; (b) run a type checker over your code. You will only get the benefits of type hints of you do both, and there is no pressure to opt in. If someone else adds type hints to their code, which you import, but you don't run the type checker, nothing changes for you when you run your code (you can still get away with things that happen to work even the type checker would reject them). When you are reading their source code, you will see their type hints -- like every other aspect of writing good code, some people's type hints will be readable, while others' less so. At least nobody will be writing type hints in Cyrillic. :-)

 

2. Clearly, great thought has been put into this PEP. If anyone has a good analysis of the potential impact on Python 3 adoption, please do pass along. I would be interested in reading the information.

I wish I had a crystal ball, but this is hard to predict. Anecdotally, some people believe this will be catnip, while others believe it to be poison. The truth will surely be somewhere in the middle. At this point we don't know what drives Python 3 adoption except time -- it's definitely going up. :-)

[Paul Sokolovsky]

Hello,

On Tue, 21 Apr 2015 09:50:59 -0700
Ethan Furman <et...@stoneleaf.us> wrote:

> On 04/21, Paul Sokolovsky wrote:
> >
> > And for example yesterday's big theme was people blackmailing that
> > they stop contributing to stdlib if annotations are in [...]
>
> A volunteer's honest reaction is not blackmail, and categorizing it
> as such is not helpful to the discussion.

Sure, that was rather humoresque note. Still, one may wonder why
"honest reaction" is like that, if from reading PEP484 it's clear that
it doesn't change status quo: https://www.python.org/dev/peps/pep-3107
added annotations long ago, and PEP484 just provides default
semantics for them. Note that *default*, not the "only". PEP484 is full
of conditions and long transition windows. Did PEP3107 shutter
everyone's world? No. And PEP484 is nothing but a logical continuation
of PEP3107, coming forward with real use for annotations, but not
intended to shutter everyone's world.

Well, hopefully Guido now clarified what was already written PEP484 (or
not written, like nowhere it says "we add requirement of mandatory
typehints in version 3.x [of stdlib or otherwise]"). But now people try
to come up with *anti*-patterns on how to use type annotation and argue
that these anti-patterns are not useful. Surely, annotations are useful
in some places and not useful in other.

requests doesn't need them? Good. But it's quite useful to annotate FFT
routines and subroutines as taking arrays of floats, we can't get
"faster than C"(tm) without that, like some other languages did (or
claim to have done, and now look attractive). (You don't do FFT in
Python? OMG, that's old.)

--
Best regards,
Paul mailto:pmi...@gmail.com

[Alexander Belopolsky]

On Tue, Apr 21, 2015 at 2:23 PM, Guido van Rossum <gu...@python.org> wrote:

At least nobody will be writing type hints in Cyrillic. :-)

Why not?  It works just fine:

>>> Список = list

>>> def sum(x: Список):

...     pass

...

>>>

(See https://en.wikipedia.org/wiki/Rapira  for some prior art.)

[Łukasz Langa]

On Apr 21, 2015, at 11:23 AM, Guido van Rossum <gu...@python.org> wrote:

2. Clearly, great thought has been put into this PEP. If anyone has a good analysis of the potential impact on Python 3 adoption, please do pass along. I would be interested in reading the information.

I wish I had a crystal ball, but this is hard to predict. Anecdotally, some people believe this will be catnip, while others believe it to be poison. The truth will surely be somewhere in the middle. At this point we don't know what drives Python 3 adoption except time -- it's definitely going up. :-)

Anecdotal evidence shows that some organizations perceive this feature as one that justifies migration. Some of those organizations are pretty serious about open-sourcing things. That makes me believe that by sheer volume of the code they’re producing, Python 3 adoption will continue to increase.

As Gregory Smith rightfully pointed out, nobody wants ugly code. I understand why people are afraid of that and it warms my heart that they are. The community cares so much about aesthetics and readability, it’s great!

We will evolve this over time. This will be a learning process for everybody but we can’t learn to swim by only theorizing about it. We thought of the evolving nature of the solution from Day 1, hence the *provisional* nature of it. The wordy syntax is another example of that. Not requiring changes to the interpreter and the standard library was very high on the list of priorities. Once the *concept* proves itself, then we can improve on the syntax.

Acknowledging PEP 484 being just the second step^ in a long journey is why some “obvious” parts are left out for now (hello, duck typing; hello, multiple dispatch; hello, runtime type checks in unit tests; etc. etc.). Those are often big enough to warrant their own PEPs.

^ PEP 3107 being the first.

-- 
Lukasz Langa | Facebook
Production Engineer | Global Consistency 
(+1) 650-681-7811