type hinting backward compatibility with python 3.0 to 3.4

[Edward Ned Harvey (python)]

I think it's great that for built-in types such as int and str, backward compatibility of type hinting annotations is baked into python 3.0 to 3.4. In fact, I *thought* python 3.0 to 3.4 would *ignore* annotations, but it doesn't...

I'm struggling to create something backward compatible that requires the 'typing' module. For example, the following program is good in python 3.5, but line 11 is a syntax error in python 3.4:

1 import sys
2
3 if sys.version_info[0] < 3:
4 raise RuntimeError("Must use at least python version 3")
5
6 # The 'typing' module, useful for type hints, was introduced in python 3.5
7 if sys.version_info[1] >= 5:
8 from typing import Optional
9
10
11 def divider(x: int, y: int) -> Optional[float]:
12 if y == 0:
13 return None
14 return x / y
15
16 print("22 / 7 = " + str(divider(22, 7)))
17 print("8 / 0 = " + str(divider(8, 0)))
18

When I run this program in python 3.4, I get this:
Traceback (most recent call last):
File "./ned.py", line 11, in <module>
def divider(x: int, y: int) -> Optional[float]:
NameError: name 'Optional' is not defined

[Peter Otten]

Luckily it's a NameError. To backport you only have to define a symbol. The
easiest way to do that is to get hold of a copy of 3.5's typing.py, so let's
try that first:

$ cp /usr/local/lib/python3.5/typing.py .
$ python3.4
Python 3.4.3 (default, Nov 17 2016, 01:08:31)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing import Optional
>>> def divider(x: int, y: int) -> Optional[float]: pass
...
>>>

Works...

[Edward Ned Harvey (python)]

This pattern seems to work:

import sys

if sys.version_info[0] < 3:

raise RuntimeError("Must use at least python version 3")

# The 'typing' module, useful for type hints, was introduced in python 3.5

if sys.version_info[1] >= 5:

from typing import Optional
optional_float = Optional[float]
else:
optional_float = object

def divider(x: int, y: int) -> optional_float:
if y == 0:
return None
return x / y

print("3 / 0 = " + str(divider(3,0)))

[Steve D'Aprano]

On Fri, 19 May 2017 11:35 pm, Edward Ned Harvey (python) wrote:

> I think it's great that for built-in types such as int and str, backward
> compatibility of type hinting annotations is baked into python 3.0 to 3.4.
> In fact, I *thought* python 3.0 to 3.4 would *ignore* annotations, but it
> doesn't...

Why would you think that? Was there something in the documentation or PEP
for annotations that lead you to believe that they were ignored by the
interpreter? If so, that documentation needs improvement.

> I'm struggling to create something backward compatible that requires the
> 'typing' module. For example, the following program is good in python 3.5,
> but line 11 is a syntax error in python 3.4:

You say "syntax error". But it isn't a syntax error, the actual error is:

> When I run this program in python 3.4, I get this:
> Traceback (most recent call last):
> File "./ned.py", line 11, in <module>
> def divider(x: int, y: int) -> Optional[float]:
> NameError: name 'Optional' is not defined

NameError.

You solve this NameError the same way you solve any other NameError: by
making sure that the name "Optional" is defined.

You should actually read the error message you get, and pay attention to it.


(By the way, nobody should be using Python 3.0, it is so heavily bug-ridden
that it is not supported. Most Python 3 devs consider Python 3.3 the first
one worth supporting.)



--
Steve
Emoji: a small, fuzzy, indistinct picture used to replace a clear and
perfectly comprehensible word.

[Gregory Ewing]

Steve D'Aprano wrote:
> On Fri, 19 May 2017 11:35 pm, Edward Ned Harvey (python) wrote:
>
>> I *thought* python 3.0 to 3.4 would *ignore* annotations, but it
>> doesn't...
>
> Why would you think that?

Ever since Guido retconned the purpose of annotations to be
for static type hinting *only*, it would make more sense for
the interpreter to ignore them, or at least not evaluate them
immediately at run time (since it would avoid all the problems
of forward references, etc).

So I can see how someone relying on the principle of least
surprise might assume that.

--
Greg

[Chris Angelico]

They're function metadata. What would the principle of least surprise
say about this?

print("Spam")
def func(arg: print("Foo") = print("Quux")):
print("Blargh")
print("Fred")
func()
print("Eggs")

What should be printed, and in what order?

Actually, Python does violate least-surprise in one area here. There's
one message that gets printed "out of order" compared to my
expectation. I wonder if it's the same one that other people will be
surprised at.

ChrisA

[oliver]

makes sense that the RHS of the equality (deafult value) should be
evaluated before the LHS (arg name + type) but if you hadn't pointed out
that something was not as you expected, i would not have paid attention and
anticipated as you did. Then again RHS vs LHS might have nothing to do with
the reason that the interpreter evaluates in this order :)

On Fri, 19 May 2017 at 21:59 Chris Angelico <ros...@gmail.com> wrote:

> On Sat, May 20, 2017 at 11:42 AM, Gregory Ewing
> <greg....@canterbury.ac.nz> wrote:

> They're function metadata. What would the principle of least surprise
> say about this?
>
> print("Spam")
> def func(arg: print("Foo") = print("Quux")):
> print("Blargh")
> print("Fred")
> func()
> print("Eggs")
>
> What should be printed, and in what order?
>
> Actually, Python does violate least-surprise in one area here. There's
> one message that gets printed "out of order" compared to my
> expectation. I wonder if it's the same one that other people will be
> surprised at.
>
> ChrisA

> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
Oliver
My StackOverflow contributions
My CodeProject articles
My Github projects
My SourceForget.net projects

[Dan Stromberg]

On Fri, May 19, 2017 at 6:35 AM, Edward Ned Harvey (python)
<pyt...@nedharvey.com> wrote:
> I think it's great that for built-in types such as int and str, backward compatibility of type hinting annotations is baked into python 3.0 to 3.4. In fact, I *thought* python 3.0 to 3.4 would *ignore* annotations, but it doesn't...

Perhaps you'd be interested in the typing backport?
https://pypi.python.org/pypi/typing

[Steve D'Aprano]

On Sat, 20 May 2017 11:42 am, Gregory Ewing wrote:

> Steve D'Aprano wrote:
>> On Fri, 19 May 2017 11:35 pm, Edward Ned Harvey (python) wrote:
>>
>>> I *thought* python 3.0 to 3.4 would *ignore* annotations, but it
>>> doesn't...
>>
>> Why would you think that?
>
> Ever since Guido retconned the purpose of annotations to be
> for static type hinting *only*, it would make more sense for
> the interpreter to ignore them, or at least not evaluate them
> immediately at run time (since it would avoid all the problems
> of forward references, etc).

You mean treat them as syntactically comments?

def function(arg:I can put ***ANYTHING*** I like here!!!):
...

I don't think that's a good idea.

Ever since they were introduced, annotations have always been evaluated and
recorded in the function.__annotations__ attribute. That makes them
available at runtime for decorator to do additional processing, or for
introspection.

> So I can see how someone relying on the principle of least
> surprise might assume that.

Oh I don't know, given that everything else apart from #comments is
evaluated at runtime[1] in Python, I think it would be surprising if
annotations weren't.




[1] Apart from some constant folding done as an optimization.

[Steve D'Aprano]

On Sat, 20 May 2017 11:57 am, Chris Angelico wrote:

> They're function metadata. What would the principle of least surprise
> say about this?
>
> print("Spam")
> def func(arg: print("Foo") = print("Quux")):
> print("Blargh")
> print("Fred")
> func()
> print("Eggs")
>
> What should be printed, and in what order?

My prediction:

Spam
Foo
Quux
Fred
Blargh
Eggs

but I wouldn't be the least bit astonished if Foo and Quux are in the
opposite order. As in fact they are.

> Actually, Python does violate least-surprise in one area here. There's
> one message that gets printed "out of order" compared to my
> expectation. I wonder if it's the same one that other people will be
> surprised at.

The reason appears to be that the default arguments are evaluated first,
from left to right, followed by the annotations:


py> def func(a:print(1)=print(2), b:print(3)=print(4), c:print(4)=print(5)):
... pass
...
2
4
5
1
3
4

[Gregory Ewing]

Chris Angelico wrote:
> They're function metadata. What would the principle of least surprise
> say about this?
>
> print("Spam")
> def func(arg: print("Foo") = print("Quux")):
> print("Blargh")
> print("Fred")
> func()
> print("Eggs")

Most languages that have static type declarations wouldn't
let you write something like that in the first place, so the
fact that Python does is surprising to begin with.

--
Greg

[Gregory Ewing]

Steve D'Aprano wrote:
> You mean treat them as syntactically comments?
>
> def function(arg:I can put ***ANYTHING*** I like here!!!):

They could be parsed as expressions and stored as an AST.
That would allow introspection, and you could evaluate them
if you wanted

> Ever since they were introduced, annotations have always been evaluated and
> recorded in the function.__annotations__ attribute.

Yes, but I'm not sure how much good that does you. Because
of the forward-reference thing, you need to be prepared to
get a string instead of a type object, in which case you
need to do some evaluation on it yourself anyway.

--
Greg

[Chris Angelico]

This is true. But since Python _does_ work with dynamic evaluation
(which consequently demands that these kinds of things be expressions
evaluated at compile time), it must by definition be possible to have
side effects. Of course, you would never actually do this in
production code, but I've often made use of the fact that a function's
default argument isn't technically a constant - for example:

DEFAULT_URL = "http://yada.yada.example/"
if testing:
DEFAULT_URL = "http://yada.yada.localhost/"
if staging:
DEFAULT_URL = "http://yada.yada.test.example/"

...

def do_stuff(thing, *, url=DEFAULT_URL):

It has to be evaluated at run time, but from the programmer's point of
view, it's a constant. In C, for instance, this kind of thing would
generally be done with #if and #define.

Like everything in Python, there is a well-defined order of
evaluation, just in case it matters. I wouldn't hazard a guess as to
how often it will, but it's good to know that if it does, you can just
test it once and then be confident :)

ChrisA

[Steve D'Aprano]

On Sun, 21 May 2017 12:14 am, Gregory Ewing wrote:

> Steve D'Aprano wrote:
>> You mean treat them as syntactically comments?
>>
>> def function(arg:I can put ***ANYTHING*** I like here!!!):
>
> They could be parsed as expressions and stored as an AST.
> That would allow introspection, and you could evaluate them
> if you wanted

Ooh, that's a nice idea!

[Gregory Ewing]

Chris Angelico wrote:
> But since Python _does_ work with dynamic evaluation
> (which consequently demands that these kinds of things be expressions
> evaluated at compile time), it must by definition be possible to have
> side effects.

In most languages, type declarations are not expressions
and aren't evaluated at all in the usual sense, so questions
of evaluation order and side effects just don't apply.

If we were designing a static typing system for Python from
scratch, I would be strongly arguing that type hints shouldn't
be expressions evaluated at run time, because it's such a
poor fit for the intended use. Introspectable by all means,
but evaluated by default, no.

--
Greg

[Chris Angelico]

How do you declare that a parameter must be an instance of some class?
Classes are themselves created at run time. Or would your typing
system require that all types be created in some declarable way?

ChrisA

[Gregory Ewing]

Chris Angelico wrote:
> How do you declare that a parameter must be an instance of some class?
> Classes are themselves created at run time. Or would your typing
> system require that all types be created in some declarable way?

Types that you want statically checked have to be described
in a declarative way, because the description is going to be
processed by a tool that is not executing the program.

You wouldn't be able to put a type that can't be described
declaratively into a type annotation, but there would be no
reason to do so in the first place.

--
Greg

[bartc]

They might be /created/ at runtime, but it's a pretty good bet that the
name A in this declaration:

class A...

is the name of a class. The question in Python, as always, is whether an
A used as the name of a type in a type, is still this same A. And
presumably such a type hint can precede the declaration of A.

In fact the declaration of A might be in a different module from its use
in a type hint, which means that, in the CPython byte-code compiler
anyway, it is not visible at compile-time, when type hints could best be
put to good effect.

Furthermore, both A, and the type-hinting code, might be conditional. So
that on Tuesdays, A is a class, the rest of the week it's the name of a
module.

Python doesn't make things easy.

--
bartc

[Chris Angelico]

That kind of difference is fine with an external tool, but it'd make
the language itself inconsistent. For (say) mypy to be able to check
your code, it needs to be able to figure out what classes exist. Fine,
no problem. But for CPython to be able to run your code, it needs to
be able to syntactically verify everything. With mypy, if your classes
are too dynamic, you might have to create a stub file that's more
static (but non-functional) just for the type checking. Can you do
that if your type checker is part of the language? What happens if the
stub file is flat-out wrong? (Again, with mypy, all you break is the
type checker.)

There's a lot of difference between PEP 484 type hints and C-style
static typing. Or even PEP 484 and Pike-style type hinting, where you
can say "object(implements yada yada)" and you'll generally mean "yada
yada or a subclass thereof" (though not necessarily). Pike's static
type checks are only useful if the type in question exists in a static
context - otherwise you have to just use "object" or some other parent
class. (Or just ignore type checking for that. It's a
dynamically-typed language, so static checking is as optional as it is
in Python.)

ChrisA

[Chris Angelico]

On Sun, May 21, 2017 at 7:29 PM, bartc <b...@freeuk.com> wrote:
>
> They might be /created/ at runtime, but it's a pretty good bet that the name
> A in this declaration:
>
> class A...
>
> is the name of a class. The question in Python, as always, is whether an A
> used as the name of a type in a type, is still this same A. And presumably
> such a type hint can precede the declaration of A.
>
> In fact the declaration of A might be in a different module from its use in
> a type hint, which means that, in the CPython byte-code compiler anyway, it
> is not visible at compile-time, when type hints could best be put to good
> effect.

That isn't a problem - mypy follows imports. It'd be pretty useless if
it didn't :)

> Furthermore, both A, and the type-hinting code, might be conditional. So
> that on Tuesdays, A is a class, the rest of the week it's the name of a
> module.

Or, more plausible example: on platforms that have a system-provided
source of entropy, random.random is an instance of SystemRandom, but
on those that don't, it's an instance of DeterministicRandom with an
arbitrarily-chosen seed. The two will have slightly different APIs.

> Python doesn't make things easy.

Python makes things flexible, which has a cost.

ChrisA

[justin walters]

On Sun, May 21, 2017 at 2:38 AM, Chris Angelico <ros...@gmail.com> wrote:

> On Sun, May 21, 2017 at 7:29 PM, bartc <b...@freeuk.com> wrote:
> >

> > They might be /created/ at runtime, but it's a pretty good bet that the
> name
> > A in this declaration:
> >
> > class A...
> >
> > is the name of a class. The question in Python, as always, is whether an
> A
> > used as the name of a type in a type, is still this same A. And
> presumably
> > such a type hint can precede the declaration of A.
> >
> > In fact the declaration of A might be in a different module from its use
> in
> > a type hint, which means that, in the CPython byte-code compiler anyway,
> it
> > is not visible at compile-time, when type hints could best be put to good
> > effect.
>

> That isn't a problem - mypy follows imports. It'd be pretty useless if
> it didn't :)
>

> > Furthermore, both A, and the type-hinting code, might be conditional. So
> > that on Tuesdays, A is a class, the rest of the week it's the name of a
> > module.
>

> Or, more plausible example: on platforms that have a system-provided
> source of entropy, random.random is an instance of SystemRandom, but
> on those that don't, it's an instance of DeterministicRandom with an
> arbitrarily-chosen seed. The two will have slightly different APIs.
>

> > Python doesn't make things easy.
>

> Python makes things flexible, which has a cost.
>
> ChrisA

> --
> https://mail.python.org/mailman/listinfo/python-list
>

Perhaps a good way to distinguish between a class that can be used as a
type and a class
that cannot be used as a type would be to require some sort of dunder
method be
defined on the type class. At first I was thinking `__type__`, but then I
remembered that's
already in use. maybe something as simple as `__hint__`.

That or only allow classes that inherit from `type` to be used in type
annotations.

I'm just spit balling ideas here.

[oliver]

Could something like this support forward declarations without the hackish
use of strings?

On Sun, 21 May 2017 at 12:01 justin walters <walters....@gmail.com>
wrote:

> On Sun, May 21, 2017 at 2:38 AM, Chris Angelico <ros...@gmail.com> wrote:
>
> > On Sun, May 21, 2017 at 7:29 PM, bartc <b...@freeuk.com> wrote:
> > >

> > > They might be /created/ at runtime, but it's a pretty good bet that the
> > name
> > > A in this declaration:
> > >
> > > class A...
> > >
> > > is the name of a class. The question in Python, as always, is whether
> an
> > A
> > > used as the name of a type in a type, is still this same A. And
> > presumably
> > > such a type hint can precede the declaration of A.
> > >
> > > In fact the declaration of A might be in a different module from its
> use
> > in
> > > a type hint, which means that, in the CPython byte-code compiler
> anyway,
> > it
> > > is not visible at compile-time, when type hints could best be put to
> good
> > > effect.
> >

> > That isn't a problem - mypy follows imports. It'd be pretty useless if
> > it didn't :)
> >

> > > Furthermore, both A, and the type-hinting code, might be conditional.
> So
> > > that on Tuesdays, A is a class, the rest of the week it's the name of a
> > > module.
> >

> > Or, more plausible example: on platforms that have a system-provided
> > source of entropy, random.random is an instance of SystemRandom, but
> > on those that don't, it's an instance of DeterministicRandom with an
> > arbitrarily-chosen seed. The two will have slightly different APIs.
> >

> > > Python doesn't make things easy.
> >

> > Python makes things flexible, which has a cost.
> >
> > ChrisA
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
>
> Perhaps a good way to distinguish between a class that can be used as a
> type and a class
> that cannot be used as a type would be to require some sort of dunder
> method be
> defined on the type class. At first I was thinking `__type__`, but then I
> remembered that's
> already in use. maybe something as simple as `__hint__`.
>
> That or only allow classes that inherit from `type` to be used in type
> annotations.
>
> I'm just spit balling ideas here.

[Gregory Ewing]

bartc wrote:
> In fact the declaration of A might be in a different module from its use
> in a type hint, which means that, in the CPython byte-code compiler
> anyway, it is not visible at compile-time, when type hints could best be
> put to good effect.

The static type checker would have to understand enough about
imports to deal with that.

> Furthermore, both A, and the type-hinting code, might be conditional. So
> that on Tuesdays, A is a class, the rest of the week it's the name of a
> module.

But that situation is hardly *static*, so you can't blame a
static type checker for not coping with it.

--
Greg

[Gregory Ewing]

Chris Angelico wrote:
> With mypy, if your classes
> are too dynamic, you might have to create a stub file that's more
> static (but non-functional) just for the type checking. Can you do
> that if your type checker is part of the language?

I think you may have misunderstood. I'm not suggesting that
the runtime interpreter should perform static type checking.
I'm only suggesting that it should treat type annotations
completely passively -- essentially as comments that are
constrained to have the syntax of expressions.

The type checking itself would still be done as a separate
operation that can use stub files, etc. just the same as
mypy does now.

--
Greg