[Python-Dev] In support of PEP 649

[Samuel Colvin]

I've read the recent discussions regarding PEP 649 and PEP 563 with interest, Larry Hastings recently contacted me when prompted to do so in a related discussion.

I maintain pydantic which uses type annotations to provide runtime data validation.

I'm extremely worried that if PEP 649 gets rejected, pydantic will not be able to fully support python 3.10 and may even end up getting abandoned, at best it will be much slower and more brittle.

Please, please put pragmatism over principle and accept PEP 649. As discussed in this issue, using `typing.get_type_hints()` is not a good replacement for type annotations that are accessible as python objects.

I know this is not a popularity contest, but I think it's worth explaining how widely used pydantic is for those who don't use it or haven't heard of it:

• it's downloaded ~7m times a month and is the most popular standalone python data validation library by stars
• it's used (in production and open source projects) by most of the big tech companies you've heard of - it's widely used both in web development and data science
• it is one of the foundations of FastAPI which is by far the fastest growing python web framework and recently came third (behind django and flask) in web frameworks by usage in the python developer survey

Pydantic is not the only one, there are numerous other libraries that use type annotations at runtime.

In short, you may not like it, but runtime inspection of type annotations makes python better, and a massive and growing number of people rely on that.

Rejecting PEP 649 would put that in peril, please accept it.

Samuel

[Christopher Barker]

I haven't commented on these, as I don't use type annotations in the "usual" way.,

but:

On Thu, Apr 15, 2021 at 12:06 PM Samuel Colvin <samc...@gmail.com> wrote:

I maintain pydantic which uses type annotations to provide runtime data validation.

I maintain in in-house system kinda similar to pydantic that doesn't use type annotations for validation, but does use them for other things (mostly serializing/deserializing to/from JSON), and other kinds of data validation.

> using `typing.get_type_hints()` is not a good replacement for type annotations that are accessible as python objects.

Absolutely. It is very, very, handy to simply have the object itself easily accessible.

-CHB

--

Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython

[Bluenix]

Please accept PEP 649!

Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:

```py
import discord
from discord.ext import commands

bot = commands.Bot(command_prefix='>')

@bot.command()
# discord.py reads the typehints and converts the arguments accordingly
async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed
await ctx.send(f'{member.mention}! {text}')

bot.run('token')
```

I must say, this is extremely ergonomic design! Don't break it :)
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/S2O7SE4QZQARAYSCOT7PQUEPNENHDJTQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Paul Ganssle]

I should point out that "accept PEP 649" and "break pydantic" are not the only options here. The thing that will break pydantic is the end of PEP 563's deprecation period, not a failure to implement PEP 649.

Other viable options include:

- Leave PEP 563 opt-in until we can agree on a solution to the problem.
- Leave PEP 563 opt-in forever.
- Deprecate PEP 563 and go back to status quo ante.

I haven't followed this closely enough — if PEP 649 were accepted today, would it even be ready for use before the 3.10 code freeze (which is in a few weeks)?

Assuming this is a real problem (and based in part on how long it took for attrs to get what support it has for PEP 563, I wouldn't be surprised if PEP 563 is quietly throwing a spanner in the works in several other places as well), my vote is to leave PEP 563 opt-in until at least 3.11 rather than try to rush through a discussion on and implementation of PEP 649.

Best,
Paul

[Paul Bryan]

I've been working on a similar framework, Fondat [1], that shares similar objectives as Pydantic and FastAPI, albeit implementing in a different manner. 

I believe other frameworks will have to "run a gauntlet" (as I already have)  to deal with the affects of PEP 563. If anyone on these teams are interested in how I've dealt with such issues, please feel free to contact me directly.

I also recommend members of these teams monitor and/or participate in the typing-sig (and maybe the python-dev) mailing list, as discussions in these lists are likely to affect their work in the future.

Paul 

[1] https://github.com/fondat/fondat-core

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/GT2HQDH2HOZFSOTA5LHTFL5OV46UPKTB/

[Larry Hastings]

On 4/15/21 2:49 PM, Paul Ganssle wrote:

I haven't followed this closely enough — if PEP 649 were accepted today, would it even be ready for use before the 3.10 code freeze (which is in a few weeks)?

Perhaps I'm a poor judge of the quality of my own code.   But I'd say I have a working prototype and it seems... fine.  It didn't require a lot of changes, it just needed the right changes in the right spots.  Getting my interactions with symtable and compile were the hardest parts and I think those are all sorted out now.

Cheers,

/arry

[Bernat Gabor]

Hello,

I'm a bit surprised that this topic is brought up just days before the feature freeze of Python 3.10.  How did tools work until now with from future import annotations? This feature has been in the language for years (since 3.7), and I haven't heard until today anyone complaining about this feature gap. Did everyone just told users not to use lazy evaluation of type hints until now? Also there's been advertised for at least a year that next python will make it default 🤔

Thanks,

Bernat

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/PBGFXPC7D3NFLJSFQBDAQQCW6JCMVCFC/

[Christopher Barker]

On Thu, Apr 15, 2021 at 3:29 PM Bernat Gabor <gaborj...@gmail.com> wrote:

> I'm a bit surprised that this topic is brought up just days before the feature freeze of Python 3.10.

I have not idea about the technical details, but I think there is a bit of a discomment in the community:

Annotations have been around for years.

More recently, there was an official specification of how to use them for type hinting (was that PEP 484? a bit confused as that appears to still be provisional -- but anyway)

Since the specification for type hinting, a lot of work has been done on/with static type checkers (e.g. MyPy).

That work has informed the changes / improvements / proposals to the annotation and type system in Python.

However, those of us that are not actively involved (and maybe not that interested) in static typing have been not paying a great deal of attention to those efforts.

But some of us may, in fact, be using annotations for other reasons, but since we haven't been involved in the discussion, some issues may have been missed.

-CHB

[Brett Cannon]

On Thu, Apr 15, 2021 at 12:01 PM Samuel Colvin <samc...@gmail.com> wrote:

I've read the recent discussions regarding PEP 649 and PEP 563 with interest, Larry Hastings recently contacted me when prompted to do so in a related discussion.

I maintain pydantic which uses type annotations to provide runtime data validation.

I'm extremely worried that if PEP 649 gets rejected, pydantic will not be able to fully support python 3.10 and may even end up getting abandoned, at best it will be much slower and more brittle.

Please, please put pragmatism over principle and accept PEP 649.

Please don't phrase the decision in these terms. While for Pydantic PEP 649 is more pragmatic, that does not mean PEP 563 isn't pragmatic for other people for other reasons. Making this an "us versus them" discussion just makes the whole situation feel confrontational when instead everyone is trying to figure out the best thing for everybody when there's no perfect answer.

[Paul Ganssle]

It seems evident that the problems with PEP 563 have been well-known at least to pydantic for several years, as you can see in the issue Samuel Colvin linked: https://github.com/samuelcolvin/pydantic/issues/2678

That said, while I do think that "please contact upstream when you see a problem developing, not just before a major deadline" is a good lesson to take away from this, it is probably not worth litigating the question of the particular manner of the notification. As much as I think it would have been good for this discussion to happen 6 months, 1 year, 2 years or 3 years ago, "before the code freeze" is a heck of a lot better than "after the release" (which is often when the notification comes in, and why we have been encouraging people to test against alphas and betas).

Hopefully we can let it rest by saying that the timing of learning about this apparently urgent situation could have been much better, but it could have been worse as well.

Best,
Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/5TGKKRZXGQROQKS2WX6WFGCHTMNUJYBF/

[Christopher Barker]

On Thu, Apr 15, 2021 at 4:20 PM Brett Cannon <br...@python.org> wrote:

 Making this an "us versus them" discussion just makes the whole situation feel confrontational when instead everyone is trying to figure out the best thing for everybody when there's no perfect answer. 

I agree that that was strong language, but from the perspective of folks not interested in Static Typing, there has been a lot of activity, or at least discussion,  that makes it seem like this optional feature is driving the evolution of the language. 

So folks are going to get concerned, and maybe upset, if it looks like changes might be made that may break other features in order to make static typing work better.

And as I noted in my last post — many folks have not been paying attention to the typing discussions because they didn’t realize it concerned them.

-CHB

[Guido van Rossum]

On Thu, Apr 15, 2021 at 16:48 Christopher Barker <pyth...@gmail.com> wrote:

And as I noted in my last post — many folks have not been paying attention to the typing discussions because they didn’t realize it concerned them.

It seems a little disingenuous to claim discussions about annotations don’t concern you when you’re actively using them (for typing, no less, in the case of pydantic). And I am sure a project as popular (by their own description) as pydantic will find a way forward if PEP 649 is rejected, despite overdramatized claims.

That said, I agree it is better that this came up before the feature freeze than after the release. And I am willing to accept that the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week) is less important than backwards compatibility.

--

--Guido (mobile)

[Oscar Benjamin]

On Fri, 16 Apr 2021 at 01:13, Guido van Rossum <gu...@python.org> wrote:
>
> On Thu, Apr 15, 2021 at 16:48 Christopher Barker <pyth...@gmail.com> wrote:
>>
>> And as I noted in my last post — many folks have not been paying attention to the typing discussions because they didn’t realize it concerned them.
>
> It seems a little disingenuous to claim discussions about annotations don’t concern you when you’re actively using them (for typing, no less, in the case of pydantic). And I am sure a project as popular (by their own description) as pydantic will find a way forward if PEP 649 is rejected, despite overdramatized claims.

It looks to me like pydantic were well aware that this affected them
and have been working over time to try and support this. They probably
also had their eye on PEP 649 or hoped that something else would come
along to fix this but they can now see the crunch point approaching.

As Chris says though other libraries/developers have been using
annotations for some time and might not have realised that typing
considerations could undermine their other uses for annotations since
annotations predate typing and typing is an optional feature.

> That said, I agree it is better that this came up before the feature freeze than after the release. And I am willing to accept that the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week) is less important than backwards compatibility.

Would it be problematic to postpone making __future__.annotations the default?


Oscar

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/VCFHDQFOBQXZDKKPSTU5FNKP2AZIJMX2/

[Barry Warsaw]

On Apr 15, 2021, at 17:47, Oscar Benjamin <oscar.j....@gmail.com> wrote:
>
> Would it be problematic to postpone making __future__.annotations the default?

This is a good question, and I think the SC would really like to know if anybody has objections to postponing this to 3.11. We haven’t discussed it yet (this whole topic is on our agenda for next Monday), but it might be the best thing to do given where we are in the 3.10 release cycle. It would give everyone a chance to breathe and come up with the right long term solution.

Cheers,
-Barry

[Inada Naoki]

On Fri, Apr 16, 2021 at 9:49 AM Oscar Benjamin
<oscar.j....@gmail.com> wrote:
>
>
> > That said, I agree it is better that this came up before the feature freeze than after the release. And I am willing to accept that the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week) is less important than backwards compatibility.
>
> Would it be problematic to postpone making __future__.annotations the default?
>

__future__.annotation is the default since 2020-10-04.
https://github.com/python/cpython/commit/044a1048ca93d466965afc027b91a5a9eb9ce23c#diff-ebc983d9f91e5bcf73500e377ac65e85863c4f77fd5b6b6caf4fcdf7c0f0b057

After that, many changes are made on compiler and other places.
So reverting the change is not so simple.

And personally, I love static typing but I don't use type hint for
performance/memory usage reason.
I spend much effort to optimize PEP 563 to minimize type hinting overhead.
So it's very sad that if I can not use type hinting when I can drop
Python 3.9 support.

So if PEP 649 is accepted, I want to use it since Python 3.10.
Otherwise, I can not use type hinting even after I dropped Python 3.9
support.

But it is up to release manager and steering council.

--
Inada Naoki <songof...@gmail.com>

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/6ZYCD63KBP2EPDZIYJD2IPHCVRV4LQGP/

[Guido van Rossum]

As long as you know that’s not zero cost either. Undoing the work will be complex (merge conflicts will abound) and and all testing and implementation people have done assuming PEP 563 semantics will still have to be redone.

And note that there are some corner cases where the “PEP 563 semantics” in 3.10 could not be made fully compatible with the behavior under `from __future__ Import annotations` in 3.9 (in truth, because not enough testing was done under those semantics in 3.7-3.9).

But it could be done, and that’s the way PEP 649 is currently written (using `from __future__ import co_annotations`).

--

--Guido (mobile)

[Larry Hastings]

I don't have any objections, and I certainly see the wisdom in such a decision.

Best wishes,

/arry

[Inada Naoki]

On Fri, Apr 16, 2021 at 6:03 AM Bluenix <bluen...@gmail.com> wrote:
>
> Please accept PEP 649!
>
> Python's type hinting has become so much more useful than originally thought, and without this change much of that will be hindered. For example (you already know about Pydantic and FastAPI) [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you to use typehinting to specify how arguments should be converted. Take the following code:
>
> ```py
> import discord
> from discord.ext import commands
>
> bot = commands.Bot(command_prefix='>')
>
> @bot.command()
> # discord.py reads the typehints and converts the arguments accordingly
> async def reply(ctx, member: discord.Member, *, text: str): # ctx is always passed
> await ctx.send(f'{member.mention}! {text}')
>
> bot.run('token')
> ```
>
> I must say, this is extremely ergonomic design! Don't break it :)

Are you sure about PEP 563 break it and discord.py can not fix it?

As far as my understanding, PEP 563 don't hurt this use case so much:

* This use case evaluates runtime type information only once. So
eval() overhead is not a problem.
* Currently, annotation is very very complex and varies. For example,
List[int] will be `List[int]`, `List['int']`, `'List[int]'`,
`'List["int"]'`, `List[ForwardRef('int')]` etc...
After PEP 563, only `'List[int]'` is practical so we can stop
supporting `List["int"]` and others at some point.
So playing with runtime type will become easier in the future.

Am I wrong?

--
Inada Naoki <songof...@gmail.com>

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/NSZGWCABWFYWZZTNCE5VE5ZVC3OUJCNU/

[Antoine Pitrou]

On Fri, 16 Apr 2021 10:35:19 +0900
Inada Naoki <songof...@gmail.com> wrote:
>
> And personally, I love static typing but I don't use type hint for
> performance/memory usage reason.
> I spend much effort to optimize PEP 563 to minimize type hinting overhead.
> So it's very sad that if I can not use type hinting when I can drop
> Python 3.9 support.

AFAIU, we're not talking about dropping PEP 563 (yet), but about making
it opt-in again.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/NVJOKSXRFEUKWHF4USVOB2U3YTCL3PZL/

[Tin Tvrtković]

Guido:

> It seems a little disingenuous to claim discussions about annotations don’t
> concern you when you’re actively using them (for typing, no less, in the
> case of pydantic). And I am sure a project as popular (by their own
> description) as pydantic will find a way forward if PEP 649 is rejected,
> despite overdramatized claims.

I also maintain a library that uses type annotations in a runtime context (cattrs), but in a slightly different way than Pydantic. My project is much less popular than Pydantic (to be expected, since it only deals with de/serialization and the class definition layer is attrs/dataclasses), and the first issue filed on my bug tracker about future annotations was created at the end of 2019, so that's when I started thinking about it.

To be honest, supporting stringified annotations was a lot of work to do in an efficient way (and that's the main reason cattrs has two converter classes now), but there was plenty of time and I simply did the work. The new design is even faster than the old one.

[Jukka Lehtosalo]

On Fri, Apr 16, 2021 at 8:27 AM Inada Naoki <songof...@gmail.com> wrote:

  After PEP 563, only `'List[int]'` is practical so we can stop
supporting `List["int"]` and others at some point.

There's a lot of code written before PEP 563 was available (and code that needs to support earlier Python versions) that uses string literal quoting in various places. If Python stops supporting List["X"], it will be a significant backward compatibility break. Some of these string literal escapes are made unnecessary by PEP 563, but there are still cases where string literals are needed that neither PEP 563 nor PEP 649 address. Examples include type aliases, type variable definitions and base classes, but there are others [1]. These aren't evaluated in a type annotation context, so manual string literal escaping would still be needed.

 

  So playing with runtime type will become easier in the future.

Am I wrong?

In some cases things will become easier, but other common use cases still seem to be unresolved. I haven't seen any proposal that can completely replace all uses of string literal escapes. I guess one option would be to only allow string literal escaping to be used outside type annotations in the future. I think that this could be feasible with a suitable deprecation period.

Also, the use of "if TYPE_CHECKING" breaks the runtime use of annotations. PEP 563 and PEP 649 reduce the need for string literal escaping in this use case, but it mostly helps static type checkers. This is pretty common in codebases that use static type checking. My main issue with PEP 649 is that it only addresses a subset of remaining issues, i.e. it doesn't go far enough. It's also not clear to me if PEP 649 (or PEP 563) can be extended to cover the remaining runtime issues, or if we'd need a *third* approach to solve them.

Jukka

[1] https://www.python.org/dev/peps/pep-0563/#forward-references

[Danny]

> Are you sure about PEP 563 break it and discord.py can not fix it?

PEP 563 has been supported by discord.py since the PEP was introduced -- the change was not particularly hard to support on my end. Newer versions of the library are even more supportive of things like `ForwardRef` and the like.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/Z4FDBM2NKDSY4LKAJ7JY2ZOXJQVL22YD/

[Samuel Colvin]

Thank you everyone for your responses.

I entirely accept that I should have brought this up earlier, perhaps much earlier.

In my defence, when PEP 563 first came on my radar I assumed that get_type_hint() would be improved before it became the default behaviour, AFAIK it hasn't really changed. In particular the subtle changes to scope described in detail in PEP 649 [1] are still there and are still a big headache for pydantic.

As Paul Ganssle says, my timing was not great, but it could also have been worse.

I also never meant to create an "us versus them" style discussion, I was just worried about the narrow time window I had and the likelihood that my voice would not be heard, hence sounding as impassioned as I did.

For a fairly comprehensive description of why PEP 563 represents challenges to pydantic, I defer to Łukasz Langa (the instigator of PEP 563) who has explained it very clearly in another thread "[Python-Dev] PEP 563 in light of PEP 649" [2].

Samuel

[1] - https://www.python.org/dev/peps/pep-0649/#id9
[2] - https://mail.python.org/archives/list/pytho...@python.org/message/ZBJ7MD6CSGM6LZAOTET7GXAVBZB7O77O/

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/CJTAJJEXZXM5WKJ47POLK3YWWNPKK5NB/

[Christopher Barker]

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I did find this SO post:

https://stackoverflow.com/questions/52171982/new-annotations-break-inspection-of-dataclasses

which is related, but not quite the same -- THAT issue was fixed.

My issue does show up in this BPO:

https://bugs.python.org/issue39442

Somewhere in the middle of that thread, Eric wrote: "At this point, this seems more like fodder for python-ideas."

Then there's a bit more to the thread, which peters out without resolution.

On to the issue:

dataclasses may be the only part of the stdlib that uses annotations.

dataclasses store the annotations in Field objects type attribute, in the __dataclass_fields__ attribute. 

We can see that with this code:

@dataclass
class Simple:
    x : int
    y : float
    name : str

s = Simple(3, 4.0, "fred")

print("The field types:")
for f in s.__dataclass_fields__.values():
    print(f"name: {f.name}, type: {f.type}, type of type: {type(f.type)}")

Which results in:

The field types:
name: x, type: <class 'int'>, type of type: <class 'type'>
name: y, type: <class 'float'>, type of type: <class 'type'>
name: name, type: <class 'str'>, type of type: <class 'type'>

with:

from __future__ import annotations

The result is:

The field types:
name: x, type: int, type of type: <class 'str'>
name: y, type: float, type of type: <class 'str'>
name: name, type: str, type of type: <class 'str'>

This of course is completely as expected.

I have no idea how dataclasses uses the Field type attribute -- as far as I can tell, for nothing at all. However, it is there, and it is called "type", rather than say, "annotation".

And I have a whole pile of code that fully expects the Fields' type attribute to be an actual type object that I can call to crate an instance of that type (or call a method on it, which is what I am actually doing)

So my code will very much break with this change.

I fully understand that the __dataclass_fields__ attribute was probably never intended to be part of the public API, so I get what I deserve. 

However, the Field object is documented, as such:

"""

class dataclasses.Field

Field objects describe each defined field. These objects are created internally, and are returned by the fields() module-level method (see below). Users should never instantiate a Field object directly. Its documented attributes are:

name: The name of the field.
type: The type of the field.
default, default_factory, init, repr, hash, compare, and metadata have the identical meaning and values as they do in the field() declaration.

Other attributes may exist, but they are private and must not be inspected or relied on.

"""

That last sentence implies that the type attribute CAN be inspected and relied upon, which is what I am doing.

And I haven't tried to solve this problem in my use case, but I'm not sure it can be done -- when I get around to inspecting the type of the Field objects, I have no idea what namespace they are in -- so I can't reconstruct them from the string. I really need the type object itself.

So I'll probably need to re-write much of the dataclasses decorator, to call eval() early -- though even then I'm not sure I'll have access to the proper namespace.

Anyway -- one more data point:  PEP 563 changes the (semi-public?) API of dataclasses.

Though *maybe* that could be addressed with a dataclasses update -- again, I've only started to think about it -- there was some discussion of that in the BPO, though Eric didn't seem particularly interested.

-CHB

[Eric V. Smith]

On 4/17/2021 12:28 AM, Christopher Barker wrote:

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I have!

In retrospect, that field probably should have been named "annotation". Regardless, the intent was always "store what's in __annotations__[field_name]", or what the user specified in field(..., type=whatever, ...).

And I have a whole pile of code that fully expects the Fields' type attribute to be an actual type object that I can call to crate an instance of that type (or call a method on it, which is what I am actually doing)

So my code will very much break with this change.

True, unfortunately. To be clear to everyone not paying close attention, "this change" is PEP 563.

I fully understand that the __dataclass_fields__ attribute was probably never intended to be part of the public API, so I get what I deserve. 

However, the Field object is documented, as such:

"""

class dataclasses.Field

Field objects describe each defined field. These objects are created internally, and are returned by the fields() module-level method (see below). Users should never instantiate a Field object directly. Its documented attributes are:

name: The name of the field.
type: The type of the field.
default, default_factory, init, repr, hash, compare, and metadata have the identical meaning and values as they do in the field() declaration.

Other attributes may exist, but they are private and must not be inspected or relied on.

"""

That last sentence implies that the type attribute CAN be inspected and relied upon, which is what I am doing.

Yes, Field.type is very much part of the public dataclasses API as available through dataclasses.fields(), not through cls.__dataclass_fields__.

And I haven't tried to solve this problem in my use case, but I'm not sure it can be done -- when I get around to inspecting the type of the Field objects, I have no idea what namespace they are in -- so I can't reconstruct them from the string. I really need the type object itself.

@dataclass pretty much has the same problem with respect to calling typing.get_type_hints().

So I'll probably need to re-write much of the dataclasses decorator, to call eval() early -- though even then I'm not sure I'll have access to the proper namespace.

Anyway -- one more data point:  PEP 563 changes the (semi-public?) API of dataclasses.

Though *maybe* that could be addressed with a dataclasses update -- again, I've only started to think about it -- there was some discussion of that in the BPO, though Eric didn't seem particularly interested.

I still think it's working as intended: it uses what's in __annotations__ (or passed in to field()). As everyone who has tried to call typing.get_type_hints() knows, it's hard to get right, if it's even possible, because, as you say "when I get around to inspecting the type of the Field objects, I have no idea what namespace they are in". My opinion is that the person who wants a real type object probably has a better idea of that namespace than @dataclass does, but there's a very good chance they don't know, either.

@dataclass goes out of its way to not call typing.get_type_hints(). The original reason for this is not wanting to force typing to be imported, if it wasn't already being used. That may have been addressed with PEP 560, but I've never really checked on the impact.

Another reason for not calling typing.get_type_hints(): there's really only one thing [*] dataclasses wants to know, with regard to the type/annotation of the field: is the type of this field typing.ClassVar? It doesn't seem that the performance issues and possible failures make it worth calling typing.get_type_hints() just for this case. @dataclass uses other tricks (not described here).

In any event, all of this mess is one reason I'd like to see PEP 649 get accepted: there would never be a reason to call typing.get_type_hints(), and the values in the Field object would again be type objects.

Back to my original point: If you ignore the test for ClassVar, then dataclasses completely ignores the values in __annotations__ or Field.type. It's no different from typing.NamedTuple in that regard.

I do have sympathy for users looking at Field.type and getting a string instead of a type object: but that's really no different from non-dataclass users looking at any occurrence of __annotations__ and now getting a string: that's a result of PEP 563 across the board, not just with dataclasses. As I said in bpo-39442:

Isn't that the entire point of "from __future__ import annotations"?

Eric

[*]: Actually two things: the other being "is the field of type dataclasses.InitVar?". It has some of the same problems as ClassVar, but we know that dataclasses has been imported, so it's less of a big deal.

[Jelle Zijlstra]

El vie, 16 abr 2021 a las 21:32, Christopher Barker (<pyth...@gmail.com>) escribió:

...

dataclasses may be the only part of the stdlib that uses annotations.

There's another one, actually: functools.singledispatch, at https://github.com/python/cpython/blob/3.9/Lib/functools.py#L860. It uses the annotations on the first argument to an `@register`ed function to figure out how to dispatch. Currently it uses `typing.get_type_hints` to get the type. Presumably, this would break with PEP 563 semantics if you try to create a singledispatch function in a nested namespace.

[jacob....@gmail.com]

Related, `inspect.Parameter.annotation` is affected too, but at least this attribute is called `annotation` instead of `type`.

I noticed this today with `multipledispatch` (though [reported](https://github.com/mrocklin/multipledispatch/issues/104) in 2019) and some other internal code, both using `inspect.signature`. `multipledispatch` could reasonably swap to `typing.get_type_hints`, but the other code I was working with also used `Parameter.default` and `Parameter.kind`, so would require both `inspect.signature` and `get_type_hints`. Not the worst and arguably necessary now/pre PEP 563 with explicit string annotations, but the PEP 649 semantics would be a welcome improvement.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/RW7NUODC5YLSEXDNE4DBN6TGFGMGVF4L/

[Christopher Barker]

On Sat, Apr 17, 2021 at 6:12 AM Eric V. Smith <er...@trueblade.com> wrote:

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I have!

Thanks Eric. 

 

In retrospect, that field probably should have been named "annotation". Regardless, the intent was always "store what's in __annotations__[field_name]", or what the user specified in field(..., type=whatever, ...).

intent is all well and good, but what was the intent of __annotations__ In the first place. Static typing was certainly one of the intents, but as it DID store a value, not a string from the beginning, we can only assume that there was some intent that the value might be used.

As for dataclasses -- why store it at all? It seems that dataclasses use the fact that a class attribute has an annotation to determine what to add to the field -- which is a nifty use of teh syntax. But presumably the intent of storing the annotation in the Field object, and providing a public attribute for it, was so that it could be used.

I suppose your point is that the type attribute stored the annotation, and PEP 653 changes what the annotation will be so there's nothing specific at all about dataclasses here. Which is true. But users of dataclasses may have been less aware of all the implications as they didn't write that annotation etrating code themselves. I know I wasn't.

In any event, all of this mess is one reason I'd like to see PEP 649 get accepted: 

Indeed -- that is the title of this thread, after all :-)

And see others' notes, there seems to be two other places in the stdlib that will be affected.

[Eric V. Smith]

On 4/17/2021 6:44 PM, Christopher Barker wrote:

On Sat, Apr 17, 2021 at 6:12 AM Eric V. Smith <er...@trueblade.com> wrote:

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I have!

Thanks Eric. 

 

In retrospect, that field probably should have been named "annotation". Regardless, the intent was always "store what's in __annotations__[field_name]", or what the user specified in field(..., type=whatever, ...).

intent is all well and good, but what was the intent of __annotations__ In the first place. Static typing was certainly one of the intents, but as it DID store a value, not a string from the beginning, we can only assume that there was some intent that the value might be used.

Yes, but wouldn't that say that PEP 563 could never be implemented then, since it would break that intent? We already knew it broke all code that was expecting a type object in __annotations__. I'm just saying my expectation is 563 will break any code expecting a type object in __annotations__, or anything that's derived from __annotations__ (specifically in my case, dataclass users looking at Field.type). My probably unarticulated intent was that Field.type agree with __annotations__[field_name].

As for dataclasses -- why store it at all? It seems that dataclasses use the fact that a class attribute has an annotation to determine what to add to the field -- which is a nifty use of teh syntax. But presumably the intent of storing the annotation in the Field object, and providing a public attribute for it, was so that it could be used.

I didn't want to predict what someone was using dataclasses for. Especially someone using field(..., metadata=something) might want access to any of the field() values, including .type. I supposed I could have said "for defaults, default_factory, repr, etc., look at the Field object, but for the type look at __annotations__[field_name]", but I decided it would be easier if everything about a field would be in the Field object. Not that it makes much difference: I believe that Field.type always agrees with __annotations__[field_name].

This allows someone to say (where some_dataclass_field_object is a Field):

process_field(some_dataclass_field_object)

instead of this, if the .type field weren't present:

process_field(some_dataclass_field_object, SomeDataclass.__annotations__[some_dataclass_field_object.name])

I suppose your point is that the type attribute stored the annotation, and PEP 653 changes what the annotation will be so there's nothing specific at all about dataclasses here. Which is true. But users of dataclasses may have been less aware of all the implications as they didn't write that annotation etrating code themselves. I know I wasn't.

In any event, all of this mess is one reason I'd like to see PEP 649 get accepted: 

Indeed -- that is the title of this thread, after all :-)

Well, not everyone is not in support of the subject!

Eric

And see others' notes, there seems to be two other places in the stdlib that will be affected.

-CHB

--

Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython

-- 
Eric V. Smith

[Alex Hall]

I'd like to chime in with an example of how PEP 563 breaks code that uses dataclasses.

I've written a library instant_api (https://github.com/alexmojaki/instant_api) that is heavily inspired by FastAPI but uses dataclasses for complex types instead of pydantic. The example at the beginning of the README is short and demonstrates it nicely. Basically it lets you write code on both the client and server sides that work seamlessly with standard dataclasses, type hints, and type checkers without any plugins, instead of untyped dicts parsed from the JSON that is communicated behind the scenes.

`from __future__ import annotations` breaks that README example, even though there are no locally defined types, because as mentioned the dataclass field now contains a string instead of a type.

Going a bit deeper, instant_api is powered by https://github.com/alexmojaki/datafunctions, which is more generic than instant_api so that others can build similar tools. Again, the idea is that you can write code with nice dataclasses and type hints, but call it with basic JSON serializable types like dicts. For example:

```
from dataclasses import dataclass
from datafunctions import datafunction

@dataclass
class Point:
x: int
y: int

@datafunction
def translate(p: Point, dx: int, dy: int) -> Point:
return Point(p.x + dx, p.y + dy)

assert translate({"x": 1, "y": 2}, 3, 4) == {"x": 4, "y": 6}

# This is equivalent to the following without @datafunction
# assert translate(Point(1, 2), 3, 4) == Point(4, 6)
```

In the same way as before, `from __future__ import annotations` breaks this code. The reason is that datafunctions itself is powered by https://github.com/lovasoa/marshmallow_dataclass. Here's an example:

```
from dataclasses import dataclass
from marshmallow_dataclass import class_schema

@dataclass
class Point:
x: int
y: int

schema = class_schema(Point)()
assert schema.load({"x": 1, "y": 2}) == Point(1, 2)
```

Again, in the same way as before, `from __future__ import annotations` breaks this code. Specifically `class_schema(Point)` breaks trying to deal with the string `'int'` instead of a type.

This problem was raised in https://github.com/lovasoa/marshmallow_dataclass/issues/13 two years ago. It's by far the oldest open issue in the repo. It was clear from the beginning that it's a difficult problem to solve. Little progress has been made, there's one PR that's not in good shape, and it seems there's been no activity there for a while. A couple of other issues have been closed as duplicates. One of those issues is about being unable to use recursive types at all.

marshmallow_dataclass has 266 stars. It builds on https://github.com/marshmallow-code/marshmallow, an extremely popular and important data (de)serialization and validation library. Here's a little timeline:

- 2013: marshmallow 0.1.0 first released in 2013
- 2014: marshmallow 1.0.0 released
- 2015: attrs (precursor to dataclasses) first released
- 2016: Python 3.6.0 final released, allowing the variable annotations which make pydantic and dataclasses possible.
- 2017: First version of pydantic released
- 2018: Python 3.7.0 final released, introducing dataclasses

Nowadays pydantic is the natural successor/alternative to marshmallow - Google autocompletes "pydantic vs " with marshmallow as the first option, and vice versa. But marshmallow is clearly well established and entrenched, and thanks to marshmallow_dataclass it was the better fit for my particular use case just last year when I made instant_api.

If someone wants to keep combining dataclasses and marshmallow, but without marshmallow_dataclass (e.g. if PEP 563 goes through before marshmallow_dataclass is ready) then they need to revert to the raw marshmallow API which doesn't use type hints. The previous example becomes much uglier:

```
from dataclasses import dataclass
from marshmallow import Schema, fields, post_load

@dataclass
class Point:
x: int
y: int

class PointSchema(Schema):
x = fields.Int()
y = fields.Int()

@post_load
def make_point(self, data, **kwargs):
return Point(**data)

schema = PointSchema()
assert schema.load({"x": 1, "y": 2}) == Point(1, 2)
```

This post turned out longer than I initially planned! In summary, my point is that type hints and dataclasses as they work right now make it possible to write some really nice code - nice for humans to both write and read, nice for type checkers and other static analysis, and providing very nice features using annotations at runtime. And despite clear demand and benefits and ample time, people haven't managed to make this code continue working with stringified type annotations. Clearly doing so is not easy. So there's a good case for the dataclasses module to resolve these annotations to actual types, especially if PEP 563 goes through but even if it doesn't.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/GPPWPKQ2RHABB64REWHE6HDHRNRFUXXM/

[Inada Naoki]

As far as I know, both Pydantic and marshmallow start using annotation
for runtime type after PEP 563 is accepted. Am I right?
When PEP 563 is accepted, there are some tools using runtime type in
annotation, but they are not adopted widely yet.

But we didn't have any good way to emit DeprecationWarning for use
cases that PEP 563 breaks.
Maybe, we should have changed the default behavior soon in Python 3.8.
With long preparation period without DeperecationWarning, use cases
broken by PEP 563 beccome much.

I still love PEP 563. Python is dynamic language so runtime type and
static type can not be 100% consistent.
Annotation is handy tool. But there is only one annotation per object.
When we start using annotation for multiple purposes, it becomes ugly
monster soon.

Using annotation syntax only for static typing like TypeScript is the ideal.
TypeScript is far more succeeded than Python about Gradual Typing.

But, current status is went to where I hate already. Annotation is
used for multiple purposes already.
I'm sad and disappointed about it, but that's that.

I'm OK to keep PEP 563 opt-in. And I think we should do it. (off
course, I'm not a SC member. I follow the SC decision).

And if we keep PEP 563 opt-in, no need to compare PEP 563 with PEP 649.
PEP 649 should be compared with "stock semantics + opt-in PEP 563 semantics".

Of course, one semantics is better than two semanticses.
But we need to have three semanticses until we can remove stock + PEP
563 semantices.
We should think about PEP 649 very carefully, spending more time.

Regards,

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/BYSKN5TXEVIOHN6WVMSFRIVMF66OEPOT/

[Guido van Rossum]

Hi Samuel,

I have a naive question about pydantic. IIRC there's something in your tracker about not being able to handle recursive definitions correctly with PEP 563. But I must be misremembering, because PEP 563 actually makes that easier, not harder. Without PEP 563, how does it handle classes like this?

@<pydantic decorator>

class C:

    def method(self) -> C: ...

(My apologies for not doing more research, but I figured this would be quicker for you to answer than for me to research, given that I've never used pydantic.)

--

--Guido van Rossum (python.org/~guido)

Pronouns: he/him (why is my pronoun here?)

[Samuel Colvin]

There are a number of issues around recursive types, in general PEP 563 doesn't make a big difference either way in this case.

I think you mean something like

from pydantic import BaseModel
from typing import Optional

class Foo(BaseModel):
    x: int
    foo: Optional['Foo']

Foo.update_forward_refs()
print(Foo(x=1, foo={'x': 1}))

The only difference with "from __future__ import annotations" is you can unquote Foo, e.g. Optional[Foo]. In all cases you still need "Foo.update_forward_refs()" since Foo is not defined while creating Foo.

You could also use ForwardRef: "FooType = ForwardRef('Foo')...Optional[FooType]", but that's slightly more verbose and doesn't add anything.

(Sidenote: as I'm writing this, I realise you could know what Foo is while creating Foo (same name -> reference to itself) and perhaps automatically call update_forward_refs() at the end of ModelMetaclass.__new__(), but you still need the public update_forward_refs() for more complex cases; not least bodging around PEP 563 scopes, see here.)

Samuel

[Samuel Colvin]

As far as I know, both Pydantic and marshmallow start using annotation
for runtime type after PEP 563 is accepted. Am I right?

Not quite, pydantic was released in June 2017 (see HN post: https://news.ycombinator.com/item?id=14477222) and always used annotations, PEP 563 was created in September 2017, a few months later. 

I can't speak about Marshmallow, back then I don't think it used annotations at all, but maybe it does now.

When PEP 563 is accepted, there are some tools using runtime type in
annotation, but they are not adopted widely yet.

Yes, that's completely fair.

Samuel