[Python-Dev] PEP 646 (Variadic Generics): final call for comments

[Matthew Rahtz via Python-Dev]

Hi everyone,

We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council.

If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.com/python/peps/pull/1880 clarifies which grammar changes would be required, now that PEP 637 has been rejected. We also have a second PR in progress at https://github.com/python/peps/pull/1881 clarifying some of the motivation.)

Thanks!

Matthew and Pradeep

[Stephan Hoyer]

I just wanted to thank Matthew & Pradeep for writing this PEP and for clarifications to the broader context of PEP 646 for array typing in https://github.com/python/peps/pull/1904.

As someone who is heavily involved in the Python numerical computing community (e.g., NumPy, JAX, Xarray), but who is not so familiar with the details of Python's type system, it is reassuring to see that a broad range of use-cases related to type checking of named axes & shapes have been considered, and could build upon the infrastructure in this PEP.

Type checking for shapes is something the NumPy community is very interested in -- there are more thumbs up on the relevant issue on NumPy's GitHub than any others (https://github.com/numpy/numpy/issues/7370) and we recently added a "typing" module that is under active development.

It will certainly require experimentation to figure out the best ways to use type checking for ndarrays, but this PEP looks like an excellent foundation for such work.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/5GWS2GGVJ4PAMOWM6YVVKZVMR5BRFRGV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Filipe Laíns]

Hey,

Any updates on this? I am very excited about this PEP :)

Cheers,
Filipe Laíns

[Matthew Rahtz via Python-Dev]

Still working on it :) Specifically the grammar changes in cpython, though I haven't found much time to work on it recently. Tune into the tensor typing meeting on Monday for a more detailed update.

[Dan Moldovan via Python-Dev]

I'd be interested in using this the mechanisms defined in this PEP to define rank-generic Tensor types in TensorFlow, which are important in specifying `tf.function` signatures in a Pythonic way, using type annotations (rather than the custom `input_signature` mechanism we have today - see this issue: https://github.com/tensorflow/tensorflow/issues/31579). Variadic generics are among the last few missing pieces to create an elegant set of type definitions for tensors and shapes.

Best,

Dan Moldovan
Senior Software Engineer, TensorFlow Dev Team

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/HTCARTYYCHETAMHB6OVRNR5EW5T2CP4J/

[Petr Viktorin]

Hi,
I'm very late to the discussion -- I relied on the typing-sig and SC to
handle this, but now that I'm on the SC, I no longer have that luxury :)
This mail has my own opinions, not necessarily the SC's.


I've read the PEP, and I quite like it! It's clear that typing-sig
thought this through very well.
The thing that surprised me is the proposed changes that affect more
than typing annotations. Quite deep in the PEP, the "Grammar Changes"
section explains the (quite exciting) change to make star-unpacking
possible in normal indexing operations, e.g.::

idxs_to_select = (1, 2)
array[0, *idxs_to_select, -1] # Equivalent to [0, 1, 2, -1]

However, the PEP is silent about indexing assignment, e.g.::

array[0, *idxs_to_select, -1] = 1

IMO, it would be very confusing to not keep these in sync. If they are,
the assignment change should be documented and tested appropriately. Is
that the plan?



For a second point, the PEP says:

> this PEP disallows multiple unpacked TypeVarTuples within a single type parameter list. This requirement would therefore need to be implemented in type checking tools themselves rather than at the syntax level.

Typing annotations are sometimes used for other things than *static*
typing, and I wouldn't be surprised if type checkers themselves started
allowing this (as a non-standard extension in cases where things aren't
ambiguous):

def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ...

If multiple unpackings in a tuple aren't blocked by the compiler, they
should be tested and documented as syntactically valid annotations --
just not valid static typing annotations (even though other uses are
currently deprecated). In particular, once the compiler allows multiple
unpackings, disallowing them at the syntax level later would mean
breaking backwards compatibility.
Do we share that view?



And after reading the PEP again, I'm unclear on some details in the
Aliases section. Could you please clarify these examples for me?

SplitDataset = Tuple[Array[*Ts], Array[*Ts]]
SplitDataset[Height] # Valid? What would this be equivalent to?


TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]]
TwoArrays[Height] # Valid? Equivalent to what? How to specify this fully?

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/5YLXBCYX4VAGXWFVYLLL5RMTBXCKWPVU/

[Guido van Rossum]

The previous SC approved the PEP despite this.

If you want to convince the SC to request this feature parity in the PEP, I won't stop you.

But unless that happens I would rather not update the PEP again (it's been tough to get to this point).

Maybe you can write a separate PEP? That would probably be simpler for all involved (the PEP 646 authors would not have to be involved, and the separate PEP would be very straightforward.

 

For a second point, the PEP says:

> this PEP disallows multiple unpacked TypeVarTuples within a single type parameter list. This requirement would therefore need to be implemented in type checking tools themselves rather than at the syntax level.

Typing annotations are sometimes used for other things than *static*
typing, and I wouldn't be surprised if type checkers themselves started
allowing this (as a non-standard extension in cases where things aren't
ambiguous):

     def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ...

I don't think that sentence is trying to forbid this. The problem appears in things like

def foo(*args: tuple[*Ts1, *Ts2]) -> ...

Maybe the wording in the PEP can be imrpoved?

 

If multiple unpackings in a tuple aren't blocked by the compiler, they
should be tested and documented as syntactically valid annotations --
just not valid static typing annotations (even though other uses are
currently deprecated). In particular, once the compiler allows multiple
unpackings, disallowing them at the syntax level later would mean
breaking backwards compatibility.
Do we share that view?

Agreed that the syntax with multiple stars will not be deprecated at runtime, but type checkers may reject it. (Just as type checkers reject many other programs that would run fine.)

 

And after reading the PEP again, I'm unclear on some details in the
Aliases section. Could you please clarify these examples for me?

SplitDataset = Tuple[Array[*Ts], Array[*Ts]]
SplitDataset[Height]  # Valid? What would this be equivalent to?


TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]]
TwoArrays[Height]  # Valid? Equivalent to what? How to specify this fully?

I'll leave this to the PEP authors to address.

--

--Guido van Rossum (python.org/~guido)

Pronouns: he/him (why is my pronoun here?)

[Petr Viktorin]

As I learned after drafting that question, at least some members of the
previous SC feel like this went without saying. So I'm a bit surprised
by this reply. Guess it goes to show that PEPs need to be precise :)

Looking at the current grammar, I see that the proposed change to
`slices` will affect indexed assignment. And the reference
implementation also allows it.

With that in mind, can we treat this as this as a clarification, rather
than a substantial change? A simple note that indexed assignment is also
affected should be enough. (Tests/docs in the eventual implementation
won't be so simple, but they'll be needed anyway.)

Of course, if __setitem__ isn't meant to be affected, the proposed
grammar change needs to be revisited.

> For a second point, the PEP says:
>
> > this PEP disallows multiple unpacked TypeVarTuples within a
> single type parameter list. This requirement would therefore need to
> be implemented in type checking tools themselves rather than at the
> syntax level.
>
> Typing annotations are sometimes used for other things than *static*
> typing, and I wouldn't be surprised if type checkers themselves started
> allowing this (as a non-standard extension in cases where things aren't
> ambiguous):
>
>      def tprod(Generic[*T1], Generic[*T2]) -> Generic[*T1, *T2]: ...
>
>
> I don't think that sentence is trying to forbid this. The problem
> appears in things like
>
> def foo(*args: tuple[*Ts1, *Ts2]) -> ...
>
> Maybe the wording in the PEP can be imrpoved?

Looks like it can -- the current wording does forbid it:

> As of this PEP, only a single type variable tuple may appear in a type parameter list

But improving that smells like a rabbit hole. It's probably better to
keep it disallowed (as a static typing expression) in the current PEP.

> If multiple unpackings in a tuple aren't blocked by the compiler, they
> should be tested and documented as syntactically valid annotations --
> just not valid static typing annotations (even though other uses are
> currently deprecated). In particular, once the compiler allows multiple
> unpackings, disallowing them at the syntax level later would mean
> breaking backwards compatibility.
> Do we share that view?
>
>
> Agreed that the syntax with multiple stars will not be deprecated at
> runtime, but type checkers may reject it. (Just as type checkers reject
> many other programs that would run fine.)

+1, glad we're on the same page here.

> And after reading the PEP again, I'm unclear on some details in the
> Aliases section. Could you please clarify these examples for me?
>
> SplitDataset = Tuple[Array[*Ts], Array[*Ts]]
> SplitDataset[Height]  # Valid? What would this be equivalent to?
>
>
> TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]]
> TwoArrays[Height]  # Valid? Equivalent to what? How to specify this
> fully?
>
>
> I'll leave this to the PEP authors to address.

Sounds good. Thank you!

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OFQ27ZAQWPYLZAVPZFJH3K6LL3I7FWDR/

[Kevin Millikin via Python-Dev]

The wording there probably should be improved.  I had a different interpretation when I read that, so that suggests it needs to be clarified.

We should ensure to draw a clear distinction between type parameters and type arguments.  (Generic classes and functions are parameterized over type parameters and they have a type parameter list that is implicit in the syntax.  Generic classes can be explicitly instantiated by giving them type arguments, and an instantiation has a (explicit or implicit) type argument list.)

So when I read:

"""
As of this PEP, only a single type variable tuple may appear in a type parameter list:

class Array(Generic[*Ts1, *Ts2]): ...  # Error\
"""

I interpreted it to mean that the error is that the type _parameters_ of the generic class Array include *Ts1 and *Ts2 (not that they were used as type arguments to Generic).  Similarly, this should be an error:

class Array(dict[*Ts1], Generator[*Ts2]): ...

even though there is only a single type variable tuple appearing in a type _argument_ list.

The reason for the restriction is that the tupling of Array's type arguments is not explicit in an instantiation of Array, so we rely on this restriction so that they can be unambiguously tupled.

I don't think there is necessarily a similar restriction on a generic function's type parameters, because we don't have the ability to explicitly instantiate generic functions anyway.

An alternative wording is along the lines of: "As of this PEP, only a single type variable tuple may appear among a generic class's type parameters."

def foo(*args: tuple[*Ts1, *Ts2]) -> ...

is already prohibited by "Multiple Unpackings in a Tuple: Not Allowed".

There are three other occurrences of "type parameter list" in the PEP.  Two of them talk about instantiating generic type aliases and should be changed to "type argument list".  The last one is less clear, I can't quite parse out what it's trying to say.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/ZHBIGZQPK6Y5MSAOV3BHU4VRPIUKSJHJ/

[Matthew Rahtz via Python-Dev]

Thanks for this feedback, Petr!

First point (indexing assignment)

Great catch; we hadn't thought about this. I agree it would be better to keep these in sync.

I just tested this in our current CPython implementation, and can confirm it looks like this already works fine. So as much as I agree with Guido in preferring not to make too many more updates to the PEP, I guess we can indeed just fix this with a small clarification. I'll also add some tests for this to our CPython implementation.

Second point (multiple TypeVarTuples)

In terms of the wording in the PEP - Guido, our intention actually was to prohibit even the straightforward cases for now. Iirc, our reasoning was that we thought the decision boundary between "straightforward to infer type assignment" and "nontrivial to infer type assignment" (and of course "no unique type assignment") was tricky enough that we shouldn't complicate the already-long PEP by trying to describe all the cases where it was and wasn't ok.

Petr, do I understand that the crux for you is basically that we should commit to multiple-stars at the syntax level - the main implication being to make sure we've properly documented and tested it? I'm happy with this; we plan to follow up with another PEP that does talk about when multiple unpackings are ok anyway. I guess we should just a) clarify in the PEP that allowing multiple unpackings in the grammar isn't accidental, and b) test this in our CPython implementation?

Third point (aliases)

This is actually a great point - I had to think about this myself.

Since PEP 484 doesn't explicitly spell out how assignment of type to type variables in generic aliases works either, I had to try some things with mypy playground to figure out what the current rules are. I think the logic is, if we have an alias like

Foo = tuple[T1, T1, T2]

Foo[int, str]

then we construct a list of unique type variables in the alias (here [T1, T2]), then assign types to those variables by going left to right through the type argument list when the alias is instantiated. Guido, can you remember from your time with mypy whether this is correct? Pradeep, I guess you'll also know about this?

Based on that precedent, I believe that:

SplitDataset = Tuple[Array[*Ts], Array[*Ts]]

SplitDataset[Height]  # Valid; equivalent Tuple[Array[Height], Array[Height]]

(indeed, I just tested this with Pyre, and that matches our implementation there)

For this one:

TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]]
TwoArrays[Height]

Since we we allow a TypeVarTuple to be bound to an empty list of types, when we try to assign type arguments [Height] to the unique list of type variables [Ts1, Ts2], I think we should end up with Ts1 containing Height and Ts2 being empty; thus:

TwoArrays[Height]  # Valid; equivalent to Tuple[Array[Height], Array[()]]

But I actually can't get this to type check correctly in Pyre. Pradeep, this is the test I was using: https://gist.github.com/mrahtz/cc86c29538de1d4a80a2e8958ae71c5a Am I doing something wrong?

Also, by that logic, in this situation all the type arguments would be assigned to Ts1, and Ts2 would always end up being empty. That seems a bit weird but...shrug it also seems correct.

In any case, this is definitely something we should explain better in the PEP. I'll make a TODO for myself to write something on this once Pradeep and Guido have confirmed whether my understanding is correct.

[Guido van Rossum]

On Thu, Jan 13, 2022 at 7:23 AM Matthew Rahtz <mra...@google.com> wrote:

Thanks for this feedback, Petr!

First point (indexing assignment)

Great catch; we hadn't thought about this. I agree it would be better to keep these in sync.

I just tested this in our current CPython implementation, and can confirm it looks like this already works fine. So as much as I agree with Guido in preferring not to make too many more updates to the PEP, I guess we can indeed just fix this with a small clarification. I'll also add some tests for this to our CPython implementation.

Agreed. I just misremembered this, my bad! Please do the clarification etc.

 

Second point (multiple TypeVarTuples)

In terms of the wording in the PEP - Guido, our intention actually was to prohibit even the straightforward cases for now. Iirc, our reasoning was that we thought the decision boundary between "straightforward to infer type assignment" and "nontrivial to infer type assignment" (and of course "no unique type assignment") was tricky enough that we shouldn't complicate the already-long PEP by trying to describe all the cases where it was and wasn't ok.

Petr, do I understand that the crux for you is basically that we should commit to multiple-stars at the syntax level - the main implication being to make sure we've properly documented and tested it? I'm happy with this; we plan to follow up with another PEP that does talk about when multiple unpackings are ok anyway. I guess we should just a) clarify in the PEP that allowing multiple unpackings in the grammar isn't accidental, and b) test this in our CPython implementation?

I would love it for the cases where it's *not* ambiguous to just work (once type checkers support it). I'd like the PEP to be written so as not to disallow those. Type checkers that only support a single star even when it's unambiguous should be considered out of compliance. (But could still be fully compliant otherwise.) User pressure will then sort things out.

That seems better than requiring a follow-up PEP just to allow non-ambiguous cases that follow the PEP's semantics.

FWIW I still feel we don't have a simple enough example to distinguish the situations.

This should clearly be disallowed:

def f(*args: *tuple[*Ts1, *Ts2])...

This (and similar variants) should also be disallowed:

def f(*args: *tuple[*Ts, *Ts])...

This should be allowed:

def f(*args: *tuple[int, *Ts, str, T])...

But not this:

def f(*args: *tuple[*Ts1, str, *Ts2])...

(Even though there might be cases where it's unambiguous, like f(1, 2, 3, "", 4, 5).)

It seems you are saying that this would be disallowed, even though there's no ambiguity?

def f(a: Array[*Ts1], b: Array[*Ts2])...

I'm failing to see the point. (Other than that perhaps Pyre doesn't support it? :-) The two positional (non-variadic!) arguments have different types, each of which is an Array with variadic length. I can see nothing ambiguous about this. If you indeed want to disallow this, can you show a scenario where there's an ambiguity?

Also, in call sites I feel that multiple stars shouldn't be an issue. Example:

def f(*args: *Ts)...

f(*Array[int, int], *Array[str, str])  # Ts becomes [int, int, str, str]

 

Third point (aliases)

This is actually a great point - I had to think about this myself.

Since PEP 484 doesn't explicitly spell out how assignment of type to type variables in generic aliases works either, I had to try some things with mypy playground to figure out what the current rules are. I think the logic is, if we have an alias like

Foo = tuple[T1, T1, T2]

Foo[int, str]

then we construct a list of unique type variables in the alias (here [T1, T2]), then assign types to those variables by going left to right through the type argument list when the alias is instantiated. Guido, can you remember from your time with mypy whether this is correct? Pradeep, I guess you'll also know about this?

Yes, that's how it works. Also, Foo[int] would be an error in that case.

 

Based on that precedent, I believe that:

SplitDataset = Tuple[Array[*Ts], Array[*Ts]]

SplitDataset[Height]  # Valid; equivalent Tuple[Array[Height], Array[Height]]

(indeed, I just tested this with Pyre, and that matches our implementation there)

Yeah, looks fine.

As a corollary, this would also be correct, right?

SplitDataSet[Height, Width]

and it would make Ts be (Height, Width). Right?

 

For this one:

TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]]
TwoArrays[Height]

Given the previous example and my corollary, this seems ambiguous, just like `def f(*tuple[*Ts1, *Ts2])`.

 

Since we we allow a TypeVarTuple to be bound to an empty list of types, when we try to assign type arguments [Height] to the unique list of type variables [Ts1, Ts2], I think we should end up with Ts1 containing Height and Ts2 being empty; thus:

TwoArrays[Height]  # Valid; equivalent to Tuple[Array[Height], Array[()]]

I hope not.

 

But I actually can't get this to type check correctly in Pyre. Pradeep, this is the test I was using: https://gist.github.com/mrahtz/cc86c29538de1d4a80a2e8958ae71c5a Am I doing something wrong?

Also, by that logic, in this situation all the type arguments would be assigned to Ts1, and Ts2 would always end up being empty. That seems a bit weird but...shrug it also seems correct.

No?

 

In any case, this is definitely something we should explain better in the PEP. I'll make a TODO for myself to write something on this once Pradeep and Guido have confirmed whether my understanding is correct.

[Guido van Rossum]

On Thu, Jan 13, 2022 at 8:18 AM Matthew Rahtz <mra...@google.com> wrote:

Thanks also Kevin for this feedback!

Good point about being careful to distinguish type parameters vs type arguments. If I understand correctly, you're making two points:

1. The wording of the 'Multiple Type Variable Tuples: Not Allowed' section - you're saying that we're being a bit imprecise here in saying that we disallow multiple TypeVarTuples in a type parameter list, given that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the parameter list for the function f, but there it's unambiguous, so in fact it is allowed.

That looks like a syntax error. We only allow *Ts if the argument is of the form *a. `def f(x: *Ts1)` is not allowed.

Maybe you were thinking of `def f(x: Array[*Ts1], y: Array[*Ts2]) as the example? That seems unambiguous since the two positional arguments are given separately.

[Matthew Rahtz via Python-Dev]

Thanks also Kevin for this feedback!

Good point about being careful to distinguish type parameters vs type arguments. If I understand correctly, you're making two points:

1. The wording of the 'Multiple Type Variable Tuples: Not Allowed' section - you're saying that we're being a bit imprecise here in saying that we disallow multiple TypeVarTuples in a type parameter list, given that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the parameter list for the function f, but there it's unambiguous, so in fact it is allowed.

2. Use of wrong terminology elsewhere in the PEP. Agreed.

I'll draft a PR tweaking the wording to fix both these points.

[Matthew Rahtz via Python-Dev]

[Matthew]

> 1. The wording of the 'Multiple Type Variable Tuples: Not Allowed' section - you're saying that we're being a bit imprecise here in saying that we disallow multiple TypeVarTuples in a type parameter list, given that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the parameter list for the function f, but there it's unambiguous, so in fact it is allowed.

[Guido]

> That looks like a syntax error. We only allow *Ts if the argument is of the form *a. `def f(x: *Ts1)` is not allowed.

> Maybe you were thinking of `def f(x: Array[*Ts1], y: Array[*Ts2]) as the example? That seems unambiguous since the two positional arguments are given separately.

Sorry, yes!

[Kevin Millikin via Python-Dev]

Yes, exactly.

Specifically, the "wrong" example in section 'Multiple Type Variable Tuples: Not Allowed' suggests that maybe what is wrong is that `Generic` was given more than one unpacked type variable tuple.  The actual problem is a consequence of that: `class Array` has more than one type variable tuple as type parameters.  (But there are other ways that could happen and all of them should be wrong.)

I think it might be good to say that explicitly in that section.

[Matthew Rahtz via Python-Dev]

First point (indexing assignment)

[Guido]

> Agreed. I just misremembered this, my bad! Please do the clarification etc.

Will do.

Second point (multiple TypeVarTuples)

[Guido]

> I would love it for the cases where it's *not* ambiguous to just work (once type checkers support it). I'd like the PEP to be written so as not to disallow those. Type checkers that only support a single star even when it's unambiguous should be considered out of compliance. (But could still be fully compliant otherwise.) User pressure will then sort things out. That seems better than requiring a follow-up PEP just to allow non-ambiguous cases that follow the PEP's semantics.

I agree that would be nice, but I feel a bit worried this could result in additional churn.

I was going to propose adding just a single sentence to the PEP along the lines of "Multiple unpacking are ok when the result would be unambiguous", but as you point out, there's no simple rule for deciding what would and wouldn't be ambiguous - so we'd probably have to spell out it out with examples in the PEP, which would a) make the PEP even longer, and b) maybe put us at risk of the SC wanting to re-re-review.

So overall I'd strongly prefer to just add a sentence clarifying support at the grammar level to satisfy Petr, but leave it at that. Wdyt?

To be clear, though:

> This should clearly be disallowed:
> def f(*args: *tuple[*Ts1, *Ts2])...
> This (and similar variants) should also be disallowed:
> def f(*args: *tuple[*Ts, *Ts])...
> This should be allowed:
> def f(*args: *tuple[int, *Ts, str, T])...
> But not this:
> def f(*args: *tuple[*Ts1, str, *Ts2])...

Right, I think the current wording in the PEP ("Multiple Unpackings in a Tuple: Not Allowed") is sufficient to make it clear these are disallowed.

> It seems you are saying that this would be disallowed, even though there's no ambiguity?
> def f(a: Array[*Ts1], b: Array[*Ts2])...

This should definitely be fine, and I don't think there's any wording in the PEP which would suggest they be disallowed.

Third point (aliases)

[Guido]

> As a corollary, this would also be correct, right?
> SplitDataSet[Height, Width]
> and it would make Ts be (Height, Width). Right?

Right, exactly.

> TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]]

> TwoArrays[Height]

> Given the previous example and my corollary, this seems ambiguous, just like `def f(*tuple[*Ts1, *Ts2])`.

Oh, right, good point. So it sounds like the algorithm I described for aliases isn't quite accurate - it's not that we assign types to type variables by going from left to right and assigning greedily, it's that we assign types to type variables using the same mechanisms as in other contexts.

So I guess the result is that multiple TypeVarTuples in aliases should be disallowed, for the same reasons as disallowing multiple TypeVarTuples in parameter lists elsewhere? (I'll wait for confirmation on this from you and Pradeep, then draft a PR clarifying this.)

[Petr Viktorin]

On 13. 01. 22 16:23, Matthew Rahtz wrote:
> Thanks for this feedback, Petr!
>

> *First point (indexing assignment)*

>
> Great catch; we hadn't thought about this. I agree it would be better to
> keep these in sync.
>
> I just tested this in our current CPython implementation, and can
> confirm it looks like this already works fine. So as much as I agree
> with Guido in preferring not to make too many more updates to the PEP, I
> guess we can indeed just fix this with a small clarification. I'll also
> add some tests for this to our CPython implementation.

Thanks.

> *Second point (multiple TypeVarTuples)*

>
> In terms of the wording in the PEP - Guido, our intention actually

> /was/ to prohibit even the straightforward cases for now. Iirc, our

> reasoning was that we thought the decision boundary between
> "straightforward to infer type assignment" and "nontrivial to infer type
> assignment" (and of course "no unique type assignment") was tricky
> enough that we shouldn't complicate the already-long PEP by trying to
> describe all the cases where it was and wasn't ok.
>
> Petr, do I understand that the crux for you is basically that we should
> commit to multiple-stars at the syntax level - the main implication
> being to make sure we've properly documented and tested it? I'm happy

> with this; we plan to follow up with another PEP that /does/ talk about

> when multiple unpackings are ok anyway. I guess we should just a)
> clarify in the PEP that allowing multiple unpackings in the grammar
> isn't accidental, and b) test this in our CPython implementation?

Even less, actually.
The PEP doesn't make a very clear distinction between invalid Python
syntax vs. invalid type annotation, so I wanted to check if we're on the
same page here: the newly valid syntax will be subject to PEP 387.
We clearly are on the same page, and I don't think you need to update
the PEP.


> *Third point (aliases)*
> *
> *

> This is actually a great point - I had to think about this myself.
>
> Since PEP 484 doesn't explicitly spell out how assignment of type to
> type variables in generic aliases works either, I had to try some things
> with mypy playground to figure out what the current rules are. I think
> the logic is, if we have an alias like
>
> Foo = tuple[T1, T1, T2]
> Foo[int, str]
>
> then we construct a list of unique type variables in the alias (here
> [T1, T2]), then assign types to those variables by going left to right
> through the type argument list when the alias is instantiated. Guido,
> can you remember from your time with mypy whether this is correct?
> Pradeep, I guess you'll also know about this?
>
> Based on that precedent, I believe that:
>
> SplitDataset = Tuple[Array[*Ts], Array[*Ts]]
> SplitDataset[Height]  # Valid; equivalent Tuple[Array[Height],
> Array[Height]]
>
> (indeed, I just tested this with Pyre, and that matches our
> implementation there)
>
> For this one:
>
> TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]]
> TwoArrays[Height]
>

> Since we we allow a TypeVarTuple to be bound to an /empty/ list of

> types, when we try to assign type arguments [Height] to the unique list
> of type variables [Ts1, Ts2], I think we should end up with Ts1
> containing Height and Ts2 being empty; thus:
>
> TwoArrays[Height]  # Valid; equivalent to Tuple[Array[Height], Array[()]]
>
> But I actually can't get this to type check correctly in Pyre. Pradeep,
> this is the test I was using:
> https://gist.github.com/mrahtz/cc86c29538de1d4a80a2e8958ae71c5a
> <https://gist.github.com/mrahtz/cc86c29538de1d4a80a2e8958ae71c5a> Am I
> doing something wrong?
>

> Also, by that logic, in this situation /all/ the type arguments would be

> assigned to Ts1, and Ts2 would always end up being empty. That seems a

> bit weird but.../shrug/ it also seems correct.

>
> In any case, this is definitely something we should explain better in
> the PEP. I'll make a TODO for myself to write something on this once
> Pradeep and Guido have confirmed whether my understanding is correct.

When I asked my curious question, I thought I misread a piece of text,
not that it's a detail that went unnoticed, and could delay the PEP.


I can't speak for the whole SC, but on the Monday meeting I'll suggest
accepting the PEP with a note that
- index assignment is also affected, and
- the details around multiple unpackings in a type expression aren't
specified precisely. This gives individual type checkers some leeway,
but can be tightened in future PEPs.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/UWZ33C67QT2FV5RDUUHLRTMVF7HA2VEZ/

[Matthew Rahtz via Python-Dev]

> Even less, actually.
> The PEP doesn't make a very clear distinction between invalid Python
> syntax vs. invalid type annotation, so I wanted to check if we're on the
> same page here: the newly valid syntax will be subject to PEP 387.
> We clearly are on the same page, and I don't think you need to update
> the PEP.

Ok, fair enough.

> When I asked my curious question, I thought I misread a piece of text,
> not that it's a detail that went unnoticed, and could delay the PEP.
> I can't speak for the whole SC, but on the Monday meeting I'll suggest
> accepting the PEP with a note that
> - index assignment is also affected, and
> - the details around multiple unpackings in a type expression aren't
> specified precisely. This gives individual type checkers some leeway,
> but can be tightened in future PEPs.

Cool. Thanks, Petr!

[Jim J. Jewett]

I'm seeing enough different interpretations to think things aren't quite specified -- but I'm not sure if it matters.

(1) Is any of this something that should affect computation, or is it really just a question of how to interpret possibly ambiguous documentation?

(2) Are any of these troubling cases something that a person should actually write for a normal situation? Or are they just arguments about which abbreviations are acceptable? Or about how automatically-generated (inferred) type descriptions should be written?

(3) Are the slice-expansion questions all assumed to be indexing an n-dimensional array, as opposed to [start, stop, step]? Is that explicit in the PEP, and just not in the extracts here?

(4) Expanding multiple * shouldn't be ambiguous; the problem is figuring out what to condense into which if two are adjacent. So
s1, s2 =[a,b], (1,2,3)
[*s1, *s2] should turn into [a, b, 1, 2, 3]
The problem is that
[*s3, *s4] = (a, b, 1, 2, 3)
is ambiguous ... and I didn't really get that distinction from Petr's question or the answers. I can't tell whether I've missed something crucial, or others are arguing over angels on a pinhead ... so whatever the PEP ends up deciding, it should be explicit. (I *think* the earlier parts of this thread are consistent with this, and discussing whether to say explicitly that certain formats are forbidden (but maybe not enforced by the grammar), meaningless, or valid but currently meaningless outside of typing.)

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/YH4URO5EDQODG4QMGOCSXHV6RYTMLK5M/

[Glenn Linderman]

On 1/18/2022 2:35 PM, Jim J. Jewett wrote:
> The problem is that
> [*s3, *s4] = (a, b, 1, 2, 3)
> is ambiguous ...

It wouldn't have to be... but as you say, it needs to be explicit.

*s3 could get all the content, and *s4 a tuple with no members.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/CNTEQGZRHLPODTIR74SUHD7OWSEZ4FL3/