[Python-ideas] with-statement syntactic quirk

[Barry Warsaw]

with-statements have a syntactic quirk, which I think would be useful to fix.
This is true in Python 2.7 through 3.3, but it's likely not fixable until 3.4,
unless of course it's a bug <wink>.

Legal:

>>> with open('/etc/passwd') as p1, open('/etc/passwd') as p2: pass

Not legal:

>>> with (open('/etc/passwd') as p1, open('/etc/passwd') as p2): pass

Why is this useful? If you need to wrap this onto multiple lines, say to fit
it within line length limits. IWBNI you could write it like this:

with (open('/etc/passwd') as p1,
open('/etc/passwd') as p2):
pass

This seems analogous to using parens to wrap long if-statements, but maybe
there's some subtle corner of the grammar that makes this problematic (like
'with' treating the whole thing as a single context manager).

Of course, you can wrap with backslashes, but ick!

Cheers,
-Barry

[Nick Coghlan]

It's not an especially subtle corner of the grammar, it's
tuples-as-context-managers (i.e. the case with no as clauses) that
causes hassles:

with (cmA, cmB):
pass

This is: a) useless (because tuples aren't context managers); but also
b) legal syntax (it blows up at runtime, complaining about a missing
__enter__ or __exit__ method rather than throwing SyntaxError at
compile time)

Adding support for line continuation with parentheses to import
statements was easier, because they don't accept arbitrary
subexpressions, so there was no confusion with tuples.

I do think it makes sense to change the semantics of this, but I ain't
volunteering to figure out the necessary Grammar changes :P

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas

[Devin Jeanpierre]

On Wed, Oct 31, 2012 at 6:38 AM, Barry Warsaw <ba...@python.org> wrote:
> This seems analogous to using parens to wrap long if-statements, but maybe
> there's some subtle corner of the grammar that makes this problematic (like
> 'with' treating the whole thing as a single context manager).

This seemed kind of icky when I read it, and I think Nick Coghlan
stated the reason best.

Is there a reason the tokenizer can't ignore newlines and
indentation/deindentation between with/etc. and the trailing colon?
This would solve the problem in general, without ambiguous syntax.

-- Devin

[Eli Bendersky]

On Wed, Oct 31, 2012 at 5:17 AM, Devin Jeanpierre <jeanpi...@gmail.com> wrote:

On Wed, Oct 31, 2012 at 6:38 AM, Barry Warsaw <ba...@python.org> wrote:
> This seems analogous to using parens to wrap long if-statements, but maybe
> there's some subtle corner of the grammar that makes this problematic (like
> 'with' treating the whole thing as a single context manager).

This seemed kind of icky when I read it, and I think Nick Coghlan
stated the reason best.

Is there a reason the tokenizer can't ignore newlines and
indentation/deindentation between with/etc. and the trailing colon?
This would solve the problem in general, without ambiguous syntax.

At the expense of making the tokenizer context dependent?

Eli
 

[Devin Jeanpierre]

On Wed, Oct 31, 2012 at 8:33 AM, Eli Bendersky <eli...@gmail.com> wrote:
>> Is there a reason the tokenizer can't ignore newlines and
>> indentation/deindentation between with/etc. and the trailing colon?
>> This would solve the problem in general, without ambiguous syntax.
>
> At the expense of making the tokenizer context dependent?

It's already context-dependent in some sense, but this wouldn't make
it any moreso. For example, the tokenizer already ignores
indents/dedents when inside parens/braces/brackets, and handling this
only slightly more complex than that. In particular, the trailing
colon is the one not inside braces or brackets.

Also, I'd avoid the term "context-dependent". It sounds too similar to
"context-sensitive" !

Anyway, it looks like this isn't how the tokenizer treats
braces/brackets (it ignores indent/dedent, but not newlines (I guess
the grammar handles those)). What I meant to suggest was, treat "with
... :" similarly to how the OP suggests treating "with (...) :".

[Eli Bendersky]

On Wed, Oct 31, 2012 at 5:45 AM, Devin Jeanpierre <jeanpi...@gmail.com> wrote:

On Wed, Oct 31, 2012 at 8:33 AM, Eli Bendersky <eli...@gmail.com> wrote:
>> Is there a reason the tokenizer can't ignore newlines and
>> indentation/deindentation between with/etc. and the trailing colon?
>> This would solve the problem in general, without ambiguous syntax.
>
> At the expense of making the tokenizer context dependent?

It's already context-dependent in some sense, but this wouldn't make
it any moreso. For example, the tokenizer already ignores
indents/dedents when inside parens/braces/brackets, and handling this
only slightly more complex than that. In particular, the trailing
colon is the one not inside braces or brackets.

Also, I'd avoid the term "context-dependent". It sounds too similar to
"context-sensitive" !

I use the two as rough synonyms. Shouldn't I?
 

Anyway, it looks like this isn't how the tokenizer treats
braces/brackets (it ignores indent/dedent, but not newlines (I guess
the grammar handles those)). What I meant to suggest was, treat "with
... :" similarly to how the OP suggests treating "with (...) :".

If this gets accepted, then, is there a reason to stop at "with"? Why not ignore newlines between "if" and its trailing ":" as well? [playing devil's advocate here]

Eli

[Devin Jeanpierre]

On Wed, Oct 31, 2012 at 8:52 AM, Eli Bendersky <eli...@gmail.com> wrote:
>> Also, I'd avoid the term "context-dependent". It sounds too similar to
>> "context-sensitive" !
>
> I use the two as rough synonyms. Shouldn't I?

"context sensitive" has a technical meaning, in the same way that
"regular" or "recursively enumerable" does. In this particular case,
the technical meaning doesn't align very well with the lay / intuitive
meaning, but gets used in the same place as where one might use the
phrase in the lay / intuitive sense -- if you'd said "context
sensitive" I would've assumed you meant it in the technical sense.

I guess I can't say that you should avoid the term unless I have a
replacement. Maybe just using more words would help, like saying "then
the actions of the tokenizer would depend on the context"?

>>
>> Anyway, it looks like this isn't how the tokenizer treats
>> braces/brackets (it ignores indent/dedent, but not newlines (I guess
>> the grammar handles those)). What I meant to suggest was, treat "with
>> ... :" similarly to how the OP suggests treating "with (...) :".
>
>
> If this gets accepted, then, is there a reason to stop at "with"? Why not
> ignore newlines between "if" and its trailing ":" as well? [playing devil's
> advocate here]

I'd be very confused if newlines were acceptable inside `with` but not
`if` and those.

I'm not seeing a downside to changing them as well, except that it
makes the workload (maybe significantly?) larger. I'm not sure if it's
made that much larger. In the tokenizer it's easy, maybe in the
grammar it's not so easy, and I don't know if this has to be in the
grammar. The last time I ever tried editing python's parsing rules it
ended very very poorly.

[Nick Coghlan]

Note that I agreed with Barry that we probably *should* change it from
a principle-of-least-surprise point of view. I just called "not it" on
actually figuring out how to make it work given the current Grammar
definition as a starting point :)

Between expression precedence control, singleton tuples, generator
expressions, function calls, function parameter declarations, base
class declarations, import statement grouping and probably a couple of
other cases that have slipped my mind, parentheses already have plenty
of different meanings in Python, and we also have plenty of places
where the syntactical rules aren't quite the same as those in an
ordinary expression.

The thing that makes Python's parser simple is the fact that we have
*prefixes* in the Grammar that make it clear when the parsing rules
should change, so you don't need much lookahead at parsing time (it's
deliberately limited to only 1 token, in fact). The challenge in this
particular case is to avoid a Grammar ambiguity relative to ordinary
expression syntax without duplicating large sections of the grammar
file.

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Guido van Rossum]

Honestly, is a backslash going to kill you?


--
--Guido van Rossum (python.org/~guido)

[Nick Coghlan]

On Thu, Nov 1, 2012 at 1:13 AM, Guido van Rossum <gu...@python.org> wrote:
> Honestly, is a backslash going to kill you?

Aye, given the cost-benefit ratio on this one, I'll be rather
surprised if anyone ever actually fixes it. I just wanted to be clear
that I'm not *philosophically* opposed to fixing it (since I think
Barry's proposed behaviour makes more sense from a user perspective),
I'm just fairly sure it's likely to be hard to fix without making the
Grammar harder to maintain, which *would* be a difficult sell for
something that's a relatively trivial wart :)

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Guido van Rossum]

On Wed, Oct 31, 2012 at 8:28 AM, Nick Coghlan <ncog...@gmail.com> wrote:
> On Thu, Nov 1, 2012 at 1:13 AM, Guido van Rossum <gu...@python.org> wrote:
>> Honestly, is a backslash going to kill you?
>
> Aye, given the cost-benefit ratio on this one, I'll be rather
> surprised if anyone ever actually fixes it. I just wanted to be clear
> that I'm not *philosophically* opposed to fixing it (since I think
> Barry's proposed behaviour makes more sense from a user perspective),
> I'm just fairly sure it's likely to be hard to fix without making the
> Grammar harder to maintain, which *would* be a difficult sell for
> something that's a relatively trivial wart :)

Yeah, the problem is that when you see a '(' immediately after 'with',
you don't know whether that's just the start of a parenthesized
expression or the start of a (foo as bar, blah as blabla) syntactic
construct.

--
--Guido van Rossum (python.org/~guido)

[Joachim König]

On 31/10/2012 16:42, Guido van Rossum wrote:
> Yeah, the problem is that when you see a '(' immediately after 'with',
> you don't know whether that's just the start of a parenthesized
> expression or the start of a (foo as bar, blah as blabla) syntactic
> construct.

but couldn't "with" be interpreted as an additional kind of opening
parantheses (and "if", "for", "while",
"elif", "else" too) and the ":" as the closing one?

I'm sure this has been asked a number of times but I couldn't find an
answer.

Joachim

[Barry Warsaw]

On Oct 31, 2012, at 09:55 PM, Nick Coghlan wrote:

>It's not an especially subtle corner of the grammar, it's
>tuples-as-context-managers (i.e. the case with no as clauses) that
>causes hassles:
>
> with (cmA, cmB):
> pass
>
>This is: a) useless (because tuples aren't context managers); but also
>b) legal syntax (it blows up at runtime, complaining about a missing
>__enter__ or __exit__ method rather than throwing SyntaxError at
>compile time)

So clearly we need to make tuples proper context managers <wink>.

-Barry

[Benjamin Peterson]

Nick Coghlan <ncoghlan@...> writes:
> I do think it makes sense to change the semantics of this, but I ain't
> volunteering to figure out the necessary Grammar changes :P

It would not be difficult to special in AST construction. We do this for some
other things already.

[MRAB]

Another possibility could be to allow a tuple of context managers and a
tuple of names:

with (open('/etc/passwd'), open('/etc/passwd')) as (p1, p2):
pass

meaning:

with open('/etc/passwd') as p1:
with open('/etc/passwd')) as p2:
pass

or perhaps more correctly:

with open('/etc/passwd') as temp_1:
with open('/etc/passwd')) as temp_2:
p1, p2 = temp_1, temp_2
pass

It would also allow:

with (cmA, cmB):
pass

meaning:

with cmA:
with cmB:
pass

[Antoine Pitrou]

On Wed, 31 Oct 2012 11:38:53 +0100
Barry Warsaw <ba...@python.org> wrote:
> with-statements have a syntactic quirk, which I think would be useful to fix.
> This is true in Python 2.7 through 3.3, but it's likely not fixable until 3.4,
> unless of course it's a bug <wink>.
>
> Legal:
>
> >>> with open('/etc/passwd') as p1, open('/etc/passwd') as p2: pass
>
> Not legal:
>
> >>> with (open('/etc/passwd') as p1, open('/etc/passwd') as p2): pass
>
> Why is this useful? If you need to wrap this onto multiple lines, say to fit
> it within line length limits. IWBNI you could write it like this:
>
> with (open('/etc/passwd') as p1,
> open('/etc/passwd') as p2):
> pass

This bit me a couple of days ago. +1 for supporting it.

Regards

Antoine.

[Arnaud Delobelle]

No need for backslashes, just put the brackets in the right place:

with (
open('/etc/passwd')) as p1, (

open('/etc/passwd')) as p2:
pass

;)

--
Arnaud

[Raymond Hettinger]

On Oct 31, 2012, at 3:38 AM, Barry Warsaw <ba...@python.org> wrote:

 IWBNI you could write it like this:

   with (open('/etc/passwd') as p1,
         open('/etc/passwd') as p2):
         pass

This seems analogous to using parens to wrap long if-statements, but maybe
there's some subtle corner of the grammar that makes this problematic (like
'with' treating the whole thing as a single context manager).

Of course, you can wrap with backslashes, but ick!

I would rather live a bashslash than have yet another change to the grammar

or have context-manager semantics added to tuples.

ISTM that this isn't a problem worth solving.  

Raymond

[Mathias Panzenböck]

On 10/31/2012 05:37 PM, Joachim König wrote:
> On 31/10/2012 16:42, Guido van Rossum wrote:
>> Yeah, the problem is that when you see a '(' immediately after 'with',
>> you don't know whether that's just the start of a parenthesized
>> expression or the start of a (foo as bar, blah as blabla) syntactic
>> construct.
>
> but couldn't "with" be interpreted as an additional kind of opening parantheses (and "if", "for",
> "while",
> "elif", "else" too) and the ":" as the closing one?
>

What about "def", "class", "lambda" and "except"?

[Mathias Panzenböck]

On 10/31/2012 10:03 PM, Arnaud Delobelle wrote:
> On 31 October 2012 10:38, Barry Warsaw <ba...@python.org> wrote:
>> with-statements have a syntactic quirk, which I think would be useful to fix.
>> This is true in Python 2.7 through 3.3, but it's likely not fixable until 3.4,
>> unless of course it's a bug <wink>.
>>
>> Legal:
>>
>>>>> with open('/etc/passwd') as p1, open('/etc/passwd') as p2: pass
>>
>> Not legal:
>>
>>>>> with (open('/etc/passwd') as p1, open('/etc/passwd') as p2): pass
>>
>> Why is this useful? If you need to wrap this onto multiple lines, say to fit
>> it within line length limits. IWBNI you could write it like this:
>>
>> with (open('/etc/passwd') as p1,
>> open('/etc/passwd') as p2):
>> pass
>>
>> This seems analogous to using parens to wrap long if-statements, but maybe
>> there's some subtle corner of the grammar that makes this problematic (like
>> 'with' treating the whole thing as a single context manager).
>>
>> Of course, you can wrap with backslashes, but ick!
>
> No need for backslashes, just put the brackets in the right place:
>
> with (
> open('/etc/passwd')) as p1, (
> open('/etc/passwd')) as p2:
> pass
>
> ;)
>

Because that's not confusing. Why not write:

with open('/etc/passwd'
) as p1, open(
'/etc/passwd') as p2:
pass

[Barry Warsaw]

On Oct 31, 2012, at 06:47 PM, Raymond Hettinger wrote:

>I would rather live a bashslash than have yet another change to the grammar
>or have context-manager semantics added to tuples.

Yeesh. Despite the wink I guess people took me seriously about the tuples
thing. I mean, really, c'mon! :)

>ISTM that this isn't a problem worth solving.

There are certainly other ways to refactor away the long lines. It's an odd
corner of syntactic quirkery that really only came about as I began removing
nested() calls, and nested context managers are already probably pretty rare.
It doesn't bother me that it's not worth "fixing" but I'm glad the discussion
is now captured for eternity in a mail archive.

-Barry

[Antoine Pitrou]

Uh, what people seem to miss is that it's not only about nested context
managers. It can happen with a single context manager:

with (some_context_manager(many_arguments...)
as my_variable):
...

# SyntaxError!


Regards

Antoine.

[Steven D'Aprano]

On 01/11/12 21:48, Antoine Pitrou wrote:

> Uh, what people seem to miss is that it's not only about nested context
> managers. It can happen with a single context manager:
>
> with (some_context_manager(many_arguments...)
> as my_variable):
> ...
>
> # SyntaxError!

Have I missed something?

with some_context_manager(
many_arguments,
and_more_arguments,
and_still_more_arguments
) as my_variable:
...


I'm still not seeing a problem that needs fixing.


--
Steven

[Nick Coghlan]

On Wed, Oct 31, 2012 at 8:38 PM, Barry Warsaw <ba...@python.org> wrote:

I've been remiss in not mentioning the new alternative in 3.3 for
handling nesting of complex context management stacks:

with contextlib.ExitStack() as cm:
p1 = cm.enter_context(open('/etc/passwd'))
p2 = cm.enter_context(open('/etc/passwd'))

(Note: ExitStack is really intended for cases where the number of
context managers involved varies dynamically, such as when you want to
make a CM optional, but you *can* use it for static cases if it seems
appropriate)

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Terry Reedy]

On 11/1/2012 6:48 AM, Antoine Pitrou wrote:

> Uh, what people seem to miss is that it's not only about nested context
> managers. It can happen with a single context manager:
>
> with (some_context_manager(many_arguments...)
> as my_variable):
> ...
>
> # SyntaxError!

As it should be. With is not a(n overt) function and with statements
should not look like function calls. With clauses are not expressions,
so they cannot (should not) be arbitrarily surrounded by parentheses as
expressions can. Multiple with clauses do not form a tuple and do not
need parentheses to set off the tuple and modify expression precedence.

I understand better now why I did not like adding optional \-avoidance
parentheses to import statements. Sprinkling statements with optional,
non-expression parentheses does not fit the nature of statement syntax.

--
Terry Jan Reedy

[Yuval Greenfield]

On Thu, Nov 1, 2012 at 1:40 PM, Nick Coghlan <ncog...@gmail.com> wrote:

I've been remiss in not mentioning the new alternative in 3.3 for
handling nesting of complex context management stacks:

    with contextlib.ExitStack() as cm:
        p1 = cm.enter_context(open('/etc/passwd'))
        p2 = cm.enter_context(open('/etc/passwd'))

(Note: ExitStack is really intended for cases where the number of
context managers involved varies dynamically, such as when you want to
make a CM optional, but you *can* use it for static cases if it seems
appropriate)

Go's "defer" is quite a neat solution for these hassles if anyone's in the mood for a time machine discussion.

http://golang.org/doc/effective_go.html#defer

Yuval Greenfield

[Cameron Simpson]

On 31Oct2012 08:17, Devin Jeanpierre <jeanpi...@gmail.com> wrote:
| Is there a reason the tokenizer can't ignore newlines and
| indentation/deindentation between with/etc. and the trailing colon?
| This would solve the problem in general, without ambiguous syntax.

To my mind one of the attractive features of the current syntax is that
forgetting the colon causes an immediate complaint. Once one allows an
arbitrary number of lines between (if,while,with) and the colon one is
into the same zone as several other languages, where a common mistake
can cause a syntax comoplain many lines beyond where the mistake was
made.

I understand the attractiveness here, but I think I would prefer staying
with the status quo (overt brackets or icky trailing sloshes) to extend
the lines in a condition over opening the syntax to complaints far beyond
the mistake.

Just 2c,
--
Cameron Simpson <c...@zip.com.au>

If you don't know what your program is supposed to do, you'd better not start
writing it. - Edsger W. Dijkstra

[Nick Coghlan]

Go was one of the reference points for the ExitStack design (it's a large part of why the API also supports providing callbacks directly to the exit stack, not just as context managers).

Cheers,
Nick.

--
Sent from my phone, thus the relative brevity :)
>
> Yuval Greenfield

[Bruce Leban]

On Thu, Nov 1, 2012 at 3:14 PM, Cameron Simpson <c...@zip.com.au> wrote:

To my mind one of the attractive features of the current syntax is that

forgetting the colon causes an immediate complaint.

I agree 

I understand the attractiveness here, but I think I would prefer staying
with the status quo (overt brackets or icky trailing sloshes) to extend
the lines in a condition over opening the syntax to complaints far beyond
the mistake.

Ditto except for the part about \ continuation being icky. I don't think it's that bad. The two things that make \ continuation less attractive than () continuation is that you can't put comments on the \-continued lines and invisible trailing white space is a syntax error.

I don't understand the reason for either restriction.

with open('/etc/passwd') as p1, \      # source

     open('/etc/passwd') as p2:        # destination

seems more readable than

with open('/etc/passwd') as p1, \

     open('/etc/passwd') as p2:        # source, destination

A reasonable restriction (to my mind) would be to require at least two spaces or a tab after a \ before a comment (although requiring just one space would also be ok with me although I personally would always use more). This change couldn't break existing code since \ is currently a syntax error if followed by whitespace or a comment.

I would ignore whitespace after a final \ in a string, but would not allow comments.

(Yes, I realize that better variable names would obviate the need for these particular comments but comments are still useful sometimes :-)

--- Bruce

Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com

[Barry Warsaw]

On Nov 02, 2012, at 08:41 AM, Nick Coghlan wrote:

>> Go's "defer" is quite a neat solution for these hassles if anyone's in
>the mood for a time machine discussion.
>>
>> http://golang.org/doc/effective_go.html#defer
>
>Go was one of the reference points for the ExitStack design (it's a large
>part of why the API also supports providing callbacks directly to the exit
>stack, not just as context managers).

Is it fair to say that the difference between Go's defer and ExitStack is that
the latter gives you the opportunity to clean up earlier than at function
exit?

Cheers,
-Barry

[Nick Coghlan]

Yep. You can also do some pretty interesting things with ExitStack
because of the pop_all() operation (which moves all of the registered
operations to a *new* ExitStack instance).

I wrote up a few of the motivating use cases as examples and recipes
in the 3.3 docs:
http://docs.python.org/3/library/contextlib#examples-and-recipes

I hope to see more interesting uses over time as more people explore
the possibilities of a dynamic tool for composing context managers
without needing to worry about the messy details of unwinding them
correctly (ExitStack.__exit__ is by far the most complicated aspect of
the implementation).

[Yuval Greenfield]

On Sun, Nov 4, 2012 at 5:41 PM, Nick Coghlan <ncog...@gmail.com> wrote:

Yep. You can also do some pretty interesting things with ExitStack

because of the pop_all() operation (which moves all of the registered
operations to a *new* ExitStack instance).

I wrote up a few of the motivating use cases as examples and recipes
in the 3.3 docs:
http://docs.python.org/3/library/contextlib#examples-and-recipes

I hope to see more interesting uses over time as more people explore
the possibilities of a dynamic tool for composing context managers
without needing to worry about the messy details of unwinding them
correctly (ExitStack.__exit__ is by far the most complicated aspect of
the implementation).

Cheers,
Nick.

Pretty interesting things indeed. It does look like the concept is very powerful. If I were to design a language I might have just given every function an optional ExitStack. I.e. an explicit, dynamic, introspectable version of defer.