Status report: the fstrings branch

[Edward K. Ream]

I have just completed a spectacular simplification of the TokenOrderGenerator class.

The tog.advance_str and and tog.advance_if machinery is gone. These methods "polluted" all visitors involving "else" tokens.  No more.

This would never have happened were it not for a hole I found yesterday.  Once again, the parse tree can not distinguish between two different sets of tokens. The following generate exactly the same parse trees:

a = [ b : ]

a = [ b : : ]

My first thought was to implement tog.advance_slice, but then, Aha!, I saw that a single pointer suffices to sync tokens with parse tree nodes.  This pointer is tog.px, set only by tog.sync_tokens.

It's just a bit tricky to use tog.px to look ahead in the token list. That's what tog.find_next_significant_token and tog.get_concatenated_string_tokens do.

Details

Instead of three or four indices that step through the token list, a single index, tog.px suffices. Multiple indices suffered from theoretical as well as practical problems. It was hardly obvious that the various pointers would play well together.

Only visitors for ast.Bytes, ast.If, ast.JoinedStr, ast.Slice and ast.Str ever need to look ahead in the token list.  This simplifies several other visitors.

Summary

The fstringify branch now converts all Leo files to f-strings without obvious problems.

The new code should be much easier to understand and maintain.

Edward