Go compiler - syntax tree vs AST?

[Vincent Blanchon]

Hi all,

The compiler documentation mentions a syntax tree in the parsing phase when the AST transformation phase mentions a conversion from the syntax tree to the compiler's AST representation.

If I do not misunderstand, the command "go tool compile -W" will display the AST.

I was wondering how far is different the syntax tree from the AST? Where could I get documentation or an example of this syntax tree?

Also, is this syntax tree mandatory? Is it not possible to build the AST directly from the lexer+parser?

Thanks in advance

[Ian Lance Taylor]

On Tue, Sep 3, 2019 at 6:23 AM Vincent Blanchon
<blanchon...@gmail.com> wrote:
>
> The compiler documentation mentions a syntax tree in the parsing phase when the AST transformation phase mentions a conversion from the syntax tree to the compiler's AST representation.
> If I do not misunderstand, the command "go tool compile -W" will display the AST.
>
> I was wondering how far is different the syntax tree from the AST? Where could I get documentation or an example of this syntax tree?

Note that AST just means Abstract Syntax Tree, so it's a little
confusing to talk converting a syntax tree to an AST. What we really
have is two different ASTs.

Unfortunately none of this stuff is well documented. The parser
generates a syntax tree as defined in cmd/compile/internal/syntax.
That is then converted to the Node tree defined in
cmd/compile/internal/gc/syntax.go. The -W option dumps the latter. I
don't know of a way to dump the former, but perhaps there is one.

> Also, is this syntax tree mandatory? Is it not possible to build the AST directly from the lexer+parser?

The use of two different syntax trees is entirely historical due to
the evolution of the compiler. The original code base was a C
compiler written in C which became a Go compiler written in C which
was machine translated from C to Go. There has been a lot of cleanup
but there is still a lot of historical cruft. As far as I know the
most likely next evolutionary step in this area will be to replace the
Node tree with the cmd/compile/internal/syntax tree.

Hope this helps, and I hope someone will correct me if I made a mistake.

Why do you ask?

Ian

[Vincent Blanchon]

Hello Ian,

Thank you for your answer.

Why do you ask? 

Just out of curiosity, I was working with the SSA package for a tool, and since the compiler uses this package as well I wanted to go a bit deeper on that and write a small article on it. That's quite an interesting part but the usage of two different syntax tree was a bit confusing.

Hope this helps

Yes, a lot. Thanks again!

[liu si]

> As far as I know the most likely next evolutionary step in this area will be to replace the Node tree with the cmd/compile/internal/syntax tree.

hello All, 

  I have a question. As far as I know, the AST transformation phase has been in existence for several years now. So I want to know the reason why the AST transformation phase has not been eliminated now? 

  Thanks in advance.

[messi...@gmail.com]

>  I don't know of a way to dump the former, but perhaps there is one.

It seems there's no way to dump the former tree through any compiler flag by now. The dumper entry point is Fdump but one can only use it through UT. Maybe it's only for internal use for compiler developers.

>  As far as I know the most likely next evolutionary step in this area will be to replace the Node tree with the cmd/compile/internal/syntax tree.

Really? I see in master branch, all the Node tree related code has been moved to a separated package ir(cmd/compile/internal/ir/), and a lot of work has been done for this change. Do you mean the compiler will use only one kind of Syntax Tree and this part of code will be eliminated in the end?

Thanks.