Indexed export format enabled by default at master

[Matthew Dempsky]

I just submitted CL 106797 to enable the compiler's new indexed export format by default. This should be transparent to most people, except that large projects should see faster build times (e.g., Juju and Kubernetes now take 15% to 20% less time to build from scratch on my workstation).

If you do suspect problems, you can revert back to the previous export format with the compiler's -iexport=false flag. This needs to be applied across the entire build, so if you're using cmd/go this means -gcflags=all=-iexport=false.

As always, please file issues if you find any.

Thanks.

[Robert Griesemer]

Thanks, Matt, this is great!

- gri

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ma...@influxdata.com]

I'm seeing around a 10% speedup in build times for influxdb, which is a decent size project but not as big as Juju or k8s. Good work!

------- before indexed export format ---------

$ gotip version

go version devel +0cd0dc96e0 Mon Apr 23 18:18:01 2018 +0000 darwin/amd64

$ time GOCACHE=off gotip build ./cmd/influxd # open source

real 0m12.214s

user 0m30.062s

sys 0m3.562s

$ time GOCACHE=off gotip build ./cmd/influxd # closed source

real 0m15.762s

user 0m41.103s

sys 0m4.622s

-------- after indexed export format  ---------

$ gotip version

go version devel +f2316c2789 Tue Apr 24 14:14:56 2018 +0000 darwin/amd64

$ time GOCACHE=off gotip build ./cmd/influxd # open source

real 0m10.774s

user 0m27.242s

sys 0m3.273s

$ time GOCACHE=off gotip build ./cmd/influxd # closed source

real 0m13.427s

user 0m35.413s

sys 0m4.148s

[Anca Emanuel]

There is some TODO document or some "20,000 foot view" of what it needs (ideally) to be done to have the fastest compiler around ?

Like:

1. do work in parallel

2. avoid unnecessary work on data.

https://go-review.googlesource.com/c/go/+/103935/4/src/cmd/compile/README.md "convert the syntax package's AST to its own representation"

[Josh Bleecher Snyder]

> There is some TODO document or some "20,000 foot view" of what it needs
> (ideally) to be done to have the fastest compiler around ?

Not that I know of.

Though I would always like it to be faster, I don't think we're doing
too badly right now.

> 1. do work in parallel

We already do quite a lot of this.

> "convert the syntax package's AST to its own representation"

This is probably the biggest individual win available. It's a *lot* of
work, though. I don't know that anyone is actively tackling it.


Other high-level-ish things that come to mind as significant, in no
particular order:

* Monitor compiler performance. Some of us do this occasionally,
manually. A dashboard would be nice. This helps prevent both big
regressions and long slow slides, and keeps it in the minds of the
compiler contributors. (We already have pretty good attentiveness
around this, happily.)

* Keep grinding away at the details. There has been a lot of this
done, so there's not much low hanging fruit, but little improvements
do add up. (Also, poring over the compiler and runtime is a fruitful
source of compiler optimizations.)

* Improve our inlining decisions. Inlining is quite expensive from a
compilation time perspective, and we don't necessarily spend that
budget well right now.

* Make regalloc's liveness analysis non-quadratic. (This is hard.)

* File issues for (and fix!) cases in which the compiler is
inordinately slow, even if they seem unusual. (GitHub label:
ToolSpeed.) In many cases, compilation time is dominated by its
slowest component, and identifying and fixing asymptotic problems can
help broadly.

* Provide some education around how to write generated code in a way
that is compiler-friendly. I've been meaning to write a blog post
about this for ages...

-josh

[Anca Emanuel]

Thanks. If there is not already, maybe create an bug with all these ideas to track ?

[ra...@develer.com]

Il giorno martedì 24 aprile 2018 19:29:26 UTC+2, Josh Bleecher Snyder ha scritto:

> "convert the syntax package's AST to its own representation"

This is probably the biggest individual win available. It's a *lot* of
work, though. I don't know that anyone is actively tackling it.

Can you elaborate on this? Can this be done piecewise? Is there any example of existing CL that goes in this direction?

Also (maybe related): what's the big plan with walk? Is walk something that should eventually go away? Would it make sense to start moving things off walk into SSA?

Giovanni

[Matthew Dempsky]

On Wed, Apr 25, 2018 at 9:17 AM <ra...@develer.com> wrote:

Also (maybe related): what's the big plan with walk?

https://github.com/golang/go/issues/17728

[Robert Griesemer]

On Wed, Apr 25, 2018 at 9:16 AM, <ra...@develer.com> wrote:

Il giorno martedì 24 aprile 2018 19:29:26 UTC+2, Josh Bleecher Snyder ha scritto:

> "convert the syntax package's AST to its own representation"

This is probably the biggest individual win available. It's a *lot* of
work, though. I don't know that anyone is actively tackling it.

Can you elaborate on this? Can this be done piecewise? Is there any example of existing CL that goes in this direction?

Various people (incl. Josh, Ian, Matt, etc.) have done piece-wise work over the years and made excellent progress in simplifying the existing node structure and making it more light-weight and consistent. But this is more about the whole-sale replacement of the legacy node data structure with something that's more modern and Go idiomatic. Conceivably this could be done incrementally if one has a very clear picture of where one wants to end up with; but it would be increasingly painful because the node structure is pervasive in the the compiler front-end. That is, I don't see this as a piece-wise operation, and I also don't see that this is the kind of work that should be done outside the Go Team or somebody who is not full time on the compiler team.

Also (maybe related): what's the big plan with walk? Is walk something that should eventually go away? Would it make sense to start moving things off walk into SSA?

See Matt's e-mail. 

Giovanni

[Josh Bleecher Snyder]

> Thanks. If there is not already, maybe create an bug with all these ideas to
> track ?

Most of these have tracking bugs. For ease of future reference, I'll
list them below.

Oh, and I remembered one more old idea, which I am still intrigued by,
but which Russ isn't keen on (and he's usually right about most
things). Here it is, for the curious:
https://github.com/golang/go/issues/15734. The discussion there also
contains some interesting links about cmd/go's scheduling, which might
also hold some promise.

>> > "convert the syntax package's AST to its own representation"
>>
>> This is probably the biggest individual win available. It's a *lot* of
>> work, though. I don't know that anyone is actively tackling it.

https://github.com/golang/go/issues/17728

Also, Robert wrote:

> That is, I don't see this as a piece-wise operation, and I also don't see that this is the kind of work that should be done outside the Go Team or somebody who is not full time on the compiler team.

I agree that the final, wholesale replacement will definitely need
that level of attention. However, FWIW, some amount can still be
chipped away at. The most obvious is migrating optimizations and
lowerings from walk to SSA construction, and perhaps even adding
higher level SSA ops, to make SSA construction simpler. (As an aside,
higher level SSA ops could also theoretically improve compiler
performance a little. The high water mark of number of SSA Values
determines the cost of many later passes, and that high water mark is
frequently at initial construction. If we could use higher level SSA
ops initially, do phi elim and copy elim, and then expand those higher
level SSA ops, we might get a performance boost as well as a
comprehensibility improvement.)

>> Other high-level-ish things that come to mind as significant, in no
>> particular order:
>>
>> * Monitor compiler performance. Some of us do this occasionally,
>> manually. A dashboard would be nice. This helps prevent both big
>> regressions and long slow slides, and keeps it in the minds of the
>> compiler contributors. (We already have pretty good attentiveness
>> around this, happily.)

No tracking bug. I don't plan to file one.

>> * Keep grinding away at the details. There has been a lot of this
>> done, so there's not much low hanging fruit, but little improvements
>> do add up. (Also, poring over the compiler and runtime is a fruitful
>> source of compiler optimizations.)

No tracking bug; it wouldn't be actionable anyway.

>> * Improve our inlining decisions. Inlining is quite expensive from a
>> compilation time perspective, and we don't necessarily spend that
>> budget well right now.

https://github.com/golang/go/issues/17566

>> * Make regalloc's liveness analysis non-quadratic. (This is hard.)

Filed https://github.com/golang/go/issues/25120.

>> * File issues for (and fix!) cases in which the compiler is
>> inordinately slow, even if they seem unusual. (GitHub label:
>> ToolSpeed.) In many cases, compilation time is dominated by its
>> slowest component, and identifying and fixing asymptotic problems can
>> help broadly.

GitHub label: ToolSpeed

>> * Provide some education around how to write generated code in a way
>> that is compiler-friendly. I've been meaning to write a blog post
>> about this for ages...

No tracking bug. I'll do it eventually, if no one beats me to it.


-josh