Go 1.7 planning

[Russ Cox]

The Go 1.6 release candidate is looking good. Hopefully in a few weeks it will be out, and we'll open the tree for Go 1.7 development. That makes now a good time to talk about what we want to do in Go 1.7 as far as technical work.

Keith intends to merge the SSA-based compiler back end (amd64 only right now) into the main tree at the beginning of the Go 1.7 cycle. It works, and it generates code that on average runs about as fast as the current compiler. In many cases the SSA back end generates significantly better code than the old back end with little effect on performance: x86-64 chips (at least the server-class ones) are remarkably good at executing bad code quickly. We expect that the effect on non-x86 chips will be more pronounced, and there is still significant room for improvement in the x86 code quality, so we may yet see significant wins there. Unfortunately the SSA back end slows down the overall compilation by about 2x compared to using the old back end. This is not fundamental: we've spent almost all the development time focused on correctness and completeness, not on making it fast. The compiler work in the Go 1.7 cycle will focus on recovering much of that compile time loss.

Robert has a new, more compact export format for the compilers nearly ready. This should speed compilation a small amount and save a bit of disk space, and it fixes a few small problems in the old format. But most importantly it gives us a much cleaner base on which to make future changes to the export data, which will make it easier to do more interprocedural optimizations like escape analysis and inlining that work no matter where package boundaries are.

Rick and Austin have been working on further improvements to the garbage collector, especially in terms of throughput. That's a long effort and may not be ready for Go 1.7.

Another effort we'd like to do for Go 1.7 (but don't have concrete plans for yet) is collection and display of historical performance data, as a replacement for the old performance dashboard we had that no longer runs. The part that I believe is most important is the separation of collection from display. We want to make it possible for anyone to run a benchmarking system, just as anyone can run a builder, and have all that data feed into a database of some kind that anyone can fetch (perhaps as a Git repository) and analyze. It requires significant expertise to collect useful data and it requires significant but different expertise to analyze and display it well, and we think that separating these two parts will help let experts make progress on their halves independently.

In general I would encourage people once again to focus on cleanup and polish for Go 1.7. There are many parts of the tree that work but are not as good as they could be. For example we know that math/big is missing assembly support to varying degrees on some architectures. We also know there are many minor problems, missing examples, and so on that are significant collectively, even if not taken one at a time.

What are you planning to do in the Go 1.7 cycle? Or what GitHub issues do we not have marked with a Go 1.7 milestone yet but merit one, in your opinion?

Please reply to this email thread and we'll make a doc out of the responses.

Thanks.

Russ

[Brad Fitzpatrick]

 On Thu, Jan 28, 2016 at 7:15 AM, Russ Cox <r...@golang.org> wrote:

What are you planning to do in the Go 1.7 cycle? Or what GitHub issues do we not have marked with a Go 1.7 milestone yet but merit one, in your opinion?

I intend to do significantly less in Go 1.7 compared to Go 1.6. I have work I'd like to do on ACME, gRPC, GCP APIs, and other things not in the Go tree.

I will also be away at the end of the cycle.

If http2 bugs come up, those will be my main priority. I also have a few minor cleanups in mind to make the http2 code easier to follow and share more code between the client & server.

Unfortunately the SSA back end slows down the overall compilation by about 2x compared to using the old back end. This is not fundamental: we've spent almost all the development time focused on correctness and completeness, not on making it fast. The compiler work in the Go 1.7 cycle will focus on recovering much of that compile time loss.

This may be an unpopular opinion, but: perhaps we need to recover the 2x slowdown we already suffered from Go 1.4 to Go 1.5 before we get twice as slow yet again with the wishful thinking that we'll recover it. Go 1.5 and Go 1.6 are still painful compared to Go 1.4. I'd like to say we won't merge SSA until our compiler is fast again.

4x slower builds & tests than Go 1.4 will not be pleasant.

[David Chase]

There are three bugs related to escape analysis, two with CLs ready, one not.

1) https://go-review.googlesource.com/#/c/18041/1 -- explaining why stuff escapes. (-m -m)

2) https://go-review.googlesource.com/#/c/17581/ -- (slightly) better escape analysis for recursive calls

3) https://github.com/golang/go/issues/14018 -- we know the "real" type but only use the declared interface type, and punt on escape (and inlining)

#3 is upstream of escape analysis and could in theory iterate a few times against inlining, since each can enable more of the other.

There have been some ugly hacks in allocation-critical Go application code that this optimization would have obviated.

Current work is on SSA, reviewing CLs and working on making bounds check elimination more aggressive in hopes

that we'll ever see anything more than a mere 1% reduction in text size.  Early benchmarking suggests that it is not

terrible.  The code will need a round of serious reviewing.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Griesemer]

- I'm going to complete the new export format. It's a binary format, so there will also be a mechanism to display it in a human-readable form.

- I'm hoping to spend more time on cleaning up the compiler front-end. Making nodes smaller should help memory use and eventually performance, and also improve maintainability.

- There are several lose ends in math/big that require some careful attention. There's also some low-hanging fruit in some of the core assembly routines that should have significant impact across many math/big functions.

- Ongoing spec maintenance (mostly outstanding issues).

- There will be time required for bug fixes across existing libraries.

- There may be Google3 work (TBD).

- I will be away for up to one month, possibly a bit longer (wedding of my brother in Europe, summer vacation, GopherCon).

- gri

[Aram Hăvărneanu]

For 1.7 I will on working on SPARC64 and ARM64.

For SPARC64, I will finish the port and try to get it in in the first
half of the cycle.

For ARM64, I will do SSA, add fast assembly versions for things like
math/big, and generally do other optimizations.

--
Aram Hăvărneanu

[Michael Monashev]

Hi, Russ.

What about multy-valued channels?

my_chan:=make(chan (int, err), 100)
...
my_chan <- val, err
...
val, err <- my_chan

It can be useful to return values and error from goroutine. Like
'return' statement in function.

--

Michael

[Brad Fitzpatrick]

Language changes are out of scope.

[Matthew Dempsky]

On Thu, Jan 28, 2016 at 9:15 AM, Michael Monashev <softs...@gmail.com> wrote:

What about multy-valued channels?

These already exist.  The syntax for them is:

    myChan := make(chan struct{x int; err error}, 100)

    ...

    myChan <- struct{x int; err error}{val, err}

    ...

    s := <-myChan

    val, err = s.x, s.err

(If golang.org/issue/12854 is accepted/implemented, the send statement could be written as "myChan <- {val, err}".)

[Aram Hăvărneanu]

On Thu, Jan 28, 2016 at 6:15 PM, Michael Monashev <softs...@gmail.com> wrote:
> ...

This thread is for people working *on* Go. For feature requests and
proposals please go through the usual channels, not on this thread.
Thank you.

--
Aram Hăvărneanu

[Ian Lance Taylor]

On Thu, Jan 28, 2016 at 7:15 AM, Russ Cox <r...@golang.org> wrote:
>

> What are you planning to do in the Go 1.7 cycle? Or what GitHub issues do we
> not have marked with a Go 1.7 milestone yet but merit one, in your opinion?

I would like to provide guarantees that make it possible to use
finalizers with confidence (issue 13347).

I would like to decide how we are going to decide about the memory
model (issues 7948, 9442, 5045, plus various other communications).
People who care about memory models do not like it.

I would like to fix the object file format to make linking test
programs much more efficient but I doubt I will get to it.

Ian

[roger peppe]

I'm +1 on this. Although historically I've always used Go tip, with
sadness I usually use Go1.4.3 now because it saves me so
much time. As an example, in one code base that I work on,
if I touch a deep dependency
and rebuild a test binary, under 1.4.3 it takes about 50s,
but under 1.6 it takes >180s. That extra 2 minutes of thumb
twiddling feels painful. If the time difference goes up to 5 minutes,
I won't be happy.

As for development, I'd like to try to get encoding/xml fixed again.

cheers,
rog.

[Matthew Dempsky]

On Thu, Jan 28, 2016 at 9:29 AM, Robert Griesemer <g...@golang.org> wrote:

- I'm going to complete the new export format. It's a binary format, so there will also be a mechanism to display it in a human-readable form.

I'm interested in helping with this, and I've already prepared a patch series at https://github.com/golang/go/compare/master...mdempsky:binary-builtin to eliminate the compiler's internal uses of the legacy text format.  (Will move to Gerrit once Go 1.7 tree opens up.)

I'm also generally interested in helping to further cleanup the compiler and runtime to make them more Go-ey.  I already have a long list of random loose ends to look at, but I'm open to suggestions if anyone has them.

[Seb Binet]

Hi,

I'd like to go forward with the 'reflect.StructOf' proposal:

https://golang.org/cl/9251

And perhaps, work on the 'plugin' package.

-s

sent from my droid

--

[Brad Fitzpatrick]

On Thu, Jan 28, 2016 at 10:36 AM, Seb Binet <seb....@gmail.com> wrote:

Hi,

I'd like to go forward with the 'reflect.StructOf' proposal:

https://golang.org/cl/9251

And perhaps, work on the 'plugin' package.

What's the plugin package?

[Joe Tsai]

In 1.6, I made several changes to archive/tar to harden that library up. I would like to continue that work and fix up the remaining issues with that library (#9647, #9683, #11171, #12594, #13548). Most of the work is already done (github.com/dsnet/tar), and I have pretty much been merging the changes slowly back. Thanks to @bradfitz for reviewing most of the past changes; would you be interesting in reviewing the future ones too?

Also, I am interested in merging in a bzip2 encoder (#4828) and fixing minor bzip2 problems (#13941). I have a plan to submit the new code in two-phases. First phases will focus on an working encoder by re-using index/suffixarray for the BWT construction. It won't be particularly fast, but will be correct. The second phase will be improving the speed of the BWT construction; that may be a future Go version.

If time permits, I would love to merge in some of the performance optimizations I have for flate and bzip2 decompression rates, which improves performance by 1.5x to 3x. Though, I doubt I'll get to them this time around.

[Brad Fitzpatrick]

I can probably review the archive/tar stuff. Or dsymonds.

[klau...@gmail.com]

I plan to submit my performance and compression optimizations for deflate. They are compatible, and brings the native Go implementation very close to cgo-based ones in perms of performance. This should increase performance of zip, gzip, png and direct use of the deflate library.

[Caleb Spare]

I intend to work on some small encoding/json improvements (6492, 12529, and 12146) unless someone beats me to them.

If I have time, I also want to work on the testing features in 10149 and 4899. The latter is tricky and may prove out of reach for me, and a solution probably needs a formal proposal at this point.

[Dave Cheney]

I want to second Brad's call for a focus on recovering the speed of the current compiler before merging SSA. 

Taking a 2x hit in 1.5 was palatable for Go users with the proviso that it was temporary. To take another 2x would stretch that promise beyond the point I think most people will reasonably accept.

Thanks

Dave

[Keith Randall]

My original plan was to spend 20% extra compiler time for the SSA backend.  We would then see how much of that we would get back because of better generated code for the compiler itself.

We're at 100% extra only because no optimization at all has happened yet.  I still intend to get back down to that 20%.

Feel free to jump in and help out on the ssa branch.  Any contributions would be appreciated.

[Richard Gooch]

On Thursday, 28 January 2016 07:15:07 UTC-8, rsc wrote:

The Go 1.6 release candidate is looking good. Hopefully in a few weeks it will be out, and we'll open the tree for Go 1.7 development. That makes now a good time to talk about what we want to do in Go 1.7 as far as technical work.

 I'm interested in having a streaming interface added to encoding/gob and encoding/json so that there is no need to marshal/unmarshal data into/from a buffer before/after writing/reading. The current implementation is causing us some pain. If I can get in-principle agreement on this enhancement, I'll plan and scope out the work and either do it myself or assign one of my team to do it.

Regards,

Richard....

[Andrew Gerrand]

On 29 January 2016 at 08:55, Richard Gooch <rg+go...@safe-mbox.com> wrote:

 I'm interested in having a streaming interface added to encoding/gob and encoding/json so that there is no need to marshal/unmarshal data into/from a buffer before/after writing/reading. The current implementation is causing us some pain. If I can get in-principle agreement on this enhancement, I'll plan and scope out the work and either do it myself or assign one of my team to do it.

Have you filed an issue about this? https://golang.org/s/proposal-process

[Richard Gooch]

No. I just cried a little when I skimmed over it, before my eyes glazed over :-/
Hopefully a full design doc isn't needed?

Regards,

Richard....

[Andrew Gerrand]

The first step is to file an issue describing what you'd like to add.

After that we can determine whether a full proposal is required. (probably not)

--

[Matt Silverlock]

I'd like to see net/context + net/http integration as per https://groups.google.com/forum/#!topic/golang-dev/TerfinvPffw (an associated issue does not seem to exist for it).

[Richard Gooch]

On Thursday, 28 January 2016 14:11:42 UTC-8, Andrew Gerrand wrote:

The first step is to file an issue describing what you'd like to add.

After that we can determine whether a full proposal is required. (probably not)

 https://github.com/golang/go/issues/14140

Regards,

Richard....

[Ian Lance Taylor]

I assume (or I hope) that Seb is referring to the plugin package
described at https://golang.org/s/execmodes .

Ian

[areg.meli...@intel.com]

Our team is definitely going to contribute to SSA.

Also lots of IA specific optimizations both for RT and compiler.

[David Norton]

I agree with Brad, Rog, and Dave that compiler speed should be one of the priorities.  I see a 2.7x slow down going from 1.4 to 1.5.  Doubling that means staying on 1.4 for daily dev work for another cycle of Go.

GC performance:  +1 anything that helps that work and -1 anything that impedes it.  We ended up rolling back to 1.4 for official releases due to performance.  1.6 looks promising but still slower than 1.4.

On Thursday, January 28, 2016 at 10:15:07 AM UTC-5, rsc wrote:

The Go 1.6 release candidate is looking good. Hopefully in a few weeks it will be out, and we'll open the tree for Go 1.7 development. That makes now a good time to talk about what we want to do in Go 1.7 as far as technical work.

Keith intends to merge the SSA-based compiler back end (amd64 only right now) into the main tree at the beginning of the Go 1.7 cycle. It works, and it generates code that on average runs about as fast as the current compiler. In many cases the SSA back end generates significantly better code than the old back end with little effect on performance: x86-64 chips (at least the server-class ones) are remarkably good at executing bad code quickly. We expect that the effect on non-x86 chips will be more pronounced, and there is still significant room for improvement in the x86 code quality, so we may yet see significant wins there. Unfortunately the SSA back end slows down the overall compilation by about 2x compared to using the old back end. This is not fundamental: we've spent almost all the development time focused on correctness and completeness, not on making it fast. The compiler work in the Go 1.7 cycle will focus on recovering much of that compile time loss.

Robert has a new, more compact export format for the compilers nearly ready. This should speed compilation a small amount and save a bit of disk space, and it fixes a few small problems in the old format. But most importantly it gives us a much cleaner base on which to make future changes to the export data, which will make it easier to do more interprocedural optimizations like escape analysis and inlining that work no matter where package boundaries are.

Rick and Austin have been working on further improvements to the garbage collector, especially in terms of throughput. That's a long effort and may not be ready for Go 1.7.

Another effort we'd like to do for Go 1.7 (but don't have concrete plans for yet) is collection and display of historical performance data, as a replacement for the old performance dashboard we had that no longer runs. The part that I believe is most important is the separation of collection from display. We want to make it possible for anyone to run a benchmarking system, just as anyone can run a builder, and have all that data feed into a database of some kind that anyone can fetch (perhaps as a Git repository) and analyze. It requires significant expertise to collect useful data and it requires significant but different expertise to analyze and display it well, and we think that separating these two parts will help let experts make progress on their halves independently.

In general I would encourage people once again to focus on cleanup and polish for Go 1.7. There are many parts of the tree that work but are not as good as they could be. For example we know that math/big is missing assembly support to varying degrees on some architectures. We also know there are many minor problems, missing examples, and so on that are significant collectively, even if not taken one at a time.

What are you planning to do in the Go 1.7 cycle? Or what GitHub issues do we not have marked with a Go 1.7 milestone yet but merit one, in your opinion?

[Brad Fitzpatrick]

David, do you mean gc or GC? (Capitals means garbage collector, lower is go compiler)

If GC, I don't know which bug you're talking about.

[David Norton]

Brad, sorry, I should have been specific.  I did mean GC as in garbage collector.  Not a specific bug but just saying +1 to the throughput work already in progress and -1 to any other work that might slow their progress.

[minux]

I will bring external linking to linux/ppc64 and linux/mips64{,le}.

I will also upstream the disassembler into x/arch and then

integrate it into cmd/objdump. These should make linux/ppc64

port more complete.

Cherry and I also have plans for the linux/mips64 cgo support.

[ron minnich]

There is a riscv port in progress, although if we hit 1.7 we'd be amazed. There are also two more non-Unix OS ports in progress, and we're chasing behind ToT just as fast as we can.

ron

--

[Seb Binet]

yep, that's the one.
thanks for clearing that up and apologies for the vagueness of my first email.

-s

[Keith Randall]

A week of low-hanging-fruit fixes by a few of us and SSA has improved to about 40% slower than tip.  Progress...

[daniel...@gmail.com]

As a user of Go (currently using 1.4.3 because don't really need to upgrade right now, but certainly want to eventually. I love the compile speed, and that is actually one of the main reasons I haven't updated yet), I wanted to chime in and agree with this statement:

"This may be an unpopular opinion, but: perhaps we need to recover the 2x slowdown we already suffered from Go 1.4 to Go 1.5 before we get twice as slow yet again with the wishful thinking that we'll recover it. Go 1.5 and Go 1.6 are still painful compared to Go 1.4. I'd like to say we won't merge SSA until our compiler is fast again.

4x slower builds & tests than Go 1.4 will not be pleasant."

Totally something that makes sense to me. Before slowing it down further, improve, and introduce next step. Unless of course this new thing helps overall. (I have no idea how SSA works currently, and it might be double effort to improve before SSA is introduced, but as I said, no idea how SSA works)

However, if including things (SSA) that makes it 2x slower, wouldn't it be a point to possibly include it early, and work on improving it before final release? Meaning it won't be 2x worse at the release?

Regards,
Daniel

[jserv...@gmail.com]

I second Brad's call for a focus on recovering the speed of the current compiler. 

This is imperative to maintain Go’s web development demographic. 

According to the commander in chief, "most Go programmers come from languages like Python and Ruby" and here is why: "Python and Ruby programmers come to Go because they don't have to surrender much expressiveness, but gain performance and get to play with concurrency." (http://commandcenter.blogspot.com/2012/06/less-is-exponentially-more.html)

When developing a web application or a software prototype, people need a "tight feedback loop". 

Python and Ruby provide this near 0 second waiting time to see the results via interpretation. 

2 seconds is a feedback loop too long. 

Please do not drive these web developers back to use Python or Ruby. 

I am not sure why Go is a must to compile Go. If the 2x slowdown cannot be recouped due to a switch from C to Go, then switching back to C does not seem like a crazy idea to me. 

Even Swift uses C++ to compile itself. Can we be pragmatic and keep the developer (the end user) happy for what the language was designed for? 

Thanks,

JPro