execution speed in 1.5 vs 1.4 (was Re: [golang-dev] Re: Go 1.7 planning)

[Russ Cox]

On Fri, Jan 29, 2016 at 11:07 AM, David Norton <dgno...@gmail.com> wrote:

GC performance:  +1 anything that helps that work and -1 anything that impedes it.  We ended up rolling back to 1.4 for official releases due to performance.  1.6 looks promising but still slower than 1.4.

 

Wait a second. You and others were talking about slow compile speed. But you swerved here and seem to be saying that you are using 1.4 for running production code not because of compile speed but because you found that your programs run faster in 1.4 than in 1.5. In general that is not expected. If I'm reading that correctly, can you please elaborate about what you're seeing as far as your own program's performance being worse in 1.5 (and still in 1.6) than in 1.4?

Thanks very much.

Russ

[Caleb Spare]

​In case it helps, the other example I know of people sticking with 1.4 due to performance issues with 1.5 is InfluxDB. See

https://github.com/influxdata/influxdb/issues/5217

-Caleb​

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brad Fitzpatrick]

On Fri, Jan 29, 2016 at 9:27 AM, Caleb Spare <ces...@gmail.com> wrote:

​In case it helps, the other example I know of people sticking with 1.4 due to performance issues with 1.5 is InfluxDB. See

https://github.com/influxdata/influxdb/issues/5217

Did anybody from those bugs bring this up on the Go mailing list or Go bug tracker? I at least don't remember seeing anything about this previously.

[Rick Hudson]

Paul,

If we could get the output from GODEBUG=gctrace=1 for 1.4.x, 1.5.x and 1.6rc1 it would be useful. In addition if you could record and send profiles of the tests on 1.4.2, 1.5.x and 1.6rc1 (unfortunately, pprof can't diff profiles from different binaries, but they're usually easy enough to compare by hand).

Sorry this dropped through the cracks, could you open a Go issue so that this gets the appropriate visibility.

On Fri, Jan 29, 2016 at 2:09 PM, <pa...@influxdb.com> wrote:

We should have posted to the list, sorry about that. I had sent an email directly to Rick Hudson back in December, but I think it got lost in the cracks.

We're happy to do much more extensive testing and help out any way we can.

Paul

[David Norton]

Russ,

Just to clarify, I was citing two different performance cases...

1.  Compiler performance mentioned by Brad, Rog, & Dave.  I agree with them and would like to see that improved in 1.7.

2.  The second is the run time performances issues we saw at InfluxDB, mentioned by Paul, Jason, & Caleb.

On Fri, Jan 29, 2016 at 3:27 PM, <ja...@influxdb.com> wrote:

We switched InfluxDB from 1.5 back to 1.4 due to performance and stability issues.   The performance issues we saw showed much higher cpu utilization and load on our 1.5 builds.  When we tested 1.4 vs 1.5, we saw significant decreases in write throughput and latency for the database.

https://github.com/influxdata/influxdb/issues/5217

https://github.com/influxdata/influxdb/issues/4977#issuecomment-170201007

Stability issues usually showed up on customers running on larger hardware.  For example, in this issue https://github.com/influxdata/influxdb/issues/5283 we were not sure if it was a bug in our code or go 1.5 at the time.  Rolling back to 1.4 resolved it though.  This was actually the second time we had to roll back from 1.5 due to stability issues: https://github.com/golang/go/issues/13176#issuecomment-155947904.   We were also struggling with these two issues at various times while using 1.5: https://github.com/influxdata/influxdb/issues/4554 and https://github.com/golang/go/issues/12932.  I believe those were resolved w/ 1.5.2 though.

We'd love to see runtime performance and stability back on par with 1.4 in go 1.6 or 1.7.

Jason

 

Thanks very much.

Russ

[to...@influxdb.com]

Rick,

I can take care of getting you the gtrace output and profiles for the different versions. I've been doing a ton of internal testing on InfluxDB, as well as fielding the performance reports from users, do I'm probably closest to the issue. If there's not already an issue open, I'll start a new one with those files.

Thanks!

Todd

[ja...@influxdb.com]

On Friday, January 29, 2016 at 10:06:18 AM UTC-7, rsc wrote:

[pa...@influxdb.com]

We should have posted to the list, sorry about that. I had sent an email directly to Rick Hudson back in December, but I think it got lost in the cracks.

We're happy to do much more extensive testing and help out any way we can.

Paul

On Friday, January 29, 2016 at 9:32:39 AM UTC-8, Brad Fitzpatrick wrote:

[Brad Fitzpatrick]

Please please please... it really would be helpful going forward if you participate in the testing process and give feedback rather than just silently reverting to old versions. By our benchmarks, releases keep getting better. If you have data that says otherwise, file bugs:

    https://golang.org/issues/new

Some people seem quite happy with Go 1.6: https://twitter.com/brianhatfield/status/692778741567721473

And the same people quite happy with Go 1.5: https://twitter.com/brianhatfield/status/692778741567721473

There were many of those going around for Go 1.5.

If you're not similarly happy, that is very interesting. Again: https://golang.org/issues/new

Testing Go 1.5 is pointless at this point, but if you can file bugs with GC traces comparing Go 1.4 to Go 1.6rc1, you might still help reveal a problem in the GC and get it fixed before the release. Maybe. rc1 is pretty close to totally frozen, but if the fix is small enough, it's possible. Filing bugs earlier is always better, though.

[andrey mirtchovski]

> Testing Go 1.5 is pointless at this point

the people who know the GC algorithm intimately may disagree: one more
regression data point would probably be quite useful in diagnosing the
issue.

i just wish they wrote it up after they're done :)

[Austin Clements]

On Fri, Jan 29, 2016 at 9:08 PM, andrey mirtchovski <mirtc...@gmail.com> wrote:

> Testing Go 1.5 is pointless at this point

the people who know the GC algorithm intimately may disagree: one more
regression data point would probably be quite useful in diagnosing the
issue.

"Pointless" may be a bit strong, but Brad's right that effort should be focused on the latest version and the last version that worked well for Influx. It's possible data on an intermediate version may be enlightening, but not very likely.

i just wish they wrote it up after they're done :)

Which "they" and "it" do you mean?

[andrey mirtchovski]

> Which "they" and "it" do you mean?

that one is easy: "they" is you, "it" is the regression issue
currently discussed :)

i have a voracious appetite for everything published with regards to
the GC algorithm. i find it fascinating. i wish the solution to a
regression such as this (even if it turns out to be trivial) would be
discussed somewhere.

[Keith Randall]

If there's a bug, there will be discussion on the bug and on any CLs referenced from it.

[andrey mirtchovski]

> If there's a bug, there will be discussion on the bug and on any CLs
> referenced from it.

right. with "write-up" i was wishfully thinking for something similar
to this: http://www.infoq.com/presentations/go-gc-performance

no pressure. and i'm by no means saying that any discussion is being
hidden. i'm just not following the CL list...

[Rick Hudson]

Could you send us a gctrace (GODEBUG=gctrace=1)? If you could also let us know how much RAM is in this machine. Finally have you tried increasing GOGC? If so what was the result? Any 1.6rc1 numbers would also be appreciated.

On Tue, Feb 2, 2016 at 12:41 AM, <mhal...@gmail.com> wrote:

Just to give another example (similar in application to Influx) where 1.5 is definitely slower than 1.4 - Running a UDP synchronous client-server and maxing looping through a send and receive routine is slower in 1.5.  Granted that 1.4 had quite a bit of variability in speed as a result of the garbage collection, but on average 1.5 ended up being 20% slower on windows that 1.4 had been.  This is a pretty specific case and I haven't tested on Ubuntu.  You might say that this is not a good example, but I am developing towards visualization of megahertz data and having my server needing to work 20% harder to do the same thing is a bit of a drag.

[Austin Clements]

Let's move technical discussion to https://github.com/golang/go/issues/14189.

[Austin Clements]

I don't think there's any single place everything is written down, but there are a few documents that may be of interest. There's a large comment at the top of https://github.com/golang/go/blob/master/src/runtime/mgc.go that explains a lot of the algorithm at a high level (though I just noticed parts of that are out of date; I suppose that's what happens). There's also a pretty thorough design document for how the garbage collector is scheduled, which is still quite accurate: https://golang.org/s/go15gcpacing

[mhal...@gmail.com]

Just to give another example (similar in application to Influx) where 1.5 is definitely slower than 1.4 - Running a UDP synchronous client-server and maxing looping through a send and receive routine is slower in 1.5.  Granted that 1.4 had quite a bit of variability in speed as a result of the garbage collection, but on average 1.5 ended up being 20% slower on windows that 1.4 had been.  This is a pretty specific case and I haven't tested on Ubuntu.  You might say that this is not a good example, but I am developing towards visualization of megahertz data and having my server needing to work 20% harder to do the same thing is a bit of a drag.

On Friday, January 29, 2016 at 9:06:18 AM UTC-8, rsc wrote:

[Matthew Halladay]

Sorry, I didn't read the thread enough to understand that this was compile speed and not performance speed.  With the small size of the programs I am working with I have no problem with compile times.

I wrote a couple tests for raw UDP throughput using ServerConn.ReadFromUDP(buf) inside a loop.  I saw a difference between home and work computers that made me blink a couple time before realizing I was running 1.4.3 still at work.  When I upgraded to 1.5 it resulted in slightly degraded average performace, much better minimum performance, and slightly worse maximum performance.

I haven't tested this any further, but have tested against C# with a simple Read, Depacketize, Compare message number routine on a continuous ~50 MBps stream.  I found that with the same implementation in both C# and Go that Go required ~7.5% CPU while C# needed 4.5% average CPU.  I will refrain from hypothesizing on reasons.  I haven't run the same test on Linux.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/sfwfQTu7eM4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.