Using -O3 for performance testing

[Scott Jones]

I saw a comment about how, with -O3, that a number of things that had been slower than v0.4.5 were now faster.

That worries me a lot, since that doesn't seem like a fair comparison, and can gloss over real performance regressions.

For performance regression testing, if we want to test with -O3, we should be comparing it to a v0.4.5 also compiled with -O3

(*and* make sure there are tests added of compile time, as that can be important for some applications where code is generated,

and effects startup-time)

Yes, it's good news, but we don't want to lose sight of things that actually have gotten slower, and need to have some attention paid to them to see

exactly where the slow down is coming from (and there's been a lot of good work recently to focus on those regressions that have been identified)

-Scott

[Jameson Nash]

v0.4 was using something closer to -O3, hence our discussion of the comparison in the performance-tracker issue

[Scott Jones]

Could you elaborate how it was closer to -O3?  I'd read that the default for v0.4.x was -O2, and was concerned because of that.

Also, regression testing against v0.4.x is good for knowing when v0.5 will be ready for a release, but are we doing regular regression tests on master now,

so that we can more easily detect when new regressions (or massive speedups!) are introduced?

Thanks for the response!

-Scott

[Jacob Quinn]

For regular regression tests, see a recent comment here: https://github.com/JuliaLang/julia/issues/13893#issuecomment-225089178

-Jacob

[Stefan Karpinski]

The point of that comparison is not to be fair – it's to find out how LLVM changes and different optimization passes are affecting performance. We can change what optimization passes run by default if it makes sense.

[Scott Jones]

Ah, I hadn't seen that yet, that is great news!

Kudos to all those who contributed to improving the benchmarking framework, in particular @jrevels and @johnmyleswhite (we use Benchmarks.jl for our own performance testing at work), and @iainNZ & @staticfloat for the perf tests.

[Tobi]

There have been huge changes between 0.4 and 0.5 so that it seems to be difficult determining the cause of them. In particular the switch in LLVM version has been made.

If i understand it correctly 0.4 was shipped with -O2 since -O3 did not gain direct advantages. If this has changed with recent versions of LLVM I don't see that it would be unfair to switch to -O3.

Cheers

Tobias

[Scott Jones]

I was not saying that it would be unfair to switch to -O3, nor that it is unfair to compare things with different optimization levels to determine which one should be used in v0.5,

but rather that care should be taken that any performance improvements from switching to -O3 do not hide real performance regressions.

For example, if PR #27 makes splatting twice as slow, and unrelated PR #42 makes everything twice as fast, then if you aren't careful, the performance regression in PR #27

may go unnoticed.  That's all I was concerned about, but from Jacob's comment, that has very recently been mostly addressed, by doing frequent performance regression testing (kudos again to J. Revels!).

(People here know that I complained about a number of things like this over a year ago, but steadily they have been getting addressed - it's amazing to see the improvements in documentation (internal and external, and the whole framework), unit testing, performance testing, testing for package breakage, etc. that has occurred in the last 14 months!)