[LLVMdev] The performance of LLVM vs GCC

[Jianzhou Zhao]

Hi all,

This talk includes the performance comparison between LLVM and GCC
(page 30/31): LLVM wins a lot for both compilation and execution time.
http://llvm.org/pubs/2008-10-04-ACAT-LLVM-Intro.pdf
That talk and data were in 2008, I was wondering if there is any
updated performance evaluation between the latest LLVM and GCC?
Thanks.

--
Jianzhou
_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Duncan Sands]

Hi Jianzhou Zhao,

> This talk includes the performance comparison between LLVM and GCC
> (page 30/31): LLVM wins a lot for both compilation and execution time.
> http://llvm.org/pubs/2008-10-04-ACAT-LLVM-Intro.pdf
> That talk and data were in 2008, I was wondering if there is any
> updated performance evaluation between the latest LLVM and GCC?

Vladimir Makarov did some comparisons and reported the results here:

http://gcc.gnu.org/ml/gcc/2011-09/msg00043.html

Ciao, Duncan.

[Martin Whitaker]

[and copy to list]

Jianzhou Zhao wrote:
> Hi all,
>
> This talk includes the performance comparison between LLVM and GCC
> (page 30/31): LLVM wins a lot for both compilation and execution time.
> http://llvm.org/pubs/2008-10-04-ACAT-LLVM-Intro.pdf
> That talk and data were in 2008, I was wondering if there is any
> updated performance evaluation between the latest LLVM and GCC?
> Thanks.
>

Here's a more recent one:

http://www.phoronix.com/scan.php?page=article&item=gcc_46_llvm29&num=1

Martin

[Rotem, Nadav]

And this one, with LLVM ~3.0:

http://www.phoronix.com/scan.php?page=article&item=amd_bulldozer_compilers&num=1

http://www.phoronix.com/scan.php?page=article&item=gcc_46_llvm29&num=1

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[Duncan Sands]

Hi Martin,

> Here's a more recent one:
>
> http://www.phoronix.com/scan.php?page=article&item=gcc_46_llvm29&num=1

a few comments:

(1) The smallpt benchmark uses openmp which dragonegg supports but clang
does not. That's why on the machines with N processors the clang compiled
binary takes N times as long as the others: it is only using one processor
unlike the others.
(2) On the Himeno benchmark dragonegg beats the pants off everyone else. But
the benchmark was compiled at -O0 (no optimization!) so this is of no interest.
(3) GraphicsMagick also uses openmp, see point (1).

Ciao, Duncan.

[Duncan Sands]

On 11/03/11 10:11, Rotem, Nadav wrote:
> And this one, with LLVM ~3.0:
>
> http://www.phoronix.com/scan.php?page=article&item=amd_bulldozer_compilers&num=1

What, no dragonegg?! :)

Ciao, Duncan.

[陳韋任]

> > http://www.phoronix.com/scan.php?page=article&item=amd_bulldozer_compilers&num=1
>
> What, no dragonegg?! :)

+1

--
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667

[Chris Lattner]

On Nov 3, 2011, at 3:27 AM, Duncan Sands wrote:

> On 11/03/11 10:11, Rotem, Nadav wrote:
>> And this one, with LLVM ~3.0:
>>
>> http://www.phoronix.com/scan.php?page=article&item=amd_bulldozer_compilers&num=1
>
> What, no dragonegg?! :)

It is worth pointing out that the methodology of Phoronix makes their results basically useless. They willfully test -O0 codegen against different compilers, don't mention when a test is using OpenMP or not, and (in this case) are testing on a micro architecture that only one of the compilers is tuned for.

YMMV, but I consider their results to be complete garbage. It's nice to see Clang beating the pants off GCC in some tests though ;-)

-Chris

[Duncan Sands]

Hi Chris,

>>> And this one, with LLVM ~3.0:
>>>
>>> http://www.phoronix.com/scan.php?page=article&item=amd_bulldozer_compilers&num=1
>>
>> What, no dragonegg?! :)
>
> It is worth pointing out that the methodology of Phoronix makes their results basically useless. They willfully test -O0 codegen against different compilers, don't mention when a test is using OpenMP or not,

yes, I mentioned this in another email in this thread. I particularly enjoyed
the use of -O0 :) They know about these issues since they added notes about
-O0 and openmp to one of their previous sets of benchmarks, probably because of
feedback. They never seemed to realize that smallpt uses openmp though! I
notice that they do have a note about openmp after the graphicsmagick
benchmark, and don't report results for himeno (they were using -O0 for this
before), so I guess that's progress.

Ciao, Duncan.

and (in this case) are testing on a micro architecture that only one of the
compilers is tuned for.
>
> YMMV, but I consider their results to be complete garbage. It's nice to see Clang beating the pants off GCC in some tests though ;-)

[Eric Christopher]

On Nov 3, 2011, at 10:02 AM, Duncan Sands wrote:

 They know about these issues since they added notes about
-O0 and openmp to one of their previous sets of benchmarks, probably because of
feedback.

Yeah, I sent them feedback before… doesn't seem to have done any lasting good though.

-eric

[Martin Whitaker]

Hi Duncan,

Duncan Sands wrote:
> Hi Martin,
>
>> Here's a more recent one:
>>
>> http://www.phoronix.com/scan.php?page=article&item=gcc_46_llvm29&num=1
>
> a few comments:
>
> (1) The smallpt benchmark uses openmp which dragonegg supports but clang
> does not. That's why on the machines with N processors the clang compiled
> binary takes N times as long as the others: it is only using one processor
> unlike the others.
> (2) On the Himeno benchmark dragonegg beats the pants off everyone else. But
> the benchmark was compiled at -O0 (no optimization!) so this is of no interest.
> (3) GraphicsMagick also uses openmp, see point (1).
>

All benchmarks have to be treated with a healthy degree of skepticism!

For points 1 and 3, I think this is still a justifiable comparison to make -
the fact that clang doesn't support openmp means it genuinely can't perform as
well for these particular applications. For point 2, I agree this is a totally
useless benchmark.

What did strike me is that for the Intel processors there didn't seem to be a
clear winner or loser, but for the Opteron clang always came last. This
matches my own experiments on my Athlon based PC. Does this match other
peoples' experiences?

Martin

[Martin Whitaker]

Duncan Sands wrote:
> Vladimir Makarov did some comparisons and reported the results here:
>
> http://gcc.gnu.org/ml/gcc/2011-09/msg00043.html
>

I notice this one also has a test compiled with -O0 ;-)

Martin

[陳韋任]

Hi Duncan

Although people would like to know the performance comparison between
LLVM and GCC. Dragonegg has an advantage of combining both LLVM and GCC
opt. I remember Jack posted some interesting result on the list. [1]

Maybe you can post those result on the Dragonegg website so that
people know how they can play with Dragonegg. ;-)

Regards,
chenwj

[1] http://lists.cs.uiuc.edu/pipermail/llvmdev/2011-October/043931.html

--
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667

_______________________________________________