Partial Draft of Journal Paper on Embench

[David PATTERSON]

Monday morning is the first day of the computer architecture conference (ISCA) in Florida, and a key session exactly overlaps with our scheduled meeting (11AM ET), so I'll miss it.

Attached is the draft. I've highlighted in red the proposed experiments "we" would need to run to complete the paper (the last table has a summary of the experiments).

In each case we want to know the answer for CoreMark and Dhrystone as well to contrast the result with the more accurate Embench

1. ARM Performance gain over 7 GCC generations [DONE, in paper]
2. Code size different ARM vs RISC-V [Embench done, need for CoreMark & Dhrystone]
3. Performance change if optimize for code size for ARM and RISC-V
4. Code size change if optimize for performance for ARM and RISC-V
5. Performance/Code size  change if use CLANG for ARM and RISC-V
6. Performance/Code size  change if use LLVM for ARM and RISC-V
7. Performance/Code size  change if add M extension to RV32I
   
8. Performance/Code size  change if add C extension to RV32IM
   
9. Performance/Code size  change if add BitManip extension to RV32IMC
   
10. Performance/Code size  change if add F extension to RV32IMCB (should be zero)
    
11. Performance/Code size  change if add D extension to RV32IMCBF (should be zero)

Maybe you could discuss: is this reasonable or too much, and of OK, by when would the experiments be done?

I any case, please add comments to the Google Doc or to the PDF

In writing this up, a question arose for our naming policy for Embench, which I've copied below: 

"I wonder if it is confusing to call it Embench 1.0, 1.1, ... for integer and Embench 2.0, 2.1, ... for DSP. The other option would be Embench Integer 1.0, 1.1, ... and Embench DSP 0.5, 1.0, 1.1, ... . The latter is what MLPerf did for training, inference, embedded, ..."

Best,
Dave

 Embench Journal Paper Text.docx

[Ray Simar]

Hi all,

For today’s agenda, let’s discuss Dave’s below list of experiments.  As he suggests, let’s discuss the feasibility of answering each question and when we could have the results of the experiments.  

It would be great to hear any thoughts on how we might divide and conquer this problem, perhaps expanding beyond the usual, excellent, contributors we have!

Our call will be at the usual time and Zoom link:

• 10 AM US Central time, 8 AM US Pacific

[Mark Hill]

Hi everyone,

 

Sorry I didn’t make the call yesterday. One thing I’ve discovered when doing cross-platform comparisons for a toolchain is that care needs to be taken to ensure that all the detailed optimisations parameters are set to comparable values. For example, I have seen that armclang enables loop unrolling at O2 whereas this optimisation is not enabled for RISC-V clang even at -O3.  Jeremy, is there a way to get clang to report the settings of all the optimisation parameters so that they can be compared to check they match? If we don’t do this we risk comparing compiler optimisations rather than architectures/microarchitectures.

 

Best regards

 

Mark

--
You received this message because you are subscribed to the Google Groups "Embench" group.
To unsubscribe from this group and stop receiving emails from it, send an email to embench+u...@lists.librecores.org.
To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/9ACEE1EB-064A-4D21-ACF4-320460D1F44E%40rice.edu.

•  

 

Thanks in advance for all of your help.

 

And a belated Happy Father’s Day to all of the fathers we have in our group.  It’s a US holiday, but why not wish all our fathers well. :-)

 

All the best,

Ray

Begin forwarded message:

 

From: David PATTERSON <pat...@cs.berkeley.edu>

Subject: [embench] Partial Draft of Journal Paper on Embench

Date: June 17, 2023 at 8:10:19 AM CDT

To: emb...@lists.librecores.org

Reply-To: pat...@cs.berkeley.edu

 

Monday morning is the first day of the computer architecture conference (ISCA) in Florida, and a key session exactly overlaps with our scheduled meeting (11AM ET), so I'll miss it.

 

Attached is the draft. I've highlighted in red the proposed experiments "we" would need to run to complete the paper (the last table has a summary of the experiments).

 

In each case we want to know the answer for CoreMark and Dhrystone as well to contrast the result with the more accurate Embench

1. ARM Performance gain over 7 GCC generations [DONE, in paper]
2. Code size different ARM vs RISC-V [Embench done, need for CoreMark & Dhrystone]
3. Performance change if optimize for code size for ARM and RISC-V
4. Code size change if optimize for performance for ARM and RISC-V
5. Performance/Code size  change if use CLANG for ARM and RISC-V
6. Performance/Code size  change if use LLVM for ARM and RISC-V
7. Performance/Code size  change if add M extension to RV32I
8. Performance/Code size  change if add C extension to RV32IM
9. Performance/Code size  change if add BitManip extension to RV32IMC
10. Performance/Code size  change if add F extension to RV32IMCB (should be zero)
11. Performance/Code size  change if add D extension to RV32IMCBF (should be zero)

Maybe you could discuss: is this reasonable or too much, and of OK, by when would the experiments be done?

 

In any case, please add comments to the Google Doc or to the PDF

 

In writing this up, a question arose for our naming policy for Embench, which I've copied below: 

"I wonder if it is confusing to call it Embench 1.0, 1.1, ... for integer and Embench 2.0, 2.1, ... for DSP. The other option would be Embench Integer 1.0, 1.1, ... and Embench DSP 0.5, 1.0, 1.1, ... . The latter is what MLPerf did for training, inference, embedded, ..."

 

 

 

Best,
Dave

 

 

 Embench Journal Paper Text.docx

 

--
You received this message because you are subscribed to the Google Groups "Embench" group.
To unsubscribe from this group and stop receiving emails from it, send an email to embench+u...@lists.librecores.org.
To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/CAHis7pkcf%2BE96pVo_AXQX5YL%3DGXv2Fw%3DGHBKVZoSDm-L7ey5DQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Embench" group.
To unsubscribe from this group and stop receiving emails from it, send an email to embench+u...@lists.librecores.org.
To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/9ACEE1EB-064A-4D21-ACF4-320460D1F44E%40rice.edu.

 

--
You received this message because you are subscribed to the Google Groups "Embench" group.
To unsubscribe from this group and stop receiving emails from it, send an email to embench+u...@lists.librecores.org.
To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/9ACEE1EB-064A-4D21-ACF4-320460D1F44E%40rice.edu.

[Paolo Savini]

Hi Mark,

Jeremy is on vacation this week. Good point Mark, thanks for this. Here's a way you can tell clang to print the flags used:

echo "int;" | clang -xc -O2  - -o /dev/null -\#\#\#

We could use that to monitor and compare the -O2 and Os sets across the different Clang/LLVM versions.

Best,

Paolo

To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/eb022ce67fef4ef5a690ae8fb41af444%40huawei.com.

[Mark Hill]

Thanks for the tip Paolo, that’s useful to know.

To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/70d26930-b90f-d5f9-c376-be1895bae3ee%40embecosm.com.