Draft of IEEE Computer paper

[David PATTERSON]

We have what I believe are the latest results and a full draft. At the minimum, please take a look to be sure I included your name and update the one sentence bio and email address on page 7. (I'd prefer if you updated the Google Doc so that i don't have to do it, but I included a PDF as well.)

I also attached the (large) spreadsheet with many tabs if you want to look at the data. The yellow cells have the latest data from Jennifer.

It is currently a short paper (~4850 words out of 6000) in part because we listed some questions that we didn't answer as exercises for the reader, so there is room to add more.

Below are the official IEEE guidelines for being an author. My view is that it's much better to err on the side of inclusion vs exclusion.

Best,
Dave

 Embench Journal Paper Text.docx

 Embench 1.0 Results

https://journals.ieeeauthorcenter.ieee.org/become-an-ieee-journal-author/publishing-ethics/ethical-requirements/

IEEE considers individuals who meet all of the following criteria to be authors:

1. made a significant intellectual contribution to the theoretical development, system or experimental design, prototype development, and/or the analysis and interpretation of data associated with the work contained in the article;
2. contributed to drafting the article or reviewing and/or revising it for intellectual content;
3. approved the final version of the article as accepted for publication, including references.

Contributors who do not meet all of the above criteria may be included in the Acknowledgment section of the article. Omitting an author who contributed to your article or including a person who did not fulfill all of the above requirements is considered a breach of publishing ethics.

[Ray Simar]

David,

So great to see this update!  Thanks for your persistence, patience, and leadership in pulling this together.

All,

Please see Dave’s request for inputs.  And, if at all possible, update Dave’s Google Doc with your one-sentence bio and email address on page 7.  If I have the Google Doc link correct, it should be here: https://drive.google.com/file/d/1dUjVoaAUuMFqDzfItZTk1xu7aEsDGtnV/view

As we have a meeting one week from today, let’s see if we can have given Dave all of the feedback we can by that meeting.  We’ll use the meeting to review our progress.

Thanks to everyone for their support!

All the best,

Ray

--
You received this message because you are subscribed to the Google Groups "Embench" group.
To unsubscribe from this group and stop receiving emails from it, send an email to embench+u...@lists.librecores.org.
To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/CAHis7pkLE2AWgCu1iy%3DsSL_L9mutEvGR0XEM6xVu8PQa1VXC5Q%40mail.gmail.com.

[Jeremy Bennett]

On 11-12-2023 17:32, Ray Simar wrote:
> David,
>
> So great to see this update!  Thanks for your persistence, patience, and
> leadership in pulling this together.

Hi Dave,

Thanks for pulling this all together. I'm disappearing on vacation for a
few days tomorrow, but I'll take a look before I leave in the morning.

Best wishes,


Jeremy

>> Embench Journal Paper Text.docx
>> <https://urldefense.com/v3/__https://drive.google.com/file/d/1hsHLI08RKC_W3wnJWt15B33u3LqEp4MW/view?usp=drive_web__;!!BuQPrrmRaQ!gCNom6V1HR9E_5r3mP8Ij_pYLdFekk21v-3Z1xsLc2S6wLxmNKx7mT-TXF2TOKZVVXXx3tKvCk8Z3MHmsVQJzg$>
>>
>> Embench 1.0 Results
>> <https://docs.google.com/spreadsheets/d/1UhmOgGy8_ujWjylmgtL4mWh5bRGeTUQ2126T7Nl99a4/edit?usp=drive_web>
>>
>>
>>
>> https://journals.ieeeauthorcenter.ieee.org/become-an-ieee-journal-author/publishing-ethics/ethical-requirements/ <https://urldefense.com/v3/__https://journals.ieeeauthorcenter.ieee.org/become-an-ieee-journal-author/publishing-ethics/ethical-requirements/__;!!BuQPrrmRaQ!gCNom6V1HR9E_5r3mP8Ij_pYLdFekk21v-3Z1xsLc2S6wLxmNKx7mT-TXF2TOKZVVXXx3tKvCk8Z3MEtIQO7UA$>
>>
>> IEEE considers individuals who meet *all* of the following criteria to
>> be authors:
>>
>> 1. made a significant intellectual contribution to the theoretical

>> development, system or experimental design, prototype development,
>> and/or the analysis and interpretation of data associated with the
>> work contained in the article;

>> 2. contributed to drafting the article or reviewing and/or revising
>> it for intellectual content;
>> 3. approved the final version of the article as accepted for

>> publication, including references.
>>
>> Contributors who do not meet all of the above criteria may be included
>> in the Acknowledgment section of the article. Omitting an author who
>> contributed to your article or including a person who did not fulfill
>> all of the above requirements is considered a breach of publishing ethics.
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Embench" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to embench+u...@lists.librecores.org

>> <mailto:embench+u...@lists.librecores.org>.

>> To view this discussion on the web visit

>> https://groups.google.com/a/lists.librecores.org/d/msgid/embench/CAHis7pkLE2AWgCu1iy%3DsSL_L9mutEvGR0XEM6xVu8PQa1VXC5Q%40mail.gmail.com <https://urldefense.com/v3/__https://groups.google.com/a/lists.librecores.org/d/msgid/embench/CAHis7pkLE2AWgCu1iy*3DsSL_L9mutEvGR0XEM6xVu8PQa1VXC5Q*40mail.gmail.com?utm_medium=email&utm_source=footer__;JSU!!BuQPrrmRaQ!gCNom6V1HR9E_5r3mP8Ij_pYLdFekk21v-3Z1xsLc2S6wLxmNKx7mT-TXF2TOKZVVXXx3tKvCk8Z3MG1crYtnw$>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Embench" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to embench+u...@lists.librecores.org

> <mailto:embench+u...@lists.librecores.org>.

> To view this discussion on the web visit

> https://groups.google.com/a/lists.librecores.org/d/msgid/embench/4F0FC84F-FBE0-4ACC-AA7B-D6647EA606DE%40rice.edu <https://groups.google.com/a/lists.librecores.org/d/msgid/embench/4F0FC84F-FBE0-4ACC-AA7B-D6647EA606DE%40rice.edu?utm_medium=email&utm_source=footer>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Embench" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to embench+u...@lists.librecores.org

> <mailto:embench+u...@lists.librecores.org>.

> To view this discussion on the web visit

> https://groups.google.com/a/lists.librecores.org/d/msgid/embench/4F0FC84F-FBE0-4ACC-AA7B-D6647EA606DE%40rice.edu <https://groups.google.com/a/lists.librecores.org/d/msgid/embench/4F0FC84F-FBE0-4ACC-AA7B-D6647EA606DE%40rice.edu?utm_medium=email&utm_source=footer>.

--
Cell: +44 7970 676050 (UK), +49 172 683 5745 (Germany)
SkypeID: jeremybennett
Twitter: @jeremypbennett
Email: jeremy....@embecosm.com
Web: www.embecosm.com
PGP key: 1024D/BEF58172FB4754E1 2009-03-20

[Jeremy Bennett]

On 09-12-2023 15:25, David PATTERSON wrote:
> We have what I believe are the latest results and a full draft. At the
> minimum, please take a look to be sure I included your name and update
> the one sentence bio and email address on page 7. (I'd prefer if you
> updated the Google Doc so that i don't have to do it, but I included a
> PDF as well.)

Hi Dave,

The paper looks great. I have only made 3 changes/suggestions to the
Google Doc.
- I have added my bio at the end
- I have fixed a typo in the spelling of "Embecosm" in the bios
- I have added a comment about whether section 5 should also reference
BEEBS.

I've asked Paolo to make sure all the Embecosm authors i) have their
bios completed ASAP and ii) review the paper.

Thanks for putting this all together.

Best wishes,


Jeremy

> I also attached the (large) spreadsheet with many tabs if you want to
> look at the data. The yellow cells have the latest data from Jennifer.
>
> It is currently a short paper (~4850 words out of 6000) in part because
> we listed some questions that we didn't answer as exercises for the
> reader, so there is room to add more.
>
> Below are the official IEEE guidelines for being an author. My view is
> that it's much better to err on the side of inclusion vs exclusion.
>
> Best,
> Dave
>

> Embench Journal Paper Text.docx
> <https://drive.google.com/file/d/1hsHLI08RKC_W3wnJWt15B33u3LqEp4MW/view?usp=drive_web>

>
> Embench 1.0 Results
> <https://docs.google.com/spreadsheets/d/1UhmOgGy8_ujWjylmgtL4mWh5bRGeTUQ2126T7Nl99a4/edit?usp=drive_web>
>
>
>

> https://journals.ieeeauthorcenter.ieee.org/become-an-ieee-journal-author/publishing-ethics/ethical-requirements/ <https://journals.ieeeauthorcenter.ieee.org/become-an-ieee-journal-author/publishing-ethics/ethical-requirements/>
>
> IEEE considers individuals who meet *all* of the following criteria to
> be authors:
>
> 1. made a significant intellectual contribution to the theoretical

> development, system or experimental design, prototype development,
> and/or the analysis and interpretation of data associated with the
> work contained in the article;

> 2. contributed to drafting the article or reviewing and/or revising it
> for intellectual content;
> 3. approved the final version of the article as accepted for

> publication, including references.
>
> Contributors who do not meet all of the above criteria may be included
> in the Acknowledgment section of the article. Omitting an author who
> contributed to your article or including a person who did not fulfill
> all of the above requirements is considered a breach of publishing ethics.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Embench" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to embench+u...@lists.librecores.org
> <mailto:embench+u...@lists.librecores.org>.
> To view this discussion on the web visit

> https://groups.google.com/a/lists.librecores.org/d/msgid/embench/CAHis7pkLE2AWgCu1iy%3DsSL_L9mutEvGR0XEM6xVu8PQa1VXC5Q%40mail.gmail.com <https://groups.google.com/a/lists.librecores.org/d/msgid/embench/CAHis7pkLE2AWgCu1iy%3DsSL_L9mutEvGR0XEM6xVu8PQa1VXC5Q%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Ray Simar]

Hi all,

I am looking forward to our December meeting this Monday, December 18th.

First, here’s our Zoom link for our meeting (8 AM US Pacific time): https://riceuniversity.zoom.us/j/94249046147?pwd=YzBHTzdHWWVFY09YNHdvTjREME84Zz09 

I have appended Dave’s email about the great progress he has made.  Please read over this.  Let’s use this meeting to do a check on our overall progress and find out what we can do to help Dave get this over the line.

For example, everyone please follow up on this (minimum) action:

• take a look to be sure Dave included your name and update the one sentence bio and email address on page 7. (Dave would prefer if you updated the Google Doc so that he doesn't have to do it, but I included a PDF as well.)

Dave and I followed up with Stefan and he will be working on a real use case of Embench for inclusion in the paper.  If he makes the meeting, he can give us an update.

See you tomorrow!

Happy Holidays, Merry Christmas,
and Best Wishes for the New Year,
Ray

Begin forwarded message:

--
You received this message because you are subscribed to the Google Groups "Embench" group.

To unsubscribe from this group and stop receiving emails from it, send an email to embench+u...@lists.librecores.org.
To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/CAHis7pkLE2AWgCu1iy%3DsSL_L9mutEvGR0XEM6xVu8PQa1VXC5Q%40mail.gmail.com.

[David Patterson]

I also wonder if we should talk about updating Embench 1.0 given how long it has been since we released in AND the known flaws in Cubic and Nbody

Best, Dave

P.S. If this email arrives outside of regular hours, I'm not expecting a reply then. I just work weird hours.

[Ray Simar]

Hi David,

I agree!  Thanks for mentioning this.

Happy Holidays, Merry Christmas,
and Best Wishes for the New Year,
Ray

To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/CAH_T6%3D%2BDUTjqMR-C1o%2B_3Xrvuin1JmqomNr9KGkXhcso76PkdA%40mail.gmail.com.

[Roger Shepherd]

Ray,

On 18 Dec 2023, at 00:26, Ray Simar <ray....@rice.edu> wrote:

Hi all,

I am looking forward to our December meeting this Monday, December 18th.

First, here’s our Zoom link for our meeting (8 AM US Pacific time): https://riceuniversity.zoom.us/j/94249046147?pwd=YzBHTzdHWWVFY09YNHdvTjREME84Zz09 

Apologies. I’m away at the moment and it is unlikely I can join the call today.

I have appended Dave’s email about the great progress he has made.  Please read over this.  Let’s use this meeting to do a check on our overall progress and find out what we can do to help Dave get this over the line.

For example, everyone please follow up on this (minimum) action:

• take a look to be sure Dave included your name and update the one sentence bio and email address on page 7. (Dave would prefer if you updated the Google Doc so that he doesn't have to do it, but I included a PDF as well.)

I’ve updated the Google doc with my info.

I have a number of points arising from the latest draft which I’d hoped to raise on the call. As it seems I won’t make the call, I’ll try to put a short mail together with the main points - not 100% sure I’ll manage this before the meeting.

Regards

Roger

To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/70F56E28-BE7F-481F-8CC3-60CF0F69C721%40rice.edu.
<Embench Journal Paper Text.pdf>

--
You received this message because you are subscribed to the Google Groups "Embench" group.
To unsubscribe from this group and stop receiving emails from it, send an email to embench+u...@lists.librecores.org.

To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/70F56E28-BE7F-481F-8CC3-60CF0F69C721%40rice.edu.
<drive_2020q4_32dp.png>

--
You received this message because you are subscribed to the Google Groups "Embench" group.
To unsubscribe from this group and stop receiving emails from it, send an email to embench+u...@lists.librecores.org.

To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/70F56E28-BE7F-481F-8CC3-60CF0F69C721%40rice.edu.
<AHkbwyJsWj78v_FwCab2UokXbMYWjkcYulm3GgQ9D5JhWH1y0Uscidh5S8dpL_2UWXTwtSmrbqaeyqLEWa1nnItWMHF3UXMPOYnT0HqgfNZDDfDnIOo=w1200-h630-p.png>

--
You received this message because you are subscribed to the Google Groups "Embench" group.
To unsubscribe from this group and stop receiving emails from it, send an email to embench+u...@lists.librecores.org.

To view this discussion on the web visit https://groups.google.com/a/lists.librecores.org/d/msgid/embench/70F56E28-BE7F-481F-8CC3-60CF0F69C721%40rice.edu.

Roger Shepherd

roger.s...@chipless.eu

[Roger Shepherd]

Ray et al,

I’ve made a couple of changes to the Google doc as “suggestions” (I think - I’m not familiar with Google docs so excuse me if it hasn’t worked as intended). I also have the following comments which I hope read in the supportive manner in which they were intended - 1) I think the paper is looking pretty good, 2) I know I can be very picky.

Enjoy your call.

Roger

1. The compiler optimisation point could be expended to illustrate the utility of having a suite rather than a single program. It looks like nearly all the performance improvement comes from a single one of the component benchmarks - statement. 

2, Was there actually a goal of no floating point? We certainly didn’t achieve it! And nobody is problematic for other reasons as well.

3. Regarding code size. In deeply embedded/very low-end systems, the issue of “smaller code size is cheaper” is often less important than “smaller code size means more function for a fixed cost” - how much function can I offer for $1?

4. Benchmarking code size. I think this really important and the more I think about this, the more important I think it is. In my experience, measuring code size is rocket science - speed is comparatively easy to measure - you just need a stopwatch. Code size is inherently harder - first you need to decide what you want to measure (e.g. libraries - what about f.p. on a fixed point machine…..) - then you need to be able to measure it (e.g. striping symbols and debugging information….). I think saying that Embench sidesteps the issue of libraries underplays what has been done. I’d prefer to say Embench has solved the issue!

5. Benchmark size and library size.

I think this topic deserves to be covered more. We should say that a benefit of the Embench suite is that it has a lot more code than either of the legacy benchmarks. We give the LOC for Embench but don’t give the figures for  Dhrystone and Coremark. If we do look at the (code) size of the benchmarks we can then move on to the library comparison. I think a code size comparison between the three libraries, Embench (total), Coremark and Dhrystone would be useful - whether text, a table or a graph. I think it’s worth stating that glib has 10x the code of Drystone.

6. Related work.

Concerning Dhrystone. “The library calls within the benchmark consume the majority of the time, but they are not officially part of the benchmark”. Up to a point…. Because of its (flawed) method of construction - a synthetic benchmark designed to match STATIC properties of programs being used to measure DYNAMIC (execution) properties - the bulk of the work in Dhrystone are string operations - comparison and copy. These operations ARE part of the program but are implemented by the library. So, I think “not officially part of the benchmark” isn’t really correct. The comment in Weiss’s paper is  “While you may think you are benchmarking a processor, you are really benchmarking are the compiler writer’s optimizations of the C library functions for a particular platform.” I think the real problem is that real programs aren’t hyperoptimised, libraries are, and so a benchmark dominated by library performance can’t reflect what real programs do. [Unless real programs are dominated by library performance - which might be the case for Objective C or Swift programs, but won’t be the case for most deeply embedded programs].

7. Other

I think we could say more about the benefit of Embench being a suite (more than just my point 1 above). But perhaps that should be a separate paper? 

Roger Shepherd

roger.s...@chipless.eu

[Ray Simar]

Hi all,

Here’s a quick list of items and actions from today:

First, The paper is in great shape and ready to launch for review.  Might be a 1-3 months time frame till we hear back.  Many thanks to Dave for taking the leadership on this effort!

• Dave thinks he can incorporate Rogers recent comments.

Want to move toward on Embench 1.0/integer/IOT suite

• Asked Jeremy to come up with a list of the four or so benchmarks we might delete.  
• Nbody and Cubic are on the list.
• Per Jennifer, the ud benchmark has a * 2.0 in it that should be changed to a * 2 to avoid an unnecessary floating-point operation.
• Jeremy has a good bitmanip operation he’d like to add to the suite.
• We could use help cleaning up the flows to support the change in benchmarks.  Jeremy created these, but he could use some help!

Dave raised the thought of including an embedded ML benchmark

• Ray will take a look at this based on some recent work his students have done on TinyML.

Finally, we have moved our Jan 15th meeting to Jan 22nd.

That looks to me like the high points.  Please shoot me a note if you’d like to make any corrections and additions.

Happy Holidays, Merry Christmas,
and Best Wishes for the New Year,
Ray