Congratulations - MLPerf Tiny v0.5 release with results!

69 views
Skip to first unread message

David Kanter

unread,
Jun 16, 2021, 1:01:12 PM6/16/21
to public, members, tiny
Hi Everyone,

The brand new MLPerf Tiny benchmark suite and results went live this morning - you can find the results here: https://www.mlcommons.org/en/inference-tiny-05/

We have the press release here: https://mlcommons.org/en/news/mlperf-tiny-v05/

This is one of our last benchmark suites to go live and at this point, we can now safely say that MLPerf spans microwatts to megawatts :) Kudos to anyone who can get us down to nanowatts...

The first v0.5 round included five submissions from academic, industry organizations, and national labs, producing 17 peer-reviewed results. Submissions this round included software and hardware innovations from Latent AI, Syntiant, PengCheng Labs, Columbia, UCSD, CERN, and Fermilab.

This was the first of many releases for MLPerf Tiny. The benchmark suite was a collaboration over the last 18 months, including representatives from: Harvard University, EEMBC, CERN, Columbia, Digital Catapult, Fermilab, Google, Infineon, Latent AI, ON Semiconductor, Peng Cheng Laboratories, Qualcomm, Renesas, SambaNova Systems, Silicon Labs, STMicroelectronics,  Synopsys, Syntiant, UCSD, and Voicemed.

We had a joint press briefing two days ago with about half a dozen press and look forward to great coverage in the coming days.

A big thanks to all of our contributors for your hard work and congratulations to the submitters on your results!

I want to emphasize a number of folks for their outsize contributions:

Colby and Vijay who led the MLPerf Tiny effort for MLCommons.

Peter Torelli from EEMBC who helped with a lot of coding and support, contributing the EnergyBench framework for energy measurement.

To our benchmarks owners: Csaba from Digital Catapult, Jeremy from Syntiant, Nat from Google, and Pietro from VoiceMed.

Let's give all of these folks and our submitters a big congratulations, we are looking forward to more great work in the future!

Also, since this is a brand new benchmark suite - if it looks interesting, please get involved at https://groups.google.com/u/4/a/mlcommons.org/g/tiny

Thanks and have a fantastic day!

David Kanter
Executive Director



Umeton, Renato,Ph.D.

unread,
Jun 18, 2021, 8:54:05 PM6/18/21
to David Kanter, public, members, tiny

Amazing result!

 

Congratulations everybody!

 

Best,

Renato

 

 

 

 

– 

Renato Umeton, Ph.D.

renato...@dfci.harvard.edu

Associate Director of Artificial Intelligence Operations and Data Science Services

Dana-Farber Cancer Institute, a teaching affiliate of Harvard Medical School

+ Harvard T.H. Chan School of Public Health

+ Massachusetts Institute of Technology

+ Weill Cornell Medicine

450 Brookline Avenue, OS355, Boston, MA 02215

Phone (617) 582 - 9754

 

From: David Kanter <da...@mlcommons.org>
Date: Wednesday, June 16, 2021 at 1:01 PM
To: public <pub...@mlcommons.org>, members <mem...@mlcommons.org>, tiny <ti...@mlcommons.org>
Subject: Congratulations - MLPerf Tiny v0.5 release with results!

        External Email - Use Caution        

--
You received this message because you are subscribed to the Google Groups "public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public+un...@mlcommons.org.
To view this discussion on the web visit https://groups.google.com/a/mlcommons.org/d/msgid/public/CA%2B700FG1heHLQDO%3DkUEXbujo3mAOLpvS%2Bg3ZhNDfoQuzcJKcig%40mail.gmail.com.

The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at http://www.massgeneralbrigham.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

Please note that this e-mail is not secure (encrypted).  If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately.  Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail. 

Peter Mattson

unread,
Jun 18, 2021, 8:54:05 PM6/18/21
to David Kanter, public, members, tiny
Great work everyone! Really exciting! :-)

--
You received this message because you are subscribed to the Google Groups "members" group.
To unsubscribe from this group and stop receiving emails from it, send an email to members+u...@mlcommons.org.
To view this discussion on the web visit https://groups.google.com/a/mlcommons.org/d/msgid/members/CA%2B700FG1heHLQDO%3DkUEXbujo3mAOLpvS%2Bg3ZhNDfoQuzcJKcig%40mail.gmail.com.
For more options, visit https://groups.google.com/a/mlcommons.org/d/optout.

Janapa Reddi, Vijay

unread,
Jun 18, 2021, 8:54:05 PM6/18/21
to Peter Mattson, David Kanter, public, members, tiny
Many thanks to Peter Mattson and David Kanter! Both of them worked behind the scenes to help us get this benchmark done collaboratively across organizations.

And for those curious, here is the coverage in the press, thanks to the working group's effort and the administration at MLCommons:

(thank you all for your effort)

Coverage Total: 12 Articles


Notable Tweets: 


NEW: AI News Feed @AINewsFeed

MLCommons launches machine learning benchmark for devices like smartwatches and voice ... bit.ly/3gDLc8R #ai #ml #dl


NEW: Ben Wodecki @benwodecki

@commons_ml, the open engineering consortium behind the MLPerf benchmark test, has launched a new measurement suite aimed at ‘tiny’ devices like smartwatches and voice assistants aibusiness.com/document.asp?d…


NEW: Sally Ward-Foxton @sallywf

MLPerf now includes a benchmark for #TinyML systems. Developed with @eembc_org, the new benchmark has been carefully thought out and will hopefully offer a meaningful comparison for the smallest #AI systems going forward. @commons_ml @mlperf1 eetimes.com/mlperf-launche…


Datanami @datanami

MLCommons Releases MLPerf Tiny Inference Benchmark

ow.ly/1Li850FbWhm


Dave's News @davesnewscom

To measure ultra-low power AI, MLPerf gets a TinyML benchmark 

davesnews.co.uk/media/to-measu…


Edge Impulse @EdgeImpulse

MLPerf Tiny v0.5 is @commons_ml's first inference benchmark suite that targets ML use cases on embedded devices, measuring neural network model accuracy, performance latency and system power consumption. Check out the newly released results: bit.ly/3xtCyQ2


EnterpriseAI @EnterpriseAI_

MLCommons Releases MLPerf Tiny Inference Benchmark 

enterpriseai.news/2021/06/16/mlc…


Hackster.io @Hacksterio

MLPerf Tiny Inference is a new benchmarking suite for tinyML devices. hackster.io/news/how-does-…


HPCwire @HPCwire

MLCommons Releases MLPerf Tiny Inference Benchmark

ow.ly/Q7I450Fc1Y4 #HPC #AI


Neurons.AI - The Global AI Ecosystem #intoAI #AI @into_AI

To measure ultra-low power AI, MLPerf gets a TinyML benchmark - What's called TinyML, a broad neurons.ai/blog/news-stor… #ai #intoAInews


Stratodesk @stratodesk

To measure ultra-low power #AI, #MLPerf gets a #TinyML benchmark zdnet.com/article/to-mea… by @TiernanRayTech #EdgeComputing


Tiera @TieraOAtOSM

@commons_ml launched a new benchmark, #MLPerf #Tiny Inference, to measure how a trained #neural #network can process new data for low-power devices in small form factors and included an optional power measurement.

embeddedcomputing.com/technology/ai-…


ZDNet @ZDNet

To measure ultra-low power AI, MLPerf gets a TinyML benchmark zd.net/3gshSCe by @TiernanRayTech


Cheers! 

To view this discussion on the web visit https://groups.google.com/a/mlcommons.org/d/msgid/members/CAM7%2BEqc8%2B5bP1VmCXeK59oHa0UFE0DeuCbQVfYPsty%2B7WXKxhQ%40mail.gmail.com.

For more options, visit https://groups.google.com/a/mlcommons.org/d/optout.


--
Vijay Janapa Reddi, Ph. D. | Associate Professor 
John A. Paulson School of Engineering and Applied Sciences
Science and Engineering Complex (SEC)
150 Western Ave, Room #5.305
Boston, MA 02134
Harvard University

Dylan Zika

unread,
Jun 21, 2021, 9:09:44 PM6/21/21
to Janapa Reddi, Vijay, David Kanter, Peter Mattson, members, public, tiny
Great job everyone - good to see the weekly meetings turn into real results! 

You received this message because you are subscribed to the Google Groups "public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public+un...@mlcommons.org.
To view this discussion on the web visit https://groups.google.com/a/mlcommons.org/d/msgid/public/CAJe%3DzstnFgBLkSjLWu2aeHw2gfMC4Uy3K9jLhcjLmAR%3DH7e%2B4Q%40mail.gmail.com.

Alex Karargyris

unread,
Jul 2, 2021, 8:51:10 PM7/2/21
to Dylan Zika, Janapa Reddi, Vijay, David Kanter, Peter Mattson, members, public, tiny
Congratulations to the team! I can only imagine the hard work... Right on time before summer vacation :) 

-Alex

To view this discussion on the web visit https://groups.google.com/a/mlcommons.org/d/msgid/public/CAK%3DObKND33zyDen9fua3Fxd2QW4qThsf3EJNzvYwKo4h1zxRMw%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages