Congratulations to Tiny and Training on new benchmarks and results!

[David Kanter]

Good morning everyone!

I want to offer a huge congratulations to the entire MLCommons® community. Today we released the 2Q23 MLPerf™ Training and Tiny results. Collectively, these benchmark suites scale from ultra-low power devices that draw just a few microwatts for inference all the way up to the most powerful multi-megawatt data center training platforms. Overall we saw performance gains of up to 1.54x compared to just a few months ago.

I am particularly thrilled by the excellent MLPerf Training team, which has added two new benchmarks this time. A large language model using GPT3 and a recommender using DLRM-dcnv2. The GPT3 benchmark in particular was a heavy lift, and required more compute than a single run of the original GPT3 - it’s a great example of how we can come together as a community and build things. NB: Our actually GPT3 benchmark is not the whole thing, but ~0.3% of GPT3 :)

The press release is here: https://mlcommons.org/en/news/mlperf-training-tiny-2q2023/

You can help promote us via Linkedin: https://www.linkedin.com/feed/update/urn:li:activity:7079492888371306499

or twitter: https://twitter.com/MLCommons/status/1673723035835322368?s=20

I’d especially like to congratulate several teams for their first submissions including CoreWeave, IEI, and Quanta Cloud Technology in MLPerf Training and Bosch, cTuning, fpgaConvNet, Kai Jiang, Krai, Nuvoton, and Skymizer for their first submissions in MLPerf Tiny.

A few individuals I'd like to call out for their contributions to these rounds:

• Eric Han and Ritika Borkar for leading the Training WG

• Csaba Kiraly and Jeremy Holleman for leading the Tiny WG

• Jan Lasek, Sami Wilf for leading the development of our new recommender benchmark (DLRM-dcnv2), and many participants including John Tran, Tayo Oguntebi, and Tong Xuan Liu

• Shriya Palsamudram, Yuechao Pan, and Anmol Gupta for leading the development of our LLM benchmark (GPT3), and many participants

• Pablo Gonzalez Mesa, Kongtao Chen, and Xinyuan Huang for improving the submission checker to make everything smoother

• Kelly Berschauer, Nathan Wasson, and David Tafur for driving the entire marketing and results process on their own!

Again congratulations to all the submitters and everyone else who has worked on the submissions. Please take a bit of the day to relax and celebrate doing your part to make ML better for everyone and as always let me know if you have any questions!

David Kanter

[David Kanter]

Great question Uday - we are using a smaller dataset than the full C4. The benchmark starts from a saved checkpoint and then measures the time to hit the target accuracy. Generally that should be about 1.3B tokens worth of training.

https://docs.google.com/presentation/d/1V8e8ODthM9_u4M8Mn4MX0dBPO43Zb-8hx2KZSldzQ58/edit has more information.

David

On Tue, Jun 27, 2023 at 12:57 PM Uday Kurkure <ukur...@vmware.com> wrote:

• Our actually GPT3 benchmark is not the whole thing, but ~0.3% of GPT3 :)

 

What does ~0.3% of GPT3 mean?

 

-Uday

 

From: David Kanter <da...@mlcommons.org>
Date: Tuesday, June 27, 2023 at 11:15 AM
To: public <pub...@mlcommons.org>, community <comm...@mlcommons.org>, Voting Representatives <voting-repr...@mlcommons.org>, Representative Contacts <represe...@mlcommons.org>, tiny-chairs <tiny-...@mlcommons.org>, training <trai...@mlcommons.org>, board <bo...@mlcommons.org>
Subject: Congratulations to Tiny and Training on new benchmarks and results!

!! External Email

--
You received this message because you are subscribed to the Google Groups "Voting Representatives" group.
To unsubscribe from this group and stop receiving emails from it, send an email to voting-representa...@mlcommons.org.
To view this discussion on the web visit https://groups.google.com/a/mlcommons.org/d/msgid/voting-representatives/CA%2B700FGMuVVR7sLE50THQCE34Jd5pygpqtm0h36k57wxYtwZDA%40mail.gmail.com.
For more options, visit https://groups.google.com/a/mlcommons.org/d/optout.

 

!! External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.

 

[Uday Kurkure]

• Our actually GPT3 benchmark is not the whole thing, but ~0.3% of GPT3 :)

 

What does ~0.3% of GPT3 mean?

 

-Uday

 

From: David Kanter <da...@mlcommons.org>
Date: Tuesday, June 27, 2023 at 11:15 AM
To: public <pub...@mlcommons.org>, community <comm...@mlcommons.org>, Voting Representatives <voting-repr...@mlcommons.org>, Representative Contacts <represe...@mlcommons.org>, tiny-chairs <tiny-...@mlcommons.org>, training <trai...@mlcommons.org>, board <bo...@mlcommons.org>
Subject: Congratulations to Tiny and Training on new benchmarks and results!

!! External Email

Good morning everyone!

--
You received this message because you are subscribed to the Google Groups "Voting Representatives" group.
To unsubscribe from this group and stop receiving emails from it, send an email to voting-representa...@mlcommons.org.
To view this discussion on the web visit https://groups.google.com/a/mlcommons.org/d/msgid/voting-representatives/CA%2B700FGMuVVR7sLE50THQCE34Jd5pygpqtm0h36k57wxYtwZDA%40mail.gmail.com.
For more options, visit https://groups.google.com/a/mlcommons.org/d/optout.

[Uday Kurkure]

Thanks, David.

[itay hubara]

The GPT benchmark runs over approx. 1.2B tokens (at least for batch sizes 1536 and 2048) instead of 400B so it is 0.003 fraction of the full GPT3 training

Itay

You received this message because you are subscribed to the Google Groups "public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public+un...@mlcommons.org.
To view this discussion on the web visit https://groups.google.com/a/mlcommons.org/d/msgid/public/BY5PR05MB7105C5D17C76D5A7418719AFBF27A%40BY5PR05MB7105.namprd05.prod.outlook.com.

[Uday Kurkure]

Thanks, Itay.

On Jun 28, 2023, at 4:55 AM, itay hubara <itayh...@gmail.com> wrote:



!! External Email