Power System description - Invitation to edit

David Kanter (via Google Sheets)

unread,

Dec 15, 2020, 5:50:05 PM12/15/20

to pub...@mlcommons.org

da...@mlcommons.org has invited you to edit the following spreadsheet:

Google Sheets: Create and edit spreadsheets online.
Google LLC, 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA
You have received this email because da...@mlcommons.org shared a spreadsheet with you from Google Sheets.

David Kanter

unread,

Jan 12, 2021, 4:03:47 PM1/12/21

to Jeff Liu, public

Jeff,

We absolutely want our results to be reproducible, so thank you for pointing out these problems.

Can you share a little bit more detail here? Which data sets are a problem?

What kind of error messages are you getting?

Thanks,

David

On Tue, Jan 12, 2021 at 12:59 PM Jeff Liu <yajun...@gmail.com> wrote:

We have all kind of issues to reproduce MLPerf Inferencing 0.7 both for Intel and NVIDIA, the key issues are: 1) repo be moved or version changed 2) dataset, either no access or too slow to download.

There are customers who ask for MLPerf benchmark numbers prior to making the decision to procure GPU systems. So my goal is to make MLPerf a standard spec on all our AI/DL/ML related systems. My humble opinion: we need to make the MLPerf procedures repeatable on each submitter, otherwise, it will not differentiate from other benchmarking apps.

Thanks

jeff

--

David Kanter

Executive Director

MLCommons

Jeff Liu

unread,

Jan 16, 2021, 1:02:28 PM1/16/21

to public, da...@mlcommons.org

We have all kind of issues to reproduce MLPerf Inferencing 0.7 both for Intel and NVIDIA, the key issues are: 1) repo be moved or version changed 2) dataset, either no access or too slow to download.

There are customers who ask for MLPerf benchmark numbers prior to making the decision to procure GPU systems. So my goal is to make MLPerf a standard spec on all our AI/DL/ML related systems. My humble opinion: we need to make the MLPerf procedures repeatable on each submitter, otherwise, it will not differentiate from other benchmarking apps.

Thanks

jeff

On Tuesday, December 15, 2020 at 2:50:05 PM UTC-8 da...@mlcommons.org wrote:

Jeff Y. Liu

unread,

Jan 16, 2021, 1:02:37 PM1/16/21

to David Kanter, public

David,

For the dataset, ImageNet is no longer processing requests for downloads of specific images. I have to use Academic Bittorrent to get ImageNet 2012. In DLRM training, the datasets mentioned in the procedure is not there anymore, it might be trivial in this case because we can use a similar one. If MLCommons can come up with a way to centralize datasets and repos for each submission, that will be ideal.

In my case, I am trying to do inferencing(0.7) with NVIDIA, at the very beginning, here is the message:

"make prebuild"

#16 6.016 Err:22 http://archive.ubuntu.com/ubuntu eoan Release

#16 6.016 404 Not Found [IP: 91.189.88.142 80]

#16 6.154 Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2298 kB]

#16 6.336 Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [45.6 kB]

#16 6.337 Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [304 kB]

#16 6.341 Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2140 kB]

#16 6.507 Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]

#16 6.507 Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]

#16 6.978 Reading package lists...

#16 7.888 E: The repository 'http://archive.ubuntu.com/ubuntu eoan Release' does not have a Release file.

------

executor failed running [/bin/bash -c echo 'deb http://archive.ubuntu.com/ubuntu eoan main restricted universe multiverse' | tee -a /etc/apt/sources.list.d/eoan.list && echo 'Package: *\nPin: release a=eoan\nPin-Priority: -10\n' | tee -a /etc/apt/preferences.d/eoan.pref && apt-get update && apt-get install --no-install-recommends -t eoan -y libjemalloc2 libtcmalloc-minimal4]: exit code: 100

Makefile:174: recipe for target 'build_docker' failed

make[1]: *** [build_docker] Error 1

make[1]: Leaving directory '/home/ubuntu/inference_results_v0.7/closed/NVIDIA'

Makefile:141: recipe for target 'prebuild' failed

make: *** [prebuild] Error 2

Then "make download_model" spews out many errors... is there a place we can file those errors I have seen just to avoid others to experience the same?

Thanks for answering my email and looking forward to hearing back from you!

Jeff

Po-Han Huang

unread,

Jan 19, 2021, 12:38:32 PM1/19/21

to David Kanter, Jeff Y. Liu, public

Hi Jeff,

> #16 7.888 E: The repository 'http://archive.ubuntu.com/ubuntu eoan Release' does not have a Release file.

Please replace all the eoan with focal in the docker/Dockerfile. Eoan is no longer supported.

> Then "make download_model" spews out many errors... is there a place we can file those errors I have seen just to avoid others to experience the same?

Could you share with us what errors you saw? Also, it would be great to file a Issue on GitHub repo (https://github.com/mlcommons/inference_results_v0.7/issues) so that others who encountered the same issue can search and get the answer.

Thanks,

Po-Han

From: Jeff Y. Liu <yajun...@gmail.com>
Sent: Wednesday, January 13, 2021 11:22 PM
To: David Kanter <da...@mlcommons.org>
Cc: public <pub...@mlcommons.org>
Subject: Re: Power System description - Invitation to edit

External email: Use caution opening links or attachments

--
You received this message because you are subscribed to the Google Groups "public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public+un...@mlcommons.org.
To view this discussion on the web visit https://groups.google.com/a/mlcommons.org/d/msgid/public/377257694.1652669.1610551374946%40mail.yahoo.com.

Vasudev, Rakshith

unread,

Jan 19, 2021, 12:38:32 PM1/19/21

to Jeff Y. Liu, David Kanter, public

Hello Jeff,

I saw this error before, this is what I tried, it worked for me:

sudo rm -f /etc/apt/sources.list.d/eoan.list

Hope this helps.

Thanks,

Rakshith

From: Jeff Y. Liu <yajun...@gmail.com>
Sent: Wednesday, January 13, 2021 9:23 AM
To: David Kanter
Cc: public
Subject: Re: Power System description - Invitation to edit

[EXTERNAL EMAIL]

Power System description

Open in Sheets

Google Sheets: Create and edit spreadsheets online.
Google LLC, 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA
You have received this email because da...@mlcommons.org shared a spreadsheet with you from Google Sheets.

--

David Kanter

Executive Director

MLCommons

--

Vasudev, Rakshith

unread,

Jan 19, 2021, 12:38:32 PM1/19/21

to Jeff Y. Liu, David Kanter, public

Jeff,

Actually, I want to take it back. That solution works when I’m not able to install a new package inside the container and not during the prebuild.

Thanks,

Rakshith

Anton Lokhmotov

unread,

Jan 20, 2021, 5:29:28 PM1/20/21

to Jeff Y. Liu, David Kanter, public

Hi Jeff,

Sorry you've been having issues with reproducing v0.7 submissions. I'd suggest to file them in the v0.7 results repo: https://github.com/mlcommons/inference_results_v0.7/issues

Please provide more details about which submission you are trying to reproduce, what's your system, etc. I'll take a look if it's something I can help you with. (I have experience reproducing both NVIDIA and Intel submissions.) Otherwise, I'll bring it to attention of the right people.

Thanks, Anton.

--

Jeff Liu

unread,

Jan 22, 2021, 8:51:01 PM1/22/21

to Anton Lokhmotov, David Kanter, public

Hi Anton,

NVIDIA close division, currently, I am trying Inferencing 0.7. I have been pulling away for other project, should be able to get back to it shortly.

Regards

Jeff

On Jan 20, 2021, at 1:30 PM, Anton Lokhmotov <an...@dividiti.com> wrote:

Anton Lokhmotov

unread,

Jan 22, 2021, 8:51:04 PM1/22/21

to Jeff Liu, David Kanter, public

Jeff,

Maybe the issue is that your Ubuntu release (Eoan, 19.10) has reached its End of Life?

https://askubuntu.com/questions/1299440/ubuntu-eoan-release-does-not-have-a-release-file