Important: HPCC Benchmarks clarification

[Verónica Vergara Larrea (SCC13 Science Lead)]

Hi all,

To clarify a couple of questions we've received about HPCC and LINPACK: 

For the HPCC portion of the competition, points will be awarded across the overall benchmark, not just the LINPACK score. 

Once HPCC results are submitted Monday, they may not be re-run. The HPC Challenge benchmarks [1] must be run as-is, no modifications are allowed. If teams wish to run a modified or optimized version of HPL, they can. But it is important to note that ONLY the HPCC results will count towards your final score. If a modified version of HPL is run, those results will count towards the "Highest LINPACK" and "FLOPS per dollar LINPACK" awards, in Standard and Commodity tracks, respectively. 

As in previous years, in the Standard Track the LINPACK score submitted towards the "Highest LINPACK" award must use double precision arithmetic and adhere to the LINPACK rules [2]. 

In the Commodity Track, the LINPACK score submitted towards the "FLOPS per dollar LINPACK" may use single, double, or mixed-precision arithmetic. 

Thanks,

Verónica

[1] http://icl.cs.utk.edu/hpcc/ 

[2] http://www.netlib.org/utk/people/JackDongarra/faq-linpack.html

[hartma...@gmail.com]

Hi Veronica,

Making sure I understand this…

1, We all need to run HPCC, as downloaded from icl.cs.utk.edu/hpcc/, and turn in some results on Monday. Once the results are turned in, no changes can be made. The results here will go toward the team's overall score.

2, For those on the standard track with GPU-enabled clusters, we can also run the double-precision HPL given by NVIDIA (available from NVIDIA Developers website), and our results there will go towards the "Highest LINPACK" award, but not towards the overall score. (Is this correct?)

Thanks in advance for the clarifications!

Regards,
-Rebecca

[Verónica Vergara Larrea (SCC13 Science Lead)]

(reposting in Google group)

Hi Rebecca,

Yep, that is correct. In the Standard track, teams are allowed to use optimized HPL code that best suits their architecture as long as it uses double precision arithmetic, and it is open source (*).

(*) If the code is not open source, it can be used only if ALL teams can get access to the source for non-commercial purposes.

Hope this helps! If you have any questions, please let me know.

Best,
Verónica

--
Verónica Vergara L.

[Jeff Shafer]

Hi Veronica,

Regarding HPCC, how will the overall benchmark score be calculated?  (There are many specific test results displayed in the HPCC summary section at the end. I can pick out some of the important ones, but it would be nice to be sure if we're optimizing for the right set.)

Thanks,

Jeff

[Jeff Shafer]

To follow up on my question a bit further, these are the HPCC result metrics collected for public rankings, so I assume you're doing something similar.  But how are these inputs weighted and used to rank teams?

HPL_Tflops --- G-HPL

PTRANS_GBs --- G-PTRANS

MPIRandomAccess_GUPs --- G-RandomAccess

MPIFFT_Gflops --- G-FFT

StarSTREAM_Triad * CommWorldProcs --- EP-STREAM Sys

StarSTREAM_Triad --- EP-STREAM Triad

StarDGEMM_Gflops --- EP-DGEMM

RandomlyOrderedRingBandwidth_GBytes --- RandomRing Bandwidth

RandomlyOrderedRingLatency_usec --- RandomRing Latency

Thanks,

Jeff

[Verónica Vergara Larrea (SCC13 Science Lead)]

Hi Jeff,

The following categories will be scored for HPCC:

MPI-RandomAccess
MPI-FFT
HPL
PTRANS
EP-RandomAccess ("Star RandomAccess")
EP-FFT ("Star-FFT")
EP-DGEMM ("Star DGEMM")
EP-STREAM ("Star STREAM")

The team with the best score in each category will get maximum points for that category. All other teams will be given a score based on the percentage from the best score achieved.

Best,

Verónica

[Skyline Supercomputing]

Veronica,

Clarification: while HPCC benchmarks must be run without code modifications, are Commodity Track teams using GPUs allowed to compile HPCC using a GPU-accelerated version of BLAS? I understand that any optimized versions of HPL must be run separately. 

Nathan Moos

Skyline Supercomputing

[Skyline Supercomputing]

Additional follow-up question: for Commodity Track teams, is there a penalty of any type associated with running separate LINPACK in lower precision?

Nathan Moos

Skyline Supercomputing

[Verónica Vergara Larrea (SCC13 Science Lead)]

Hi Nathan,

You are free to use any BLAS library to build HPCC. 

If the library you are looking at requires you to modify the code to work, then you will not be able to use it for the HPCC submission. You could, however, use it for your LINPACK submission that will count towards the "FLOPS per dollar LINPACK" award.

Best,

Verónica

[Verónica Vergara Larrea (SCC13 Science Lead)]

Hi Nathan,

In Commodity Track, both FLOPS and precision will count towards the score, though FLOPS will carry more weight. The score breakdowns will be given at the competition.

Best,

Verónica