Bandwidth Difference Optane 128 GB vs. 256 GB DIMMs
49 views
Skip to first unread message
Lawrence Benson
Apr 30, 2021, 4:56:13 AM4/30/21
to pmem
Hey,
while running some measurements on PMem, I came across some interesting performance numbers for sequential writes. The workload represents a "logging" workload, in which each thread sequentially writes to its own log file. We are using 8 threads, each writing ~4 GiB (30 GiB in total) in 1024 Byte chunks. Each chunk consists of 16x 64 Byte ntstores (using _mm512_stream_si512). After each chunk, we perform an sfence (_mm_sfence) to ensure the completion of the chunk. So far, this is pretty standard as done in various benchmarking papers. We run all threads on the same socket using numactl.
However, we get (significantly) different results on two different PMem servers (see server configuration below). On our server with 128 GB DIMMs, we get ~11.7 GB/s while we get ~15.7 GB/s on our server with 256 GB DIMMs. As most work I am aware of reports numbers around 12 GB/s, the results on our 256 GB server confuse me slightly. I've verified the numbers our application reports with VTune, as shown in the images below.
128 GB DIMM Results:
256 GB DIMM Results:
We have one server with 128 GB DIMMs (fully stocked, i.e., 2 sockets x 6 DIMMs), and an Intel Xeon Gold 5220S with 18 Cores @ 2.7 GHz. The second server has fully stocked 256 GB DIMMs and an Intel Xeon Gold 6240L with 18 Cores @ 2.6 GHz. Otherwise, both servers are running Ubuntu 20.04 with a 5.4 Kernel and memory speed is set to 2666 MT/s.
The main difference between the servers is the CPU and the DIMMs, which makes it hard to exactly pinpoint the cause for the performance difference. One major difference is that the 128 GB server has 4 NUMA nodes, i.e., two smaller ones per CPU while the 256 GB server has only 2 NUMA nodes, i.e., one per CPU. Also, as we are using AVX-512 instructions for the ntstores, the 256 GB server has 2 AVX-512 FMA units, while the 128 GB server only has one according to wikichip.org.
So my main question is: Why are we seeing this major performance difference? Is it because the DIMMs are actually better, is it because the CPU is better, or is it a combination of both? I'd appreciate any help in investigating this or maybe some links to official performance specification from Intel (which I have not really found yet). I'm mainly comparing to results published in research papers.
As a side note: We are also seeing performance differences for other workloads (e.g., random reads), but the logging one was the first that we started investigating in detail.
I'm happy to provide any additional information if this helps.
Best,
Lawrence
steve
Apr 30, 2021, 7:05:16 AM4/30/21
to Lawrence Benson, pmem
Hi Lawrence,
See this document: https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/optane-persistent-memory-200-series-brief.pdf, which
indicatesthat the write bandwidth for the 256 GB parts is much higher than that for the 128 GB or 512 GB parts.
Exactly WHY this happens is a question for Intel, of course.
On Fri, 30 Apr 2021 01:56:13 -0700 (PDT), Lawrence Benson <lawrence...@gmail.com> wrote:
>Hey,
>
>while running some measurements on PMem, I came across some interesting
>performance numbers for sequential writes. The workload represents a
>"logging" workload, in which each thread sequentially writes to its own log
>file. We are using 8 threads, each writing ~4 GiB (30 GiB in total) in 1024
>Byte chunks. Each chunk consists of 16x 64 Byte ntstores (using
>_mm512_stream_si512). After each chunk, we perform an sfence (_mm_sfence)
>to ensure the completion of the chunk. So far, this is pretty standard as
>done in various benchmarking papers. We run all threads on the same socket
>using numactl.
>
>However, we get (significantly) different results on two different PMem
>servers (see server configuration below). On our server with 128 GB DIMMs,
>we get ~11.7 GB/s while we get ~15.7 GB/s on our server with 256 GB DIMMs.
>As most work I am aware of reports numbers around 12 GB/s, the results on
>our 256 GB server confuse me slightly. I've verified the numbers our
>application reports with VTune, as shown in the images below.
>
>*128 GB DIMM Results:*
>[image: 128-dimm.png]
>
>*256 GB DIMM Results:*
>server only has one <https://en.wikichip.org/wiki/intel/xeon_gold/5220s>
>according to wikichip.org.
>
>So my main question is: *Why are we seeing this major performance
>difference?* Is it because the DIMMs are actually better, is it because
Steve Heller
See this document: https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/optane-persistent-memory-200-series-brief.pdf, which
indicatesthat the write bandwidth for the 256 GB parts is much higher than that for the 128 GB or 512 GB parts.
Exactly WHY this happens is a question for Intel, of course.
On Fri, 30 Apr 2021 01:56:13 -0700 (PDT), Lawrence Benson <lawrence...@gmail.com> wrote:
>Hey,
>
>while running some measurements on PMem, I came across some interesting
>performance numbers for sequential writes. The workload represents a
>"logging" workload, in which each thread sequentially writes to its own log
>file. We are using 8 threads, each writing ~4 GiB (30 GiB in total) in 1024
>Byte chunks. Each chunk consists of 16x 64 Byte ntstores (using
>_mm512_stream_si512). After each chunk, we perform an sfence (_mm_sfence)
>to ensure the completion of the chunk. So far, this is pretty standard as
>done in various benchmarking papers. We run all threads on the same socket
>using numactl.
>
>However, we get (significantly) different results on two different PMem
>servers (see server configuration below). On our server with 128 GB DIMMs,
>we get ~11.7 GB/s while we get ~15.7 GB/s on our server with 256 GB DIMMs.
>As most work I am aware of reports numbers around 12 GB/s, the results on
>our 256 GB server confuse me slightly. I've verified the numbers our
>application reports with VTune, as shown in the images below.
>
>[image: 128-dimm.png]
>
>*256 GB DIMM Results:*
>[image: 256-dimm.png]
>
>We have one server with 128 GB DIMMs (fully stocked, i.e., 2 sockets x 6
>DIMMs), and an Intel Xeon Gold 5220S with 18 Cores @ 2.7 GHz. The second
>server has fully stocked 256 GB DIMMs and an Intel Xeon Gold 6240L with 18
>Cores @ 2.6 GHz. Otherwise, both servers are running Ubuntu 20.04 with a
>5.4 Kernel and memory speed is set to 2666 MT/s.
>
>The main difference between the servers is the CPU and the DIMMs, which
>makes it hard to exactly pinpoint the cause for the performance difference.
>One major difference is that the 128 GB server has 4 NUMA nodes, i.e., two
>smaller ones per CPU while the 256 GB server has only 2 NUMA nodes, i.e.,
>one per CPU. Also, as we are using AVX-512 instructions for the ntstores,
>the 256 GB server has 2 AVX-512 FMA units
><https://en.wikichip.org/wiki/intel/xeon_gold/6240l>, while the 128 GB
>
>We have one server with 128 GB DIMMs (fully stocked, i.e., 2 sockets x 6
>DIMMs), and an Intel Xeon Gold 5220S with 18 Cores @ 2.7 GHz. The second
>server has fully stocked 256 GB DIMMs and an Intel Xeon Gold 6240L with 18
>Cores @ 2.6 GHz. Otherwise, both servers are running Ubuntu 20.04 with a
>5.4 Kernel and memory speed is set to 2666 MT/s.
>
>The main difference between the servers is the CPU and the DIMMs, which
>makes it hard to exactly pinpoint the cause for the performance difference.
>One major difference is that the 128 GB server has 4 NUMA nodes, i.e., two
>smaller ones per CPU while the 256 GB server has only 2 NUMA nodes, i.e.,
>one per CPU. Also, as we are using AVX-512 instructions for the ntstores,
>the 256 GB server has 2 AVX-512 FMA units
>server only has one <https://en.wikichip.org/wiki/intel/xeon_gold/5220s>
>according to wikichip.org.
>
>So my main question is: *Why are we seeing this major performance
>difference?* Is it because the DIMMs are actually better, is it because
>the CPU is better, or is it a combination of both? I'd appreciate any help
>in investigating this or maybe some links to official performance
>specification from Intel (which I have not really found yet). I'm mainly
>comparing to results published in research papers.
>
>As a side note: We are also seeing performance differences for other
>workloads (e.g., random reads), but the logging one was the first that we
>started investigating in detail.
>
>I'm happy to provide any additional information if this helps.
>
>Best,
>Lawrence
------------
>in investigating this or maybe some links to official performance
>specification from Intel (which I have not really found yet). I'm mainly
>comparing to results published in research papers.
>
>As a side note: We are also seeing performance differences for other
>workloads (e.g., random reads), but the logging one was the first that we
>started investigating in detail.
>
>I'm happy to provide any additional information if this helps.
>
>Best,
>Lawrence
Steve Heller
steve
Apr 30, 2021, 7:12:57 AM4/30/21
to st...@steveheller.org, Lawrence Benson, pmem
Hi Lawrence,
After rereading your message, based on your CPU IDs, I believe you are using series 100 pmem, whereas the document I cited is for series 200 pmem.
However, the difference in performance between 128 GB and 256 GB in series 200 is very similar to what you are seeing, so that may still be relevant.
------------
Steve Heller
After rereading your message, based on your CPU IDs, I believe you are using series 100 pmem, whereas the document I cited is for series 200 pmem.
However, the difference in performance between 128 GB and 256 GB in series 200 is very similar to what you are seeing, so that may still be relevant.
Steve Heller
ppbb...@gmail.com
Apr 30, 2021, 7:38:30 AM4/30/21
to pmem
Hi,
Here's the same document for the PMem 100 series:
And, as you can see from the datasheet, this difference in write bandwidth between individual SKUs is the likely reason for the behavior you are observing. Just as Steve suggested earlier.
Piotr
Lawrence Benson
Apr 30, 2021, 7:53:40 AM4/30/21
to pmem
Hey Steve and Piotr,
thanks for your replies. I was looking for this type of document but sometimes admittedly struggle with finding the correct Intel documentation.
The numbers in the documents are very interesting. I really wonder what causes these strong differences. But at leat this seems to be the explanation I was looking for.
Thanks,
Lawrence
Reply all
Reply to author
Forward
0 new messages