PMEM memory alignment and prefetching

[Raqib Islam]

Hi All,

Hope you have enjoyed a great weekend.

I like to know the recommendation and best practice of the following two important parameter for Optane PMEM in App Direct Mode:

1) Memory Alignment: We know for DRAM in 64-bit cpu the preferred memory alignment is the cache line size (e.g. 64-bit). We also know Optane PMEM has a 256-Byte physical media access granularity. My question is, what should be the proper memory alignment if we use Optane PMEM in App Direct Mode?

2) CPU Prefetching: What's the rule of CPU Prefetching for Optane PMEM? How does it differ from DRAM while multiple cpu core access prefetcher?

Thanks in advance.

--

Abdullah Al Raqibul Islam

[Andy Rudoff]

Hi,

On your first question, it really depends on the access pattern.  It is true the current Optane persistent memory product has an ECC block of 256 bytes.  If the device receives four sequential 64-byte stores, it will combine them into a single ECC block and you get the "sequential" performance numbers that have been reported by various publications.  On the other hand, if the device receives random 64-byte stores, its ability to combine them into full ECC block writes is impacted and you'll see the "random" performance numbers (roughly one-fourth of sequential, since only one cache line out of four is getting updated with each ECC block update).  Reads behave similarly, where four sequential cache lines give the best read bandwidth and random reads are roughly one fourth of that.  You also have to take into account the caching done by the CPU and whether writes are happening due to CPU cache evictions (which tend to look more random) or by non-temporal stores, which bypass the CPU cache.

On your second question, I'm not aware of any differences in the prefetch logic between DRAM and Optane media in the Cascade Lake CPU, but my understanding of the prefetch logic is that it is micro-architectural and not documented in any public specs, so different products are free to change how prefetches are done.

Hope that helps,

-andy

[Abdullah Al Raqibul Islam]

Hi Andy,

Thanks for your response. Combining sequential reads in ECC block seems like prefetch 4 cache line size data from the PMEM device. If that is not the case then, how the sequential read actually get the benefit? If it takes much of your time, can you please share any resource (regarding how sequential read actually perform) with me?

Thank you again for your great effort.

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/ea304ce2-ac88-4b46-9223-55e44637846e%40googlegroups.com.

[Andy Rudoff]

Hi,

I'm not sure how you're using "prefetch" in your question, whether you are talking about prefetching to the CPU caches, or what happens down on the memory module itself.  My comments about sequential accesses were about what happens on the memory module itself.  Since the ECC block size is 256 bytes on the current product, there must be a 256-byte buffer down on the device that it uses to check ECC.  When the CPU fetches the first cache line, the entire 256 bytes is read into the buffer on the device in order to check ECC, and then the single cache line requested is returned.  If the CPU then fetches the next three sequential cache lines, the device will find the contents in its buffer and return those cache lines very quickly, without any additional media reads.  On the other hand, if the CPU jumps around (random access instead of sequential), then three out of the four cache lines read into the device's internal buffer are not used and every read must wait for the media to return a full 256-byte ECC block.  None of this discussion has anything to do with CPU cache prefetches, other than the fact that those can often anticipate sequential accesses and pipeline the data transfer, further improving performance.

Hope that helps.

-andy

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+unsubscribe@googlegroups.com.

[Piotr Balcer]

Hi,

What I would add is that software should *not* assume any specific size of the underlying ECC block size, but rather employ the common principles of cache-oblivious algorithm design.

Doing manual software prefetching is difficult, and in practice unlikely to yield significant benefit - unless you tune your software for one exact hardware platform. You should design your software to take advantage of builtin hardware prefetching instead.

If you want to learn more, I encourage your to read chapters 3, for general advice about performance, and 11, for Intel Optane PMem specific advice, from Intel® 64 and IA-32 ArchitecturesOptimization Reference Manual.

https://software.intel.com/sites/default/files/managed/9e/bc/64-ia-32-architectures-optimization-manual.pdf

Piotr

To unsubscribe from this group and stop receiving emails from it, send an email to pm...@googlegroups.com.

[Abdullah Al Raqibul Islam]

Hi Andy,

In my question, I used “prefetch” in wrong way,  but you replied exactly what I was looking for. Thanks for your detailed explanation.

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/e8a0f547-5e5d-48c7-92c2-acf053e71015%40googlegroups.com.

[Abdullah Al Raqibul Islam]

Thanks Piotr. I will check the document.

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/f3a5e651-d551-4b1a-8710-ee7a3d3d770e%40googlegroups.com.

[steve]

Hi all,

I'm looking to buy a server with the following main options:

2x Xeon Gold 5215L
192GB DRAM
12x Optane DIMMs (512s preferred but 256 if that's all I can afford)
A couple of TB of SSD.

Dell seems to have reasonable pricing on everything except for the Optane DIMMs. I can buy 256 GB parts (NMA1XBD256GQS) on ebay for about $400 each as
compared with almost $6000 each(!) from Dell.

Dell says that I can put my own Optane DIMMs in but they won't guarantee that they will work because Dell buys special OEM parts that are different in
some unspecified way from the standard part with the above part number.

Does anyone have any suggestions as to how I can put together a server like this for a reasonable amount of money? In particular, are Dell's concerns
about Optane DIMM compatibility legitimate or can I assume that the ones I buy on ebay will be compatible?

Any reports of experience with another server vendor would also be welcomed, of course.
------------
Steve Heller