Optane DIMM bandwidth under multicores

[Qing Wang]

Hi all,

I read the " Intel® 64 and IA-32 Architectures Optimization Reference Manual", which says when mutliple threads write Optane PM in a  sequential manner with 256B  granularity, the bandwidth will drop.

However, I cannot reproduce it. 

I use 3 X 256GB Optane DIMM and each thread writes PM sequentially with 256B  granularity.

This is my nstore code:

===========================================

void fastMemcpy(void *pvDest, void *pvSrc, size_t nBytes) {

  //   assert(nBytes % 32 == 0);

  //   assert((intptr_t(pvDest) & 31) == 0);

  //   assert((intptr_t(pvSrc) & 31) == 0);

  const __m256i *pSrc = reinterpret_cast<const __m256i *>(pvSrc);

  __m256i *pDest = reinterpret_cast<__m256i *>(pvDest);

  int64_t nVects = nBytes / sizeof(*pSrc);

  for (; nVects > 0; nVects--, pSrc++, pDest++) {

    const __m256i loaded = _mm256_stream_load_si256(pSrc);

    _mm256_stream_si256(pDest, loaded);

  }

  _mm_sfence();

}

===============================================

When using 18 threads, the write bandwidth is 6.3GB/s, which is close to peak bandwidth.

Yet, when I remove  _mm_sfence(), the write bandwidth under 18 threads is 3.7GB/s.

It confuses me, can anyone explain it?

Thanks,

Qing Wang

[ppbb...@gmail.com]

This is exactly the phenomenon that the 12.2.3 section of the optimization manual talks about. By adding the fence, you make the traffic more "orderly", increasing the chance that stores will be write-combined into larger 256-byte blocks.

[Jan K]

> 12.2.3 section of the optimization manual

Could you please point me to the "optimization manual"? It seems I
must have overlooked it, and i failed to make up the right keywords to
find such a manual myself.

> --
> You received this message because you are subscribed to the Google Groups
> "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pmem+uns...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pmem/e9ede516-2161-420e-ac3f-fad95d1f60f4n%40googlegroups.com.
>

[ppbb...@gmail.com]

I was referencing the docs mentioned in the first question, it's the "Intel® 64 and IA-32 Architectures Optimization Reference Manual"

https://software.intel.com/content/www/us/en/develop/download/intel-64-and-ia-32-architectures-optimization-reference-manual.html