Reads/writes from/to DCPMM in Memory mode

[Anton Gavriliuk]

Hi team

As far as I understand, if request data not found in DRAM (DCPMM cache), next step is trying to find data in DCPMM.  That means in this case total time to find data will be sum of DRAM latency + DCPMM latency.  And if still have no requested data in DCPMM, data must be read from disk and will be write to DRAM or DCPMM ?   Can we read data directly to CPU L1 cache bypassing L3/L2 CPU caches ?

 

Anton

[Andy Rudoff]

Hi Anton,

As far as I understand, if request data not found in DRAM (DCPMM cache), next step is trying to find data in DCPMM.  That means in this case total time to find data will be sum of DRAM latency + DCPMM latency.

Correct.  Memory Mode uses persistent memory capacity as the system's main memory.  But DCPMM is slower than DRAM, so Memory Mode helps make up for the performance difference by using DRAM as a cache.  This is all done in hardware, transparent to the OS or applications.  When you hit in the DRAM cache, the latency is the same as DRAM.  When you miss, the latency will be the time it took to do the DRAM lookup plus the time it took to fetch the missed line from persistent memory, just as you state.

 

And if still have no requested data in DCPMM, data must be read from disk and will be write to DRAM or DCPMM ?

There's no such thing as "still have no requested data".  As I wrote above, in Memory Mode, the DCPMM capacity is the system's main memory.  So if you have 6TB of DCPMM, configured in memory mode, and 768GB of DRAM, when you boot the system, the OS will see 6TB of main memory.  It won't see the 768GB of DRAM, since that is managed transparently by hardware.  And every cache line of that 6TB has location where it lives in DCPMM, and at any time is potentially cached in DRAM.  The CPU caches behave the same as always, treating Memory Mode as if it is just a huge pool of DRAM.

Of course, the OS also supports paging, where you can make virtual memory appear to be larger than physical memory and fetch pages from storage as necessary.  But that has nothing to do with the physical address space exposed by Memory Mode above.  It is up to you to decide if you want a system to do paging to storage (with or without Memory Mode).

 

  Can we read data directly to CPU L1 cache bypassing L3/L2 CPU caches ?

No, I'm not aware of a way to do that.  If you say more about your goal here I might have a more useful answer.

Thanks,

-andy 

[Anton Gavriliuk]

Hi Andy

Latencies matter to peak performance.  That's why I'm asking for it.

So with DRAM miss we have latency = DRAM latency + DCPMM latency

But in worst case with (DRAM miss and DCPMM miss) we have latency = 

(DRAM latency + DCPMM latency) + ((DRAM latency + DCPMM latency + DISK latency) or (DRAM latency + DISK latency) or (DCPMM latency + DISK latency)) ??

Anton

пт, 5 апр. 2019 г. в 23:04, Andy Rudoff <an...@rudoff.com>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/1d29c4ce-94aa-4c67-ad23-ca6f05e23f89%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adrian Jackson]

This is not how memory systems work, on a memory mode miss you should only have the dcpmm latency, the memory controller will know if it's in cache or not, it doesn't have to search through both memory spaces.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/CAAiJnjrQQiA%2BsCozacypeuHytM_8QekE_9W%2ByAsSW%2BpRdLJkNg%40mail.gmail.com.

[Andy Rudoff]

Hi Anton,

Like I said, there's no disk involved in Memory Mode and there's no such thing as a DCPMM miss in Memory Mode.  In that mode, DCPMM is not a cache, it is the system's main memory.  If the lookup in the DRAM cache indicates a miss, then the line is fetched from DCPMM, there's no "searching" for the line in DCPMM, it is just addressed directly.

Doing a quick google search, I see an article written by Storage Review has a nice picture showing the flows for Memory Mode:

https://www.storagereview.com/intel_optane_dc_persistent_memory_module_pmm

(scroll down about halfway into the article to the section titled Optane DC Persistent Memory - Memory Mode)

Hope that helps.

-andy

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+unsubscribe@googlegroups.com.

[Anton Gavriliuk]

Hi team

> Like I said, there's no disk involved in Memory Mode and there's no such thing as a DCPMM miss in Memory Mode.

I'm not sure if you correctly understand my question.

Ok, I'm not talking anymore about DCPMM in Memory Mode only, instead, I'm talking about I/O performance with DCPMM in Memory Mode.

So if double miss happened (DRAM and DCPMM, or it's not possible ?) could you please describe or show me data flow like on the link above you provided.

Anton 

сб, 6 апр. 2019 г. в 19:07, Andy Rudoff <an...@rudoff.com>:

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/1d29c4ce-94aa-4c67-ad23-ca6f05e23f89%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "pmem" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/18110a14-f1f8-4d5b-bf5a-57c1ce8c74e3%40googlegroups.com.

[Anton Gavriliuk]

I switched DCPMMs to Memory Mode and now after boot I can check the data flow in case double miss (DRAM miss & DCPMM miss).

I used sequential read from local root disk,

linux-4185:~ # numactl --cpunodebind=1 --membind=1 fio --filename=/dev/sda --rw=read --ioengine=sync --bs=128k --iodepth=1 --numjobs=1 --runtime=60 --group_reporting --name=perf_test

perf_test: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=sync, iodepth=1

fio-3.13-27-gef32d

Starting 1 process

Jobs: 1 (f=1): [R(1)][100.0%][r=2121MiB/s][r=16.0k IOPS][eta 00m:00s]

perf_test: (groupid=0, jobs=1): err= 0: pid=3315: Tue Apr  9 18:49:58 2019

  read: IOPS=14.2k, BW=1777MiB/s (1863MB/s)(104GiB/60001msec)

    clat (usec): min=22, max=2939, avg=70.07, stdev=122.08

     lat (usec): min=22, max=2939, avg=70.10, stdev=122.08

 

So each time due to double miss it reads data from DISK -> DCPMM and then writes to DRAM.  That means total latency should be equal (DISK latency + (2 x DRAM latency + 2 x DCPMM latency)).  That's what I asked for. 

|---------------------------------------||---------------------------------------|

|--             Socket  0             --||--             Socket  1             --|

|---------------------------------------||---------------------------------------|

|--     Memory Channel Monitoring     --||--     Memory Channel Monitoring     --|

|---------------------------------------||---------------------------------------|

|-- Mem Ch  0: Reads (MB/s):     4.91 --||-- Mem Ch  0: Reads (MB/s):   754.15 --|

|--            Writes(MB/s):     6.16 --||--            Writes(MB/s):  1128.36 --|

|--      PMM Reads(MB/s)   :     0.00 --||--      PMM Reads(MB/s)   :   371.25 --|

|--      PMM Writes(MB/s)  :     0.00 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  1: Reads (MB/s):     2.57 --||-- Mem Ch  1: Reads (MB/s):   753.91 --|

|--            Writes(MB/s):     3.12 --||--            Writes(MB/s):  1127.98 --|

|--      PMM Reads(MB/s)   :     0.00 --||--      PMM Reads(MB/s)   :   371.25 --|

|--      PMM Writes(MB/s)  :     0.00 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  2: Reads (MB/s):     2.28 --||-- Mem Ch  2: Reads (MB/s):   754.09 --|

|--            Writes(MB/s):     1.89 --||--            Writes(MB/s):  1128.32 --|

|--      PMM Reads(MB/s)   :     0.00 --||--      PMM Reads(MB/s)   :   371.25 --|

|--      PMM Writes(MB/s)  :     0.00 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  3: Reads (MB/s):     1.66 --||-- Mem Ch  3: Reads (MB/s):   754.86 --|

|--            Writes(MB/s):     1.81 --||--            Writes(MB/s):  1130.06 --|

|--      PMM Reads(MB/s)   :     0.00 --||--      PMM Reads(MB/s)   :   371.25 --|

|--      PMM Writes(MB/s)  :     0.00 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  4: Reads (MB/s):     2.17 --||-- Mem Ch  4: Reads (MB/s):   755.66 --|

|--            Writes(MB/s):     2.96 --||--            Writes(MB/s):  1131.09 --|

|--      PMM Reads(MB/s)   :     0.00 --||--      PMM Reads(MB/s)   :   371.25 --|

|--      PMM Writes(MB/s)  :     0.00 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  5: Reads (MB/s):     1.75 --||-- Mem Ch  5: Reads (MB/s):   756.39 --|

|--            Writes(MB/s):     2.17 --||--            Writes(MB/s):  1132.16 --|

|--      PMM Reads(MB/s)   :     0.00 --||--      PMM Reads(MB/s)   :   371.25 --|

|--      PMM Writes(MB/s)  :     0.00 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- NODE 0 Mem Read (MB/s) :    15.34 --||-- NODE 1 Mem Read (MB/s) :  4529.07 --|

|-- NODE 0 Mem Write(MB/s) :    18.10 --||-- NODE 1 Mem Write(MB/s) :  6777.98 --|

|-- NODE 0 PMM Read (MB/s):      0.00 --||-- NODE 1 PMM Read (MB/s):   2227.51 --|

|-- NODE 0 PMM Write(MB/s):      0.00 --||-- NODE 1 PMM Write(MB/s):      0.00 --|

|-- NODE 0.0 NM read hit rate :  0.82 --||-- NODE 1.0 NM read hit rate :  0.51 --|

|-- NODE 0.1 NM read hit rate :  0.71 --||-- NODE 1.1 NM read hit rate :  0.51 --|

|-- NODE 0 Memory (MB/s):       33.45 --||-- NODE 1 Memory (MB/s):    13534.56 --|

|---------------------------------------||---------------------------------------|

Anton 

пн, 8 апр. 2019 г. в 19:22, Anton Gavriliuk <antos...@gmail.com>:

[steve]

Hi all,

If anyone has been trying to find a company to sell them a server with Optane DC Persistent Memory, this might be a good place to look:
https://www.avadirect.com/Supermicro-SuperWorkstation-7039A-i-Dual-Xeon-Scalable-3U-Rack-Tower-Workstation-PC/Configure/12703624

Note: I have no connection with this company other than as a potential customer.

They do carry the PM chips and have quoted me a good price but they are out of stock at the moment. The configuration at that link doesn't include the
PM chips and we haven't yet verified that the CPU is one that supports Optane PM, so just take that as a starting point.

The good news is that they don't have any problem dealing with small customers who might make only one purchase.

The sales guy was very nice and said he would get me a quote that would give me the lowest total cost, which is what I asked for.

Here's his contact info:
Kyle Mizer | Sales & Support
2045 Midway Drive, Twinsburg, OH 44087
Office: (866) 766-4629 | Direct: (216) 503-6362

I've been looking for a company that actually wants to do business with individuals, and this seems to be a good candidate.
------------
Steve Heller

[Anton Gavriliuk]

Good/low price doesn't mean best pmem solution.  Beginning with 2nd April I have seen many publishes about DCPMM.

The problem is there are many lies too, especially in performance area.

Don't trust anyone.  Me too.  Verify.  Always.

DCPMM is just single ingridiante.  It's mandatory, but far not all you need for best performance.

Anton

чт, 11 апр. 2019 г. в 15:31, steve <st...@steveheller.org>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/hjcuaettkvfucuv6ovg6o55d2esbfvh2ll%404ax.com.

[steve]

On Thu, 11 Apr 2019 15:45:40 +0300, Anton Gavriliuk <antos...@gmail.com> wrote:

>Good/low price doesn't mean best pmem solution. Beginning with 2nd April I
>have seen many publishes about DCPMM.
>The problem is there are many lies too, especially in performance area.
>
>Don't trust anyone. Me too. Verify. Always.
>
>DCPMM is just single ingridiante. It's mandatory, but far not all you need
>for best performance.

Sure, but it's also important to actually be able to get the hardware. No one else has been willing to sell it to me.

And I have to consider cost to be much more significant than people who aren't paying for it themselves.
------------
Steve Heller

[Otto Chow]

Hi Anton,

For the publishes, I won’t call it lies but have not published the whole story or highlight specific areas readers should pay attention to for various reasons.  One of the reasons ... this product is something very new to media as well.  I have seen post that just treat DCPMM as a very fast SSD.  May be this is easier for some readers to understand and try without application re-Engineering/optimization.

We are learning on the usage of this products and to improve this product everyday.  PMDK developers try to engage with you and listen from you here, so they can improve PMDK to meet your needs.

Agree with your comment ... verify the performance by yourself, with your workload/application.  Benchmark is benchmark, nothing more.  For the performance of DCPMM, it is highly related to the workload, dataset size, weight of read/write access, the pattern of read/write access and the operation mode using.

DCPMM is not a replacement to DDR4. It opens up new usage models for applications. The outcome may vary ... could be cost reduction, capacity increase etc..  For performance, it depends on what you are trying to compare.  Understanding the performance and limitation of DDR4, DCPMM, Optane DC SSD and NVMe NAND based SSD is important and pick the right technology or the combination of technology for the job!

Good luck to everyone who are exploring on the path of Persistent Memory!

Regards,

Otto

Disclose: I am working for Intel!

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/CAAiJnjqZ9J9oa5dGVq9ZAH2dFcivZywF97mwbR3YmPpK%3DmhyKw%40mail.gmail.com.

[steve]

Hi Otto,

I think the name is somewhat of a barrier to understanding DCPMM, because it is too easily confused with Optane DC SSDs, not to mention Optane Memory.

Maybe Intel should call it "Optane DIMMs", like everyone else does. :-)

But as a more direct response to your comments, yes, it is a new type of storage that needs new approaches.

That's why I'm working on my own key-value store that is intended to make optimal use of whatever type(s) of storage happens to be available, with the
DCPMM obviously at the top rank of storage devices.

So far I'm getting microsecond-range access to a billion records.

I'm working on ordering my own server so I can do more intensive optimizations with source code-level tools like Intel VTune.

>> ??, 11 ???. 2019 ?. ? 15:31, steve <st...@steveheller.org>:

>>
>>> Hi all,
>>>
>>> If anyone has been trying to find a company to sell them a server with
>>> Optane DC Persistent Memory, this might be a good place to look:
>>>
>>> https://www.avadirect.com/Supermicro-SuperWorkstation-7039A-i-Dual-Xeon-Scalable-3U-Rack-Tower-Workstation-PC/Configure/12703624
>>>
>>> Note: I have no connection with this company other than as a potential
>>> customer.
>>>
>>> They do carry the PM chips and have quoted me a good price but they are
>>> out of stock at the moment. The configuration at that link doesn't include
>>> the
>>> PM chips and we haven't yet verified that the CPU is one that supports
>>> Optane PM, so just take that as a starting point.
>>>
>>> The good news is that they don't have any problem dealing with small
>>> customers who might make only one purchase.
>>>
>>> The sales guy was very nice and said he would get me a quote that would
>>> give me the lowest total cost, which is what I asked for.
>>>
>>> Here's his contact info:
>>> Kyle Mizer | Sales & Support
>>> 2045 Midway Drive, Twinsburg, OH 44087

>>> <https://maps.google.com/?q=2045+Midway+Drive,+Twinsburg,+OH+44087++%0D%0AOffice:+(866&entry=gmail&source=g>
>>> Office: (866
>>> <https://maps.google.com/?q=2045+Midway+Drive,+Twinsburg,+OH+44087++%0D%0AOffice:+(866&entry=gmail&source=g>)

>>> 766-4629 | Direct: (216) 503-6362
>>>
>>> I've been looking for a company that actually wants to do business with
>>> individuals, and this seems to be a good candidate.
>>> ------------
>>> Steve Heller
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "pmem" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to pmem+uns...@googlegroups.com.
>>> To post to this group, send email to pm...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/pmem/hjcuaettkvfucuv6ovg6o55d2esbfvh2ll%404ax.com
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "pmem" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to pmem+uns...@googlegroups.com.
>> To post to this group, send email to pm...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pmem/CAAiJnjqZ9J9oa5dGVq9ZAH2dFcivZywF97mwbR3YmPpK%3DmhyKw%40mail.gmail.com

>> <https://groups.google.com/d/msgid/pmem/CAAiJnjqZ9J9oa5dGVq9ZAH2dFcivZywF97mwbR3YmPpK%3DmhyKw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .

>> For more options, visit https://groups.google.com/d/optout.
>>

------------
Steve Heller

[Anton Gavriliuk]

Hi Otto

I have started work on pmem solutions since they appeared in the market 3 years ago, such as NVDIMM-N, Scalable Persistent Memory etc.

Yes, I understand that pmem is something completely new.  But I'm talking for example about simplest workloads such as non-DAX access 4k random reads/writes.

Reading publishes with not correct results will just confuse people.

Anton

[steve]

What confuses me is why anyone would care about such results, no matter how accurately they are reported.

People who don't want or can't use sub-microsecond access times surely aren't going to pay the big premium for it compared to Optane SSD, for example.
------------
Steve Heller

[Anton Gavriliuk]

>  People who don't want or can't use sub-microsecond access times surely aren't going to pay the big premium for it compared to Optane SSD, for example.

Sub-microsecond range means DAX (load/store/flush).  Even with non-DAX access, through standart I/O pmem provides huge advantage in latency, becasue I/O will go as fast as OS I/O system calls fast, for 4k random reads/write typically <=2us.  And this is without any modification in Db/App.

But again, you are right Steve, DAX is the best one :-)

Anton

пт, 12 апр. 2019 г. в 07:59, steve <st...@steveheller.org>:

[Iakovos Kolokasis]

Hello,

   I just found this paper which provides a performance evaluation of the Intel Optane DC Persisten Memory if you want to read it.

https://arxiv.org/abs/1903.05714

----

Jack Kolokasis

--

You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/CAAiJnjobktyDeO6GRT0hp5ECVYm2jxk3fT7ONJKQLf%2BV18W_Xg%40mail.gmail.com.

[Amnon Izhar]

Hello Andy/all

I found the same article you mentioned from storagereview and had a follow on question. From the flow diagram it is implied that the tags for the cache are stored in DRAM - i.e. when the memory controller needs to fetch data from DRAM/Optane it needs first to read the tags from DRAM to determine if there is a cache hit (data in DRAM) or miss (data in Optane). If this is the case, on a cache hit (data read from DRAM), the memory controller needs 2 reads - one for the tag and one for the actual data. If this is the case, wouldn't the DRAM access latency be increased when Optane memory is used, compared to the case where only DRAM is used even on DRAM cache (near memory) hits?

The article just claims :

"In terms of latency profile, anything hitting the DRAM cache (near memory) will of course deliver <100 nanosecond latency." but hard to tell if this accounts for  2 DRAM accesses or 1? 

Thanks

   Amnon 

  

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/1d29c4ce-94aa-4c67-ad23-ca6f05e23f89%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "pmem" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/18110a14-f1f8-4d5b-bf5a-57c1ce8c74e3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Amnon Izhar

[Andy Rudoff]

Hi Amnon,

I have to start by saying it is microarchitectural detail how we handle the cache tags, so don't take my answer as meaning it will always work this way on all products.  In the current Intel® Optane™ DC persistent memory product, there is only one DRAM access on a cache read hit because the data & tags are arranged so that they can be fetched together via a single DDR transaction.

-andy

p.s.  Note to any marketing folks reading this: I hope you're impressed I remembered to put the ® and ™ thingies in the answer above!

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+unsubscribe@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/1d29c4ce-94aa-4c67-ad23-ca6f05e23f89%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "pmem" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+unsubscribe@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/18110a14-f1f8-4d5b-bf5a-57c1ce8c74e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Amnon Izhar

[steve]

On Fri, 12 Apr 2019 08:17:23 -0700 (PDT), Andy Rudoff <an...@rudoff.com> wrote:

>Hi Amnon,
>
>I have to start by saying it is microarchitectural detail how we handle the
>cache tags, so don't take my answer as meaning it will always work this way
>on all products. In the current Intel® Optane™ DC persistent memory
>product, there is only one DRAM access on a cache read hit because the data
>& tags are arranged so that they can be fetched together via a single DDR
>transaction.
>
>-andy
>
>p.s. Note to any marketing folks reading this: I hope you're impressed I
>remembered to put the ® and ™ thingies in the answer above!

I'm not a marketeer, but I'm impressed anyway!
------------
Steve Heller

[Kok-Hua Chia]

Hi Andy,

I have a few questions:

"In the current Intel® Optane™ DC persistent memory product, there is only one DRAM access on a cache read hit because the data & tags are arranged so that they can be fetched together via a single DDR transaction."

1. Above answer to Amnon's question implies that the cache tags are indeed contained in the DRAM, am I right?

2. Can you please help me understand how it is possible to fetch both data and tags together via a single DDR transaction? AFAIK, every DDR transaction fetches 64 bytes (BL=8), or one complete cache line. How do you stuff extra bits in there to include the cache tags? I can only think of one way: use x72 ECC DRAM DIMMs and re-purpose the ECC bits as cache tags. In this case, each 8-cycle burst will give you an extra 64 bits for use as cache tags. However, you lose ECC capability. Am I right?

"Memory mode uses Optane DC to expand main memory capacity without persistence. It combines a Optane DCPMM with a conventional DRAM DIMM that serves as a direct-mapped cache for the Optane DC PMM. The cacheblock size is 4 KB, and the CPU’s memory controller manages the cache transparently."

3. Above quote appears on page 5 of the NVSL paper (https://arxiv.org/pdf/1903.05714.pdf). Since the CPU's L1/L2/L3 caches all have block size of 64 bytes, I infer that on a read-miss, 4KB (one block) are fetched from Optane and placed in the DRAM cache only. In other words, it is more like a transparent memory-side cache than a traditional L4 cache that also participates in the processor's cache coherency protocols. Am I right?

4. Won’t fetching a large block of 4KB and placing in DRAM on every read-miss hog the memory channel, considering Optane reads are much slower than DRAM? And, upon eventual eviction from DRAM cache, won't it take a very long time to write back 4KB blocks to Optane, since writes are even slower than reads?

5. And if the program does not take advantage of spatial locality, won't all that pre-fetched data goes to waste?

Thanks!
KH

[Andy Rudoff]

Hi KH,

I'll answer what I can.  Some of what you ask would fall under non-public details because, as I said earlier in the thread, we're into the space of microarchitectural details here.  If you have an NDA relationship with Intel and need additional details about the CPU internals, please use that avenue to get more information.

Anyway, my best effort answers are included below.

Thanks,

-andy

On Saturday, April 13, 2019 at 6:32:08 AM UTC-6, Kok-Hua Chia wrote:

Hi Andy,

I have a few questions:

"In the current Intel® Optane™ DC persistent memory product, there is only one DRAM access on a cache read hit because the data & tags are arranged so that they can be fetched together via a single DDR transaction."

1. Above answer to Amnon's question implies that the cache tags are indeed contained in the DRAM, am I right?

2. Can you please help me understand how it is possible to fetch both data and tags together via a single DDR transaction? AFAIK, every DDR transaction fetches 64 bytes (BL=8), or one complete cache line. How do you stuff extra bits in there to include the cache tags? I can only think of one way: use x72 ECC DRAM DIMMs and re-purpose the ECC bits as cache tags. In this case, each 8-cycle burst will give you an extra 64 bits for use as cache tags. However, you lose ECC capability. Am I right?

First, I'll reiterate that a memory-side cache hit means exactly one fetch from DDR.  The details on where all the required information lives is not public, but the cache lines do still have ECC protection.  I'll point out that there's many ways to do this, analogous to how directory information is stored for each cache line on various CPU architectures, but the details of Intel's exact layout are not currently public.

 

"Memory mode uses Optane DC to expand main memory capacity without persistence. It combines a Optane DCPMM with a conventional DRAM DIMM that serves as a direct-mapped cache for the Optane DC PMM. The cacheblock size is 4 KB, and the CPU’s memory controller manages the cache transparently."

3. Above quote appears on page 5 of the NVSL paper (https://arxiv.org/pdf/1903.05714.pdf). Since the CPU's L1/L2/L3 caches all have block size of 64 bytes, I infer that on a read-miss, 4KB (one block) are fetched from Optane and placed in the DRAM cache only. In other words, it is more like a transparent memory-side cache than a traditional L4 cache that also participates in the processor's cache coherency protocols. Am I right? 

4. Won’t fetching a large block of 4KB and placing in DRAM on every read-miss hog the memory channel, considering Optane reads are much slower than DRAM? And, upon eventual eviction from DRAM cache, won't it take a very long time to write back 4KB blocks to Optane, since writes are even slower than reads?

5. And if the program does not take advantage of spatial locality, won't all that pre-fetched data goes to waste?

The sentence in the NVSL paper is incorrect.  The Memory Mode cache line size is 64 bytes.  On a miss, 64 bytes are fetched from the Optane media to satisfy the request and fill the line in the DRAM cache.  Your characterization of the cache as a "transparent memory-side cache" is exactly right.

 

Thanks!
KH

[Brian Bulkowski]

Thought I would share some basic price research, thinking of this availability question ----

The best summary I've got is from ( as usual ) anand tech, here:

https://www.anandtech.com/show/14146/intel-xeon-scalable-cascade-lake-deep-dive-now-with-optane

which shows Intel has moved high density memory ( DRAM and Optane ) into upcharges in the Xeon line.

The key sentences:

• No letter = Normal Memory Support (1.5 TB)
• M = Medium Memory Support (2.0 TB)
• L = Large Memory Support (4.5 TB)

Out of the new letter configurations, users will notice that the no-letter designation now has support for 1.5 TB of memory, double the first generation. This goes up to 2 TB for M, and 4.5 TB for L. These values include parts that are fitted with Optane, and as such an ‘L’ CPU can support 3.0 TB of Optane plus 1.5 TB of DDR4 memory, to give that 4.5 TB total.

So if you want 3TB of optane, you have to make sure you get an L part. Anand doesn't say how much a base-level CPU can handle, whether anything that CAN do optane can do 1.5T, or whether there are different values ( like 0.5T for a base part ). Still looking for that info.

Further down in the list, you see the price schedule, and L rated parts have a serious premium. The cheapest is the 5215L, which Anand is showing as a $9119 part, compared to  the base models which are in the $1500  to $2k, the M which is $4k.

This also shows that there are some Cascade Lakes without Optane DIMM support, but it's mostly the "silver" and "bronze", Gold and better should do it.
And there is some leaking of pricing --- Anand again --- I can't say whether it is correct or not ( because Intel never told me ) :

https://www.anandtech.com/show/14180/pricing-of-intels-optane-dc-persistent-memory-modules-leaks

Regarding getting your own build to build a database, good luck! When I built that which became Aerospike ( which is an open source database that supports PMEM ), I had to spend about $25k in 2008 on a server substantial enough to prove 100x faster than oracle, out of my own pocket.... I hope you wouldn't expect starting a new database to be cheap.

-brian

--

You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/CAAiJnjobktyDeO6GRT0hp5ECVYm2jxk3fT7ONJKQLf%2BV18W_Xg%40mail.gmail.com.

[steve]

On Sat, 13 Apr 2019 11:18:13 -0700, Brian Bulkowski <br...@aerospike.com> wrote:

>Thought I would share some basic price research, thinking of this
>availability question ----
>
>The best summary I've got is from ( as usual ) anand tech, here:
>
>https://www.anandtech.com/show/14146/intel-xeon-scalable-cascade-lake-deep-dive-now-with-optane
>
>which shows Intel has moved high density memory ( DRAM and Optane ) into
>upcharges in the Xeon line.
>
>The key sentences:
>

> * No letter = Normal Memory Support (1.5 TB)
> * M = Medium Memory Support (2.0 TB)
> * L = Large Memory Support (4.5 TB)

>
>Out of the new letter configurations, users will notice that the
>no-letter designation now has support for 1.5 TB of memory, double the
>first generation. This goes up to 2 TB for M, and 4.5 TB for L. These
>values include parts that are fitted with Optane, and as such an ‘L’ CPU
>can support 3.0 TB of Optane plus 1.5 TB of DDR4 memory, to give that
>4.5 TB total.
>
>So if you want 3TB of optane, you have to make sure you get an L part.
>Anand doesn't say how much a base-level CPU can handle, whether anything
>that CAN do optane can do 1.5T, or whether there are different values (
>like 0.5T for a base part ). Still looking for that info.
>
>Further down in the list, you see the price schedule, and L rated parts
>have a serious premium. The cheapest is the 5215L, which Anand is
>showing as a $9119 part, compared to  the base models which are in the
>$1500  to $2k, the M which is $4k.
>
>This also shows that there are some Cascade Lakes without Optane DIMM
>support, but it's mostly the "silver" and "bronze", Gold and better
>should do it.
>And there is some leaking of pricing --- Anand again --- I can't say
>whether it is correct or not ( because Intel never told me ) :
>
>https://www.anandtech.com/show/14180/pricing-of-intels-optane-dc-persistent-memory-modules-leaks

It is odd how hard it is to get pricing and figure out which CPUs can do what, isn't it?

>Regarding getting your own build to build a database, good luck! When I
>built that which became Aerospike ( which is an open source database
>that supports PMEM ), I had to spend about $25k in 2008 on a server
>substantial enough to prove 100x faster than oracle, out of my own
>pocket.... I hope you wouldn't expect starting a new database to be cheap.

From what I can tell, it will be a minimum of $5-6k (also out of my own pocket) to build a server that will be able to run (and help me further tune)
my microsecond-range-latency key-value store. Note that I don't have an actual quote yet, so these numbers could be way off.

However, given 10 years of advancement in technology, with the attendant decrease in prices, that doesn't seem too far out of line compared to what
you spent.
------------
Steve Heller

[Otto Chow]

Hi Steve and all,

About the features supported by the processor, you can find out more from ark.intel.com

About the pricing, the listed price is for reference only.  Getting a quotation from your preferred OEM / reseller is the most accurate rate.

To build a server with a CPU model of the lowest end CPU model with Intel Optane DC Persistent Memory supported, the model is Silver 4215.  There is only 1 model in Silver support the feature.  I don’t claim that this is the lowest price CPU model .... the end user price is determined by OEMs and there are many factors where a lowest end CPU model may not be the cheapest one.

Hope this helps.

Regards,

Otto

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/07a4be12nieq46i6k7ml8c8moht592gf7m%404ax.com.

[Otto Chow]

Hi Brian,

I will help to answer some of your questions.

For 3TB of Intel Optane DC Persistent Memory (DCPMM), do you mean 3TB per CPU or per system?  The CPU model suffix you mentioned - no letter, M and L are Memory capacity supported  on a per CPU socket basis.

Besides, it also depends on the operation mode the system is running.  For memory mode (memory side cache), the memory capacity counted is only DCPMM memory size.  For Application Direct mode, the memory capacity counted is DDR4 memory capacity + DCPMM memory capacity.

Some may ask ... what happen if I installed memory capacity larger than the CPU allowed?  Will my system boot?  The system will still boot but the memory capacity exceeded than the CPU allowed will be disabled.  There will be an alert message when the system is booted.

Due to the sensitivity of pricing, I don’t want to touch in the public forum.  Please contact your preferred OEM. Don’t assume that 128GB DCPMM DIMMs  has the same $ per GB with 256GB DCPMM DIMMs.

Regards,

Otto

[Otto Chow]

Hi Steven,

Thank you for your comment on the product branding.  It is beyond by pay grade to comment it ... :). 

The name “Optane” is about the memory media used. For the DIMM and SSD form factor, they are also using Optane memory technology.

If you are based in US, hope that you will join the upcoming PMDK Summit.  Per my memory, it includes sessions of other tools as well.

There is info on this event in another posting of this forum.

Good luck to your exploration on Persistent Memory.

Regards,

Otto

[steve]

Hi Otto,

I'm not surprised that you can't comment on the branding. :-)

As for the PMDK Summit: yes, I am in the US, but that is a long way from here and going there would consume resources that I need to buy my server to
do my own research.

Maybe once I have sold my code for a lot of $, I'll go to the next one!

------------
Steve Heller

[Kok-Hua Chia]

Hi Andy,

Thank you for your answers, Appreciate it very much! Unfortunately I am retired and do not have an NDA arrangement with Intel :(

First, I'll reiterate that a memory-side cache hit means exactly one fetch from DDR.  The details on where all the required information lives is not public, but the cache lines do still have ECC protection.  I'll point out that there's many ways to do this, analogous to how directory information is stored for each cache line on various CPU architectures, but the details of Intel's exact layout are not currently public.

I presume you mean that the ECC bits have nothing to do with the cache tags?

But, in your previous answer to Amnon, you said that "tags+data" are fetched via a single DDR transaction, meaning the tags are also stored in DRAM. Did I understand that correctly?

The Storage Review article (https://www.storagereview.com/intel_optane_dc_persistent_memory_module_pmm) suggests the same thing. There is a diagram that shows the transaction flow for a DRAM cache miss: the CPU first initiates a DRAM access, presumably to look up tags in order determine hit/miss, and if it is a miss, goes to Optane to fetch the requested data. Am I interpreting the diagram correctly?

Your characterization of the cache as a "transparent memory-side cache" is exactly right.

I am curious why Intel decided to implement it as a memory-side cache? On a DRAM cache miss, you would have to fetch the requested data from Optane and place it in DRAM immediately following. It seems the whole transaction would require 3 accesses: one to lookup the DRAM tags to determine hit/miss, one to read the requested data from Optane, and another to place the requested data in DRAM. If it was instead implemented as a traditional L4 cache (exclusive victim buffer), then only 2 accesses are needed: one to lookup DRAM tags, and another to fetch the requested data from Optane. The data fetched from Optane is not immediately placed in DRAM; only in the L1/L2 caches of the CPU. Upon subsequent eviction from L1/L2, the data gets thrown into the exclusive L3. Eventually it gets evicted from L3 as well; that's when you write the data into the L4 DRAM cache. This seems like a better way (assuming heavy data re-use), do you agree?

Thanks,

KH

[Anton Gavriliuk]

>  >The key sentences:
>
>  * No letter = Normal Memory Support (1.5 TB)
>  * M = Medium Memory Support (2.0 TB)
>  * L = Large Memory Support (4.5 TB)

Are you sure in marked yellow above ?

According to ark, https://ark.intel.com/content/www/us/en/ark/products/series/192283/2nd-generation-intel-xeon-scalable-processors.html

for example let's open Gold 6244 with "No latter"

Memory Specifications

• Max Memory Size (dependent on memory type)1 TB
  
• Memory TypesDDR4-2933
  
• Maximum Memory Speed2933 MHz
  
• Max # of Memory Channels6
  
• ECC Memory Supported ‡Yes
  
• Intel® Optane™ DC Persistent Memory SupportedYes

So not 1.5TB, instead 1TB.

I suppose this 1TB can be mixed in any supportable combination of DRAM & DCPMM modules.

Other CPUs with "No Latter" show 1TB too, I didn't find 1.5TB

Anton 

сб, 13 апр. 2019 г. в 22:34, steve <st...@steveheller.org>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/07a4be12nieq46i6k7ml8c8moht592gf7m%404ax.com.

[Brian Bulkowski]

No, I am not sure, I am quoting Anandtech which may be wrong. I look forward to better information.

The problem with the ARK statements is it's unclear whether "memory" is DRAM or Optane + DRAM.

The Anand article - which already seems faulty - seems to imply otherwise.

The ARK page says "Max Memory Size ( dependent on memory type )". Are the types DRAM and Optane and the max depends on whether it is Optane?

If I read between the lines of the ARK statement, it says literally "Intel® Optane™ DC persistent memory is a revolutionary tier of non-volatile memory that sits between memory and storage to provide large, affordable memory capacity that is comparable to DRAM performance."

That sentance is wrong on the face - it doesn't sit between anything and anything, it's right on the bus. And, it can't be memory that sits between memory and storage - that's just a logically impossible statement.

It would be nice if ARK could say that "Max Memory Size" either includes, or does not include Optane?

I look forward to less confusion ....

-brian

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/CAAiJnjqWUJoifC9fTRWeWS%3D2qGDY0hfX6CDj%3DS-Wdkw5zDXz%2BQ%40mail.gmail.com.

[Anton Gavriliuk]

> It would be nice if ARK could say that "Max Memory Size" either includes, or does not include Optane?

Agree, it's  quite unclear.  However we can investigate it by comparing with 8280L from ark

Memory Specifications

• Max Memory Size (dependent on memory type)4.5 TB
  

• Memory TypesDDR4-2933
  
• Maximum Memory Speed2933 MHz
  
• Max # of Memory Channels6
  
• ECC Memory Supported ‡Yes
  
• Intel® Optane™ DC Persistent Memory SupportedYes

We know that here 3TB DCPMM + 1.5TB DRAM.  So logically assume that 1 TB for "No Latter" CPUs mean total memory size, sum of DCPMM + DRAM.  No 1.5 TB support.

Anton

пн, 15 апр. 2019 г. в 07:16, Brian Bulkowski <br...@aerospike.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/0ed1f1e5-6ed9-027a-9b97-c391021cefa5%40aerospike.com.

[Otto Chow]

Hi Anton and Brian,

Will take this opportunity to explain. 

1. That max. Memory size is per socket basis.

2. Reason why stating “depend on memory type” ... I would say depend on memory type plus memory mode.  For system installed with DDR4 memory only, it is clearly that server memory size is DDR 4 memory size. 

3.  For system installed with DDR4 and Optane DC Persistent Memory (DCPMM), the server memory size will be depended on the memory mode the server operates on. The modes are “memory mode”, “Application Direct mode” and “Mixed Mode”.  

For “memory mode”, DDR4 will become memory side cache.  Therefore DDR4 memory size will not be counted / classified as server memory size. In this case, server memory size is DCPMM memory size.

For “Application Direct mode”, DDR4 memory will be used as server memory/DRAM.  Therefore, server memory size is the sum of DDR4 memory size and DCPMM memory size.

For “mixed mode”, it means that part of DCPMM will be partitioned into memory mode and part of DCPMM will be partitioned into Application Direct mode.  Since there is DCPMM as memory mode, 100% DDR4 memory will be turned into memory side cache.  The server memory size is the memory size of DCPMM.

4. One common configuration under Application Direct mode (per socket) - 32GB DDR4 x 6 + 128GB DCPMM x 6 = 1TB. That is  Max. memory size supported by CPU SKU “with no letter”.

Hope this removes the confusion on max. memory size supported.

Regards,

Otto

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/CAAiJnjqFtbsrp%2BPYuDJGfJ0DRC_bqJOXKnfKPD_yain8%3DfEK%3Dg%40mail.gmail.com.

[steve]

Hi Otto,

Your example in 4 seems to require 12 DIMM slots per socket.

Let's say I buy a server that has 8 DIMM slots per socket and the CPU is a Xeon Silver 4215. What would be the maximum memory configuration in
AppDirect mode?

>> - Max Memory Size (dependent on memory type)4.5 TB
>> - Memory TypesDDR4-2933
>> - Maximum Memory Speed2933 MHz
>> - Max # of Memory Channels6
>> - ECC Memory Supported ‡Yes
>> - Intel® Optane™ DC Persistent Memory SupportedYes

>>
>>
>> We know that here 3TB DCPMM + 1.5TB DRAM. So logically assume that 1 TB
>> for "No Latter" CPUs mean total memory size, sum of DCPMM + DRAM. No 1.5
>> TB support.
>>
>> Anton
>>

>> ??, 15 ???. 2019 ?. ? 07:16, Brian Bulkowski <br...@aerospike.com>:

>>> - Max Memory Size (dependent on memory type)1 TB
>>> - Memory TypesDDR4-2933
>>> - Maximum Memory Speed2933 MHz
>>> - Max # of Memory Channels6
>>> - ECC Memory Supported ‡Yes
>>> - Intel® Optane™ DC Persistent Memory SupportedYes

>>>
>>>
>>> So not 1.5TB, instead 1TB.
>>>
>>> I suppose this 1TB can be mixed in any supportable combination of DRAM &
>>> DCPMM modules.
>>>
>>> Other CPUs with "No Latter" show 1TB too, I didn't find 1.5TB
>>>
>>> Anton
>>>

>>> ??, 13 ???. 2019 ?. ? 22:34, steve <st...@steveheller.org>:

>>> <https://groups.google.com/d/msgid/pmem/CAAiJnjqWUJoifC9fTRWeWS%3D2qGDY0hfX6CDj%3DS-Wdkw5zDXz%2BQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .

>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "pmem" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to pmem+uns...@googlegroups.com.
>>> To post to this group, send email to pm...@googlegroups.com.
>>> To view this discussion on the web visit

>>> https://groups.google.com/d/msgid/pmem/0ed1f1e5-6ed9-027a-9b97-c391021cefa5%40aerospike.com
>>> <https://groups.google.com/d/msgid/pmem/0ed1f1e5-6ed9-027a-9b97-c391021cefa5%40aerospike.com?utm_medium=email&utm_source=footer>
>>> .

>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "pmem" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to pmem+uns...@googlegroups.com.
>> To post to this group, send email to pm...@googlegroups.com.
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/pmem/CAAiJnjqFtbsrp%2BPYuDJGfJ0DRC_bqJOXKnfKPD_yain8%3DfEK%3Dg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/pmem/CAAiJnjqFtbsrp%2BPYuDJGfJ0DRC_bqJOXKnfKPD_yain8%3DfEK%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .

>> For more options, visit https://groups.google.com/d/optout.
>>

------------
Steve Heller

[Otto Chow]

Hi Steve,

You are right.  Each CPU /Socket supports up to 12 memory DIMM slots.  For a 2-way server, there are 24 DIMMs slots you can use.

Certainly, you may see some server models from OEMs having only 16 DIMM slots in total. You should check how the DIMM slots are associated to the memory channels of each CPU.

Please name an OEM server model, so I can propose the configuration.

Regards,

Otto

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/pgm8bepehfhj10ujtabbhihn7ni4iuh568%404ax.com.

[steve]

Hi Otto,

Thanks for your assistance.

Here's what I'm looking at purchasing:

https://www.avadirect.com/Supermicro-SuperWorkstation-7039A-i-Dual-Xeon-Scalable-3U-Rack-Tower-Workstation-PC/Configure/12707021

Also, do you have any idea what the backorder status is at Intel for the Xeon Silver 4215?

[Otto Chow]

Hi Steve,

As you can see from the screen capture of SuperMicro 7039A-i  DIMM layout ... for all 6 memory channels of each CPU/Socket, there are two channels with two DIMM slots (i.e. blue+black, P1-DIMM-A1, P1-DIMM-A2, P1-DIMM-D1 and P1-DIMM-D2).  The others memory channels have 1 DIMM slot only.

For all Intel-validated configurations, there is one named 2-1-1.  It means on per socket basis, 6 x DDR4 memory DIMM + 2 x DCPMM memory DIMM.  Since there are many combinations with the memory population rules, Intel may not validate every single one.

Here are the memory population rules (you should be able to get it from Supermicro) for DCPMM:

1. Max. 1 DCPMM DIMM per channel

2. Mixing DCPMMs of different capacities  across or within the same sockets is NOT supported. 

3. Populate DCPMM on iMC0 before iMC1.  Please note that there are two memory controllers per CPU - iMC0 (P1-DIMM-A/B/C) and iMC1 (P1-DIMM-D/E/F).

4. Place DCPMM in slot closest to CPU unless it is the only DIMM in the DDR channel.

5. Populating iMC0 only leaving iMC1 empty is supported.

Memory Mode

- Min. 1 DRAM DIMM + 1 DCPMM per iMC

- Populate DRAM across all available DDR4 channels to maximize bandwidth

Application Direct Mode

- Min 1 DRAM DIMM per iMC (i.e. 2 DRAM per CPU)

- Min. 1 DCPMM anywhere on platform

Based on the above rules, you may consider 4 x DDR4 Memory DIMM (P1-DIMM-A1/B1/D1/E1) + 4 x DCPMM Memory DIMM (P1-DIMM-A2/C1/D2/F1).  This configuration allows a balance on memory bandwidth between DDR4 and DCPMM thru the interleaving.

Please check with Supermicro on the memory configurations / layouts they have validated.

Regards,

Otto

 

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/g9mabe9mnao0oci0ulqjtvcdvuvlrndpae%404ax.com.