Interleaving odd numbers of pmem devices?
19 views
Skip to first unread message
steve
May 25, 2020, 5:45:42 PM5/25/20
to pmem
Hi all,
I now have 3 x 256 parts in my machine, and the BIOS won't let me make one interleaved volume out of them. What are the rules for interleaving? Does
the number of parts have to be even, or a power of 2, or what?
Thanks!
------------
Steve Heller
I now have 3 x 256 parts in my machine, and the BIOS won't let me make one interleaved volume out of them. What are the rules for interleaving? Does
the number of parts have to be even, or a power of 2, or what?
Thanks!
------------
Steve Heller
Steve Scargall
Jun 3, 2020, 3:46:04 PM6/3/20
to pmem
Hi Steve,
You can see that my system supports 1-way, 2-way, 3-way, 4-way, 6-way interleaving.
eg:
AD = AppDirect
Use 'ipmctl show -topology' to see where the DDR and PMem are physically installed. You should get the slot number, eg:
We can also interpret the DimmID which has an 0xABCD format, where A, B, C, and D are defined as follows:
I suspect the problem is with the way the DIMM slots have been populated. The BIOS supports interleaving PMem modules across IMCs in a socket only if they are placed symmetrically. If we have an asymmetric population (which you'll have with just 3 x PMem), the BIOS won't interleave.
To verify what interleaving support a system has, run 'ipmctl show -system PCAT' to dump the Platform Configuration Attributes Table, eg:
# ipmctl show -system PCAT---Platform Configurations Attributes Table--- Signature: PCAT Length: 136 bytes Revision: 0x2 Checksum: 0xae OEMID: INTEL OEMTableID: S2600WF OEMRevision: 0x2 CreatorID: INTL CreatorRevision: 0x20091013 ---TableType=0x0 Length: 16 bytes TypeEquals: PlatformCapabilityInfoTable DcpmmMgmtSWConfigInputSupport: 0x1 (Yes) MemoryModeCapabilities: 0x27 (1LM, 2LM, AppDirect, SubNUMA Cluster) CurrentMemoryMode: 0x14 -Current Volatile Memory Mode: 1LM -Allowed Persistent Memory Mode: AppDirect -Allowed Volatile Memory Mode: 2LM -SubNUMA Cluster Mode Disabled PersistentMemoryRASCapability: 0x0
---TableType=0x1 Length: 16 bytes TypeEquals: MemoryInterleaveCapabilityTable MemoryMode: 0x3 (AppDirect) NumberOfInterleaveFormatsSupported: 0x1 InterleaveFormatSupported(0): 0x801f4040 -Recommended -Channel interleave size: 4KB -iMC interleave size: 4KB -Channel ways: 1-way, 2-way, 3-way, 4-way, 6-way InterleaveAlignmentSize: 0x1eEach system OEM/ODM has a DIMM Population Matrix/Guide for the specific motherboard. The matrix shows the supported slots that either a DRAM or PMem can be installed.
I forget which Supermicro workstation you have, but you can find the motherboard model number with:
# dmidecode -t baseboard# dmidecode -t baseboard...
Base Board Information Manufacturer: Supermicro Product Name: X11DPT-B-G6-NI22
...Then you can find the motherboard manual on the Supermicro website. Within the manual is a section dedicated to memory. Using the above motherboard as an example, you'll find the symmetric and asymmetric population table, ie:
AD = AppDirect
MM = MemoryMode
AD + MM = Mixed Mode
You should always use the manufacturer's documentation for your specific system. However, the Intel motherboard document has a more complete matrix, ie:
Use 'ipmctl show -topology' to see where the DDR and PMem are physically installed. You should get the slot number, eg:
# ipmctl show -topology DimmID | MemoryType | Capacity | PhysicalID| DeviceLocator================================================================================ 0x0001 | Logical Non-Volatile Device | 252.438 GiB | 0x0026 | CPU1_DIMM_A2 0x0011 | Logical Non-Volatile Device | 252.438 GiB | 0x0028 | CPU1_DIMM_B2 0x0021 | Logical Non-Volatile Device | 252.438 GiB | 0x002a | CPU1_DIMM_C2 0x0101 | Logical Non-Volatile Device | 252.438 GiB | 0x002c | CPU1_DIMM_D2 0x0111 | Logical Non-Volatile Device | 252.438 GiB | 0x002e | CPU1_DIMM_E2 0x0121 | Logical Non-Volatile Device | 252.438 GiB | 0x0030 | CPU1_DIMM_F2 0x1001 | Logical Non-Volatile Device | 252.438 GiB | 0x0032 | CPU2_DIMM_A2 0x1011 | Logical Non-Volatile Device | 252.438 GiB | 0x0034 | CPU2_DIMM_B2 0x1021 | Logical Non-Volatile Device | 252.438 GiB | 0x0036 | CPU2_DIMM_C2 0x1101 | Logical Non-Volatile Device | 252.438 GiB | 0x0038 | CPU2_DIMM_D2 0x1111 | Logical Non-Volatile Device | 252.438 GiB | 0x003a | CPU2_DIMM_E2 0x1121 | Logical Non-Volatile Device | 252.438 GiB | 0x003c | CPU2_DIMM_F2 N/A | DDR4 | 32.000 GiB | 0x0025 | CPU1_DIMM_A1 N/A | DDR4 | 32.000 GiB | 0x0027 | CPU1_DIMM_B1 N/A | DDR4 | 32.000 GiB | 0x0029 | CPU1_DIMM_C1 N/A | DDR4 | 32.000 GiB | 0x002b | CPU1_DIMM_D1 N/A | DDR4 | 32.000 GiB | 0x002d | CPU1_DIMM_E1 N/A | DDR4 | 32.000 GiB | 0x002f | CPU1_DIMM_F1 N/A | DDR4 | 32.000 GiB | 0x0031 | CPU2_DIMM_A1 N/A | DDR4 | 32.000 GiB | 0x0033 | CPU2_DIMM_B1 N/A | DDR4 | 32.000 GiB | 0x0035 | CPU2_DIMM_C1 N/A | DDR4 | 32.000 GiB | 0x0037 | CPU2_DIMM_D1 N/A | DDR4 | 32.000 GiB | 0x0039 | CPU2_DIMM_E1 N/A | DDR4 | 32.000 GiB | 0x003b | CPU2_DIMM_F1
We can also interpret the DimmID which has an 0xABCD format, where A, B, C, and D are defined as follows:
A = Socket
B = Memory Controller
C = Channel
D = Slot
There's information & diagrams in the motherboard manual, labels on the physical motherboard, and commonly a sticker on the system itself to help.
HTH,
Steve
st...@steveheller.org
Jun 3, 2020, 4:47:07 PM6/3/20
to Steve Scargall, pmem
Hi Steve,
Running "ipmctl show -system PCAT" under Linux gives me the below
results, which says that it supports 3-way interleave.
But my X11DAi-N motherboard doesn't have the DIMM slots A2, B2, and C2
that the Intel documentation says I should use.
What I have on the first CPU (the one that will have 3 PMMs), is A1, A2
B1, C1 on one side of the CPU and D1, D2, E1, F1 on the other side.
The A2 and D2 slots are the black slots that support only PMM, but I
will have only 2 DRAM DIMMs per socket after I get my second CPU, so I
won't need all of the blue slots for DRAM.
Should I put the three PMMs in A1,, B1, and C1?
Thanks!
---Platform Configurations Attributes Table---
Signature: PCAT
Length: 104 bytes
Revision: 0x2
Checksum: 0xa2
OEMID: ALASKA
OEMTableID: A M I
---TableType=0x6
Length: 32 bytes
TypeEquals: SocketSkuInfoTable
SocketID: 0x0
MappedMemorySizeLimit: 1099511627776
TotalMemorySizeMappedToSpa: 678604832768
CachingMemorySize: 0
> --
> You received this message because you are subscribed to the Google
> Groups "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pmem+uns...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pmem/52f12b48-b8b4-4c29-a597-ee8622564d08%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/pmem/52f12b48-b8b4-4c29-a597-ee8622564d08%40googlegroups.com?utm_medium=email&utm_source=footer
Running "ipmctl show -system PCAT" under Linux gives me the below
results, which says that it supports 3-way interleave.
But my X11DAi-N motherboard doesn't have the DIMM slots A2, B2, and C2
that the Intel documentation says I should use.
What I have on the first CPU (the one that will have 3 PMMs), is A1, A2
B1, C1 on one side of the CPU and D1, D2, E1, F1 on the other side.
The A2 and D2 slots are the black slots that support only PMM, but I
will have only 2 DRAM DIMMs per socket after I get my second CPU, so I
won't need all of the blue slots for DRAM.
Should I put the three PMMs in A1,, B1, and C1?
Thanks!
---Platform Configurations Attributes Table---
Signature: PCAT
Revision: 0x2
Checksum: 0xa2
OEMID: ALASKA
OEMTableID: A M I
Length: 32 bytes
TypeEquals: SocketSkuInfoTable
SocketID: 0x0
MappedMemorySizeLimit: 1099511627776
TotalMemorySizeMappedToSpa: 678604832768
CachingMemorySize: 0
> AD = AppDirect
>
> MM = MemoryMode
> AD + MM = Mixed Mode
>
> You should always use the manufacturer's documentation for your
> specific system. However, the Intel motherboard document has a more
> complete matrix, ie:
>
>
> MM = MemoryMode
> AD + MM = Mixed Mode
>
> You should always use the manufacturer's documentation for your
> specific system. However, the Intel motherboard document has a more
> complete matrix, ie:
>
> You received this message because you are subscribed to the Google
> Groups "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pmem+uns...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pmem/52f12b48-b8b4-4c29-a597-ee8622564d08%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/pmem/52f12b48-b8b4-4c29-a597-ee8622564d08%40googlegroups.com?utm_medium=email&utm_source=footer
Steve Scargall
Jun 3, 2020, 5:26:29 PM6/3/20
to pmem
3 x PMem in any approved slot will not provide interleaving (due to an asymmetric config). Using multiple capacity PMem modules also disables interleaving. You want to avoid putting two PMem on the same channel (eg: A1, A2) if at all possible for performance (I know your hands are tied here). Ideally, you want an even number of modules spread across both IMC's.
With 2 x DRAM and 2 x PMem (per socket), you could opt for a 1-1-0 config:
A1 = DRAM
B1 = PMem
D1 = DRAM
E1 = PMem
All other slots will be empty
You should be able to interleave the PMem with that config.
I know you want PMem capacity, so with 2 x DRAM and 3 x PMem you could do an asymmetric inverse 1-1-1/0 and forgo interleaving:
A1 = DRAM
B1 = PMem
C1 = PMem
D1 = DRAM
E1 = PMem
F1 = Empty
[A-F]2 = Empty
An 'inverse' config is when we have more PMem modules than DRAM per IMC/Socket. Functionally, it should work (assuming the BIOS permits it). It's not the optimal config to achieve the best performance though.
/Steve
Reply all
Reply to author
Forward
0 new messages