Unable to get Optane memory mode on only one socket working

Keshav Prasad

unread,

Aug 21, 2020, 3:29:35 AM8/21/20

to pmem

Hi all,

I am having some trouble setting up memory mode for Optane attached to one of the processors. The details are as below:

Hardware config:
CPU Info
Model: Genuine Intel(R) CPU 0000%@
Cores: 24
Sockets: 2
Numa Nodes: 2
-------------
Memory Info
CPU0
0x0026; Logical Non-Volatile Device; 126.375 GiB
0x0028; Logical Non-Volatile Device; 126.375 GiB
0x002a; Logical Non-Volatile Device; 126.375 GiB
0x002c; Logical Non-Volatile Device; 126.375 GiB
0x002e; Logical Non-Volatile Device; 126.375 GiB
0x0030; Logical Non-Volatile Device; 126.375 GiB
0x0025; DDR4; 16.000 GiB
0x0027; DDR4; 16.000 GiB
0x0029; DDR4; 16.000 GiB
0x002b; DDR4; 16.000 GiB
0x002d; DDR4; 16.000 GiB
0x002f; DDR4; 16.000 GiB
CPU1
0x0032; Logical Non-Volatile Device; 126.375 GiB
0x0034; Logical Non-Volatile Device; 126.375 GiB
0x0036; Logical Non-Volatile Device; 126.375 GiB
0x0038; Logical Non-Volatile Device; 126.375 GiB
0x003a; Logical Non-Volatile Device; 126.375 GiB
0x003c; Logical Non-Volatile Device; 126.375 GiB
0x0031; DDR4; 16.000 GiB
0x0033; DDR4; 16.000 GiB
0x0035; DDR4; 16.000 GiB
0x0037; DDR4; 16.000 GiB
0x0039; DDR4; 16.000 GiB
0x003b; DDR4; 16.000 GiB
-------------

Let's start with the following state- Both the sockets' Optane in AppDirect mode:

[sdp@sdp ~]$ sudo ipmctl show -region
SocketID | ISetID | PersistentMemoryType | Capacity | FreeCapacity | HealthState
=================================================================================================
0x0000 | 0x72187f481e022ccc | AppDirect | 756.000 GiB | 756.000 GiB | Healthy
0x0001 | 0x3b167f48680d2ccc | AppDirect | 756.000 GiB | 756.000 GiB | Healthy

Now, I issue the following command to setup one of the socket's Optane in Memory mode:

[sdp@sdp ~]$ sudo ipmctl create -goal -socket 0 MemoryMode=100
WARNING! The requested memory mode size for 2LM goal is below the recommended NM:FM limit of 1:4
The following configuration will be applied:
SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
===================================================================
0x0000 | 0x0001 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0011 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0021 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0101 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0111 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0121 | 126.000 GiB | 0.000 GiB | 0.000 GiB
Do you want to continue? [y/n] y
Created following region configuration goal
SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
===================================================================
0x0000 | 0x0001 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0011 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0021 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0101 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0111 | 126.000 GiB | 0.000 GiB | 0.000 GiB
0x0000 | 0x0121 | 126.000 GiB | 0.000 GiB | 0.000 GiB
A reboot is required to process new memory allocation goals.
[sdp@sdp ~]$ sudo reboot
Connection to 192.168.x.x closed by remote host.
Connection to 192.168.x.x closed.

After reboot:

[sdp@sdp ~]$ sudo ipmctl show -region
SocketID | ISetID | PersistentMemoryType | Capacity | FreeCapacity | HealthState
=================================================================================================
0x0001 | 0x3b167f48680d2ccc | AppDirect | 756.000 GiB | 756.000 GiB | Healthy

We can observe that it shows only the AppDirect region here and when I check the memory available in the system using lsmem:

[sdp@sdp ~]$ sudo lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff 2G online no 0-1
0x0000000100000000-0x000000177fffffff 90G online yes 4-93
0x0000001780000000-0x000000183fffffff 3G online no 94-96
0x0000001840000000-0x0000002ebfffffff 90G online yes 97-186
0x0000002ec0000000-0x0000002effffffff 1G online no 187
0x0000002f00000000-0x0000002f7fffffff 2G online yes 188-189
0x0000002f80000000-0x000000303fffffff 3G online no 190-192
Memory block size: 1G
Total online memory: 191G
Total offline memory: 0B

It shows only the total of DRAM available in the system, which is 191G. I was expecting the memory mode Optane should be showing up as part of system memory. What am I doing wrong here?

Anton Gavriliuk

unread,

Aug 21, 2020, 7:19:28 AM8/21/20

to Keshav Prasad, pmem

Hi all

> What am I doing wrong here?

What is your Persistent Memory firmware version, what is your OS ?

On my test box running on latest OpenSuse 15.2 I have 6 TB PM in Memory mode and 384 GB DRAM,

linux-tg7k:~ # lsmem

RANGE SIZE STATE REMOVABLE BLOCK

0x0000000000000000-0x000000003fffffff 1G online no 0
0x0000000040000000-0x000000007fffffff 1G online yes 1
0x0000000080000000-0x00000000bfffffff 1G online no 2
0x0000000100000000-0x000000013fffffff 1G online no 4
0x0000000140000000-0x00000018bfffffff 94G online yes 5-98
0x00000018c0000000-0x00000018ffffffff 1G online no 99
0x0000001900000000-0x0000016f3fffffff 1.3T online yes 100-1468
0x0000016f40000000-0x0000016fbfffffff 2G online no 1469-1470
0x0000016fc0000000-0x000002e53fffffff 1.5T online yes 1471-2964
0x000002e540000000-0x000002f17fffffff 49G online no 2965-3013
0x000002f180000000-0x000005d63fffffff 2.9T online yes 3014-5976
0x000005d640000000-0x000005e23fffffff 48G online no 5977-6024

Memory block size: 1G
Total online memory: 5.9T
Total offline memory: 0B
linux-tg7k:~ #

Anton

пт, 21 авг. 2020 г. в 10:29, Keshav Prasad <keshav...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/2767f5be-5c96-4009-bdb0-3aa2ef231518o%40googlegroups.com.

Keshav Prasad

unread,

Aug 21, 2020, 7:42:13 AM8/21/20

to pmem

Each of the Optane modules have the following Firmware:

<FWVersion>01.02.00.5375</FWVersion>

OS info:

[sdp@sdp ~]$ uname -a
Linux sdp 5.5.8-200.fc31.x86_64 #1 SMP Thu Mar 5 21:28:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

If I omit the restriction "-socket 0", ie., making the entire Optane available from both sockets for memory mode, it works fine. It shows up 1.5TB of Optane as expected.

Thanks,

-Keshav

On Friday, August 21, 2020 at 4:49:28 PM UTC+5:30, Anton Gavriliuk wrote:

Hi all

> What am I doing wrong here?

What is your Persistent Memory firmware version, what is your OS ?

On my test box running on latest OpenSuse 15.2 I have 6 TB PM in Memory mode and 384 GB DRAM,

linux-tg7k:~ # lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000003fffffff 1G online no 0
0x0000000040000000-0x000000007fffffff 1G online yes 1
0x0000000080000000-0x00000000bfffffff 1G online no 2
0x0000000100000000-0x000000013fffffff 1G online no 4
0x0000000140000000-0x00000018bfffffff 94G online yes 5-98
0x00000018c0000000-0x00000018ffffffff 1G online no 99
0x0000001900000000-0x0000016f3fffffff 1.3T online yes 100-1468
0x0000016f40000000-0x0000016fbfffffff 2G online no 1469-1470
0x0000016fc0000000-0x000002e53fffffff 1.5T online yes 1471-2964
0x000002e540000000-0x000002f17fffffff 49G online no 2965-3013
0x000002f180000000-0x000005d63fffffff 2.9T online yes 3014-5976
0x000005d640000000-0x000005e23fffffff 48G online no 5977-6024

Memory block size: 1G
Total online memory: 5.9T
Total offline memory: 0B
linux-tg7k:~ #

Anton

пт, 21 авг. 2020 г. в 10:29, Keshav Prasad <kesha...@gmail.com>:

To unsubscribe from this group and stop receiving emails from it, send an email to pm...@googlegroups.com.

Steve Scargall

unread,

Aug 21, 2020, 2:02:49 PM8/21/20

to pmem

It shows only the total of DRAM available in the system, which is 191G. I was expecting the memory mode Optane should be showing up as part of system memory. What am I doing wrong here?

Asymmetric configurations across the CPU sockets are not supported. During POST, the BIOS is likely throwing a memory training error similar to the following:

As you found, to create a mixed-mode environment (part Memory Mode, part AppDirect), the same config needs to be applied across all sockets. The following example uses 50% of the PMem capacity + 100% of DRAM capacity for Memory Mode, and 50% for AppDirect:

$ sudo ipmctl create -goal MemoryMode=50 PersistentMemoryType=AppDirect

Note: You can't dictate the % of DRAM used. If we use >=1% for Memory Mode, 100% of DRAM is used. However, there is a requirement that the assigned capacity of PMem must be equal to or greater than the DRAM capacity. ie: you can't have 384GB of DRAM and 192GB of PMem for Memory Mode due to the Direct Map Cache. We need at least 1:1 of DRAM:PMEM or higher (1:4, 1:8, 1:16, etc).

Keshav Prasad

unread,

Aug 22, 2020, 6:50:47 AM8/22/20

to pmem

Thank you very much for the explanation Steve!

Reply all

Reply to author

Forward