tiered memory, numa distance and dram performance

[Anton Gavriliuk]

Hi all

On my tiered setup

[root@memverge ~]# numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
node 0 size: 386616 MB
node 0 free: 336370 MB
node 1 cpus: 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
node 1 size: 387063 MB
node 1 free: 1090 MB
node 2 cpus:
node 2 size: 3082240 MB
node 2 free: 3082240 MB
node 3 cpus:
node 3 size: 3082240 MB
node 3 free: 2883648 MB
node distances:
node   0   1   2   3
  0:  10  21  17  28
  1:  21  10  28  17
  2:  17  28  10  28
  3:  28  17  28  10
[root@memverge ~]#

I have run the Redis server and allocated 500+ GB of memory.

[root@memverge src]# ./redis-server /home/anton/redis/redis.conf --daemonize yes
[root@memverge src]# ./redis-cli
127.0.0.1:6379> DEBUG POPULATE 5000000000 PHPREDIS_SESSION
OK
(3779.65s)
127.0.0.1:6379>

Checking which numa nodes are using by the Redis server

[root@memverge ~]# numastat -p 1349423

Per-node process memory usage (in MBs) for PID 1349423 (redis-server)
                           Node 0          Node 1          Node 2
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                         0.00            0.01            0.00
Stack                        0.00            0.02            0.00
Private                      2.16       335127.93            0.00
----------------  --------------- --------------- ---------------
Total                        2.16       335127.96            0.00

                           Node 3           Total
                  --------------- ---------------
Huge                         0.00            0.00
Heap                         0.09            0.10
Stack                        0.01            0.03
Private                 196215.52       531345.62
----------------  --------------- ---------------
Total                   196215.63       531345.75
[root@memverge ~]#

Redis allocated most of the memory in numa node1 and then closest by numa distance, numa node3 (Optane system-ram), which is slower than dram in numa node0.

So in a real CXL.mem/type3 setup, where local dram will be ddr5 and CXL memory will be ddr4/3, and the system knows that CXL memory is not just far memory, but also slower memory (ddr5 vs ddr4/3) like Optane/system-ram in my setup, what will be more preferable - numa distance or dram performance ?

Anton