Non-uniform load distribution noticed on interleaved namespace.

[Naresh Inna]

Hi all,

While running tests on the lone namespace created on an interleaved region with 6 x 128GB Optane PMEMs, I noticed that 3 out of the 6 memory channels that these PMEMs were on reported about 2X the throughput that the other 3 channels were reporting. The discrepancy was noticed with the 'pcm-memory' tool (PMM reads, PMM writes). The namespace is accessed in fsdax mode over an ext4 mountpoint. The socket 0 also has 6x16GB DRAM modues. This is seen with both read and r/w traffic. 

My question is would interleaving not ensure that load is distributed uniformly across all PMEMs of the region, or am I missing something?

Here are the details:

- Single namespace created on interleaved region 0:

# ipmctl create -goal -socket 0 PersistentMemoryType=AppDirect

# ndctl create-namespace -r region0

- Listing of the namespace and the PMEM modules:

# ndctl list -D -n namespace0.0

[

  {

    "dev":"nmem1",

    "id":"8089-a2-1838-000014c5",

    "handle":17,

    "phys_id":40,

    "security":"disabled"

  },

  {

    "dev":"nmem3",

    "id":"8089-a2-1838-000010c7",

    "handle":257,

    "phys_id":44,

    "security":"disabled"

  },

  {

    "dev":"nmem5",

    "id":"8089-a2-1838-00001330",

    "handle":289,

    "phys_id":48,

    "security":"disabled"

  },

  {

    "dev":"nmem0",

    "id":"8089-a2-1838-0000170b",

    "handle":1,

    "phys_id":38,

    "security":"disabled"

  },

  {

    "dev":"nmem2",

    "id":"8089-a2-1838-00000ce0",

    "handle":33,

    "phys_id":42,

    "security":"disabled"

  },

  {

    "dev":"nmem4",

    "id":"8089-a2-1838-00001426",

    "handle":273,

    "phys_id":46,

    "security":"disabled"

  }

]

- fsdax mounted

# mount | grep pmem

/dev/pmem0 on /mnt/pmem0 type ext4 (rw,relatime,seclabel,dax)

-Snippet of pcm-memory.x log - memory channels 3,4,5 report 2X MB/s of channels 0,1,2:

# ./pcm-memory.x -pmm 10

[..]

|---------------------------------------||---------------------------------------|

|--             Socket  0             --||--             Socket  1             --|

|---------------------------------------||---------------------------------------|

|--     Memory Channel Monitoring     --||--     Memory Channel Monitoring     --|

|---------------------------------------||---------------------------------------|

|-- Mem Ch  0: Reads (MB/s):    89.02 --||-- Mem Ch  0: Reads (MB/s):     3.26 --|

|--            Writes(MB/s):     6.04 --||--            Writes(MB/s):     1.33 --|

|--      PMM Reads(MB/s)   :   156.32 --||--      PMM Reads(MB/s)   :     0.00 --|

|--      PMM Writes(MB/s)  :   162.59 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  1: Reads (MB/s):    90.54 --||-- Mem Ch  1: Reads (MB/s):     2.97 --|

|--            Writes(MB/s):     5.68 --||--            Writes(MB/s):     1.17 --|

|--      PMM Reads(MB/s)   :   156.45 --||--      PMM Reads(MB/s)   :     0.00 --|

|--      PMM Writes(MB/s)  :   163.72 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  2: Reads (MB/s):    92.18 --||-- Mem Ch  2: Reads (MB/s):     3.04 --|

|--            Writes(MB/s):     5.88 --||--            Writes(MB/s):     1.20 --|

|--      PMM Reads(MB/s)   :   145.60 --||--      PMM Reads(MB/s)   :     0.00 --|

|--      PMM Writes(MB/s)  :   153.23 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  3: Reads (MB/s):    85.11 --||-- Mem Ch  3: Reads (MB/s):     2.62 --|

|--            Writes(MB/s):     5.01 --||--            Writes(MB/s):     1.17 --|

|--      PMM Reads(MB/s)   :   297.31 --||--      PMM Reads(MB/s)   :     0.00 --|

|--      PMM Writes(MB/s)  :   262.41 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  4: Reads (MB/s):    86.25 --||-- Mem Ch  4: Reads (MB/s):     2.58 --|

|--            Writes(MB/s):     5.15 --||--            Writes(MB/s):     1.18 --|

|--      PMM Reads(MB/s)   :   300.58 --||--      PMM Reads(MB/s)   :     0.00 --|

|--      PMM Writes(MB/s)  :   266.71 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- Mem Ch  5: Reads (MB/s):    80.45 --||-- Mem Ch  5: Reads (MB/s):     2.52 --|

|--            Writes(MB/s):     5.11 --||--            Writes(MB/s):     1.26 --|

|--      PMM Reads(MB/s)   :   299.35 --||--      PMM Reads(MB/s)   :     0.00 --|

|--      PMM Writes(MB/s)  :   264.32 --||--      PMM Writes(MB/s)  :     0.00 --|

|-- NODE 0 Mem Read (MB/s) :   523.57 --||-- NODE 1 Mem Read (MB/s) :    16.99 --|

|-- NODE 0 Mem Write(MB/s) :    32.87 --||-- NODE 1 Mem Write(MB/s) :     7.32 --|

|-- NODE 0 PMM Read (MB/s):   1355.61 --||-- NODE 1 PMM Read (MB/s):      0.00 --|

|-- NODE 0 PMM Write(MB/s):   1272.98 --||-- NODE 1 PMM Write(MB/s):      0.00 --|

|-- NODE 0.0 NM read hit rate :  1.00 --||-- NODE 1.0 NM read hit rate :  0.94 --|

|-- NODE 0.1 NM read hit rate :  1.00 --||-- NODE 1.1 NM read hit rate :  0.92 --|

|-- NODE 0 Memory (MB/s):     3185.03 --||-- NODE 1 Memory (MB/s):       24.31 --|

|---------------------------------------||---------------------------------------|

|---------------------------------------||---------------------------------------|

|--            System DRAM Read Throughput(MB/s):        540.56                --|

|--           System DRAM Write Throughput(MB/s):         40.19                --|

|--             System PMM Read Throughput(MB/s):       1355.61                --|

|--            System PMM Write Throughput(MB/s):       1272.98                --|

|--                 System Read Throughput(MB/s):       1896.17                --|

|--                System Write Throughput(MB/s):       1313.17                --|

|--               System Memory Throughput(MB/s):       3209.34                --|

|---------------------------------------||---------------------------------------|

Thanks,

Naresh.

[Anton Gavriliuk]

Hi  Naresh

>  The discrepancy was noticed with the 'pcm-memory' tool (PMM reads, PMM writes).  

Can you please tell how exactly you generate I/O pattern ?

I would check it on my PMEM box. 

Anton

ср, 2 сент. 2020 г. в 15:59, Naresh Inna <nar...@memhive.io>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/ab39f4f0-6eb3-4107-b8c7-9f0f84e8fc9dn%40googlegroups.com.

[Naresh Inna]

Hi Anton,

Thanks for taking a look at this. 

I am trying to find a pattern that causes this. The issue is seen when running PostgreSQL pgbench tests, but should appear with any other workload generator I suppose. Upon re-creating the fsdax namespace and restarting the tests, the load seemed to be evenly distributed among the channels for a period of time when initializing the database and for a while during the DB update tests, but started showing up after having run back-to-back insert-update-select (r/w) tests followed by select-only (reads) tests. The reads are originating from a dax mapped 400 GB file on a 768 GB (128GB x 6 Optane) namespace. The writes are directed to 1024x16MB dax mapped files on the same namespace.

Thanks,

Naresh.

[Anton Gavriliuk]

Hi Naresh

I have played with fio with different I/O patterns.  On my box I don't see any disbalance across the memory channels.

Please check latest pmem firmware, I have 01.02.00.5435.

I also notified that xfs is faster than ext4 in most cases.  So if you are looking highest possible performance numbers, you may try switch ext4 -> xfs.

Anton  

чт, 3 сент. 2020 г. в 19:33, Naresh Inna <nar...@memhive.io>:

[Eduardo Berrocal]

Can you paste the output of the following command?:

# ndctl list -RuN

[Naresh Inna]

Hi Eduardo,

Thanks for taking a look. I don't have the system right now. Once it is back, I will set it up and send you the o/p.

Regards,

Naresh.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/c230dbd6-086d-4aee-bbd3-d17c302b41dbn%40googlegroups.com.