1) the os is running on a bare-metal host.
2) it's an Intel Optane Persistent Memory module.
3) cpu SKU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
Stepping: 7
CPU MHz: 2800.036
CPU max MHz: 3900.0000
CPU min MHz: 1000.0000
BogoMIPS: 4600.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 22528K
NUMA node0 CPU(s): 0-15,32-47
NUMA node1 CPU(s): 16-31,48-63
4) server manufacturer:
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 3.0 present.
Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: Huawei
Product Name: BC11SPSCB0
Version: V100R005
Serial Number: 024AFQCNL9001636
Asset Tag: Huawei
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: Type2 - Board Chassis Location
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0
I try to run fio with PMEM_IS_PMEM_FORCE=1 and finally it work. but it seem had worse performance than nvme-ssd with the same fio arg, the workload is as below
pmem workload
root:/mnt/ssd01 # numactl -N 1 /usr/local/bin/fio libpmem.fio
libpmem-seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
libpmem-seqread: (g=1): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
fio-3.29-5-ga3e3
Starting 2 threads
libpmem-seqwrite: Prepopulating IO file (/mnt/pmem7/libpmem-seqwrite.0.0)
libpmem-seqread: Prepopulating IO file (/mnt/pmem7/libpmem-seqread.0.0)
Jobs: 1 (f=1): [_(1),R(1)][100.0%][r=2360MiB/s][r=604k IOPS][eta 00m:00s]
libpmem-seqwrite: (groupid=0, jobs=1): err= 0: pid=53548: Sun Jan 2 15:28:39 2022
write: IOPS=226k, BW=884MiB/s (927MB/s)(86.3GiB/100001msec); 0 zone resets
bw ( KiB/s): min=548768, max=916592, per=100.00%, avg=905724.60, stdev=45560.96, samples=199
iops : min=137192, max=229148, avg=226431.13, stdev=11390.24, samples=199
cpu : usr=99.32%, sys=0.64%, ctx=313, majf=0, minf=524294
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,22625099,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
libpmem-seqread: (groupid=1, jobs=1): err= 0: pid=53813: Sun Jan 2 15:28:39 2022
read: IOPS=601k, BW=2346MiB/s (2460MB/s)(229GiB/100000msec)
bw ( MiB/s): min= 1226, max= 2363, per=100.00%, avg=2346.55, stdev=107.36, samples=199
iops : min=314014, max=605076, avg=600716.55, stdev=27484.97, samples=199
cpu : usr=99.67%, sys=0.28%, ctx=309, majf=0, minf=262144
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=60069211,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=884MiB/s (927MB/s), 884MiB/s-884MiB/s (927MB/s-927MB/s), io=86.3GiB (92.7GB), run=100001-100001msec
Run status group 1 (all jobs):
READ: bw=2346MiB/s (2460MB/s), 2346MiB/s-2346MiB/s (2460MB/s-2460MB/s), io=229GiB (246GB), run=100000-100000msec
nvme-ssd workload
root:/mnt/ssd01 # numactl -N 1 /usr/local/bin/fio ssd.fio
libpmem-seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
libpmem-seqread: (g=1): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
fio-3.29-5-ga3e3
Starting 2 threads
libpmem-seqwrite: Prepopulating IO file (/mnt/ssd01/libpmem-seqwrite.0.0)
libpmem-seqread: Prepopulating IO file (/mnt/ssd01/libpmem-seqread.0.0)
Jobs: 1 (f=1): [_(1),R(1)][100.0%][r=5316MiB/s][r=1361k IOPS][eta 00m:00s]
libpmem-seqwrite: (groupid=0, jobs=1): err= 0: pid=55483: Sun Jan 2 15:37:47 2022
write: IOPS=339k, BW=1322MiB/s (1387MB/s)(129GiB/100000msec); 0 zone resets
bw ( MiB/s): min= 293, max= 1437, per=100.00%, avg=1322.70, stdev=269.29, samples=199
iops : min=75010, max=368066, avg=338612.28, stdev=68938.82, samples=199
cpu : usr=94.49%, sys=5.46%, ctx=326, majf=0, minf=1961220
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,33851221,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
libpmem-seqread: (groupid=1, jobs=1): err= 0: pid=55939: Sun Jan 2 15:37:47 2022
read: IOPS=1360k, BW=5311MiB/s (5569MB/s)(519GiB/100000msec)
bw ( MiB/s): min= 4268, max= 5321, per=100.00%, avg=5311.00, stdev=74.28, samples=199
iops : min=1092846, max=1362318, avg=1359616.31, stdev=19016.43, samples=199
cpu : usr=99.91%, sys=0.04%, ctx=311, majf=0, minf=16384
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=135952782,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1322MiB/s (1387MB/s), 1322MiB/s-1322MiB/s (1387MB/s-1387MB/s), io=129GiB (139GB), run=100000-100000msec
Run status group 1 (all jobs):
READ: bw=5311MiB/s (5569MB/s), 5311MiB/s-5311MiB/s (5569MB/s-5569MB/s), io=519GiB (557GB), run=100000-100000msec
The nvme device info
root:/mnt/ssd01 # smartctl -x /dev/nvme0n1
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-8-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke,
www.smartmontools.org=== START OF INFORMATION SECTION ===
Model Number: INTEL SSDPED1K015TA
Serial Number: PHKS913400GB1P5CGN
Firmware Version: E2010435
PCI Vendor/Subsystem ID: 0x8086
IEEE OUI Identifier: 0x5cd2e4
Controller ID: 0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,500,301,910,016 [1.50 TB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Sun Jan 2 15:39:45 2022 HKT
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0007): Security Format Frmw_DL
Optional NVM Commands (0x0006): Wr_Unc DS_Mngmt
Maximum Data Transfer Size: 32 Pages
I will try to update my kernel after NY vacation and see how it work.
That's a very old Kernel. If you have the opportunity to [temporarily] test using a newer Debian release that has a newer Kernel version, it would be helpful as we can't debug Kernel issues in this community. Debian Stretch is EOL with the LTS EOL date expiring in 6months (https://wiki.debian.org/DebianReleases).