file_name does not point to persistent memory when use fio to test pmem

248 views
Skip to first unread message

lin tanghui

unread,
Dec 29, 2021, 9:48:45 AM12/29/21
to pmem
i use fio to test pmem , but fio return err:  file_name does not point to persistent memory
root:~ # ndctl list -N
[
  {
    "dev":"namespace7.0",
    "mode":"fsdax",
    "map":"dev",
    "size":266352984064,
    "uuid":"69450bc4-8239-486b-a3d0-3018f765b560",
    "sector_size":512,
    "align":2097152,
    "blockdev":"pmem7"
  }
]
root:~ # lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda       8:0    0 222.6G  0 disk
├─sda1    8:1    0  46.6G  0 part /
├─sda2    8:2    0  11.2G  0 part [SWAP]
└─sda3    8:3    0 164.9G  0 part /data
sdb       8:16   0 893.1G  0 disk /mnt/ssd00
sdc       8:32   0  21.9T  0 disk /mnt/storage00
nvme0n1 259:0    0   1.4T  0 disk /mnt/ssd01
pmem7   259:1    0 248.1G  0 disk /mnt/pmem7
root:~ # mount -l |grep dax
/dev/pmem7 on /mnt/pmem7 type xfs (rw,relatime,attr2,dax,inode64,noquota)
root:~ # cd /mnt/ssd01
root:/mnt/ssd01 # numactl -N 1 /usr/local/bin/fio libpmem.fio
libpmem-seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
libpmem-seqread: (g=1): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
fio-3.29-5-ga3e3
Starting 2 threads
libpmem-seqwrite: Prepopulating IO file (/mnt/pmem7/libpmem-seqwrite.0.0)
libpmem-seqread: Prepopulating IO file (/mnt/pmem7/libpmem-seqread.0.0)
fio: destination does not support O_DIRECT
fio: pid=7841, err=22/file:engines/libpmem.c:114, func=file_name does not point to persistent memory, error=Invalid argument
fio: destination does not support O_DIRECT
fio: pid=7842, err=22/file:engines/libpmem.c:114, func=file_name does not point to persistent memory, error=Invalid argument
Run status group 0 (all jobs):

Run status group 1 (all jobs):
root:/mnt/ssd01 # cat libpmem.fio
[global]
bs=4k
size=10g
ioengine=libpmem
norandommap
time_based
group_reporting
invalidate=1
disable_lat=1
disable_slat=1
disable_clat=1
clat_percentiles=0

iodepth=1
iodepth_batch=1
thread
numjobs=1
runtime=300

#
# depends on direct option, flags are set for pmem_memcpy() call:
# direct=1 - PMEM_F_MEM_NONTEMPORAL,
# direct=0 - PMEM_F_MEM_TEMPORAL.
#
direct=1

#
# sync=1 means that pmem_drain() is executed for each write operation.
#
sync=1

#
# In case of 'scramble_buffers=1', the source buffer
# is rewritten with a random value every write operation.
#
# But when 'scramble_buffers=0' is set, the source buffer isn't
# rewritten. So it will be likely that the source buffer is in CPU
# cache and it seems to be high write performance.
#
scramble_buffers=1

#
# Setting for fio process's CPU Node and Memory Node.
# Set proper node below or use `numactl` command along with FIO.
#
numa_cpu_nodes=0
numa_mem_policy=bind:0

#
# split means that each job will get a unique CPU from the CPU set
#
cpus_allowed_policy=split

#
# The libpmem engine does IO to files in a DAX-mounted filesystem.
# The filesystem should be created on a Non-Volatile DIMM (e.g /dev/pmem0)
# and then mounted with the '-o dax' option.  Note that the engine
# accesses the underlying NVDIMM directly, bypassing the kernel block
# layer, so the usual filesystem/disk performance monitoring tools such
# as iostat will not provide useful data.
#
#filename=/mnt/pmem/somefile
directory=/mnt/pmem7

[libpmem-seqwrite]
rw=write
stonewall

[libpmem-seqread]
rw=read
stonewall




Łukasz S

unread,
Dec 30, 2021, 11:04:18 AM12/30/21
to pmem
Hi,

I'm trying to figure out why it didn't work on your machine, but with no effects so far.

Could you please re-run your workload with some FIO debug information enabled?

You'd have to simple add extra parameters like:
numactl -N 1 /usr/local/bin/fio libpmem.fio --debug=io,file

I believe "file" and "io" debug types should be sufficient, but FYI, all debug types are described here: https://github.com/axboe/fio/blob/master/HOWTO#L55
Perhaps this will shed some light on your problem - please share the debug log you'll get.


Could you also check (if that's possible) ext4 filesystem (with dax option as well), just to compare, please.

Regards,
Luke

lin tanghui

unread,
Dec 30, 2021, 9:02:42 PM12/30/21
to Łukasz S, pmem
hi,
i try to re-run with --debug=io,file, and get the output as below
```
root@:/mnt/ssd01 # numactl -N 1 /usr/local/bin/fio libpmem.fio --debug=io,file
fio: set debug option io
fio: set debug option file
file     19286 dup files: 0
io       19286 load ioengine libpmem
file     19286 add file libpmem-seqwrite.0.0
file     19286 resize file array to 2 files
file     19286 file 0x7f89e6427110 "/mnt/pmem7/libpmem-seqwrite.0.0" added at 0

libpmem-seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
file     19286 dup files: 0
io       19286 load ioengine libpmem
file     19286 add file libpmem-seqread.0.0
file     19286 resize file array to 2 files
file     19286 file 0x7f89e64272f0 "/mnt/pmem7/libpmem-seqread.0.0" added at 0

libpmem-seqread: (g=1): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
fio-3.29-5-ga3e3
Starting 2 threads
file     19286 setup files
file     19286 get file size for 0x7f89e6427110/0//mnt/pmem7/libpmem-seqwrite.0.0

libpmem-seqwrite: Prepopulating IO file (/mnt/pmem7/libpmem-seqwrite.0.0)
file     19286 open file /mnt/pmem7/libpmem-seqwrite.0.0, flags 41
io       19286 invalidate not supported by ioengine libpmem
file     19286 setup files
file     19286 get file size for 0x7f89e64272f0/0//mnt/pmem7/libpmem-seqread.0.0

libpmem-seqread: Prepopulating IO file (/mnt/pmem7/libpmem-seqread.0.0)
file     19286 open file /mnt/pmem7/libpmem-seqread.0.0, flags 41
io       19286 invalidate not supported by ioengine libpmem
io       19367 o->rw_min_bs 4096
 o->fsync_blocks 0
 o->fdatasync_blocks 0
io       19367 DEBUG fio_libpmem_init
file     19367 trying file /mnt/pmem7/libpmem-seqwrite.0.0 210
io       19367 DEBUG fio_libpmem_open_file
io       19367 f->io_size=10737418240
io       19367 td->o.size=10737418240
io       19367 td->o.iodepth=1
io       19367 td->o.iodepth_batch=1
io       19367 DEBUG fio_libpmem_file
io       19367 f->file_name = /mnt/pmem7/libpmem-seqwrite.0.0 td->o.verify = 0
io       19367 length = 10737418240 f->fd = -1 off = 0 file mode = 384

fio: destination does not support O_DIRECT
file     19367 error 1 on open of /mnt/pmem7/libpmem-seqwrite.0.0
file     19367 get_next_file_rr: (nil)
file     19367 get_next_file: NULL
io       19367 io_u 0x558bdf8be000, setting file failed
io       19367 get_io_u failed
io       19367 io_u_queued_complete: min=0
io       19367 getevents: 0
fio: pid=19367, err=22/file:engines/libpmem.c:114, func=file_name does not point to persistent memory, error=Invalid argument
file     19367 close files
io       19367 close ioengine libpmem
io       19367 free ioengine libpmem
io       19370 o->rw_min_bs 4096
 o->fsync_blocks 0
 o->fdatasync_blocks 0
io       19370 DEBUG fio_libpmem_init
file     19370 trying file /mnt/pmem7/libpmem-seqread.0.0 210
io       19370 DEBUG fio_libpmem_open_file
io       19370 f->io_size=10737418240
io       19370 td->o.size=10737418240
io       19370 td->o.iodepth=1
io       19370 td->o.iodepth_batch=1
io       19370 DEBUG fio_libpmem_file
io       19370 f->file_name = /mnt/pmem7/libpmem-seqread.0.0 td->o.verify = 0
io       19370 length = 10737418240 f->fd = -1 off = 0 file mode = 384

fio: destination does not support O_DIRECT
file     19370 error 1 on open of /mnt/pmem7/libpmem-seqread.0.0
file     19370 get_next_file_rr: (nil)
file     19370 get_next_file: NULL
io       19370 io_u 0x558bdf8be000, setting file failed
io       19370 get_io_u failed
io       19370 io_u_queued_complete: min=0
io       19370 getevents: 0
fio: pid=19370, err=22/file:engines/libpmem.c:114, func=file_name does not point to persistent memory, error=Invalid argument
file     19370 close files
io       19370 close ioengine libpmem
io       19370 free ioengine libpmem



Run status group 0 (all jobs):

Run status group 1 (all jobs):
io       19286 ioengine libpmem unregistered
io       19286 ioengine dev-dax unregistered
io       19286 ioengine pmemblk unregistered
io       19286 ioengine io_uring unregistered
io       19286 ioengine sg unregistered
io       19286 ioengine mtd unregistered
io       19286 ioengine splice unregistered
io       19286 ioengine e4defrag unregistered
io       19286 ioengine falloc unregistered
io       19286 ioengine posixaio unregistered
io       19286 ioengine exec unregistered
io       19286 ioengine filedelete unregistered
io       19286 ioengine filestat unregistered
io       19286 ioengine filecreate unregistered
io       19286 ioengine ftruncate unregistered
io       19286 ioengine net unregistered
io       19286 ioengine netsplice unregistered
io       19286 ioengine null unregistered
io       19286 ioengine sync unregistered
io       19286 ioengine psync unregistered
io       19286 ioengine vsync unregistered
io       19286 ioengine pvsync unregistered
io       19286 ioengine pvsync2 unregistered
io       19286 ioengine mmap unregistered
io       19286 ioengine cpuio unregistered
root@shylf-ywb-424:/mnt/ssd01 #
```

I try to remount filesystem with ext4, and get the same output wiht xfs.

Łukasz S <mr.ha...@gmail.com> 于2021年12月31日周五 00:04写道:
--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/c10512a2-231c-4f9c-a577-23e23f4fb261n%40googlegroups.com.

Łukasz S

unread,
Dec 31, 2021, 7:37:28 AM12/31/21
to pmem
Hi,

thanks for sending out the logs. Unfortunately they don't tell me anything new.

I've consulted your case with my colleague and perhaps your kernel/mounted filesystems have some issue with the O_DIRECT flag in some way.

We have few more follow-ups for you to try, please:
 - set in your workload 'direct' flag to 0, (libpmem engine should not care about this flag while laying out files)
 - you could run fio under 'strace' - perhaps you'll see some discrepancies in the calls,
 - perhaps there'll be something interesting in the kernel logs - please check 'dmesg' output (especially right after creating filesystem and mounting device; and after FIO run)
 - and finally please share your kernel, OS and installed PMDK versions.

Reagrds,
Luke

lin tanghui

unread,
Dec 31, 2021, 9:47:41 AM12/31/21
to Łukasz S, pmem
Hi
i had set the direct flag to 0, and it get the same output as before.

My env info is as below:
root:/mnt/ssd01 # uname -a
Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3 (2019-02-02) x86_64 GNU/Linux
root:/mnt/ssd01 # cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

and PDMK version is 1.11.1.

When creating filesystem, the dmesg only contain mount info as below:
EXT4-fs (pmem7): DAX enabled. Warning: EXPERIMENTAL, use at your own risk
EXT4-fs (pmem7): mounted filesystem with ordered data mode. Opts: dax
 After run with fio, there is no other output in dmesg. And i also run fio under strace and i had send the fio output in attachment



Łukasz S <mr.ha...@gmail.com> 于2021年12月31日周五 20:37写道:
strace_log

steve.s...@gmail.com

unread,
Dec 31, 2021, 1:21:43 PM12/31/21
to pmem
That's a very old Kernel. If you have the opportunity to [temporarily] test using a newer Debian release that has a newer Kernel version, it would be helpful as we can't debug Kernel issues in this community. Debian Stretch is EOL with the LTS EOL date expiring in 6months (https://wiki.debian.org/DebianReleases). 

libpmem recommends Kernel 4.19 or later that has advanced RAS features, though it should still fall back if the features aren't available. Kernel 4.15 is really the bare minimum you should be on IMHO. There have also been numerous Kernel and file system bug fixes since Kernel 4.9. which may be applicable to this situation.

Q) Is this OS running on a bare-metal host or a virtual environment such as a KVM Guest, VMWare ESXi Guest, or K8/K3 Container?
Q) What physical PMem device do you have? Is it an Intel Optane Persistent Memory module (256GB) or an emulated device using disk or DRAM?
Q) What CPU SKU/model are you using? (lscpu)
Q) What server/workstation manufacturer are you using? (dmidecode -t baseboard)

Focusing on "file_name does not point to persistent memory", what happens if you set PMEM_IS_PMEM_FORCE=1 in the shell environment? ie:

# export PMEM_IS_PMEM_FORCE=1
# numactl -N 1 /usr/local/bin/fio libpmem.fio

This will change the default behaviour of libpmem to always believe the file is pmem. If this gets us further, then we know it's a Kernel or File System bug/issue and you'll need to update the OS or talk to the Debian community for further triage. I'm thinking it could be related to the following work since you have "sync=1" in your FIO file. 


>> On Linux, pmem_is_pmem() returns true if the entire range was mapped directly from Device DAX (/dev/daxX.Y) without an intervening file system, or MAP_SYNC flag of mmap(2) is supported by the file system on Filesystem DAX.


>> MAP_SYNC (since Linux 4.15) This flag is available only with the MAP_SHARED_VALIDATE mapping type; mappings of type MAP_SHARED will silently ignore this flag. This flag is supported only for files supporting DAX (direct mapping of persistent memory). For other files, creating a mapping with this flag results in an EOPNOTSUPP error. Shared file mappings with this flag provide the guarantee that while some memory is mapped writable in the address space of the process, it will be visible in the same file at the same offset even after the system crashes or is rebooted. In conjunction with the use of appropriate CPU instructions, this provides users of such mappings with a more efficient way of making data modifications persistent.

If it is this issue -  I haven't walked the code paths to verify - it's plausible the EOPNOTSUPP becomes EINVAL which would explain the "error=Invalid argument" part of the message from FIO. 



lin tanghui

unread,
Jan 2, 2022, 2:28:16 AM1/2/22
to steve.s...@gmail.com, pmem
1)my os is running on a bare-metal host.
2) it's an  Intel Optane Persistent Memory module 

steve.s...@gmail.com <steve.s...@gmail.com> 于2022年1月1日周六 02:21写道:
--
You received this message because you are subscribed to a topic in the Google Groups "pmem" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pmem/naFG0mHDsyE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/480a84bf-b49c-4cd3-854a-cbd99d1caa10n%40googlegroups.com.

lin tanghui

unread,
Jan 2, 2022, 2:44:20 AM1/2/22
to steve.s...@gmail.com, pmem
1) the os is running on a bare-metal host.
2) it's an  Intel Optane Persistent Memory module.
3) cpu SKU:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                64
On-line CPU(s) list:   0-63
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
Stepping:              7
CPU MHz:               2800.036
CPU max MHz:           3900.0000
CPU min MHz:           1000.0000
BogoMIPS:              4600.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              22528K
NUMA node0 CPU(s):     0-15,32-47
NUMA node1 CPU(s):     16-31,48-63

4) server manufacturer:
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 3.0 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: Huawei
Product Name: BC11SPSCB0
Version: V100R005
Serial Number: 024AFQCNL9001636
Asset Tag: Huawei
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: Type2 - Board Chassis Location
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0

I try to run fio with PMEM_IS_PMEM_FORCE=1  and finally it work. but it seem had worse performance than nvme-ssd with the same fio arg, the workload is as below
pmem workload

root:/mnt/ssd01 # numactl -N 1 /usr/local/bin/fio libpmem.fio
libpmem-seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
libpmem-seqread: (g=1): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
fio-3.29-5-ga3e3
Starting 2 threads
libpmem-seqwrite: Prepopulating IO file (/mnt/pmem7/libpmem-seqwrite.0.0)
libpmem-seqread: Prepopulating IO file (/mnt/pmem7/libpmem-seqread.0.0)
Jobs: 1 (f=1): [_(1),R(1)][100.0%][r=2360MiB/s][r=604k IOPS][eta 00m:00s]
libpmem-seqwrite: (groupid=0, jobs=1): err= 0: pid=53548: Sun Jan  2 15:28:39 2022
  write: IOPS=226k, BW=884MiB/s (927MB/s)(86.3GiB/100001msec); 0 zone resets
   bw (  KiB/s): min=548768, max=916592, per=100.00%, avg=905724.60, stdev=45560.96, samples=199
   iops        : min=137192, max=229148, avg=226431.13, stdev=11390.24, samples=199
  cpu          : usr=99.32%, sys=0.64%, ctx=313, majf=0, minf=524294
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,22625099,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
libpmem-seqread: (groupid=1, jobs=1): err= 0: pid=53813: Sun Jan  2 15:28:39 2022
  read: IOPS=601k, BW=2346MiB/s (2460MB/s)(229GiB/100000msec)
   bw (  MiB/s): min= 1226, max= 2363, per=100.00%, avg=2346.55, stdev=107.36, samples=199
   iops        : min=314014, max=605076, avg=600716.55, stdev=27484.97, samples=199
  cpu          : usr=99.67%, sys=0.28%, ctx=309, majf=0, minf=262144
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=60069211,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1


Run status group 0 (all jobs):
  WRITE: bw=884MiB/s (927MB/s), 884MiB/s-884MiB/s (927MB/s-927MB/s), io=86.3GiB (92.7GB), run=100001-100001msec


Run status group 1 (all jobs):
   READ: bw=2346MiB/s (2460MB/s), 2346MiB/s-2346MiB/s (2460MB/s-2460MB/s), io=229GiB (246GB), run=100000-100000msec

nvme-ssd workload
root:/mnt/ssd01 # numactl -N 1 /usr/local/bin/fio ssd.fio

libpmem-seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
libpmem-seqread: (g=1): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libpmem, iodepth=1
fio-3.29-5-ga3e3
Starting 2 threads
libpmem-seqwrite: Prepopulating IO file (/mnt/ssd01/libpmem-seqwrite.0.0)
libpmem-seqread: Prepopulating IO file (/mnt/ssd01/libpmem-seqread.0.0)
Jobs: 1 (f=1): [_(1),R(1)][100.0%][r=5316MiB/s][r=1361k IOPS][eta 00m:00s]
libpmem-seqwrite: (groupid=0, jobs=1): err= 0: pid=55483: Sun Jan  2 15:37:47 2022
  write: IOPS=339k, BW=1322MiB/s (1387MB/s)(129GiB/100000msec); 0 zone resets
   bw (  MiB/s): min=  293, max= 1437, per=100.00%, avg=1322.70, stdev=269.29, samples=199
   iops        : min=75010, max=368066, avg=338612.28, stdev=68938.82, samples=199
  cpu          : usr=94.49%, sys=5.46%, ctx=326, majf=0, minf=1961220
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,33851221,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
libpmem-seqread: (groupid=1, jobs=1): err= 0: pid=55939: Sun Jan  2 15:37:47 2022
  read: IOPS=1360k, BW=5311MiB/s (5569MB/s)(519GiB/100000msec)
   bw (  MiB/s): min= 4268, max= 5321, per=100.00%, avg=5311.00, stdev=74.28, samples=199
   iops        : min=1092846, max=1362318, avg=1359616.31, stdev=19016.43, samples=199
  cpu          : usr=99.91%, sys=0.04%, ctx=311, majf=0, minf=16384
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=135952782,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1


Run status group 0 (all jobs):
  WRITE: bw=1322MiB/s (1387MB/s), 1322MiB/s-1322MiB/s (1387MB/s-1387MB/s), io=129GiB (139GB), run=100000-100000msec


Run status group 1 (all jobs):
   READ: bw=5311MiB/s (5569MB/s), 5311MiB/s-5311MiB/s (5569MB/s-5569MB/s), io=519GiB (557GB), run=100000-100000msec

The nvme device info 
root:/mnt/ssd01 # smartctl  -x /dev/nvme0n1
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-8-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPED1K015TA
Serial Number:                      PHKS913400GB1P5CGN
Firmware Version:                   E2010435
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,500,301,910,016 [1.50 TB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Sun Jan  2 15:39:45 2022 HKT
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Maximum Data Transfer Size:         32 Pages

I will try to update my kernel after NY vacation and see how it work.

steve.s...@gmail.com <steve.s...@gmail.com> 于2022年1月1日周六 02:21写道:
That's a very old Kernel. If you have the opportunity to [temporarily] test using a newer Debian release that has a newer Kernel version, it would be helpful as we can't debug Kernel issues in this community. Debian Stretch is EOL with the LTS EOL date expiring in 6months (https://wiki.debian.org/DebianReleases). 
--
You received this message because you are subscribed to a topic in the Google Groups "pmem" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pmem/naFG0mHDsyE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/480a84bf-b49c-4cd3-854a-cbd99d1caa10n%40googlegroups.com.

Jan K

unread,
Jan 6, 2022, 5:41:50 AM1/6/22
to lin tanghui, pmem
> I try to run fio with *PMEM_IS_PMEM_FORCE*=1 and finally it work. but it
> seem had worse performance than nvme-ssd with the same fio arg, the
> workload is as below
> *pmem workload*
> WRITE: bw=884MiB/s (927MB/s), 884MiB/s-884MiB/s (927MB/s-927MB/s),
> io=86.3GiB (92.7GB), run=100001-100001msec
>
> *nvme-ssd workload*
> WRITE: bw=1322MiB/s (1387MB/s), 1322MiB/s-1322MiB/s (1387MB/s-1387MB/s),
> io=129GiB (139GB), run=100000-100000msec

Sadly, these write throughput results look perfectly sane :(
Optane persistent memory is known to have an unacceptably low write
throughput in single-threaded access. (I cannot refer to read
throughput, my use case is already throttled by write throughput.)

Have a look here on what to expect:
https://engineering.mongodb.com/post/we-replaced-an-ssd-with-storage-class-memory-here-is-what-we-learned
Fast forwarding to the gist: "Write throughput was only 0.6 GB/s on a
single PM module". Contrary to their expectations multiple modules do
not improve single-thread write performance. Multiple threads
accessing PM module in parallel do get the advertised performance,
though.

> Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3 (2019-02-02) x86_64 GNU/Linux

That's the date of the build, not the kernel release date.
4.9 has been released 2016-12-11 according to
kernel.org/category/releases.html, and it's still maintained as a
longterm release, but that means it gets only bugfixes and no new
functionality. Please be careful around stable debian, you may
sometimes find an unexpectedly old software there.
Reply all
Reply to author
Forward
0 new messages