pmem and sector mode with btt

Anton Gavriliuk

unread,

Jun 5, 2020, 11:07:22 AM6/5/20

to pmem

Hi all

I would ask you about pmem in sector mode with btt.

Are there any plans to improve pmem sector mode btt performance under linux ?

Sector mode seems to be much faster on Windows Server 2019.

Anton

Vishal Verma

unread,

Jun 12, 2020, 1:14:18 PM6/12/20

to Anton Gavriliuk, pmem

Hi Anton,

I would be interested in seeing any overhead numbers you might have, and any comparisons with windows.

There aren't really any plans to specifically do performance tuning for the BTT, considering it is really a crutch for 'legacy' applications. But I'd at least be interested in seeing if there has been a regression in the overhead numbers with recent kernels from back when I first measured it.

Thanks,

-Vishal

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/CAAiJnjr0ViwTmvMb5qc6wGgO2-d58AzZEGpapcsxOG1AquNvcQ%40mail.gmail.com.

Anton Gavriliuk

unread,

Jun 16, 2020, 3:14:23 PM6/16/20

to Vishal Verma, pmem

Hi Vishal

Well, presently I removed MS Windows. But I remember performance numbers.

Write performance on latest OpenSuse 15.1 on pmem in sector mode degrades ~30%, while on on MS Windows 2019 only ~10% comparing to fsdax mode.

Anton

пт, 12 июн. 2020 г. в 20:14, Vishal Verma <stella...@gmail.com>:

Steve Scargall

unread,

Jun 16, 2020, 5:00:38 PM6/16/20

to pmem

On Tuesday, June 16, 2020 at 1:14:23 PM UTC-6, Anton Gavriliuk wrote:

Write performance on latest OpenSuse 15.1 on pmem in sector mode degrades ~30%, while on on MS Windows 2019 only ~10% comparing to fsdax mode.

Do you still have the test script(s) available so I can reproduce it? Or can you describe the test(s) you performed so I can write some similar tests? Any other information pertinent to the testing and system config would be very useful. I'll focus on Linux since that's where you observed the worst performance.

Q) Can you clarify the degrading performance observation, please? Did you observe a ~30% performance delta between sector mode on Linux and fsdax on Windows, or did you observe that a write test intended to fill the file system encounters slower performance as the file system fills up?

Thanks,

Steve

Anton Gavriliuk

unread,

Jun 17, 2020, 10:56:56 AM6/17/20

to Steve Scargall, pmem

Hi all

I have 2S box with 12 x 512 GB DCPMMs. 2 interleaved regions

For 1st region i created namespace with sector mode, for 2nd region namespace with fsdax mode

linux-tg7k:/home/anton # ndctl list
[
{
"dev":"namespace1.0",
"mode":"fsdax",
"map":"mem",
"size":3234108276736,
"uuid":"8a346cbb-89a4-4694-85c6-b56af4e45332",
"sector_size":4096,
"align":2097152,
"blockdev":"pmem1"
},
{
"dev":"namespace0.0",
"mode":"sector",
"size":3230948659200,
"uuid":"bba5ff3f-4d35-467b-ab61-2969f6723d74",
"sector_size":4096,
"blockdev":"pmem0s"
}
]

Then I created xfs for 2 namespaces and mount it. Keep in mind that I'm even not using "dax' mount option for the fsdax namespace.

linux-tg7k:/home/anton # mount -v|grep pmem
/dev/pmem0s on /mnt_sector type xfs (rw,relatime,attr2,inode64,noquota)
/dev/pmem1 on /mnt_fsdax type xfs (rw,relatime,attr2,inode64,noquota)

Now 2 fio with 8k direct sequential write

linux-tg7k:/home/anton # fio --filename=/mnt_sector/file --rw=write --ioengine=sync --bs=8k --iodepth=1 --numjobs=1 --runtime=30 --group_reporting --name=perf_test --direct=1
perf_test: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=sync, iodepth=1
fio-3.20-45-g1a953
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=787MiB/s][w=101k IOPS][eta 00m:00s]
perf_test: (groupid=0, jobs=1): err= 0: pid=23878: Wed Jun 17 17:38:57 2020
write: IOPS=115k, BW=902MiB/s (946MB/s)(26.4GiB/30001msec); 0 zone resets
clat (nsec): min=4810, max=69846, avg=8457.26, stdev=1765.49
lat (nsec): min=4849, max=69897, avg=8498.23, stdev=1766.84

linux-tg7k:/home/anton # fio --filename=/mnt_fsdax/file --rw=write --ioengine=sync --bs=8k --iodepth=1 --numjobs=1 --runtime=30 --group_reporting --name=perf_test --direct=1
perf_test: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=sync, iodepth=1
fio-3.20-45-g1a953
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=770MiB/s][w=98.5k IOPS][eta 00m:00s]
perf_test: (groupid=0, jobs=1): err= 0: pid=24498: Wed Jun 17 17:49:54 2020
write: IOPS=111k, BW=871MiB/s (913MB/s)(25.5GiB/30000msec); 0 zone resets
clat (nsec): min=7210, max=36760, avg=8776.35, stdev=1254.87
lat (nsec): min=7248, max=36869, avg=8816.66, stdev=1255.45

So with 8k seq write performance almost the same.

However, when I increased block size 8k -> 32k sector mode write performance significantly degraded

linux-tg7k:/home/anton # fio --filename=/mnt_sector/file --rw=write --ioengine=sync --bs=32k --iodepth=1 --numjobs=1 --runtime=30 --group_reporting --name=perf_test --direct=1
perf_test: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=sync, iodepth=1
fio-3.20-45-g1a953
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=865MiB/s][w=27.7k IOPS][eta 00m:00s]
perf_test: (groupid=0, jobs=1): err= 0: pid=24678: Wed Jun 17 17:53:14 2020
write: IOPS=27.5k, BW=858MiB/s (900MB/s)(25.1GiB/30000msec); 0 zone resets
clat (nsec): min=23161, max=99884, avg=36096.77, stdev=3039.98
lat (usec): min=23, max=100, avg=36.26, stdev= 3.04

linux-tg7k:/home/anton # fio --filename=/mnt_fsdax/file --rw=write --ioengine=sync --bs=32k --iodepth=1 --numjobs=1 --runtime=30 --group_reporting --name=perf_test --direct=1
perf_test: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=sync, iodepth=1
fio-3.20-45-g1a953
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=2235MiB/s][w=71.5k IOPS][eta 00m:00s]
perf_test: (groupid=0, jobs=1): err= 0: pid=24738: Wed Jun 17 17:54:11 2020
write: IOPS=73.1k, BW=2285MiB/s (2396MB/s)(66.0GiB/30000msec); 0 zone resets
clat (nsec): min=9813, max=58783, avg=13351.91, stdev=1359.49
lat (nsec): min=9937, max=60940, avg=13506.27, stdev=1361.48

I'm using latest OpenSuse 15.1

linux-tg7k:/home/anton # uname -r
4.12.14-lp151.28.52-default

I didn't see such degradation depending on growing block sizes on latest MS WS 2019.

In Ukraine I already know customers who are using pmem in sector mode production. And write performance is critical for them. They using SAP/Sybase old products such as ASE/IQ, so no DAX access. Just like ultra fast block based SSD.

Anton

ср, 17 июн. 2020 г. в 00:00, Steve Scargall <steve.s...@intel.com>:

--

You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/1116aa6b-07a3-4273-880e-a929425b6ea5o%40googlegroups.com.

Reply all

Reply to author

Forward