PM emulation - read/write latencies

[Dimitris Stavrakakis]

Hello everyone,

I am currently trying to emulate PM backed by DRAM.
In the guide (https://pmem.io/2016/02/22/pm-emulation.html) the emulated PM poses the read/write latencies of DRAM. Thus, any performance metrics are not quite representative.
For that reason there have been proposed works such as Quartz (https://github.com/HewlettPackard/quartz) or Mesmeric (https://github.com/takahiro-hirofuchi/mesmeric-emulator) which emulate PM latencies to get more realistic results.
However, these emulators target specific architectures and CPU models.
Is there any alternative to emulate PM latencies on CPUs of the Intel Core line?
I would like to ask about your experiences and suggestions.

Thank you very much in advance,
Dimitris

[Anton Gavriliuk]

Hi Dimitris

> Is there any alternative to emulate PM latencies on CPUs of the Intel Core line?

> I would like to ask about your experiences and suggestions.

I'm not familiar with those emulators.  However I think I could help you with pmem latencies you required.

I have 2 x 8280L with 12 x 512 GB DCPMMs.  Presently I use non-interleaved setup.

{
  "dev":"namespace5.0",
  "mode":"fsdax",
  "map":"dev",
  "size":"494.15 GiB (530.59 GB)",
  "uuid":"20046760-5932-4047-82b0-3a6d475a461b",
  "sector_size":4096,
  "align":2097152,
  "blockdev":"pmem5"
}

linux-tg7k:/home/anton # mount|grep pmem
/dev/pmem5 on /mnt type xfs (rw,relatime,attr2,dax,dax=always,inode64,logbufs=8,logbsize=32k,noquota)

If we are talking about standart read()/write() system calls, I think we can use fio with psync ioengine.

linux-tg7k:/home/anton # numactl -N 0 fio --filename=/mnt/file --rw=randread --bs=4k --ioengine=psync --iodepth=1 --numjobs=1 --time_based --group_reporting --name=readlatency-test-job --runtime=30
readlatency-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.27-57-gfaff8
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1556MiB/s][r=398k IOPS][eta 00m:00s]
readlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5138: Mon Aug  9 07:59:08 2021
  read: IOPS=402k, BW=1571MiB/s (1647MB/s)(46.0GiB/30001msec)
    clat (nsec): min=1754, max=237747, avg=2265.89, stdev=418.54
     lat (nsec): min=1774, max=237769, avg=2286.27, stdev=418.94

So the average 4kb random read time is 2286ns.  This is far behind my best results (should be ~540 kiops & 1.6us), but that includes pmem lat + I/O stack (optimized with "dax" xfs mount option).

However, own pmem latency is ~350ns

Please let me know if you will have any questions. 

Anton

вс, 8 авг. 2021 г. в 12:45, Dimitris Stavrakakis <dims...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/6b4c3838-cc5a-4c7c-8c6d-e636caa96d61n%40googlegroups.com.

[Jan K]

Dimitris,

Keep in mind that PM has also limited read and write bandwidth
compared to DRAM. In some cases this is a bigger performance blocker
than the latency.
Emulating real bandwidth of PM is definitely tricky, for instance
random writes smaller than 256B are slower than random 256B writes, so
a single write bandwidth parameter does not suffice.
Some test I've run with emulated pmem did not match test results
acquired on real pmem when it comes to answering a question which
implementation of a given data structure is faster, so beware.

Kind regards,
Jan

>> <https://groups.google.com/d/msgid/pmem/6b4c3838-cc5a-4c7c-8c6d-e636caa96d61n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .

>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pmem+uns...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pmem/CAAiJnjrkXn4%2BqWXz3wO-exQFDmx-H-%2B0_g1o-bxuOmFtyVN9XQ%40mail.gmail.com.
>