FIO devdax engine shows wrong seq read data

38 views
Skip to first unread message

vishnu satheesh

unread,
Jul 7, 2022, 7:50:15 AMJul 7
to pmem

While checking FIO with x16 gen3 PCI card with 32GB of memory, it shows 20GB/s bandwidth for sequential read in cache-enabled system. This output is way more than PCI spec.

What are all check points we need to check for this issue?
Is this a problem of FIO calculating bandwidth wrongly?


I am using dev-dax.fio config file. While debugging i saw , same physical address is being mapped for read thread but different for write threads, 


what might be the issue

dev-dax.fio.txt

Eduardo Berrocal

unread,
Jul 8, 2022, 12:39:16 PMJul 8
to pmem
Hello,

I am not sure that using dev-dax with a PCI card makes sense. Dax means direct access from the CPU, and a PCI card do not have that (at least not without CXL).
If you are caching, chances are that part of your reads hit DRAM which explains the performance you are seeing.

Cheers,
Eduardo.

vishnu satheesh

unread,
Jul 9, 2022, 1:01:14 AMJul 9
to pmem
Hi  Eduardo,

Actually, I am using a CXL device. Because it is an IP I can't share more info on the device. It has 32GB of memory. above performance is seen cache enabled system. 

I have one more observation. When read threads are running, all the threads are mapping memory to the same physical address even if the threads are using different virtual addresses.
But in the write case, all the time different physical address is seen. 

Wu, Dennis

unread,
Jul 10, 2022, 9:14:21 PMJul 10
to vishnu satheesh, pmem

I am not sure if we see the similar read performance issue on the BTT with /dev/pmem0.s. When we create the sector namespace and then do the multi-job seq read, we will see the read performance is above the limitation of the Pmem while if we are using the pcm_memory.x tool to monitor the Pmem bandwidth, it is small and close to one job bandwidth, so might the read map to the same physical address. We will start to take a look as well.

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/1ecc8fe8-25ef-4684-9384-68d39339a1b7n%40googlegroups.com.

vishnu satheesh

unread,
Jul 11, 2022, 12:09:48 AMJul 11
to pmem
Thank you Dennis. 

We are creating /dev/dax0.0 using EFI_FAKE_MEM kernel command line parameter. Can toy please elaborate on what is BTT, so that I can check? 
Is there any other experiments that I should do to check this issue ?

Wu, Dennis

unread,
Jul 11, 2022, 12:58:25 AMJul 11
to vishnu satheesh, pmem

BTT is the block translation table which can be found: https://docs.kernel.org/driver-api/nvdimm/btt.html?highlight=btt

Then you need to create the namespace with ndctl create-namespace –mode=sector then it will expose the /dev/pmem0.s in your system. If you run the read performance it will over the limitation of the PMem.

 

BR,

Dennis W

vishnu satheesh

unread,
Jul 13, 2022, 2:44:35 AMJul 13
to pmem
Hi Dennis,

Did you get a chance to try out this problem? 

If we want to check with BTT, should we use the same fio, if yes which ioengine we should use? 

Reply all
Reply to author
Forward
0 new messages