vNVDIMM performance with uncacheable mapping

[ramot]

Hi,

I am seeing perf issues with vNVDIMM ( on esx) when I map the pmem address range with uncacheable attribute (MTRR/PAT attribute). Simple plain bcopy() performance (no flush) from BSD kernel context (no user space involved) goes down drastically (100%+) when I use "UC" attribute in my mapping api compared to lets say Write-Back (WB) . Why would that be the case? Is there something in Uncacheable that makes it virtualization unfriendly?

Thanks.

[Andy Rudoff]

Hi,

Are you sure you're not just observing the performance impact of CPU caching (and the lack of it)?  I can't think of anything virtualization-related that would make this worse, but perhaps someone from VMware could weigh in on that.  But I would expect a bcopy() to be very much slower since you will get no write-combining from the CPU caches, causing each cache line to be fetched potentially multiple times from the media.  There's a reason we map persistent memory WB and then use CLWB to flush changes instead of mapping it UC -- I would expect mapping it UC to be unusably slow.

But if you're thinking the slowness you're seeing is beyond what the lack of CPU cache can explain, I recommend measuring the BW you're getting and comparing it to the expected BW of the NVDIMM, keeping in mind that many of your stores may end up being read-modify-writes of course.  Also, you can use tools like VTune or Emon to get visibility into what's going on in the system to confirm what operations your code is causing in the platform.

-andy