Does linux perf tool work well with pmem in DAX mode?

[MOHIT VERMA]

I am looking at the perf report of an application that is using pmem in DAX mode. I have a couple of questions (I don't understand perf fully, so excuse me if these are naive questions):

1. Does the perf tool understand CLWB instructions? 

• I am using clwb to write back dirty data to pmem. I expected perf to show this as the hottest instruction, and commenting out clwb buys back all of the slowdown of the applications. But "perf" doesn't show clwb as a hot instruction, rather perf shows a "mov" instruction somewhere else in the code as taking up most of the time.

2. What perf stats should I be looking at, to debug performance of application working with pmem in DAX mode? 

Thanks!

Mohit

[Andy Rudoff]

Hi Mohit,

I'm not a perf expert, so hopefully some others will chime in about perf specifically, but here are a couple comments that might help:

First, on the CLWB instruction, that instruction doesn't wait for the cache line flush to happen.  Perf probably does not show it as a "hot" instruction because it takes small amount of time to initiate the cache flush, but the placement of fences or other serializing instructions really determine when the processor has to wait for the flush to complete.  For example, if I do 8 CLWBs followed by an SFENCE, followed by a bunch of instructions changing local (not globally-visible) state, and then finally followed by an instruction that does something globally-visible, only that last instruction has to wait for the flushes to complete before it can execute.

You might find some of the talks from the VTune group on persistent memory to be helpful.  VTune is a different tool than perf (there is a free version available), but some of the information in these talks applies to both.  See:

https://spdk.io/news/2019/09/24/prc_summit_presentations/

And check out the talks in the VTune section (number 5 and number 7 might be interesting to you).

Hope that's helpful.

-andy

[Jan K]

> 1. Does the perf tool understand CLWB instructions?
>

> - I am using clwb to write back dirty data to pmem. I expected perf to

> show this as the hottest instruction, and commenting out clwb buys back
> all
> of the slowdown of the applications. But "perf" doesn't show clwb as a
> hot
> instruction, rather perf shows a "mov" instruction somewhere else in the
>
> code as taking up most of the time.

As far as I can tell, perf does not "understand" any instruction at
all. When sampling, perf-record just periodically records the current
instruction pointer (and attempts to unwind and record the call stack
if you specify -g option). Later, upon analysis, it probably just uses
external libs such as binutils/elfutils to decode the values it
recorded.

> 2. What perf stats should I be looking at, to debug performance of
> application working with pmem in DAX mode?

I'd be happy to know that. Are there any hardware counters events for
Optane Memory?

Regards,
Jan

2019-10-16 20:52 GMT+02:00, MOHIT VERMA <vmoh...@gmail.com>:
> I am looking at the perf report of an application that is using pmem in DAX
>
> mode. I have a couple of questions (I don't understand perf fully, so
> excuse me if these are naive questions):
>
> 1. Does the perf tool understand CLWB instructions?
>

> - I am using clwb to write back dirty data to pmem. I expected perf to

> show this as the hottest instruction, and commenting out clwb buys back
> all
> of the slowdown of the applications. But "perf" doesn't show clwb as a
> hot
> instruction, rather perf shows a "mov" instruction somewhere else in the
>
> code as taking up most of the time.
>
>
> 2. What perf stats should I be looking at, to debug performance of
> application working with pmem in DAX mode?
>
> Thanks!
> Mohit
>

> --
> You received this message because you are subscribed to the Google Groups
> "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pmem+uns...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pmem/3dcdd9c1-f8ad-448d-a9c3-d8268c08f102%40googlegroups.com.
>

[Piotr Balcer]

In addition to what Andy said, additional cost of using cache flushing instructions is the fact that data is evicted from the cache, and perf will show an increasing number of locations where cache misses on data that was evicted happen.

[MOHIT VERMA]

Hi Andy,

Thanks for pointing me to Vtune. I used it to profile my application and I found it missing a few details. For example: It doesn't show stats for time spent in stores to pmem.

The Vtune stats "persistent memory bandwidth bound" and "persistent memory access count" only count how many accesses to Pmem were loads. 

I have a few other questions:

1. Do stores to pmem use a different bandwidth channel that is not reported in Vtune? 

2. For pmem write-heavy applications how do we reason about performance?

Essentially, what I am looking for is stats like:

1. #clwbs issued in the system

2. #sfences issued in the system

3. Stall cycles due to pending stores to pmem. 

4. Stall cycles due to concurrent updates to pmem.

Are there ways to get to the above stats?

Thanks!

Mohit

[Maciej Maciejewski]

@Jan K

HW Performance counters you can find described in Software Development Manual:

https://software.intel.com/en-us/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4

For example offcore ones:

Table 18-45. MSR_OFFCORE_RSP_x Supplier Info Field Definition (CPUID Signature 06_55H, Steppings 0x5H - 0xFH)

LOCAL_PMM 22 Local home requests that were serviced by local PMM. 

REMOTE_HOP0_PMM 23 Hop 0 Remote supplier. 

REMOTE_HOP1_PMM 24 Hop 1 Remote supplier. 

REMOTE_HOP2P_PMM 25 Hop 2 or more Remote supplier

Example of utilization:

https://github.com/intel/workload-collocation-agent/blob/devel/wca/perf_uncore.py

Cheers,

Maciej