Performance hit from interleaving?

[steve]

Hi all,

I've recently been using my pmem in a non-interleaved AppDirect configuration. As expected, sequential writing is slower than with the interleaved
version, and slows down further as my process runs, which I imagine is due to thermal issues.

I was however surprised to see that random access appears to be noticeably faster without interleaving than with it. It looks like somewhere in the
neighborhood of 100 nsec overhead for the interleaving, although I haven't yet been able to run exactly the same experiment with and without
interleaving so I'm not sure of the exact overhead.

I guess I shouldn't have been surprised because obviously some extra work must be done to direct the write or read to the correct part for the
interleaved access. Is this documented somewhere or should I just rely on my experimental results? I'm mostly concerned that I'm measuring something
incorrectly.

Thanks!
------------
Steve Heller

[Anton Gavriliuk]

Hi Steve

> I guess I shouldn't have been surprised because obviously some extra work must be done to direct the write or read to the correct part for the

interleaved access. Is this documented somewhere or should I just rely on my experimental results? I'm mostly concerned that I'm measuring something incorrectly.

I think you are right.  Interleaving means striping with stripe size 64 bytes if I remember correctly, so yes, for interleaving some extra work must be done.

Based on my experience, interleaving or non-interleaving setup is highly workload dependent what is need.

Anton

пн, 25 мая 2020 г. в 14:32, steve <st...@steveheller.org>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/okancf9ap9g3fpm0po1ntq7rvfh91uj6bj%404ax.com.

[Andy Rudoff]

Note that interleaving granularity is larger than 256 bytes, and the Optane PMem modules provide "sequential" performance when you access 256 bytes (aligned).  So changing the interleave should not change the difference between sequential/random performance of an individual Optane module, but it will of course change what happens for larger blocks, sending a large access to a single DIMM versus spreading it out among multiple DIMMs.  Whether you have interleave sets going across all DIMMs in a socket, or have them set so each DIMM is in its own interleave set, the same decoder logic has to process each access (you don't skip the decoders either way), so I'm not sure why Steve is seeing faster random access.  Without more information (what type of access, what size of access, single thread versus multiple threads, etc) it is hard to guess.  The key to understanding performance puzzles is to get more visibility into what's going on, so I suggest using a tool like VTune (which is free) to examine the system counters.

-andy

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+unsubscribe@googlegroups.com.

[steve]

On Mon, 25 May 2020 15:04:19 +0300, Anton Gavriliuk <antos...@gmail.com> wrote:

>Hi Steve
>
>> I guess I shouldn't have been surprised because obviously some extra work
>must be done to direct the write or read to the correct part for the
>interleaved access. Is this documented somewhere or should I just rely on
>my experimental results? I'm mostly concerned that I'm measuring something
>incorrectly.
>
>I think you are right. Interleaving means striping with stripe size 64
>bytes if I remember correctly, so yes, for interleaving some extra work
>must be done.
>
>Based on my experience, interleaving or non-interleaving setup is highly
>workload dependent what is need.

Thanks. For optimal performance with my hash table, I think I'm going to need to add code to do my own interleaving for the hash table build phase,
then recommend using non-interleaved volumes for best retrieval performance.

Fortunately I don't think this new code will be too hard to add given my existing free block management code.

One additional variable I didn't mention before is that I've changed my pmem population: I now have 3x "256 GB" and 1x "128 GB" parts, along with 4x
32 GB of DRAM. So my total memory space is 1012 GB, about 1% short of the maximum possible address size for my Silver 4215 CPU.

I have also noticed that the 256 parts are about 10% faster for sequential access: about 2.3 GB/sec vs 2.1 GB/sec for the 128 parts. I'm not sure how
much that contributes to the speed results. I'm sure it helps a little bit, but I don't think it explains all of the speedup. To really test this,
I'll have to take the 128 part out and retest with an interleaved 768 GB volume made up of the 256 parts, but I haven't gotten to that yet.

P.S. I know mixing part sizes isn't supported, but it works fine on my machine, with the obvious exception of the inability to interleave the parts.
That's why I'm working with non-interleaved volumes in the first place, but I was pleasantly surprised by the performance difference.

>Anton
>
>??, 25 ??? 2020 ?. ? 14:32, steve <st...@steveheller.org>:

------------
Steve Heller

[steve]

On Mon, 25 May 2020 05:23:05 -0700 (PDT), Andy Rudoff <an...@rudoff.com> wrote:

>Note that interleaving granularity is larger than 256 bytes, and the Optane
>PMem modules provide "sequential" performance when you access 256 bytes
>(aligned). So changing the interleave should not change the difference
>between sequential/random performance of an individual Optane module, but
>it will of course change what happens for larger blocks, sending a large
>access to a single DIMM versus spreading it out among multiple DIMMs.
>Whether you have interleave sets going across all DIMMs in a socket, or
>have them set so each DIMM is in its own interleave set, the same decoder
>logic has to process each access (you don't skip the decoders either way),
>so I'm not sure why Steve is seeing faster random access. Without more
>information (what type of access, what size of access, single thread versus
>multiple threads, etc) it is hard to guess. The key to understanding
>performance puzzles is to get more visibility into what's going on, so I
>suggest using a tool like VTune (which is free) to examine the system
>counters.

Thanks for the additional info.

The pmem access pattern is single threaded, random access to 900-byte records in a 1 billion record table. The records are not aligned on 256-byte
boundaries.

I'm also doing a random access to DRAM before each of these random pmem accesses, to look up the address in the pmem table.

Maybe it's the 256 vs 128 parts: are they significantly faster? I can do an experiment today or tomorrow by taking the 128 part out and creating an
interleave set with the 256s.

VTune is part of Intel System Studio, right?

>-andy
>
>On Monday, May 25, 2020 at 6:04:33 AM UTC-6, Anton Gavriliuk wrote:
>>
>> Hi Steve
>>
>> > I guess I shouldn't have been surprised because obviously some extra
>> work must be done to direct the write or read to the correct part for the
>> interleaved access. Is this documented somewhere or should I just rely on
>> my experimental results? I'm mostly concerned that I'm measuring something
>> incorrectly.
>>
>> I think you are right. Interleaving means striping with stripe size 64
>> bytes if I remember correctly, so yes, for interleaving some extra work
>> must be done.
>>
>> Based on my experience, interleaving or non-interleaving setup is highly
>> workload dependent what is need.
>>
>> Anton
>>

>> ??, 25 ??? 2020 ?. ? 14:32, steve <st...@steveheller.org>:

>>
>>> Hi all,
>>>
>>> I've recently been using my pmem in a non-interleaved AppDirect
>>> configuration. As expected, sequential writing is slower than with the
>>> interleaved
>>> version, and slows down further as my process runs, which I imagine is
>>> due to thermal issues.
>>>
>>> I was however surprised to see that random access appears to be
>>> noticeably faster without interleaving than with it. It looks like
>>> somewhere in the
>>> neighborhood of 100 nsec overhead for the interleaving, although I
>>> haven't yet been able to run exactly the same experiment with and without
>>> interleaving so I'm not sure of the exact overhead.
>>>
>>> I guess I shouldn't have been surprised because obviously some extra work
>>> must be done to direct the write or read to the correct part for the
>>> interleaved access. Is this documented somewhere or should I just rely on
>>> my experimental results? I'm mostly concerned that I'm measuring something
>>> incorrectly.
>>>
>>> Thanks!
>>> ------------
>>> Steve Heller
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "pmem" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to pmem+uns...@googlegroups.com.

>>> To view this discussion on the web visit