Log Append Throughput

[Lucas Lersch]

Hi,

I have been running some simple tests to get the throughput of appending entries to pmemlog. I have noticed that when inserting entries with size 256 Bytes I get a throughput N, but when inserting entries with size 255 Bytes, I get throughput N/2. I am not sure I understand this behavior, or maybe it is just a silly mistake somewhere in my code. Here is the source code: http://pastebin.com/raw/wF9Y1ih0

Any help would be really appreciated.

Best regards.

[Marcin Ślusarz]

Hi Lucas,

The difference comes from 256 being a threshold value where
pmem_memcpy starts to use its optimized path - it's controllable by
PMEM_MOVNT_THRESHOLD environment variable.
Even if you lower this threshold, 256 byte appends are still faster because:
1) 256 is divisible by 64, so each write starts at cacheline boundary
(and pmem_memcpy does not need to do slow alignment)
2) 256 is divisible by 128, so pmem_memcpy can copy in 128 byte chunks
and not care about a tail

See memmove_nodrain_movnt in pmem.c for more details.

Hope this helps,
Marcin

> --
> You received this message because you are subscribed to the Google Groups
> "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pmem+uns...@googlegroups.com.
> To post to this group, send email to pm...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pmem/b449a043-078a-442b-9d4b-70592e74fa77%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Lucas Lersch]

Thanks a lot for your answer. After you mentioned, I took a look again at the code and it made more sense. Furthermore, I have noticed that using 257 Byte entries also imply lower throughput. Is it safe to assume that, in these cases, it would be better to externally round the entry size to 320 (256+64) or even 384(256+128)?

--

Lucas Lersch

[Lucas Lersch]

An additional question in the same context. I made more experiments with pmem_appendv() and entries of size 512 Bytes. If I have an iovec[1] the performance is the same as using pmem_append(), as expected. However, if I split the entry in half, and I pass the iovec[2] to pmem_appendv(), I get some loss in performance (from 1300~1400 ktps to 1000~1100 ktps). I understand that pmem_appendv() has an additional overhead (multiple calls to pmem_memcopy), but I am wondering if this is the expected price to be paid. Here is the code: http://pastebin.com/raw/1SsLRpsQ

In a moral general way, it seems that the fastest way is to append a single contiguous block of memory. However, if I want to append multiple variable size strings at once, I have two options: "serialize" these strings in a contiguous block of memory (additional cost) and then pmem_append() this block, OR use pmem_appendv() (performance loss). Just as a sanity check, am I right or am I missing something?

Best regards and thanks a lot for the help.

--

Lucas Lersch

[Marcin Ślusarz]

257 byte appends force slow alignment on each pmem_memcpy
(https://github.com/pmem/nvml/blob/94c3b6af2c109e7b1e855b8bf7ea86ad192d834a/src/libpmem/pmem.c#L731
), so 64 byte alignment is an absolute minimum for good performance.
128 alignment is less critical, but can gain you some performance if
you really have that much data.

[Marcin Ślusarz]

I don't know exactly why you get this perf drop, but if I'd have to
guess I'd say it's because of fixed costs of pmem_memcpy (function
pointer, all calculations/loops, edge cases handling, sfence). We
could probably optimize it, but I think gains will not be that great.
Will see.