Proper Alignment with PMDK

[Phil Go]

Hey everyone,

can structures encapsulated in PMDK be cache-aligned (64 Byte)? During our tests, we recognized that the distance is correct, but the starting address is not a multiple of 64.

As an example take the following snippet:

constexpr auto path = "/mnt/pmem1/test/align";
constexpr auto LAYOUT = "AlignTest";

int main() {
  alignas(64) int a[4];
  alignas(64) int b[4];

  struct alignas(64) C {
  ┆ alignas (64) p<std::array<int, 4>> c1;
  ┆ alignas (64) p<std::array<int, 4>> c2;
  };

  struct root {
  ┆ persistent_ptr<C> c;
  };

  pool<root> pop;
  if (access(path, F_OK) != 0) {
  ┆ pop = pool<root>::create(path, LAYOUT);
  ┆ transaction::run(pop, [&] { pop.root()->c = make_persistent<C>(); });
  } else {
  ┆ std::experimental::filesystem::remove_all(path);
  ┆ pop = pool<root>::create(path, LAYOUT);
  ┆ transaction::run(pop, [&] {
  ┆ ┆ delete_persistent<C>(pop.root()->c);
  ┆ ┆ pop.root()->c = make_persistent<C>();
  ┆ });
  }

  auto q = pop.root();

  printf("a at %p\n", a);
  printf("b at %p\n", b);
  printf("q at %p\n", &*q);
  printf("c at %p\n", &*q->c);
  printf("c1 at %p\n", &q->c->c1.get_ro());
  printf("c2 at %p\n", &q->c->c2.get_ro());
}

The output is the following:

a at 0x7ffd37d06780
b at 0x7ffd37d067c0
q at 0x7f3b63bc0550
c at 0x7f3b63c014d0
c1 at 0x7f3b63c014d0
c2 at 0x7f3b63c01510

As it can be seen, a and b in DRAM are both starting at a multiple of 64 and hava a distance of 64 (0x6780 and 0x67c0).

For the persistent structures, the distance is 64, but the address is always a multiple of 64 plus 16 (0x14d0 and 0x1510).

I could think of two explainations here:

1. It is not aligned properly
2. It is aligned properly and only the virtual address space of PMem starts at "uneven" offset

Can anyone elaborate on this?

Thanks and best regards,

Philipp

[Piotr Balcer]

Hi,

All allocated objects are aligned to 64 bytes. However, user-visible objects have, by default, a 16 byte header in which the object's size and type number is stored.

User-visible alignment is opt-in, using the allocation classes mechanism. See https://pmem.io/pmdk/manpages/linux/master/libpmemobj/pmemobj_ctl_get.3 for more details.

Piotr

[Phil Go]

Hey again,

ok, I now have seen that in the example above struct C is always aligned to 64 bytes. However, if part of my data structure really needs to be in a separate CL the PMDK does not align properly (in my eyes).

So to stick with the example, the expected way for me would be:

 struct C {
 ┆ alignas(64) p<std::array<int, 4>> c1;
 ┆ alignas(64) p<std::array<int, 4>> c2;
 };

When filling the array with 0xFFFFFFFF the dump looks like the following:

0x7ffff4a014c0: 0x000000c0      0x00000000      0x79550b68      0xe55edbb3
0x7ffff4a014d0: 0xffffffff      0xffffffff      0xffffffff      0xffffffff
0x7ffff4a014e0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a014f0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01500: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01510: 0xffffffff      0xffffffff      0xffffffff      0xffffffff
0x7ffff4a01520: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01530: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01540: 0x00000000      0x00000000      0x00000000      0x00000000

So again, the distance is correct, but why doesn't the sub-structures c1 and c2 start at 0x..1500 and 0x..1540? This is what I would expect.

To get the layout I need, I could change the struct like follows:

struct C {
┆ p<std::array<uint64_t, 6>> padding;
┆ p<std::array<int, 4>> c1;
┆ p<std::array<int, 4>> c2;
};

And the dump:

0x7ffff4a014c0: 0x000000c0      0x00000000      0x79550b68      0xe55edbb3
0x7ffff4a014d0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a014e0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a014f0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01500: 0xffffffff      0xffffffff      0xffffffff      0xffffffff
0x7ffff4a01510: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01520: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01530: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01540: 0xffffffff      0xffffffff      0xffffffff      0xffffffff
0x7ffff4a01550: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01560: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01570: 0x00000000      0x00000000      0x00000000      0x00000000

Is this really the intended behavior?

--

Philipp

[Piotr Balcer]

We do not claim to align user data to a cacheline. The alignas value does not propagate to the memory allocator, and so, the allocation itself cannot be made automatically aligned, even with normal new().

However, we also recognize the need for aligned allocations, hence our allocation class mechanism. It allows you to explicitly request alignment in a way that's more memory-efficient compared to simply

padding structures.

Piotr

[Phil Go]

Thanks Piotr!




For completion with the following snippet, the previous desired alignment is achieved:

struct C {
┆ alignas(64) p<std::array<int, 4>> c1;
┆ alignas(64) p<std::array<int, 4>> c2;
};

[...]

auto C_alloc = pop.ctl_set<struct pobj_alloc_class_desc>(
┆ ┆ "heap.alloc_class.new.desc", {64, 64, 2, POBJ_HEADER_COMPACT, 0});
transaction::run(pop, [&] {
┆ ┆ pop.root()->c = make_persistent<C>(allocation_flag::class_id(C_alloc.class_id));
});

The dump:

0x7ffff4a01640: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01650: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01660: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a01670: 0x000000c0      0x00000000      0x79550b68      0xe55edbb3
0x7ffff4a01680: 0xffffffff      0xffffffff      0xffffffff      0xffffffff
0x7ffff4a01690: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a016a0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a016b0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a016c0: 0xffffffff      0xffffffff      0xffffffff      0xffffffff
0x7ffff4a016d0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a016e0: 0x00000000      0x00000000      0x00000000      0x00000000
0x7ffff4a016f0: 0x00000000      0x00000000      0x00000000      0x00000000




--

Philipp


[Igor Chorążewicz]

Hello Philip,

I have just one comment to the code snippet (not related to the alignment). I would suggest using pmem::obj::array instead of p<std::array>. Pmem obj version provides the same interface as std one but allows for more control with respect to snapshotting (p<> will always snapshot the whole array on assignment). You can check the documentation here: https://pmem.io/libpmemobj-cpp/master/doxygen/structpmem_1_1obj_1_1array.html

Best regards,

Igor

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/c2b5fe48-d3fc-4835-8fb7-ac1db71982c5%40googlegroups.com.

[Phil Go]

Hey Igor,

yes, that is a good point, which I also recognized during profiling and benchmarking. Thank you! :-)

Best,

Philipp