Segmentation fault in memblock_rebuild_state

[Bhaskar Singhal]

I am using pmemkv (https://github.com/pmem/pmemkv) and emulating PM using /dev/shm.

fatal error: unexpected signal during runtime execution

signal SIGSEGV: segmentation violation code=0x1 addr=0x31 pc=0x7fd0823b3fe5

Stack Trace:

#0  0x00007ffff71c4390 in memblock_rebuild_state () from /usr/local/lib/libpmemobj.so.1
#1  0x00007ffff71c1295 in heap_get_bestfit_block () from /usr/local/lib/libpmemobj.so.1
#2  0x00007ffff71c76c9 in palloc_operation () from /usr/local/lib/libpmemobj.so.1
#3  0x00007ffff71c84e9 in pmalloc_construct () from /usr/local/lib/libpmemobj.so.1
#4  0x00007ffff71caf37 in tx_alloc_common.isra.26.part.27.constprop () from /usr/local/lib/libpmemobj.so.1
#5  0x00007ffff71cb008 in tx_alloc_common.isra.26.constprop () from /usr/local/lib/libpmemobj.so.1
#6  0x00007ffff71cc25e in pmemobj_tx_alloc () from /usr/local/lib/libpmemobj.so.1
#7  0x00007ffff7bd5799 in nvml::detail::pp_if_array::type nvml::obj::make_persistent(unsigned long) () from /usr/local/lib/libpmemkv.so
#8  0x00007ffff7bd1556 in pmemkv::KVSlot::set(unsigned char, std::string const&, std::string const&) () from /usr/local/lib/libpmemkv.so
#9  0x00007ffff7bd183d in pmemkv::KVTree::LeafFillSpecificSlot(pmemkv::KVLeafNode*, unsigned char, std::string const&, std::string const&, int) () from /usr/local/lib/libpmemkv.so
#10 0x00007ffff7bcebdf in pmemkv::KVTree::LeafFillSlotForKey(pmemkv::KVLeafNode*, unsigned char, std::string const&, std::string const&) () from /usr/local/lib/libpmemkv.so
#11 0x00007ffff7bd06a4 in pmemkv::KVTree::Put(std::string const&, std::string const&) () from /usr/local/lib/libpmemkv.so
#12 0x00007ffff7bd0e2e in kvtree_put () from /usr/local/lib/libpmemkv.so

CPU Info

root@ubuntu:~# cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 48
model name      : AMD A10 PRO-7350B R6, 10 Compute Cores 4C+6G
stepping        : 1
microcode       : 0x6003106
cpu MHz         : 2096.119
cache size      : 2048 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw xop fma4 tbm arat xsaveopt vmmcall fsgsbase bmi1
bogomips        : 4192.23
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:

OS - root@ubuntu:~# uname -a
Linux ubuntu 3.13.0-126-generic #175-Ubuntu SMP Thu Jul 20 17:33:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

NVML - built from latest sources

root@ubuntu:~/nvml# git describe
1.3-106-g7f81082

It looks like heap corruption, but can't figure out whats causing it. Valgrind doesn't flag anything.

Regards,

Bhaskar

[Piotr Balcer]

Odd...
Does this happen on the first launch of the application? Have the pool been converted from an older layout version?
Does the pool still work on subsequent opens?
Try launching your code with the debug version of the library.

Could you provide a minimal reproduction code?

Piotr

[Bhaskar Singhal]

Does this happen on the first launch of the application? Have the pool been converted from an older layout version?

--->Yes, this happens on the first launch of the application.

Does the pool still work on subsequent opens?

--->Yes, for a while, before crashing again due to some double free seg fault 

Try launching your code with the debug version of the library.

--->Sure. I did that earlier too, but didn't capture any details.

Could you provide a minimal reproduction code?

---> I am working on it. Meanwhile just to add, I am using pmemkv (https://github.com/pmem/pmemkv) and both key and value are base64 encoded. Value sizes can go upto 20+KB and can be zero too.

Thanks,

Bhaskar

[Bhaskar Singhal]

Adding some more information 

With the debug build:

Program received signal SIGSEGV, Segmentation fault.

[Switching to Thread 0x7fffedffb700 (LWP 16503)]

0x00007ffff719b800 in memblock_header_type (m=0x7fffedffa330) at memblock.c:76

76              if (hdr->flags & CHUNK_FLAG_COMPACT_HEADER)

(gdb) bt

#0  0x00007ffff719b800 in memblock_header_type (m=0x7fffedffa330) at memblock.c:76

#1  0x00007ffff719ca2f in memblock_rebuild_state (heap=0x7fffcfe01a38, m=0x7fffedffa330) at memblock.c:797

#2  0x00007ffff7195bb1 in heap_recycle_block (heap=0x7fffcfe01a38, b=0x7fffe00851d0, m=0x7fffedffa4e0, units=1) at heap.c:884

#3  0x00007ffff7195ce8 in heap_get_bestfit_block (heap=0x7fffcfe01a38, b=0x7fffe00851d0, m=0x7fffedffa4e0) at heap.c:916

#4  0x00007ffff71a3d74 in palloc_operation (heap=0x7fffcfe01a38, off=0, dest_off=0x7fffcfe2c8a8, size=62, constructor=0x7ffff71a943b <constructor_tx_alloc>, arg=0x7fffedffa8e0,

    extra_field=10959529184379665549, object_flags=0, class_id=0, ctx=0x7fffedffa650) at palloc.c:218

#5  0x00007ffff71a4974 in pmalloc_operation (heap=0x7fffcfe01a38, off=0, dest_off=0x7fffcfe2c8a8, size=62, constructor=0x7ffff71a943b <constructor_tx_alloc>, arg=0x7fffedffa8e0,

    extra_field=10959529184379665549, object_flags=0, class_id=0, ctx=0x7fffedffa650) at pmalloc.c:116

#6  0x00007ffff71a4b6e in pmalloc_construct (pop=0x7fffcfe00000, off=0x7fffcfe2c8a8, size=62, constructor=0x7ffff71a943b <constructor_tx_alloc>, arg=0x7fffedffa8e0,

    extra_field=10959529184379665549, object_flags=0, class_id=0) at pmalloc.c:166

#7  0x00007ffff71ab104 in tx_alloc_common (tx=0x7fffedffb608, size=62, type_num=10959529184379665549, constructor=0x7ffff71a943b <constructor_tx_alloc>, flags=0) at tx.c:1010

#8  0x00007ffff71ad9aa in pmemobj_tx_alloc (size=62, type_num=10959529184379665549) at tx.c:1961

#9  0x00007ffff7bcfa55 in nvml::obj::make_persistent<char []> (N=62) at /usr/local/include/libpmemobj++/make_persistent_array.hpp:82

#10 0x00007ffff7bcaab1 in pmemkv::KVSlot::set (this=0x7fffd9181350, hash=159 '\237', key="cgAAAAAAAE+CpfxuTY+TZ3wN60ClzhDOW0kTnR6A7ZHJHmCD8bFI22k=", value="wA==") at /root/pmemkv/src/pmemkv.cc:520

#11 0x00007ffff7bc9627 in pmemkv::KVTree::LeafFillSpecificSlot (this=0x7fffe0000a80, leafnode=0x7fffb81c76c0, hash=159 '\237', key="cgAAAAAAAE+CpfxuTY+TZ3wN60ClzhDOW0kTnR6A7ZHJHmCD8bFI22k=",

    value="wA==", slot=28) at /root/pmemkv/src/pmemkv.cc:292

#12 0x00007ffff7bc937b in pmemkv::KVTree::__lambda2::operator() (__closure=0x7fffb024a420) at /root/pmemkv/src/pmemkv.cc:280

#13 0x00007ffff7bcb8fe in std::_Function_handler<void(), pmemkv::KVTree::LeafFillSlotForKey(pmemkv::KVLeafNode*, uint8_t, const string&, const string&)::__lambda2>::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/4.8/functional:2071

#14 0x00007ffff7bd039c in std::function<void ()>::operator()() const (this=0x7fffedffabb0) at /usr/include/c++/4.8/functional:2471

#15 0x00007ffff7bce7e0 in nvml::obj::transaction::exec_tx<>(nvml::obj::pool_base&, std::function<void ()>) (pool=..., tx=...) at /usr/local/include/libpmemobj++/transaction.hpp:417

#16 0x00007ffff7bc952c in pmemkv::KVTree::LeafFillSlotForKey (this=0x7fffe0000a80, leafnode=0x7fffb81c76c0, hash=159 '\237', key="cgAAAAAAAE+CpfxuTY+TZ3wN60ClzhDOW0kTnR6A7ZHJHmCD8bFI22k=",

    value="wA==") at /root/pmemkv/src/pmemkv.cc:281

#17 0x00007ffff7bc8d3d in pmemkv::KVTree::Put (this=0x7fffe0000a80, key="cgAAAAAAAE+CpfxuTY+TZ3wN60ClzhDOW0kTnR6A7ZHJHmCD8bFI22k=", value="wA==") at /root/pmemkv/src/pmemkv.cc:179

#18 0x00007ffff7bcaf0e in pmemkv::kvtree_put (kv=0x7fffe0000a80, key=0x7fffb025a490 "cgAAAAAAAE+CpfxuTY+TZ3wN60ClzhDOW0kTnR6A7ZHJHmCD8bFI22k=", value=0x7fffb001f8e0 "wA==",

    valuebytes=0x7fffb023e8b0) at /root/pmemkv/src/pmemkv.cc:571

(gdb) p hdr

$1 = (struct chunk_header *) 0x8918d245f401ee50

(gdb) p hdr->flags

Cannot access memory at address 0x8918d245f401ee52

(gdb) f 1

#1  0x00007ffff719ca2f in memblock_rebuild_state (heap=0x7fffcfe01a38, m=0x7fffedffa330) at memblock.c:797

797             m->header_type = memblock_header_type(m);

(gdb) p m

$11 = (struct memory_block *) 0x7fffedffa330

(gdb) p *m

$12 = {chunk_id = 1799829826, zone_id = 1648910915, size_idx = 1147429475, block_off = 24922, header_type = HEADER_LEGACY, type = MEMORY_BLOCK_HUGE, m_ops = 0x0, heap = 0x7fffcfe01a38}

(gdb) f 0

#0  0x00007ffff719b800 in memblock_header_type (m=0x7fffedffa330) at memblock.c:76

76              if (hdr->flags & CHUNK_FLAG_COMPACT_HEADER)

(gdb) list

71      memblock_header_type(const struct memory_block *m)

72      {

73              struct zone *z = ZID_TO_ZONE(m->heap->layout, m->zone_id);

74              struct chunk_header *hdr = &z->chunk_headers[m->chunk_id];

75

76              if (hdr->flags & CHUNK_FLAG_COMPACT_HEADER)

77                      return HEADER_COMPACT;

78

79              if (hdr->flags & CHUNK_FLAG_HEADER_NONE)

80                      return HEADER_NONE;

(gdb) p m->heap->layout

$13 = (struct heap_layout *) 0x7fffd0102000

(gdb) p m->zone_id

$14 = 1648910915

(gdb) p z

$15 = (struct zone *) 0x8918d24299c82400

(gdb) p hdr

$16 = (struct chunk_header *) 0x8918d245f401ee50

(gdb) f 1

#1  0x00007ffff719ca2f in memblock_rebuild_state (heap=0x7fffcfe01a38, m=0x7fffedffa330) at memblock.c:797

797             m->header_type = memblock_header_type(m);

(gdb) p heap

$17 = (struct palloc_heap *) 0x7fffcfe01a38

(gdb) p *heap

$18 = {p_ops = {persist = 0x7ffff719db58 <obj_norep_persist>, flush = 0x7ffff719dbd2 <obj_norep_flush>, drain = 0x7ffff719dc4c <obj_norep_drain>,

    memcpy_persist = 0x7ffff719da4c <obj_norep_memcpy_persist>, memset_persist = 0x7ffff719dad4 <obj_norep_memset_persist>, base = 0x7fffcfe00000, pool_size = 268435456, remote = {read = 0x0,

      ctx = 0x0, base = 0}}, layout = 0x7fffd0102000, rt = 0x7fffe001f330, size = 265281536, run_id = 2, base = 0x7fffcfe00000}

(gdb) p *m

$19 = {chunk_id = 1799829826, zone_id = 1648910915, size_idx = 1147429475, block_off = 24922, header_type = HEADER_LEGACY, type = MEMORY_BLOCK_HUGE, m_ops = 0x0, heap = 0x7fffcfe01a38}

(gdb) f 2

#2  0x00007ffff7195bb1 in heap_recycle_block (heap=0x7fffcfe01a38, b=0x7fffe00851d0, m=0x7fffedffa4e0, units=1) at heap.c:884

884                     memblock_rebuild_state(heap, &r);

(gdb) list

879                     ASSERT(units <= UINT16_MAX);

880                     ASSERT(m->block_off + units <= UINT16_MAX);

881                     struct memory_block r = {m->chunk_id, m->zone_id,

882                             m->size_idx - units, (uint16_t)(m->block_off + units),

883                             0, 0, NULL, NULL};

884                     memblock_rebuild_state(heap, &r);

885                     bucket_insert_block(b, &r);

886             } else {

887                     heap_resize_chunk(heap, b, m->chunk_id, m->zone_id, units);

888             }

(gdb) p r

$20 = {chunk_id = 1799829826, zone_id = 1648910915, size_idx = 1147429475, block_off = 24922, header_type = HEADER_LEGACY, type = MEMORY_BLOCK_HUGE, m_ops = 0x0, heap = 0x7fffcfe01a38}

(gdb) p m

$21 = (struct memory_block *) 0x7fffedffa4e0

(gdb) p *m

$22 = {chunk_id = 1799829826, zone_id = 1648910915, size_idx = 1147429476, block_off = 24921, header_type = 859207267, type = (MEMORY_BLOCK_RUN | unknown: 894062948), m_ops = 0x346949477737796c,

  heap = 0x3d}

(gdb) p m->size_idx - units

$23 = 1147429475

(gdb) p units

$24 = 1

(gdb) f 1

#1  0x00007ffff719ca2f in memblock_rebuild_state (heap=0x7fffcfe01a38, m=0x7fffedffa330) at memblock.c:797

797             m->header_type = memblock_header_type(m);

(gdb) list

792      */

793     void

794     memblock_rebuild_state(struct palloc_heap *heap, struct memory_block *m)

795     {

796             m->heap = heap;

797             m->header_type = memblock_header_type(m);

798             m->type = memblock_detect_type(m, heap->layout);

799             m->m_ops = &mb_ops[m->type];

800     }

I am trying to come up with minimal reproduction code, but till now not able to reproduce the failure with some simple cases.

[Piotr Balcer]

It's difficult for me to judge from the limited data... But I'd say something is overwriting the allocator metadata.
If you have difficulty creating minimal repro, you can just send me the larger application.
The corrupted pool would help too.

Piotr