7.3.2 may have some memory problems and cause crash, I met "malloc_consolidate(): invalid chunk size" errors

89 views

Skip to first unread message

skycope

unread,

Jun 29, 2023, 5:45:04 AM6/29/23

to beegfs-user

Hi, I have 7.3.2 clusters, there are some problem with malloc, and then cause a deadlock in PThread::signalHandler!

Does someone has any ideas to solve this, thans very much!!

Env

CentOS Linux release 8.3.2011

beegfs-meta.log

"""

(0) Jun29 13:37:19 Worker63 [InodeDirStore.cpp:147] >> Bug: releaseDir requested, but dir not referenced! dirID: 81E8-649D13EE-2E

"""

/var/log/messages

"""

Jun 29 13:37:19 MM-IDC-10-50-1-71 beegfs-meta[2952007]: malloc_consolidate(): invalid chunk size

"""

Trace:

"""

TID 2952081:
#0 0x00007f423885296c __lll_lock_wait_private - /usr/lib64/libc-2.28.so
#1 0x00007f4238857c60 - 1 malloc - /usr/lib64/libc-2.28.so
#2 0x00007f42388dbdf1 - 1 __backtrace_symbols - /usr/lib64/libc-2.28.so
#3 0x00000000006d51c5 - 1 PThread::signalHandler(int) - /opt/beegfs/sbin/beegfs-meta
/usr/src/debug/beegfs-7.3.2-el8.x86_64/common/source/common/threading/PThread.cpp:90:40
#4 0x00007f4238809400 __restore_rt - /usr/lib64/libc-2.28.so
#5 0x00007f423880937f raise - /usr/lib64/libc-2.28.so
#6 0x00007f42387f3db5 - 1 abort - /usr/lib64/libc-2.28.so
#7 0x00007f423884c4e7 - 1 __libc_message - /usr/lib64/libc-2.28.so
#8 0x00007f42388535ec - 1 malloc_printerr - /usr/lib64/libc-2.28.so
#9 0x00007f4238853fb8 - 1 malloc_consolidate - /usr/lib64/libc-2.28.so
#10 0x00007f4238856368 - 1 _int_malloc - /usr/lib64/libc-2.28.so
#11 0x00007f4238857c72 - 1 malloc - /usr/lib64/libc-2.28.so
#12 0x00007f42388dbdf1 - 1 __backtrace_symbols - /usr/lib64/libc-2.28.so
#13 0x000000000057e5bf - 1 LogContext::logBacktrace() - /opt/beegfs/sbin/beegfs-meta
/usr/src/debug/beegfs-7.3.2-el8.x86_64/common/source/common/app/log/LogContext.h:86:46
#14 0x000000000057e5bf - 1 InodeDirStore::releaseDirUnlocked(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) - /opt/beegfs/sbin/beegfs-meta
/usr/src/debug/beegfs-7.3.2-el8.x86_64/meta/source/storage/InodeDirStore.cpp:148:40
#15 0x00000000005801e1 - 1 InodeDirStore::releaseDir(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) - /opt/beegfs/sbin/beegfs-meta
/usr/src/debug/beegfs-7.3.2-el8.x86_64/meta/source/storage/InodeDirStore.cpp:103:22
#16 0x000000000055c904 - 1 MetaStore::releaseDirUnlocked(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) - /opt/beegfs/sbin/beegfs-meta
/usr/src/debug/beegfs-7.3.2-el8.x86_64/meta/source/storage/MetaStore.cpp:54:23
#17 0x000000000055c904 - 1 MetaStore::releaseDir(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) - /opt/beegfs/sbin/beegfs-meta
/usr/src/debug/beegfs-7.3.2-el8.x86_64/meta/source/storage/MetaStore.cpp:49:22