seeing core in erlang (erl.exe) executable while shutting down rabbitMQ(3.8.14) on windows
Pooja Desai
I am seeing core in erlang (erl.exe) executable while shutting down rabbitMQ(3.8.14) on windows
Information from crash dump (including Stack trace) as below:
=====================STACK TEXT=====================================
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
.exr 0xffffffffffffffff
ExceptionAddress: 00007fff0ab64520 (beam_smp+0x0000000000004520)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000000000
Attempt to read from address 0000000000000000
FAULTING_THREAD: 00000000000002dc
PROCESS_NAME: erl.exe
ADDITIONAL_DEBUG_TEXT:
Use '!findthebuild' command to search for the target build information.
If the build information is available, run '!findthebuild -s ; .reload' to set symbol path and load symbols.
FAULTING_MODULE: 00007fff2c5b0000 ntdll
DEBUG_FLR_IMAGE_TIMESTAMP: 6070400f
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: 0000000000000000
READ_ADDRESS: 0000000000000000
FOLLOWUP_IP:
beam_smp+4520
00007fff`0ab64520 488b11 mov rdx,qword ptr [rcx]
MOD_LIST: <ANALYSIS/>
BUGCHECK_STR: APPLICATION_FAULT_NULL_POINTER_READ_WRONG_SYMBOLS
PRIMARY_PROBLEM_CLASS: NULL_POINTER_READ
DEFAULT_BUCKET_ID: NULL_POINTER_READ
LAST_CONTROL_TRANSFER: from 00007fff0ab691d9 to 00007fff0ab64520
STACK_TEXT:
0000003e`8f43f420 00007fff`0ab691d9 : 0000003e`8d699340 0000003e`91705610 0000003e`925135f0 0000003e`8d591dc0 : beam_smp+0x4520
0000003e`8f43f4c0 00007fff`0ab8e2b7 : 0000003e`8df802c0 0000003e`8d442240 00000000`00000000 00007fff`0ad803bd : beam_smp+0x91d9
0000003e`8f43f6c0 00007fff`0ad81f2d : 0000003e`8cc1dbd0 00000000`00000000 0000003e`8cc1dbd0 00000000`00000000 : beam_smp+0x2e2b7
0000003e`8f43f720 00007fff`19dfcd70 : 00000000`00000000 0000003e`8df801c0 00000000`00000000 00000000`00000000 : beam_smp!sys_primitive_init+0x5bb1d
0000003e`8f43f770 00007fff`2ba713d2 : 00000000`00000000 00007fff`19dfcd30 00000000`00000000 00000000`00000000 : ucrtbase!o__realloc_base+0x60
0000003e`8f43f7a0 00007fff`2c5c54f4 : 00007fff`2ba713b0 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x22
0000003e`8f43f7d0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x34
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: beam_smp+4520
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: beam_smp
STACK_COMMAND: ~4s ; kb
BUCKET_ID: WRONG_SYMBOLS
IMAGE_NAME: C:\PROGRA~1\xxxxxxx\erlang\erts-11.2\bin\beam.smp.dll
=====================STACK TEXT=====================================
Additional information:
I built erlang with below mentioned environment:
Packages Version used while building
Windows OS - Microsoft Windows Server 2016 Standard
Microsoft Visual Studio - 2019 (14.27.29110)
Microsoft Windows SDK - SDK 10 (10.0.18362.0)
msys (64 bit) - msys2-x86_64-20180531
NSIS - 3.06.1
OpenSSL - 1.0.2u - fips capable, loaded dynamically
JDK & wxWidget - not installed
Points to note:
* On Unix platforms we are not seeing any issue with the same build & similar environment.
Steps referred to build erlang referred from: https://erlang.org/doc/installation_guide/INSTALL-WIN32.html
Also, note that OTP 23.1 built with same steps & configuration is not showing this above described crash.
Please let me know if anyone has seen a similar issue in the past or if someone could provide me any pointers to solve this.
Pooja Desai
Hi,
Investigated this issue further by using multiple versions of Erlang and realised that crash dump is seen when Erlang version >= 23.2 and built with –enable-fips option. Crash is not generated if –enable-fips option is not used while building erlang.
After scanning code found below commits added into OTP 23.2 which are causing coredumps on windows:
https://github.com/erlang/otp/commit/392e80cd2aad6dfd4e503986f66cd10f15f9cbbb
https://github.com/erlang/otp/commit/2a954fe593e513774632abfe04aab303bff4e5f9
Also confirmed that coredump is not generated for OPT 23.2 after reverting above commits.
Note that no issue is seen on LinuxR platform for the same erlang OTP build with FIPS option. This issue is seen only for the Windows platform while shutting down RabbitMQ(3.8.14).
Environment and other details :
Erlang version : 23.2 or greater.
Referred steps to build erlang: https://erlang.org/doc/installation_guide/INSTALL-WIN32.html
Erlang is built locally from the src package with FIPS mode enabled.
Build details:
Package Version used while building
Windows OS Microsoft Windows Server 2016 Standard
Microsoft Visual Studio 2019 (14.27.29110)
Microsoft Windows SDK SDK 10 (10.0.18362.0)
msys (64 bit) msys2-x86_64-20180531
NSIS 3.06.1
OpenSSL 1.0.2u - fips capable, loaded dynamically
JDK & wxWidget not installed
Steps referred to build erlang is given below =>
./otp_build configure --enable-fips --enable-dynamic-ssl-lib --with-ssl=${ERL_TOP}/openssl_libs --without-javac --without-odbc
./otp_build boot -a
./otp_build release -a
./otp_build installer_win32
Stack trace from Coredump
=====================
CONTEXT: (.ecxr)
rax=0000000000000000 rbx=000000820ec4bd20 rcx=0000000000000000
rdx=0000000000000000 rsi=0000008209facd30 rdi=000000000001330b
rip=00007ffe120a4520 rsp=000000820b99f570 rbp=00000082099801c0
r8=0000008209facdc8 r9=0000000000000000 r10=0000000000000000
r11=00007ffe123abad0 r12=00007ffe120aa8aa r13=00007ffe120a9e5c
r14=00007ffe120a9e0b r15=0000008209facd50
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010244
beam_smp!build_stacktrace+0xc40:
00007ffe`120a4520 488b11 mov rdx,qword ptr [rcx] ds:00000000`00000000=????????????????
Resetting default scope
FAULTING_IP:
beam_smp!build_stacktrace+c40
00007ffe`120a4520 488b11 mov rdx,qword ptr [rcx]
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffe120a4520 (beam_smp!build_stacktrace+0x0000000000000c40)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000000000
Attempt to read from address 0000000000000000
DEFAULT_BUCKET_ID: NULL_POINTER_READ
PROCESS_NAME: erl.exe
FOLLOWUP_IP:
beam_smp!build_stacktrace+c40
00007ffe`120a4520 488b11 mov rdx,qword ptr [rcx]
READ_ADDRESS: 0000000000000000
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE_STR: c0000005
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: 0000000000000000
WATSON_BKT_PROCSTAMP: 60a268ce
WATSON_BKT_MODULE: beam.smp.dll
WATSON_BKT_MODSTAMP: 60a268a0
WATSON_BKT_MODOFFSET: 4520
BUILD_VERSION_STRING: 6.3.9600.17415 (winblue_r4.141028-1500)
MODLIST_WITH_TSCHKSUM_HASH: d0cb9d39af83f52d748e989a570501f95cbf1958
MODLIST_SHA1_HASH: 003194b5185591c6e4bed4b4f5bf621b23cda03b
NTGLOBALFLAG: 0
PROCESS_BAM_CURRENT_THROTTLED: 0
PROCESS_BAM_PREVIOUS_THROTTLED: 0
APPLICATION_VERIFIER_FLAGS: 0
PRODUCT_TYPE: 3
SUITE_MASK: 400
DUMP_FLAGS: 8000c07
DUMP_TYPE: 3
ANALYSIS_SESSION_HOST: IGNITERS-14
ANALYSIS_SESSION_TIME: 05-21-2021 13:21:37.0090
ANALYSIS_VERSION: 10.0.18362.1 amd64fre
THREAD_ATTRIBUTES:
OS_LOCALE: ENU
BUGCHECK_STR: APPLICATION_FAULT_NULL_POINTER_READ_INVALID_POINTER_READ
PRIMARY_PROBLEM_CLASS: APPLICATION_FAULT
PROBLEM_CLASSES:
ID: [0n313]
Type: [@ACCESS_VIOLATION]
Class: Addendum
Scope: BUCKET_ID
Name: Omit
Data: Omit
PID: [Unspecified]
TID: [0x19e4]
Frame: [0] : beam_smp!build_stacktrace
ID: [0n285]
Type: [INVALID_POINTER_READ]
Class: Primary
Scope: BUCKET_ID
Name: Add
Data: Omit
PID: [Unspecified]
TID: [0x19e4]
Frame: [0] : beam_smp!build_stacktrace
ID: [0n300]
Type: [NULL_POINTER_READ]
Class: Primary
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
Name: Add
Data: Omit
PID: [0x1a7c]
TID: [0x19e4]
Frame: [0] : beam_smp!build_stacktrace
LAST_CONTROL_TRANSFER: from 00007ffe120a91d9 to 00007ffe120a4520
STACK_TEXT:
00 00000082`0b99f570 00007ffe`120a91d9 beam_smp!build_stacktrace+0xc40
01 00000082`0b99f610 00007ffe`120ce1e7 beam_smp!process_main+0x47e9
02 00000082`0b99f810 00007ffe`122c0cbd beam_smp!sched_thread_func+0x127 [C:\msys64\XXX\otp-OTP-23.2\erts\emulator\beam\erl_process.c @ 8489]
03 00000082`0b99f870 00007ffe`15c8cd70 beam_smp!thr_wrapper+0xbd [C:\msys64\XXX\otp-OTP-23.2\erts\lib_src\win\ethread.c @ 131]
04 00000082`0b99f8c0 00007ffe`282c13d2 ucrtbase!o__realloc_base+0x60
05 00000082`0b99f8f0 00007ffe`2a9e54f4 kernel32!BaseThreadInitThunk+0x22
06 00000082`0b99f920 00000000`00000000 ntdll!RtlUserThreadStart+0x34
THREAD_SHA1_HASH_MOD_FUNC: 44df48f9269e0103c617b0eb5ee60a69f3e53b8f
THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 3ab49c0509697401705cdf18f8261c0895f8bbea
THREAD_SHA1_HASH_MOD: 6d5efadb52d634f51bb798273f8ae741ccf8d747
FAULT_INSTR_CODE: 49118b48
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: beam_smp!build_stacktrace+c40
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: beam_smp
IMAGE_NAME: beam.smp.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 60a268a0
STACK_COMMAND: ~4s ; .ecxr ; kb
FAILURE_BUCKET_ID: NULL_POINTER_READ_c0000005_beam.smp.dll!build_stacktrace
BUCKET_ID: APPLICATION_FAULT_NULL_POINTER_READ_INVALID_POINTER_READ_beam_smp!build_stacktrace+c40
FAILURE_EXCEPTION_CODE: c0000005
FAILURE_IMAGE_NAME: beam.smp.dll
=====================
Please look into this issue and provide your suggestions as I am not able to move forward.
Thanks,
Pooja
Mikael Pettersson
> Please look into this issue and provide your suggestions as I am not able to move forward.
https://github.com/erlang/otp/issues