The kernel module xt_RTPENGINE caused rtpengine-recording to become unresponsive, resulting in zombie processes and ultimately leading to a complete system freeze

[YoungLLL Chan]

rtpengine version the issue has been seen with

Version: 13.4.1.7+0~mr13.4.1.7

Used distribution and its version

Debian GNU/Linux 12 (bookworm)

Linux kernel version used

6.1.0-38-amd64  /  6.1.0-40-amd64  /  6.1.0-42-amd64

CPU architecture issue was seen on (see uname -m)

x86_64

Expected behaviour you didn't see

rtpengine-recording working fine

Unexpected behaviour you saw

system freeze, rtpengine-recording no response and has defunct process:
root@debian:/var/log# ps -ef|grep rtpengine
rtpengi+ 2280049 1 6 Feb11 ? 00:59:24 [rtpengine-recor]
rtpengi+ 2280320 1 5 Feb11 ? 00:48:43 /usr/bin/rtpengine -f --no-log-timestamps --pidfile /run/rtpengine/ngcp-rtpengine-daemon.pid --config-file /etc/rtpengine/rtpengine.conf
rtpengi+ 3562584 1 16 14:00 ? 00:04:50 /usr/bin/rtpengine-recording -f --no-log-timestamps --pidfile /run/rtpengine-recording/ngcp-rtpengine-recording-daemon.pid --config-file /etc/rtpengine/rtpengine-recording.conf

Steps to reproduce the problem

randomly

kern.log

Additional program output to the terminal or logs illustrating the issue

kern.log:

Feb 12 13:48:28 debian kernel: [153214.701047] list_del corruption. prev->next should be ffffa0399ac63a28, but was 0000000000000000. (prev=ffffa039deb9f570)
Feb 12 13:48:28 debian kernel: [153214.701156] ------------[ cut here ]------------
Feb 12 13:48:28 debian kernel: [153214.701159] kernel BUG at lib/list_debug.c:59!
Feb 12 13:48:28 debian kernel: [153214.701195] invalid opcode: 0000 [#1] PREEMPT SMP PTI
Feb 12 13:48:28 debian kernel: [153214.701230] CPU: 15 PID: 2280364 Comm: rtpengine-recor Tainted: G           OE      6.1.0-42-amd64 #1  Debian 6.1.159-1
Feb 12 13:48:28 debian kernel: [153214.701271] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 2.19.0 02/22/2024
Feb 12 13:48:28 debian kernel: [153214.701302] RIP: 0010:__list_del_entry_valid.cold+0x37/0x6f
Feb 12 13:48:28 debian kernel: [153214.701345] Code: fe ff 0f 0b 48 89 d1 48 c7 c7 78 91 fa a5 48 89 c2 e8 b1 9b fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 28 91 fa a5 e8 9d 9b fe ff <0f> 0b 48 89 fe 48 89 ca 48 c7 c7 f0 90 fa a5 e8 89 9b fe ff 0f 0b
Feb 12 13:48:28 debian kernel: [153214.701417] RSP: 0018:ffffc34484097bc8 EFLAGS: 00010046
Feb 12 13:48:28 debian kernel: [153214.701443] RAX: 000000000000006d RBX: ffffa0399ac63a10 RCX: 0000000000000000
Feb 12 13:48:28 debian kernel: [153214.701473] RDX: 0000000000000000 RSI: ffffa0409f9e03e0 RDI: ffffa0409f9e03e0
Feb 12 13:48:28 debian kernel: [153214.701501] RBP: ffffa039deb9f568 R08: 0000000000000000 R09: ffffc34484097a60
Feb 12 13:48:28 debian kernel: [153214.701526] R10: 0000000000000003 R11: ffffa040bff807a8 R12: 0000000000000282
Feb 12 13:48:28 debian kernel: [153214.701551] R13: ffffa039b0616e00 R14: 0000000000000000 R15: ffffa039b0616e00
Feb 12 13:48:28 debian kernel: [153214.701577] FS:  00007f34ed5f56c0(0000) GS:ffffa0409f9c0000(0000) knlGS:0000000000000000
Feb 12 13:48:28 debian kernel: [153214.701623] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 12 13:48:28 debian kernel: [153214.701645] CR2: 00007f1f0b414000 CR3: 00000001404fe002 CR4: 00000000003706e0
Feb 12 13:48:28 debian kernel: [153214.701671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 12 13:48:28 debian kernel: [153214.701695] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 12 13:48:28 debian kernel: [153214.701720] Call Trace:
Feb 12 13:48:28 debian kernel: [153214.701733]  <TASK>
Feb 12 13:48:28 debian kernel: [153214.701744]  remove_wait_queue+0x20/0x60
Feb 12 13:48:28 debian kernel: [153214.701778]  ep_remove+0x40/0x1e0
Feb 12 13:48:28 debian kernel: [153214.701811]  do_epoll_ctl+0x888/0xf50
Feb 12 13:48:28 debian kernel: [153214.701844]  ? __x64_sys_epoll_wait+0x6f/0x110
Feb 12 13:48:28 debian kernel: [153214.701880]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 12 13:48:28 debian kernel: [153214.701920]  __x64_sys_epoll_ctl+0x6c/0xa0
Feb 12 13:48:28 debian kernel: [153214.701953]  do_syscall_64+0x55/0xb0
Feb 12 13:48:28 debian kernel: [153214.701987]  ? do_futex+0xda/0x1b0
Feb 12 13:48:28 debian kernel: [153214.702018]  ? __x64_sys_futex+0x8e/0x1d0
Feb 12 13:48:28 debian kernel: [153214.702052]  ? exit_to_user_mode_prepare+0x44/0x240
Feb 12 13:48:28 debian kernel: [153214.702091]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 12 13:48:28 debian kernel: [153214.702126]  ? do_syscall_64+0x61/0xb0
Feb 12 13:48:28 debian kernel: [153214.702159]  ? __rseq_handle_notify_resume+0xa9/0x480
Feb 12 13:48:28 debian kernel: [153214.702198]  ? do_futex+0x106/0x1b0
Feb 12 13:48:28 debian kernel: [153214.702229]  ? exit_to_user_mode_prepare+0x16b/0x240
Feb 12 13:48:28 debian kernel: [153214.702268]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 12 13:48:28 debian kernel: [153214.702304]  ? do_syscall_64+0x61/0xb0
Feb 12 13:48:28 debian kernel: [153214.702336]  ? ksys_read+0xd4/0xf0
Feb 12 13:48:28 debian kernel: [153214.702369]  ? exit_to_user_mode_prepare+0x44/0x240
Feb 12 13:48:28 debian kernel: [153214.702407]  ? syscall_exit_to_user_mode+0x1e/0x40
Feb 12 13:48:28 debian kernel: [153214.702442]  ? do_syscall_64+0x61/0xb0
Feb 12 13:48:28 debian kernel: [153214.702474]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Feb 12 13:48:28 debian kernel: [153214.702514] RIP: 0033:0x7f34fe727c0a
Feb 12 13:48:28 debian kernel: [153214.703339] Code: 48 8b 0d f9 91 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 e9 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c6 91 0c 00 f7 d8 64 89 01 48
Feb 12 13:48:28 debian kernel: [153214.705077] RSP: 002b:00007f34ed5f3f38 EFLAGS: 00000246 ORIG_RAX: 00000000000000e9
Feb 12 13:48:28 debian kernel: [153214.705821] RAX: ffffffffffffffda RBX: ffffffffffffffff RCX: 00007f34fe727c0a
Feb 12 13:48:28 debian kernel: [153214.706561] RDX: 00000000000000bf RSI: 0000000000000002 RDI: 0000000000000004
Feb 12 13:48:28 debian kernel: [153214.707312] RBP: 00007f34bc086b40 R08: 0000000000000001 R09: 00007f34bc000840
Feb 12 13:48:28 debian kernel: [153214.708055] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f34d006ca40
Feb 12 13:48:28 debian kernel: [153214.708783] R13: 0000000000000000 R14: 00007ffce922ad00 R15: 00007f34ecdf5000
Feb 12 13:48:28 debian kernel: [153214.709483]  </TASK>
Feb 12 13:48:28 debian kernel: [153214.710178] Modules linked in: xt_RTPENGINE(OE) xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_set nft_compat nf_tables libcrc32c ip_set_hash_net ip_set nfnetlink sunrpc binfmt_misc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common sb_edac nls_ascii nls_cp437 vfat x86_pkg_temp_thermal intel_powerclamp fat ipmi_ssif coretemp kvm_intel kvm irqbypass ghash_clmulni_intel sha512_ssse3 sha512_generic sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd mgag200 rapl intel_cstate ledtrig_audio dell_smbios drm_shmem_helper dell_wmi_descriptor iTCO_wdt intel_pmc_bxt drm_kms_helper intel_uncore ipmi_si rfkill mei_me ipmi_devintf video iTCO_vendor_support evdev mei dcdbas i2c_algo_bit watchdog pcspkr mxm_wmi sg ipmi_msghandler acpi_power_meter button drm configfs loop fuse efi_pstore efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_mod sd_mod t10_pi crc64_rocksoft sr_mod crc64 crc_t10dif cdrom crct10dif_generic ahci
Feb 12 13:48:28 debian kernel: [153214.710287]  libahci ehci_pci ehci_hcd crct10dif_pclmul libata megaraid_sas tg3 crct10dif_common crc32_pclmul usbcore crc32c_intel scsi_mod libphy lpc_ich usb_common scsi_common wmi [last unloaded: xt_RTPENGINE(OE)]
Feb 12 13:48:28 debian kernel: [153214.717857] ---[ end trace 0000000000000000 ]---
Feb 12 13:48:28 debian kernel: [153214.776440] RIP: 0010:__list_del_entry_valid.cold+0x37/0x6f
Feb 12 13:48:28 debian kernel: [153214.777353] Code: fe ff 0f 0b 48 89 d1 48 c7 c7 78 91 fa a5 48 89 c2 e8 b1 9b fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 28 91 fa a5 e8 9d 9b fe ff <0f> 0b 48 89 fe 48 89 ca 48 c7 c7 f0 90 fa a5 e8 89 9b fe ff 0f 0b
Feb 12 13:48:28 debian kernel: [153214.779120] RSP: 0018:ffffc34484097bc8 EFLAGS: 00010046
Feb 12 13:48:28 debian kernel: [153214.780020] RAX: 000000000000006d RBX: ffffa0399ac63a10 RCX: 0000000000000000
Feb 12 13:48:28 debian kernel: [153214.780880] RDX: 0000000000000000 RSI: ffffa0409f9e03e0 RDI: ffffa0409f9e03e0
Feb 12 13:48:28 debian kernel: [153214.781761] RBP: ffffa039deb9f568 R08: 0000000000000000 R09: ffffc34484097a60
Feb 12 13:48:28 debian kernel: [153214.782632] R10: 0000000000000003 R11: ffffa040bff807a8 R12: 0000000000000282
Feb 12 13:48:28 debian kernel: [153214.783520] R13: ffffa039b0616e00 R14: 0000000000000000 R15: ffffa039b0616e00
Feb 12 13:48:28 debian kernel: [153214.784424] FS:  00007f34ed5f56c0(0000) GS:ffffa0409f9c0000(0000) knlGS:0000000000000000
Feb 12 13:48:28 debian kernel: [153214.785327] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 12 13:48:28 debian kernel: [153214.786251] CR2: 00007f1f0b414000 CR3: 00000001404fe002 CR4: 00000000003706e0
Feb 12 13:48:28 debian kernel: [153214.787177] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 12 13:48:28 debian kernel: [153214.788103] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 12 13:48:28 debian kernel: [153214.789025] note: rtpengine-recor[2280364] exited with irqs disabled
Feb 12 13:48:28 debian kernel: [153214.789995] note: rtpengine-recor[2280364] exited with preempt_count 1