Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Bug#966846: Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb

164 views
Skip to first unread message

Cédric Dufour

unread,
Aug 3, 2020, 4:10:02 AM8/3/20
to
Package: linux-source-4.19
Version: 4.19.132-1
Severity: important

Hello,

Since linux-image-4.19.0-10-amd64, I'm facing regular Kernel panics - "RIP: 0010:__cgroup_bpf_run_filter_skb+0x26d/0x3d0" - resulting in full (file) *server freeze*.

The issue is pretty well described and summarized in https://forum.proxmox.com/threads/kernel-5-4-44-causes-system-freeze-on-hp-microserver-gen8.72050/page-2#post-323498

The "culprit" commit - "netprio_cgroup: Fix unlimited memory leak of v2 cgroups" - is indeed included in Debian kernel (4.19) since changelog entry 4.19.131-1

It *seems* there is already a patch proposed upstream (although here for kernel 4.9): https://lkml.org/lkml/2020/7/20/883

Best regards,

Cédric

--
Cédric Dufour

Shane Frasier

unread,
Aug 10, 2020, 4:00:03 PM8/10/20
to
FWIW, I am seeing a very similar issue.  Some Debian 10 AWS instances used to run Guacamole via Docker recently started randomly freezing up on me.  I enabled kernel dumps and finally caught one of the machines misbehaving.  Looking at the kdump I see this:
      KERNEL: /usr/lib/debug/vmlinux-4.19.0-10-cloud-amd64
    DUMPFILE: dump.202008101612  [PARTIAL DUMP]
        CPUS: 2
        DATE: Mon Aug 10 16:11:47 2020
      UPTIME: 00:05:44
LOAD AVERAGE: 0.21, 0.11, 0.04
       TASKS: 261
    NODENAME: guac.env0.staging.cool.cyber.dhs.gov
     RELEASE: 4.19.0-10-cloud-amd64
     VERSION: #1 SMP Debian 4.19.132-1 (2020-07-24)
     MACHINE: x86_64  (2499 Mhz)
      MEMORY: 4 GB
       PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000010"
         PID: 1453
     COMMAND: "sshd"
        TASK: ffff8a3f695115c0  [THREAD_INFO: ffff8a3f695115c0]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 1453   TASK: ffff8a3f695115c0  CPU: 0   COMMAND: "sshd"
 #0 [ffffb37740c77800] machine_kexec at ffffffff97a4b297
 #1 [ffffb37740c77858] __crash_kexec at ffffffff97b0e7dd
 #2 [ffffb37740c77920] crash_kexec at ffffffff97b0f62d
 #3 [ffffb37740c77938] oops_end at ffffffff97a2907d
 #4 [ffffb37740c77958] no_context at ffffffff97a5858e
 #5 [ffffb37740c779b0] __do_page_fault at ffffffff97a58c42
 #6 [ffffb37740c77a20] async_page_fault at ffffffff982010be
    [exception RIP: __cgroup_bpf_run_filter_skb+189]
    RIP: ffffffff97b94ffd  RSP: ffffb37740c77ad0  RFLAGS: 00010286
    RAX: 0000000000000000  RBX: ffff8a3ff55e5ee8  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: ffff8a3ff3d49800  RDI: ffff8a3ff52fd500
    RBP: ffff8a3ff52fd500   R8: ffff8a3ff55e5ee8   R9: 0000000000010000
    R10: 0000000000000001  R11: ffff8a3ef6dd7500  R12: 0000000000000000
    R13: 0000000000000000  R14: ffff8a3ff52fd840  R15: ffff8a3ff55e5ee8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffffb37740c77b30] ip_finish_output at ffffffff97f65988
 #8 [ffffb37740c77b68] ip_output at ffffffff97f6640c
 #9 [ffffb37740c77bc0] __ip_queue_xmit at ffffffff97f65e6d
#10 [ffffb37740c77c18] __tcp_transmit_skb at ffffffff97f80557
#11 [ffffb37740c77c88] tcp_write_xmit at ffffffff97f81e34
#12 [ffffb37740c77cf0] __tcp_push_pending_frames at ffffffff97f82ae1
#13 [ffffb37740c77d00] tcp_sendmsg_locked at ffffffff97f733ac
#14 [ffffb37740c77da8] tcp_sendmsg at ffffffff97f73507
#15 [ffffb37740c77dc8] sock_sendmsg at ffffffff97ee8aa6
#16 [ffffb37740c77de0] sock_write_iter at ffffffff97ee8b47
#17 [ffffb37740c77e50] new_sync_write at ffffffff97c49bfb
#18 [ffffb37740c77ed0] vfs_write at ffffffff97c4c7d5
#19 [ffffb37740c77f00] ksys_write at ffffffff97c4ca77
#20 [ffffb37740c77f38] do_syscall_64 at ffffffff97a04140
#21 [ffffb37740c77f50] entry_SYSCALL_64_after_hwframe at ffffffff98200088
    RIP: 00007fd74beba504  RSP: 00007ffc1d456638  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000084  RCX: 00007fd74beba504
    RDX: 0000000000000084  RSI: 000055785f33bb90  RDI: 0000000000000003
    RBP: 000055785f31d630   R8: 0000000000000000   R9: 0000000000001000
    R10: 0000000000000008  R11: 0000000000000246  R12: 00000000000001dd
    R13: 000055785ddc9b00  R14: 0000000000000003  R15: 00007ffc1d4566e0
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b

crash> sym ffffffff97b94ffd
ffffffff97b94ffd (T) __cgroup_bpf_run_filter_skb+189 ./debian/build/build_amd64_none_cloud-amd64/./kernel/bpf/cgroup.c: 539

crash> log
[    0.000000] Linux version 4.19.0-10-cloud-amd64 (debian...@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.132-1 (2020-07-24)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.19.0-10-cloud-amd64 root=UUID=9ac8f5bd-5b64-48cd-9efd-2b2d35a30500 ro console=tty0 console=ttyS0,115200 earlyprintk=ttyS0,115200 nmi_watchdog=1 elevator=noop scsi_mod.use_blk_mq=Y crashkernel=384M-:128M
<SNIP>
[  478.686368] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[  478.693551] PGD 0 P4D 0
[  478.696291] Oops: 0000 [#1] SMP PTI
[  478.699431] CPU: 0 PID: 1453 Comm: sshd Kdump: loaded Not tainted 4.19.0-10-cloud-amd64 #1 Debian 4.19.132-1
[  478.706782] Hardware name: Amazon EC2 t3.medium/, BIOS 1.0 10/16/2017
[  478.711129] RIP: 0010:__cgroup_bpf_run_filter_skb+0xbd/0x1e0
[  478.715172] Code: 00 00 00 49 89 7f 18 48 89 0c 24 44 89 e1 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 b0 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 01 01 00 00 4d 8d 6f 30 bd
[  478.727711] RSP: 0018:ffffb37740c77ad0 EFLAGS: 00010286
[  478.731595] RAX: 0000000000000000 RBX: ffff8a3ff55e5ee8 RCX: 0000000000000000
[  478.736351] RDX: 0000000000000001 RSI: ffff8a3ff3d49800 RDI: ffff8a3ff52fd500
[  478.741042] RBP: ffff8a3ff52fd500 R08: ffff8a3ff55e5ee8 R09: 0000000000010000
[  478.745697] R10: 0000000000000001 R11: ffff8a3ef6dd7500 R12: 0000000000000000
[  478.750446] R13: 0000000000000000 R14: ffff8a3ff52fd840 R15: ffff8a3ff55e5ee8
[  478.755161] FS:  00007fd74bb17e40(0000) GS:ffff8a3ff7e00000(0000) knlGS:0000000000000000
[  478.761724] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  478.765853] CR2: 0000000000000010 CR3: 00000000a94e6005 CR4: 00000000007606b0
[  478.770524] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  478.775273] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  478.779984] PKRU: 55555554
[  478.782901] Call Trace:
[  478.785756]  ip_finish_output+0x228/0x270
[  478.789204]  ? nf_hook_slow+0x44/0xc0
[  478.792490]  ip_output+0x6c/0xe0
[  478.795685]  ? ip_append_data.part.49+0xd0/0xd0
[  478.799403]  __ip_queue_xmit+0x15d/0x410
[  478.802945]  ? set_fd_set.part.7+0x40/0x40
[  478.806411]  __tcp_transmit_skb+0x527/0xb10
[  478.810032]  tcp_write_xmit+0x384/0x1000
[  478.813636]  ? _copy_from_iter_full+0x94/0x240
[  478.817438]  __tcp_push_pending_frames+0x31/0xd0
[  478.821170]  tcp_sendmsg_locked+0xc1c/0xd50
[  478.824714]  tcp_sendmsg+0x27/0x40
[  478.827921]  sock_sendmsg+0x36/0x40
[  478.831280]  sock_write_iter+0x97/0x100
[  478.834714]  new_sync_write+0xfb/0x160
[  478.838010]  vfs_write+0xa5/0x1a0
[  478.841129]  ksys_write+0x57/0xd0
[  478.844250]  do_syscall_64+0x50/0xf0
[  478.847526]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  478.851385] RIP: 0033:0x7fd74beba504
[  478.854598] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 f9 61 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[  478.867315] RSP: 002b:00007ffc1d456638 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  478.873758] RAX: ffffffffffffffda RBX: 0000000000000084 RCX: 00007fd74beba504
[  478.878456] RDX: 0000000000000084 RSI: 000055785f33bb90 RDI: 0000000000000003
[  478.883176] RBP: 000055785f31d630 R08: 0000000000000000 R09: 0000000000001000
[  478.887885] R10: 0000000000000008 R11: 0000000000000246 R12: 00000000000001dd
[  478.892646] R13: 000055785ddc9b00 R14: 0000000000000003 R15: 00007ffc1d4566e0
[  478.897480] Modules linked in: xt_nat xt_tcpudp veth xt_conntrack ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nft_chain_nat_ipv4 nf_nat_ipv4 nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables nfnetlink br_netfilter bridge stp llc binfmt_misc overlay crct10dif_pclmul crc32_pclmul ghash_clmulni_intel nls_ascii nls_cp437 vfat fat intel_rapl_perf evdev serio_raw button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb crc32c_intel aesni_intel nvme aes_x86_64 crypto_simd ena nvme_core cryptd glue_helper
[  478.931979] CR2: 0000000000000010

Let me know if I can provide any other information that may be of use.

Shane Frasier

Kevin Tran

unread,
Aug 19, 2020, 6:40:03 PM8/19/20
to
Also seeing this issue. Long kernel panic backtrace at the end of the message. The kernel seems to fully crash only when I have Docker running in a volume with an NFS export, and then I mount the NFS volume. The underlying volume is ZFS, but I'm not entirely sure if that has much to do with it. I can consistently reproduce this by spamming umounts and mounts:

for i in $(seq 1 100); do echo $i: umount; sudo umount -l -f /data; echo $i: mount; mount /data; echo; done;

The spam doesn't crash the system when Docker is stopped - only when Docker is actively running.

I saw this related bug in the kernel: https://bugzilla.kernel.org/show_bug.cgi?id=208003
And this fix in mainline: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.9-rc1&id=ad0f75e5f57ccbceec13274e1e242f2b5a6397ed

Looks like the fix is in kernel 4.19.134: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=v4.19.134 ("cgroup: fix cgroup_sk_alloc() for sk_clone_lock()")

Since the latest Debian buster kernel is based on 4.19.132, I suppose the best thing to do is to wait for a new kernel version to be minted that (hopefully) contains the fix?

Aug 19 14:32:11 cascadia kernel: general protection fault: 0000 [#1] SMP PTI
Aug 19 14:32:11 cascadia kernel: CPU: 9 PID: 4424 Comm: nfsd Tainted: P           OE     4.19.0-10-amd64 #1 Debian 4.19.132-1
Aug 19 14:32:11 cascadia kernel: Hardware name: Supermicro Super Server/X10SDV-TLN4F, BIOS 1.3 02/13/2018
Aug 19 14:32:11 cascadia kernel: RIP: 0010:__cgroup_bpf_run_filter_skb+0xbd/0x1e0
Aug 19 14:32:11 cascadia kernel: Code: 00 00 00 49 89 7f 18 48 89 0c 24 44 89 e1 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 b0 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 01 01 00 00 4d 8d 6f 30 bd
Aug 19 14:32:11 cascadia kernel: RSP: 0000:ffffb5290ff9fa08 EFLAGS: 00010286
Aug 19 14:32:11 cascadia kernel: RAX: f70b7912beaa0ef3 RBX: ffff9bb45783eee8 RCX: 0000000000000000
Aug 19 14:32:11 cascadia kernel: RDX: 0000000000000001 RSI: ffff9bb45b922800 RDI: ffff9bb3f78eee80
Aug 19 14:32:11 cascadia kernel: RBP: ffff9bb3f78eee80 R08: ffff9bb45783eee8 R09: 0000000000000c32
Aug 19 14:32:11 cascadia kernel: R10: 0000000000000001 R11: ffff9bb373296780 R12: 0000000000000000
Aug 19 14:32:11 cascadia kernel: R13: 0000000000000000 R14: ffff9bb3f78ef1c0 R15: ffff9bb45783eee8
Aug 19 14:32:11 cascadia kernel: FS:  0000000000000000(0000) GS:ffff9bb45fa40000(0000) knlGS:0000000000000000
Aug 19 14:32:11 cascadia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 19 14:32:11 cascadia kernel: CR2: 000055f6f1a7b010 CR3: 00000007ccc0a004 CR4: 00000000003606e0
Aug 19 14:32:11 cascadia kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 19 14:32:11 cascadia kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 19 14:32:11 cascadia kernel: Call Trace:
Aug 19 14:32:11 cascadia kernel:  ip_finish_output+0x228/0x270
Aug 19 14:32:11 cascadia kernel:  ? nf_hook_slow+0x44/0xc0
Aug 19 14:32:11 cascadia kernel:  ip_output+0x6c/0xe0
Aug 19 14:32:11 cascadia kernel:  ? ip_append_data.part.49+0xd0/0xd0
Aug 19 14:32:11 cascadia kernel:  __ip_queue_xmit+0x15d/0x410
Aug 19 14:32:11 cascadia kernel:  __tcp_transmit_skb+0x527/0xb10
Aug 19 14:32:11 cascadia kernel:  tcp_write_xmit+0x384/0x1000
Aug 19 14:32:11 cascadia kernel:  do_tcp_sendpages+0x2d4/0x600
Aug 19 14:32:11 cascadia kernel:  tcp_sendpage_locked+0x44/0x60
Aug 19 14:32:11 cascadia kernel:  tcp_sendpage+0x37/0x50
Aug 19 14:32:11 cascadia kernel:  inet_sendpage+0x3f/0xf0
Aug 19 14:32:11 cascadia kernel:  kernel_sendpage+0x1a/0x30
Aug 19 14:32:11 cascadia kernel:  svc_send_common+0xf5/0x150 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? nfsd_destroy+0x50/0x50 [nfsd]
Aug 19 14:32:11 cascadia kernel:  svc_sendto+0x117/0x220 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? dput.part.33+0x32/0x140
Aug 19 14:32:11 cascadia kernel:  ? fh_put+0x93/0xf0 [nfsd]
Aug 19 14:32:11 cascadia kernel:  ? copy_cred+0x3f/0xb0 [nfsd]
Aug 19 14:32:11 cascadia kernel:  ? put_client_renew_locked+0xcd/0xe0 [nfsd]
Aug 19 14:32:11 cascadia kernel:  ? refcount_dec_and_lock+0xd/0x40
Aug 19 14:32:11 cascadia kernel:  ? auth_domain_put+ 0x15/0x70 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? svc_authorise+0x34/0x40 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? svc_process_common+0x11e/0x740 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  ? svc_tcp_sendto+0x36/0x50 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  svc_tcp_sendto+0x36/0x50 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  svc_send+0x73/0x160 [sunrpc]
Aug 19 14:32:11 cascadia kernel:  nfsd+0xe3/0x140 [nfsd]
Aug 19 14:32:11 cascadia kernel:  kthread+0x112/0x130
Aug 19 14:32:11 cascadia kernel:  ? kthread_bind+0x30/0x30
Aug 19 14:32:11 cascadia kernel:  ret_from_fork+0x35/0x40
Aug 19 14:32:11 cascadia kernel: Modules linked in: veth xt_nat xt_tcpudp xt_conntrack ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nft_chain_nat_ipv4 nf_nat_ipv4 nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc rpcsec_gss_krb5 devlink nf_tables nfnetlink aufs(OE) wireguard(E) ip6_udp_tunnel udp_tunnel overlay intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel pcc_cpufreq zfs(POE) kvm irqbypass zunicode(POE) intel_cstate zlua(POE) zavl(POE) intel_uncore icp(POE) ipmi_ssif mxm_wmi ast ttm drm_kms_helper intel_rapl_perf pcspkr drm intel_pch_thermal iTCO_wdt iTCO_vendor_support joydev cdc_acm evdev sg mei_me mei ioatdma ipmi_si wmi acpi_pad button nfsd auth_rpcgss nfs_acl lockd grace zcommon(POE) sunrpc znvpair(POE) spl(OE)
Aug 19 14:32:11 cascadia kernel:  ipmi_devintf ipmi_msghandler ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb btrfs xor zstd_decompress zstd_compress xxhash raid6_pq libcrc32c crc32c_generic algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid sd_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc ahci xhci_pci libahci xhci_hcd mpt3sas ehci_pci libata ehci_hcd usbcore raid_class aesni_intel ixgbe scsi_transport_sas igb aes_x86_64 scsi_mod nvme crypto_simd cryptd glue_helper lpc_ich nvme_core i2c_i801 mfd_core i2c_algo_bit dca usb_common mdio
Aug 19 14:32:11 cascadia kernel: ---[ end trace acf9c2d544d7fd2c ]---
Aug 19 14:32:11 cascadia kernel: RIP: 0010:__cgroup_bpf_run_filter_skb+0xbd/0x1e0
Aug 19 14:32:11 cascadia kernel: Code: 00 00 00 49 89 7f 18 48 89 0c 24 44 89 e1 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 b0 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 01 01 00 00 4d 8d 6f 30 bd
Aug 19 14:32:11 cascadia kernel: RSP: 0000:ffffb5290ff9fa08 EFLAGS: 00010286
Aug 19 14:32:11 cascadia kernel: RAX: f70b7912beaa0ef3 RBX: ffff9bb45783eee8 RCX: 0000000000000000
Aug 19 14:32:11 cascadia kernel: RDX: 0000000000000001 RSI: ffff9bb45b922800 RDI: ffff9bb3f78eee80
Aug 19 14:32:11 cascadia kernel: RBP: ffff9bb3f78eee80 R08: ffff9bb45783eee8 R09: 0000000000000c32
Aug 19 14:32:11 cascadia kernel: R10: 0000000000000001 R11: ffff9bb373296780 R12: 0000000000000000
Aug 19 14:32:11 cascadia kernel: R13: 0000000000000000 R14: ffff9bb3f78ef1c0 R15: ffff9bb45783eee8
Aug 19 14:32:11 cascadia kernel: FS:  0000000000000000(0000) GS:ffff9bb45fa40000(0000) knlGS:0000000000000000
Aug 19 14:32:11 cascadia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 19 14:32:11 cascadia kernel: CR2: 000055f6f1a7b010 CR3: 00000007ccc0a004 CR4: 00000000003606e0
Aug 19 14:32:11 cascadia kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 19 14:32:11 cascadia kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 19 14:32:11 cascadia kernel: general protection fault: 0000 [#2] SMP PTI
Aug 19 14:32:11 cascadia kernel: CPU: 2 PID: 13041 Comm: C2 CompilerThre Tainted: P      D    OE     4.19.0-10-amd64 #1 Debian 4.19.132-1
Aug 19 14:32:11 cascadia kernel: Hardware name: Supermicro Super Server/X10SDV-TLN4F, BIOS 1.3 02/13/2018
Aug 19 14:32:11 cascadia kernel: RIP: 0010:__cgroup_bpf_run_filter_skb+0xbd/0x1e0
Aug 19 14:32:11 cascadia kernel: Code: 00 00 00 49 89 7f 18 48 89 0c 24 44 89 e1 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 b0 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 01 01 00 00 4d 8d 6f 30 bd
Aug 19 14:32:11 cascadia kernel: RSP: 0000:ffff9bb45f883c00 EFLAGS: 00010296
Aug 19 14:32:11 cascadia kernel: RAX: f3f4ae25351c25db RBX: ffff9bb453a7c300 RCX: 0000000000000014
Aug 19 14:32:11 cascadia kernel: RDX: 0000000000000000 RSI: ffff9bb45b922800 RDI: ffff9bb3f78eee80
Aug 19 14:32:11 cascadia kernel: RBP: ffff9bb3f78eee80 R08: 0000000000000001 R09: 0000000000000801
Aug 19 14:32:11 cascadia kernel: R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000014
Aug 19 14:32:11 cascadia kernel: R13: 0000000000000020 R14: ffff9bb43d882862 R15: ffff9bb453a7c300
Aug 19 14:32:11 cascadia kernel: FS:  00007f28a1043b10(0000) GS:ffff9bb45f880000(0000) knlGS:0000000000000000
Aug 19 14:32:11 cascadia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 19 14:32:11 cascadia kernel: CR2: 000055b71d84fb10 CR3: 0000000749b5a004 CR4: 00000000003606e0
Aug 19 14:32:11 cascadia kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 19 14:32:11 cascadia kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 19 14:32:11 cascadia kernel: Call Trace:
Aug 19 14:32:11 cascadia kernel:  <IRQ>
Aug 19 14:32:11 cascadia kernel:  sk_filter_trim_cap+0xcd/0x1b0
Aug 19 14:32:11 cascadia kernel:  tcp_v4_rcv+0xa41/0xc20
Aug 19 14:32:11 cascadia kernel:  ip_local_deliver_finish+0x63/0x1e0
Aug 19 14:32:11 cascadia kernel:  ip_local_deliver+0xe0/0xf0
Aug 19 14:32:11 cascadia kernel:  ? ip_sublist_rcv_finish+0x80/0x80
Aug 19 14:32:11 cascadia kernel:  ip_rcv+0xbc/0xd0
Aug 19 14:32:11 cascadia kernel:  ? ip_rcv_finish_core.isra.18+0x360/0x360
Aug 19 14:32:11 cascadia kernel:  __netif_receive_skb_one_core+0x5a/0x80
Aug 19 14:32:11 cascadia kernel:  netif_receive_skb_internal+0x2f/0xa0
Aug 19 14:32:11 cascadia kernel:  napi_gro_receive+0xba/0xe0
Aug 19 14:32:11 cascadia kernel:  igb_poll+0x481/0xeb0 [igb]
Aug 19 14:32:11 cascadia kernel:  net_rx_action+0x149/0x3b0
Aug 19 14:32:11 cascadia kernel:  __do_softirq+0xde/0x2d8
Aug 19 14:32:11 cascadia kernel:  irq_exit+0xba/0xc0
Aug 19 14:32:11 cascadia kernel:  do_IRQ+0x7f/0xe0
Aug 19 14:32:11 cascadia kernel:  common_interrupt+0xf/0xf
Aug 19 14:32:11 cascadia kernel:  </IRQ>
Aug 19 14:32:11 cascadia kernel: RIP: 0033:0x7f28b5f3147a
Aug 19 14:32:11 cascadia kernel: Code: 89 c2 48 c1 e0 02 48 8b 4d 18 39 d3 7c 0f c7 04 01 00 00 00 00 ff c2 48 83 c0 04 eb e9 8d 43 01 89 45 00 48 8b 45 18 8b 04 98 <5a> 5b 5d c3 31 c0 48 39 f1 0f 84 d6 00 00 00 66 44 8b 41 2c 41 80
Aug 19 14:32:11 cascadia kernel: RSP: 002b:00007f28a1040560 EFLAGS: 00000297 ORIG_RAX: ffffffffffffffdb
Aug 19 14:32:11 cascadia kernel: RAX: 0000000000000000 RBX: 0000000000003a31 RCX: 000055b71dfb23a8
Aug 19 14:32:11 cascadia kernel: RDX: 000055b71dfb23a8 RSI: 0000000000003a31 RDI: 000055b7202930b0
Aug 19 14:32:11 cascadia kernel: RBP: 000055b7202930b0 R08: 000055b71e475e50 R09: 000055b71dfb2010
Aug 19 14:32:11 cascadia kernel: R10: 0000000000000018 R11: 000055b71df0b480 R12: 000055b71dfb23a8
Aug 19 14:32:11 cascadia kernel: R13: 000055b7212a1f10 R14: 0000000000000000 R15: 00007f28a1040790
Aug 19 14:32:11 cascadia kernel: Modules linked in: veth xt_nat xt_tcpudp xt_conntrack ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nft_chain_nat_ipv4 nf_nat_ipv4 nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter bridge stp llc rpcsec_gss_krb5 devlink nf_tables nfnetlink aufs(OE) wireguard(E) ip6_udp_tunnel udp_tunnel overlay intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel pcc_cpufreq zfs(POE) kvm irqbypass zunicode(POE) intel_cstate zlua(POE) zavl(POE) intel_uncore icp(POE) ipmi_ssif mxm_wmi ast ttm drm_kms_helper intel_rapl_perf pcspkr drm intel_pch_thermal iTCO_wdt iTCO_vendor_support joydev cdc_acm evdev sg mei_me mei ioatdma ipmi_si wmi acpi_pad button nfsd auth_rpcgss nfs_acl lockd grace zcommon(POE) sunrpc znvpair(POE) spl(OE)
Aug 19 14:32:11 cascadia kernel:  ipmi_devintf ipmi_msghandler ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb btrfs xor zstd_decompress zstd_compress xxhash raid6_pq libcrc32c crc32c_generic algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid sd_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc ahci xhci_pci libahci xhci_hcd mpt3sas ehci_pci libata ehci_hcd usbcore raid_class aesni_intel ixgbe scsi_transport_sas igb aes_x86_64 scsi_mod nvme crypto_simd cryptd glue_helper lpc_ich nvme_core i2c_i801 mfd_core i2c_algo_bit dca usb_common mdio
Aug 19 14:32:11 cascadia kernel: ---[ end trace acf9c2d544d7fd2d ]---

Kevin Tran

Salvatore Bonaccorso

unread,
Aug 30, 2020, 11:20:03 AM8/30/20
to
Hi Immanuel,

On Sun, Aug 30, 2020 at 04:49:51PM +0200, isc...@der-ball-ist-rund.net wrote:
> I can confirm this bug. For me the panic is not exclusively related
> with docker. I stopped my docker daemon - and I'm still suffering
> from random freezes of the kernel 4.19.0-10. Kernel is unusable.
>
> But I'm still running two lxc-containers and two virtual machines kvm/qemu.
>
> enclosed you'll find the "screenshot"

Yes, the issue would not be exclusively with docker.

The issue should be pending fixed, but if you can I would appreciate
if you can test (warning: temporary and unofficial build!) packages
rebased to 4.19.142 upstream:

https://people.debian.org/~carnil/tmp/linux/4.19.142-1/

Regards,
Salvatore

Sébastien NOBILI

unread,
Sep 3, 2020, 8:10:03 AM9/3/20
to
Hi Salvatore,

Le 2020-08-30 17:08, Salvatore Bonaccorso a écrit :
> On Sun, Aug 30, 2020 at 04:49:51PM +0200, isc...@der-ball-ist-rund.net
> wrote:
>> But I'm still running two lxc-containers and two virtual machines
>> kvm/qemu.
>
> Yes, the issue would not be exclusively with docker.

I'm facing this bug as well on a server with LXC containers (no
Docker/KVM/Qemu
at all).

> The issue should be pending fixed, but if you can I would appreciate
> if you can test (warning: temporary and unofficial build!) packages
> rebased to 4.19.142 upstream:
>
> https://people.debian.org/~carnil/tmp/linux/4.19.142-1/

I've installed it (amd64 version) and will let you know how things are
going on.

Sébastien

Sébastien NOBILI

unread,
Sep 5, 2020, 1:20:03 PM9/5/20
to
Hi Salvatore,

No crash for two days with this build. I'll send an update in a few
days.

Sébastien

Sébastien NOBILI

unread,
Sep 11, 2020, 5:50:03 AM9/11/20
to
Hi,

More than a week after, no problem with this build, working fine 24/7.

Sébastien

Salvatore Bonaccorso

unread,
Sep 11, 2020, 8:00:02 AM9/11/20
to
Hi Sébastien,

On Fri, Sep 11, 2020 at 11:41:16AM +0200, Sébastien NOBILI wrote:
> Hi,
>
> More than a week after, no problem with this build, working fine
> 24/7.

Thanks for confirming that. We are planning to rebase the version for
the next point release and so will contain the fix.

Regards,
Salvatore

Debian Bug Tracking System

unread,
Sep 18, 2020, 7:00:07 AM9/18/20
to
Your message dated Fri, 18 Sep 2020 10:47:08 +0000
with message-id <E1kJDv2-...@fasolo.debian.org>
and subject line Bug#966846: fixed in linux 4.19.146-1
has caused the Debian Bug report #966846,
regarding Kernel panic (4.19.0-10): RIP __cgroup_bpf_run_filter_skb
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)


--
966846: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=966846
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Michel Le Bihan

unread,
Sep 22, 2020, 6:10:02 PM9/22/20
to
Hello,

I'm a bit late but I also have this issue and it occurs every several
hours on my server. Here is the trace if anybody is still interested:
https://lebihan.pl/files/trace.txt

When can I expect the new package to be uploaded into stable?

Michel Le Bihan

Ben Hutchings

unread,
Sep 23, 2020, 12:20:02 PM9/23/20
to
In the point release, at the weekend.

Ben.

--
Ben Hutchings
The generation of random numbers is too important to be left to chance.
- Robert Coveyou


signature.asc
0 new messages