💥 PANICKED: Test report for kernel 5.11.0-rc6 (mainline.kernel.org-clang)

[CKI Project]

Hello,

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Commit: 3aaf0a27ffc2 - Merge tag 'clang-format-for-linux-v5.11-rc7' of git://github.com/ojeda/linux

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Selftests compile: FAILED
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/02/622905

One or more kernel tests failed:

aarch64:
❌ selinux-policy: serge-testsuite

x86_64:
❌ selinux-policy: serge-testsuite
❌ LTP lite
💥 LTP

We hope that these logs can help you find the problem quickly. For the full
detail on our testing procedures, please scroll to the bottom of this message.

Please reply to this email if you have any questions about the tests that we
ran or if you have any suggestions on how to make future tests more effective.

,-. ,-.
( C ) ( K ) Continuous
`-',-.`-' Kernel
( I ) Integration
`-'
______________________________________________________________________________

Compile testing
---------------

We compiled the kernel for 3 architectures:

aarch64:
make options: make LLVM=1 -j30 INSTALL_MOD_STRIP=1 targz-pkg

ppc64le:
make options: make CC=clang -j30 INSTALL_MOD_STRIP=1 targz-pkg

x86_64:
make options: make LLVM=1 -j30 INSTALL_MOD_STRIP=1 targz-pkg


We built the following selftests:

x86_64:
net: fail
bpf: fail
install and packaging: fail

You can find the full log (build-selftests.log) in the artifact storage above.


Hardware testing
----------------
We booted each kernel and ran the following tests:

aarch64:
Host 1:
✅ Boot test
❌ selinux-policy: serge-testsuite
✅ lvm thinp sanity
✅ storage: software RAID testing
✅ stress: stress-ng
🚧 ✅ xfstests - ext4
🚧 ✅ xfstests - xfs
🚧 ✅ Storage blktests
🚧 ❌ Storage nvme - rdma
🚧 ✅ Storage nvme - tcp
🚧 ✅ Storage: swraid mdadm raid_module test
🚧 ❌ storage: iSCSI parameters

Host 2:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 3:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
✅ ACPI table test
✅ LTP lite
⚡⚡⚡ LTP
⚡⚡⚡ Loopdev Sanity
⚡⚡⚡ Memory function: memfd_create
⚡⚡⚡ AMTU (Abstract Machine Test Utility)
⚡⚡⚡ Networking bridge: sanity
⚡⚡⚡ Networking MACsec: sanity
⚡⚡⚡ Networking socket: fuzz
⚡⚡⚡ Networking sctp-auth: sockopts test
⚡⚡⚡ Networking: igmp conformance test
⚡⚡⚡ Networking route: pmtu
⚡⚡⚡ Networking route_func - local
⚡⚡⚡ Networking route_func - forward
⚡⚡⚡ Networking TCP: keepalive test
⚡⚡⚡ Networking UDP: socket
⚡⚡⚡ Networking tunnel: geneve basic test
⚡⚡⚡ Networking tunnel: gre basic
⚡⚡⚡ L2TP basic test
⚡⚡⚡ Networking tunnel: vxlan basic
⚡⚡⚡ Networking ipsec: basic netns - transport
⚡⚡⚡ Networking ipsec: basic netns - tunnel
⚡⚡⚡ Libkcapi AF_ALG test
⚡⚡⚡ pciutils: update pci ids test
⚡⚡⚡ ALSA PCM loopback test
⚡⚡⚡ ALSA Control (mixer) Userspace Element test
⚡⚡⚡ Usex - version 1.9-29
⚡⚡⚡ storage: SCSI VPD
⚡⚡⚡ Syscalls: nrdiff
🚧 ⚡⚡⚡ CIFS Connectathon
🚧 ⚡⚡⚡ POSIX pjd-fstest suites
🚧 ⚡⚡⚡ Trinity Fuzzer - Tier1
🚧 ⚡⚡⚡ Memory function: kaslr
🚧 ⚡⚡⚡ LTP: openposix test suite
🚧 ⚡⚡⚡ Ethernet drivers sanity
🚧 ⚡⚡⚡ Networking: ipv6/Fujitsu-socketapi-test
🚧 ⚡⚡⚡ Networking firewall: basic netfilter test
🚧 ⚡⚡⚡ audit: audit testsuite test
🚧 ⚡⚡⚡ storage: dm/common
🚧 ⚡⚡⚡ trace: ftrace/tracer
🚧 ⚡⚡⚡ Libhugetlbfs - version 2.2.1
🚧 ⚡⚡⚡ kdump - kexec_boot

Host 4:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 5:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

ppc64le:
Host 1:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 2:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test
⚡⚡⚡ LTP lite
⚡⚡⚡ LTP
⚡⚡⚡ Loopdev Sanity
⚡⚡⚡ Memory function: memfd_create
⚡⚡⚡ AMTU (Abstract Machine Test Utility)
⚡⚡⚡ Networking bridge: sanity
⚡⚡⚡ Networking MACsec: sanity
⚡⚡⚡ Networking socket: fuzz
⚡⚡⚡ Networking sctp-auth: sockopts test
⚡⚡⚡ Networking route: pmtu
⚡⚡⚡ Networking route_func - local
⚡⚡⚡ Networking route_func - forward
⚡⚡⚡ Networking TCP: keepalive test
⚡⚡⚡ Networking UDP: socket
⚡⚡⚡ Networking tunnel: geneve basic test
⚡⚡⚡ Networking tunnel: gre basic
⚡⚡⚡ L2TP basic test
⚡⚡⚡ Networking tunnel: vxlan basic
⚡⚡⚡ Networking ipsec: basic netns - tunnel
⚡⚡⚡ Libkcapi AF_ALG test
⚡⚡⚡ pciutils: update pci ids test
⚡⚡⚡ ALSA PCM loopback test
⚡⚡⚡ ALSA Control (mixer) Userspace Element test
⚡⚡⚡ Usex - version 1.9-29
⚡⚡⚡ Syscalls: nrdiff
🚧 ⚡⚡⚡ CIFS Connectathon
🚧 ⚡⚡⚡ POSIX pjd-fstest suites
🚧 ⚡⚡⚡ Trinity Fuzzer - Tier1
🚧 ⚡⚡⚡ Memory function: kaslr
🚧 ⚡⚡⚡ LTP: openposix test suite
🚧 ⚡⚡⚡ Ethernet drivers sanity
🚧 ⚡⚡⚡ Networking: ipv6/Fujitsu-socketapi-test
🚧 ⚡⚡⚡ Networking firewall: basic netfilter test
🚧 ⚡⚡⚡ audit: audit testsuite test
🚧 ⚡⚡⚡ storage: dm/common
🚧 ⚡⚡⚡ trace: ftrace/tracer
🚧 ⚡⚡⚡ Libhugetlbfs - version 2.2.1

Host 3:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test
⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ lvm thinp sanity
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage nvme - rdma
🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test

Host 4:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test
⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ lvm thinp sanity
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage nvme - rdma
🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test

Host 5:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test
⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ lvm thinp sanity
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage nvme - rdma
🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test

x86_64:
Host 1:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
❌ selinux-policy: serge-testsuite
✅ lvm thinp sanity
✅ storage: software RAID testing
✅ stress: stress-ng
🚧 ✅ xfstests - ext4
🚧 ✅ xfstests - xfs
🚧 ✅ Storage blktests
🚧 ❌ Storage nvme - rdma
🚧 ❌ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage nvdimm ndctl test suite
🚧 ✅ Storage: swraid mdadm raid_module test
🚧 ❌ storage: iSCSI parameters

Host 2:
✅ Boot test
✅ kdump - sysrq-c
🚧 ✅ kdump - file-load

Host 3:
✅ Boot test
✅ ACPI table test
❌ LTP lite
💥 LTP
⚡⚡⚡ Loopdev Sanity
⚡⚡⚡ Memory function: memfd_create
⚡⚡⚡ AMTU (Abstract Machine Test Utility)
⚡⚡⚡ Networking bridge: sanity
⚡⚡⚡ Networking MACsec: sanity
⚡⚡⚡ Networking socket: fuzz
⚡⚡⚡ Networking sctp-auth: sockopts test
⚡⚡⚡ Networking: igmp conformance test
⚡⚡⚡ Networking route: pmtu
⚡⚡⚡ Networking route_func - local
⚡⚡⚡ Networking route_func - forward
⚡⚡⚡ Networking TCP: keepalive test
⚡⚡⚡ Networking UDP: socket
⚡⚡⚡ Networking tunnel: geneve basic test
⚡⚡⚡ Networking tunnel: gre basic
⚡⚡⚡ L2TP basic test
⚡⚡⚡ Networking tunnel: vxlan basic
⚡⚡⚡ Networking ipsec: basic netns - transport
⚡⚡⚡ Networking ipsec: basic netns - tunnel
⚡⚡⚡ Libkcapi AF_ALG test
⚡⚡⚡ pciutils: sanity smoke test
⚡⚡⚡ pciutils: update pci ids test
⚡⚡⚡ ALSA PCM loopback test
⚡⚡⚡ ALSA Control (mixer) Userspace Element test
⚡⚡⚡ Usex - version 1.9-29
⚡⚡⚡ storage: SCSI VPD
⚡⚡⚡ Syscalls: nrdiff
🚧 ⚡⚡⚡ CIFS Connectathon
🚧 ⚡⚡⚡ POSIX pjd-fstest suites
🚧 ⚡⚡⚡ Trinity Fuzzer - Tier1
🚧 ⚡⚡⚡ Memory function: kaslr
🚧 ⚡⚡⚡ LTP: openposix test suite
🚧 ⚡⚡⚡ Ethernet drivers sanity
🚧 ⚡⚡⚡ Networking: ipv6/Fujitsu-socketapi-test
🚧 ⚡⚡⚡ Networking firewall: basic netfilter test
🚧 ⚡⚡⚡ audit: audit testsuite test
🚧 ⚡⚡⚡ storage: dm/common
🚧 ⚡⚡⚡ trace: ftrace/tracer
🚧 ⚡⚡⚡ Libhugetlbfs - version 2.2.1
🚧 ⚡⚡⚡ Tracepoints: operational test
🚧 ⚡⚡⚡ kdump - kexec_boot

Test sources: https://gitlab.com/cki-project/kernel-tests
💚 Pull requests are welcome for new tests or improvements to existing tests!

Aborted tests
-------------
Tests that didn't complete running successfully are marked with ⚡⚡⚡.
If this was caused by an infrastructure issue, we try to mark that
explicitly in the report.

Waived tests
------------
If the test run included waived tests, they are marked with 🚧. Such tests are
executed but their results are not taken into account. Tests are waived when
their results are not reliable enough, e.g. when they're just introduced or are
being fixed.

Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't
finished running yet are marked with ⏱.

[Nathan Chancellor]

On Wed, Feb 03, 2021 at 01:58:32AM -0000, CKI Project wrote:
>
> Hello,
>
> We ran automated tests on a recent commit from this kernel tree:
>
> Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> Commit: 3aaf0a27ffc2 - Merge tag 'clang-format-for-linux-v5.11-rc7' of git://github.com/ojeda/linux
>
> The results of these automated tests are provided below.
>
> Overall result: FAILED (see details below)
> Merge: OK
> Compile: OK
> Selftests compile: FAILED
> Tests: PANICKED
>
> All kernel binaries, config files, and logs are available for download here:
>
> https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/02/622905
>
> One or more kernel tests failed:
>
> aarch64:
> ❌ selinux-policy: serge-testsuite
>
> x86_64:
> ❌ selinux-policy: serge-testsuite
> ❌ LTP lite

These should be the failures for this particular set of tests correct?

https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/datawarehouse-public/2021/02/02/622905/build_x86_64_redhat%3A1092370/tests/LTP_lite/9507327_x86_64_1_RHELKT1LITE.FILTERED.fail.log

I am curious, is there any way to get a side by side comparison of the
test results between GCC and clang? In other words, I would like to know
if a test is failing with both compilers versus just clang. This would
help us understand if it is a compiler specific issue or something that
broke the kernel.

> 💥 LTP

From what I can tell:

[12262.428593] general protection fault, probably for non-canonical address 0x8bc08b308b608b5: 0000 [#1] SMP PTI
[12262.428810] CPU: 1 PID: 465515 Comm: proc01 Not tainted 5.11.0-rc6 #1
[12262.428810] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[12262.428810] RIP: 0010:string+0x50/0x110
[12262.428810] Code: ce 48 c1 fe 30 85 f6 0f 84 be 00 00 00 31 c0 eb 15 66 0f 1f 84 00 00 00 00 00 48 83 c0 01 39 c6 0f 84 ac 00 00 00 4c 8d 0c 07 <45> 0f b6 14 00 45 84 d2 0f 84 a2 00 00 00 49 39 d1 73 dd 45 88 11
[12262.428810] RSP: 0018:ffffb1ea012bbc40 EFLAGS: 00010246
[12262.428810] RAX: 0000000000000000 RBX: ffff9ea25e9ea03e RCX: ffff0a00ffffff04
[12262.428810] RDX: ffff9ea25e9eb000 RSI: ffffffffffffffff RDI: ffff9ea25e9ea03e
[12262.428810] RBP: ffffffff8f5c95f1 R08: 08bc08b308b608b5 R09: ffff9ea25e9ea03e
[12262.428810] R10: 0000001000000000 R11: ffffffff8e212401 R12: ffff9ea25e9eb000
[12262.428810] R13: ffffffff8f5c95f3 R14: 0000000000000002 R15: ffffb1ea012bbca0
[12262.428810] FS: 00007fab43075740(0000) GS:ffff9ea35bd00000(0000) knlGS:0000000000000000
[12262.428810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12262.428810] CR2: 00005584bb6dba68 CR3: 0000000111bbe000 CR4: 00000000000006e0
[12262.428810] DR0: 0000000000000001 DR1: 0000000000000000 DR2: 0000000000000000
[12262.428810] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[12262.428810] Call Trace:
[12262.428810] vsnprintf+0x570/0x710
[12262.428810] seq_printf+0x6b/0x90
[12262.428810] ? lock_acquire+0x27/0x280
[12262.428810] ? mod_objcg_state+0xd2/0x160
[12262.428810] ? vsnprintf+0x32f/0x710
[12262.428810] print_name+0x46/0xc0
[12262.428810] ? seq_printf+0x6b/0x90
[12262.428810] ? __kmalloc_node+0xaf/0x340
[12262.428810] ? lock_acquire+0x27/0x280
[12262.428810] ? kvmalloc_node+0x4b/0x90
[12262.428810] lc_show+0x82/0xe0
[12262.428810] seq_read_iter+0x311/0x420
[12262.428810] proc_reg_read_iter+0x3f/0x80
[12262.428810] vfs_read+0x2c3/0x340
[12262.428810] ksys_read+0x5f/0xb0
[12262.428810] do_syscall_64+0x33/0x40
[12262.428810] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[12262.428810] RIP: 0033:0x7fab4318791d
[12262.428810] Code: 0f 1f 44 00 00 48 8b 0d 51 45 0b 00 45 31 c0 64 83 39 0b 75 d2 eb c0 e8 b1 fb ff ff 90 f3 0f 1e fa 48 39 ca 77 2b 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 0b c3 66 2e 0f 1f 84 00 00 00 00 00 48 8b 15
[12262.428810] RSP: 002b:00007ffcaeb265b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[12262.428810] RAX: ffffffffffffffda RBX: 00007ffcaeb27730 RCX: 00007fab4318791d
[12262.428810] RDX: 0000000000000400 RSI: 000000000044b820 RDI: 0000000000000005
[12262.428810] RBP: 0000000000008000 R08: 0000000000000000 R09: 00007ffcaeb27520
[12262.428810] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[12262.428810] R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
[12262.428810] Modules linked in: snd_seq_dummy minix snd_hrtimer snd_seq snd_seq_device authenc pcrypt crypto_user sha3_generic algif_hash salsa20_generic binfmt_misc n_gsm pps_ldisc slcan ppp_synctty n_hdlc mkiss ax25 ppp_async ppp_generic slip slhc nfsv3 nfs_acl nfs nfs_ssc fscache lockd grace rds tun brd overlay exfat vfat fat xfs sctp tcp_diag udp_diag inet_diag rfkill sunrpc snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation snd_soc_core snd_compress snd_pcm_dmaengine soundwire_cadence snd_hda_codec snd_hda_core ac97_bus snd_hwdep snd_pcm joydev snd_timer snd virtio_net soundcore net_failover pcspkr virtio_balloon failover i2c_piix4 fuse zram ip_tables x_tables cirrus drm_kms_helper cec virtio_blk drm serio_raw ata_generic pata_acpi floppy btrfs raid6_pq xor [last unloaded: can]
[12262.462039] ---[ end trace f683b95858968bee ]---

I cannot reproduce this in QEMU with a Debian rootfs with the supplied
configuration file though, proc01 -m 128 passes for me.

Cheers,
Nathan

> --
> You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clang-built-li...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/cki.C90653A6EF.V9S14CF2MY%40redhat.com.

[Rachel Sibley]

Hi Nathan,

We recently moved the mainline tree to use Fedora Rawhide as the base distro, this helps out kdump and other tests which require
updated user space for testing. As a result, there were a few tests that shouldn't have run like LTP Lite. So the log you linked
above is not related to the panic:
https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/datawarehouse-public/2021/02/02/622905/build_x86_64_redhat%3A1092370/tests/LTP_lite/9507327_x86_64_1_RHELKT1LITE.FILTERED.fail.log

This is the version of LTP we run for internal RHEL kernels and therefore isn't supported against upstream kernels. I'm working on a fix
for this now and should be resolved soon, sorry about that.

As far as the panic, it was was triggered on our upstream LTP test, I don't have a corresponding x86_64/LTP mainline(GCC) job since there is a BPF
bug which is causing our jobs to abort while updating the kernel for x86. You will see the following in the logs:

'failed to validate module [something] BTF: -22 '

Here is a regular mainline job for reference where you can it:
https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/datawarehouse-public/2021/02/02/622904/build_x86_64_redhat%3A1092352/tests/9507513_x86_64_1_console.log

As far as comparing tests for mainline run with gcc vs clang, we don't email an upstream ML for regular mainline, so nothing archived.
However, the logs are still published externally and we can always link them here for comparison.

If you like, we could start forwarding you reports for both ? Otherwise, we are working on publishing our results database we use internally,
but this will take some time.

Thanks,
Rachel

[Sedat Dilek]

Guess this is with CONFIG_DEBUG_INFO_BTF=y and LLVM/Clang?

Today, I played with above and diverse DWARF version settings.

Even recent pahole from Git in combination with up-to-date LLVM/Clang is incompatible with DWARF-v4 and DWARF-v5.

Means all my builds were broken.

Mark Wielaard says:

"I would try to avoid using clang producing DWARF5. It clearly has some
incompatibilities with dwarves/pahole. It should work if you don't set
DEBUG_INFO_DWARF5. Try GCC 11 (which defaults to -gdwarf-5) or an
earlier version (probably at least GCC 8 or higher) using -gdwarf-5
explicitly."

If this is with CONFIG_DEBUG_INFO_BTF=y you should make it unavailable when CC_IS_CLANG=y.

- Sedat -

 

To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/5877bb33-ba17-6459-410c-1c51bd193481%40redhat.com.

[Nick Desaulniers]

I'm attending BPF office hours tomorrow (morning, California time) to discuss this with the pahole maintainers.

https://lore.kernel.org/bpf/CAADnVQ+weKq=-=KB7j=0FfCE0bLanJ_ppn...@mail.gmail.com/

 

To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/CA%2BicZUWTia0CS5nrdGx7oE%2BZj_J3oC4id6U3jpVbUgqVosoFxw%40mail.gmail.com.

--

Thanks,

~Nick Desaulniers

[Sedat Dilek]

Great.

Bonne chance.

Just reported to linux-bpf ML.

- Sedat -

[1]  https://lore.kernel.org/bpf/CAEf4BzZ-uU3vkMA1RPt1f2Hb...@mail.gmail.com/T/#m2a5f11c5bb6ce66b3c59a1be2005dd40724d1d8b

[Nathan Chancellor]

On Wed, Feb 03, 2021 at 10:36:32PM +0100, Sedat Dilek wrote:
> On Wed, Feb 3, 2021 at 9:45 PM Rachel Sibley <rasi...@redhat.com> wrote:

[snip]

> > As far as the panic, it was was triggered on our upstream LTP test, I
> > don't have a corresponding x86_64/LTP mainline(GCC) job since there is a BPF
> > bug which is causing our jobs to abort while updating the kernel for x86.
> > You will see the following in the logs:
> >
> > 'failed to validate module [something] BTF: -22 '
> >
>
> Guess this is with CONFIG_DEBUG_INFO_BTF=y and LLVM/Clang?

I read that sentence as there are issues with BPF issues with GCC, not
LLVM/Clang.

CONFIG_DEBUG_INFO_BTF is already disabled for the LLVM jobs:

https://gitlab.com/cki-project/pipeline-definition/-/commit/7d138e9a3aede46f7674476fa1b3ca02a391767b#90e1e97a102a5713d6a68df323738846b425341a_1358_1369

Cheers,
Nathan

[Nathan Chancellor]

Hi Rachel,

Thanks for clarifying :)

> As far as the panic, it was was triggered on our upstream LTP test, I don't have a corresponding x86_64/LTP mainline(GCC) job since there is a BPF
> bug which is causing our jobs to abort while updating the kernel for x86. You will see the following in the logs:
>
> 'failed to validate module [something] BTF: -22 '
>
> Here is a regular mainline job for reference where you can it:
> https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/datawarehouse-public/2021/02/02/622904/build_x86_64_redhat%3A1092352/tests/9507513_x86_64_1_console.log
>
> As far as comparing tests for mainline run with gcc vs clang, we don't email an upstream ML for regular mainline, so nothing archived.
> However, the logs are still published externally and we can always link them here for comparison.
>
> If you like, we could start forwarding you reports for both ? Otherwise, we are working on publishing our results database we use internally,
> but this will take some time.
>

If just adding a link to the GCC logs in the clang reports is not too
much effort for you guys, that would be mighty helpful. I do not mind
sifting through the logs to try and compare results manually (as I just
need to compare the failing clang tests to the GCC run).

If that is not possible for various reasons, please explicitly CC me on
the GCC mainline reports (nat...@kernel.org) and I can just grab the
link myself. Our list gets enough reports as it stands so we can keep
the GCC ones off of this list.

Cheers,
Nathan

[Ondrej Mosnacek]

On Wed, Feb 3, 2021 at 11:06 PM Nathan Chancellor <nat...@kernel.org> wrote:
> On Wed, Feb 03, 2021 at 10:36:32PM +0100, Sedat Dilek wrote:
> > On Wed, Feb 3, 2021 at 9:45 PM Rachel Sibley <rasi...@redhat.com> wrote:
>
> [snip]
>
> > > As far as the panic, it was was triggered on our upstream LTP test, I
> > > don't have a corresponding x86_64/LTP mainline(GCC) job since there is a BPF
> > > bug which is causing our jobs to abort while updating the kernel for x86.
> > > You will see the following in the logs:
> > >
> > > 'failed to validate module [something] BTF: -22 '
> > >
> >
> > Guess this is with CONFIG_DEBUG_INFO_BTF=y and LLVM/Clang?
>
> I read that sentence as there are issues with BPF issues with GCC, not
> LLVM/Clang.

Yes, it's probably this issue:
https://lore.kernel.org/bpf/CAADnVQKbku+Mv++h2TKYZfFN...@mail.gmail.com/T/

>
> CONFIG_DEBUG_INFO_BTF is already disabled for the LLVM jobs:
>
> https://gitlab.com/cki-project/pipeline-definition/-/commit/7d138e9a3aede46f7674476fa1b3ca02a391767b#90e1e97a102a5713d6a68df323738846b425341a_1358_1369
>
> Cheers,
> Nathan
>

--
Ondrej Mosnacek
Software Engineer, Linux Security - SELinux kernel
Red Hat, Inc.

[CKI Project]

Hello,

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Commit: 3afe9076a7c1 - Merge tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Selftests compile: FAILED
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/03/623040

One or more kernel tests failed:

aarch64:
❌ selinux-policy: serge-testsuite

x86_64:

❌ LTP lite
💥 LTP
❌ selinux-policy: serge-testsuite

✅ Boot test
❌ selinux-policy: serge-testsuite
✅ lvm thinp sanity
✅ storage: software RAID testing
✅ stress: stress-ng
🚧 ✅ xfstests - ext4
🚧 ✅ xfstests - xfs
🚧 ✅ Storage blktests
🚧 ❌ Storage nvme - rdma
🚧 ✅ Storage nvme - tcp
🚧 ✅ Storage: swraid mdadm raid_module test
🚧 ❌ storage: iSCSI parameters

Host 4:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 5:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

ppc64le:
Host 1:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test

Host 2:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test

⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ lvm thinp sanity
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage nvme - rdma
🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test

Host 3:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 2:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test

❌ selinux-policy: serge-testsuite
✅ lvm thinp sanity
✅ storage: software RAID testing
✅ stress: stress-ng
🚧 ✅ xfstests - ext4
🚧 ✅ xfstests - xfs
🚧 ✅ Storage blktests
🚧 ✅ Storage nvme - rdma
🚧 ✅ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage nvdimm ndctl test suite
🚧 ✅ Storage: swraid mdadm raid_module test
🚧 ❌ storage: iSCSI parameters

Host 3:

✅ Boot test
✅ kdump - sysrq-c
🚧 ✅ kdump - file-load

[CKI Project]

Hello,

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Commit: 61556703b610 - Merge tag 'for-linus-5.11-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/rw/uml

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Selftests compile: FAILED
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/03/623045

❌ selinux-policy: serge-testsuite
✅ lvm thinp sanity
✅ storage: software RAID testing
✅ stress: stress-ng
🚧 ✅ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage nvme - rdma
🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
🚧 ⚡⚡⚡ storage: iSCSI parameters

Host 3:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 4:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test

⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 5:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

ppc64le:
Host 1:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test

⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ lvm thinp sanity
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage nvme - rdma
🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test

Host 2:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

⚡⚡⚡ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 3:

[Sedat Dilek]

On Wed, Feb 3, 2021 at 11:06 PM Nathan Chancellor <nat...@kernel.org> wrote:
>

OK, Thanks.

Can you give me the link of your (above) response in the
ClangBuiltLinux mailing-list?

- Sedat -

[Nathan Chancellor]

https://groups.google.com/g/clang-built-linux/c/de_mNh23FOc/m/oNIZyE6fBQAJ

Cheers,
Nathan

[Sedat Dilek]

Thanks for the link.

You need a Gmail-account to be able to read the mailing-list?
Asking for offline readers.

- Sedat -

[1] https://groups.google.com/g/clang-built-linux/

[Nick Desaulniers]

We can move to vger+lore. Google groups was faster to setup, but has
many downsides. I don't have resources to pursue archiving the
existing list, and corralling the folks to set up vger and lore, but I
can point people in the right direction and would be supportive.

>
> - Sedat -
>
> [1] https://groups.google.com/g/clang-built-linux/
>

> --
> You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clang-built-li...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/CA%2BicZUVV3q3Jr8HEi%3DLmqYucOWK8b3zOEvfGsk8Mn5FS--8bnQ%40mail.gmail.com.



--
Thanks,
~Nick Desaulniers

[Nathan Chancellor]

It should not require a Gmail account. I can view that link no problem
in incognito mode in Chrome.

Cheers,
Nathan

[Sedat Dilek]

Logged-out from Gmail and I was able to read "offline".
Thanks for confirming.

- Sedat -

[Veronika Kabatova]

Hi,

not sure which email to reply so I'm going with this one...

Confirmed with BPF folks that the BTF boot issue from the gcc run is due
to the dwarves incompatibility that was already mentioned and tested a new
dwarves rawhide build which fixes it. According to them it should be
released in a few days.

> > As far as comparing tests for mainline run with gcc vs clang, we don't
> > email an upstream ML for regular mainline, so nothing archived.
> > However, the logs are still published externally and we can always link
> > them here for comparison.
> >
> > If you like, we could start forwarding you reports for both ? Otherwise, we
> > are working on publishing our results database we use internally,
> > but this will take some time.
> >
>
> If just adding a link to the GCC logs in the clang reports is not too
> much effort for you guys, that would be mighty helpful. I do not mind
> sifting through the logs to try and compare results manually (as I just
> need to compare the failing clang tests to the GCC run).
>
> If that is not possible for various reasons, please explicitly CC me on
> the GCC mainline reports (nat...@kernel.org) and I can just grab the
> link myself. Our list gets enough reports as it stands so we can keep
> the GCC ones off of this list.
>

The two test runs are separate, so the gcc run may not yet be finished
when the clang reports are ready, which makes providing links hard and
we'd have to implement some sort of half-reliable syncing mechanism.
So if you don't mind, we'd just go with the cc for now (once the dwarves
issue is resolved as the kernels just can't boot due to it).

We're hoping to have the public result database within a few months,
and potentially also fully publicly available runs (independent of the
database), both of which should make the comparison process easier.


Veronika

> Cheers,
> Nathan
>
>
>

[Miguel Ojeda]

On Thu, Feb 4, 2021 at 7:09 PM 'Nick Desaulniers' via Clang Built
Linux <clang-bu...@googlegroups.com> wrote:
>
> We can move to vger+lore. Google groups was faster to setup, but has
> many downsides. I don't have resources to pursue archiving the
> existing list, and corralling the folks to set up vger and lore, but I
> can point people in the right direction and would be supportive.

I can submit the request like I did for Rust for Linux if you want.

Cheers,
Miguel

[Sedat Dilek]

+1

I am fine with that move to vger + lore.

- Sedat -

[Nick Desaulniers]

Moving our CKI friends to BCC, which I perhaps should have done sooner.

If Nathan's cool with it, I would appreciate it. I'm proud that we
have folks that would volunteer to do so, and more than happy to owe
more beers. I look forward to buying those! Being cooped up for so
long due to the pandemic has given me cabin fever.

> >
>
> +1
>
> I am fine with that move to vger + lore.
>
> - Sedat -

--
Thanks,
~Nick Desaulniers

[Nathan Chancellor]

Yes, that is fine with me! We tried to submit a request for a
vger.kernel.org list once but I do not think that we got any response.
Maybe now that the project is bigger and more relevant, they will be
more responsive to our request.

Migrating off of Google Groups would be nice (although vger has its own
problems at times).

Cheers,
Nathan

[Miguel Ojeda]

On Thu, Feb 4, 2021 at 11:22 PM Nick Desaulniers
<ndesau...@google.com> wrote:
>
> If Nathan's cool with it, I would appreciate it. I'm proud that we
> have folks that would volunteer to do so, and more than happy to owe
> more beers. I look forward to buying those! Being cooped up for so
> long due to the pandemic has given me cabin fever.

No worries (I'll just choose a very expensive import beer ;-)

Joking aside, it is just a few emails, the guys doing the work are
others (and they are quite busy, so this will likely take weeks or
months), so we can buy them the beers.

Cheers,
Miguel

[Miguel Ojeda]

On Thu, Feb 4, 2021 at 11:35 PM Nathan Chancellor <nat...@kernel.org> wrote:
>
> Yes, that is fine with me! We tried to submit a request for a
> vger.kernel.org list once but I do not think that we got any response.
> Maybe now that the project is bigger and more relevant, they will be
> more responsive to our request.

I will give it a try then.

Cheers,
Miguel

[Nick Desaulniers]

Now would be the time to use llvm-linux@vger or linux-llvm@vger if we
want to consider a rename.

I'm not sure precisely what the previous list was:
https://lists.linuxfoundation.org/mailman/listinfo/llvmlinux. I guess
it was llvm...@lists.linuxfoundation.org, not @vger?

--
Thanks,
~Nick Desaulniers

[Miguel Ojeda]

On Thu, Feb 4, 2021 at 11:51 PM Nick Desaulniers
<ndesau...@google.com> wrote:
>
> Now would be the time to use llvm-linux@vger or linux-llvm@vger if we
> want to consider a rename.

Let me know if you prefer that (I already sent the request with the
same name as the google groups one, but these things usually take
time, so I guess you have time to change).

On one hand, linux-llvm looks best (and follows the other linux-*
lists). On the other, people know the project by clang-built-linux...
So, up to you!

Cheers,
Miguel

[CKI Project]

Hello,

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Commit: 927002ed29e2 - Merge tag 'acpi-5.11-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Selftests compile: FAILED
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/04/623139

One or more kernel tests failed:

aarch64:
❌ selinux-policy: serge-testsuite

x86_64:

Host 2:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 3:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test

❌ selinux-policy: serge-testsuite
✅ lvm thinp sanity
✅ storage: software RAID testing
⚡⚡⚡ stress: stress-ng
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage nvme - rdma
🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
🚧 ⚡⚡⚡ storage: iSCSI parameters

Host 4:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load

Host 5:

⚡ Internal infrastructure issues prevented one or more tests (marked
with ⚡⚡⚡) from running on this architecture.
This is not the fault of the kernel that was tested.

✅ Boot test
⚡⚡⚡ kdump - sysrq-c
🚧 ⚡⚡⚡ kdump - file-load