Bug#856474: stap: include runtime_defines.h not found

[Jack Henschel]

Package: systemtap
Version: 3.0-7
Severity: normal
Tags: stretch

Hi,

after I set up systemtap, I ran the example command provided in the documentation:
https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#testing
> stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'

GCC (6) complains upon compiling the C source code into the kernel module that it is missing C Headers:
> /tmp/stapHaiI5M/stap_a31ee9d9da16ac912b90eb18969508b5_1547_src.c:10:29: fatal error: runtime_defines.h: No such file or directory
> #include "runtime_defines.h"

For a more verbose log, please see the attachment.

-- System Information:
Debian Release: 9.0
APT prefers testing-debug
APT policy: (500, 'testing-debug'), (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages systemtap depends on:
ii libavahi-client3 0.6.32-2
ii libavahi-common3 0.6.32-2
ii libc6 2.24-9
ii libdw1 0.168-0.2
ii libelf1 0.168-0.2
ii libgcc1 1:6.3.0-6
ii libnspr4 2:4.12-6
ii libnss3 2:3.26.2-1
ii libsqlite3-0 3.16.2-2
ii libstdc++6 6.3.0-6
ii make 4.1-9.1
ii systemtap-common 3.0-7
ii systemtap-runtime 3.0-7

$ dpkg -l | grep linux-
ii linux-base 4.5 all Linux image base package
ii linux-compiler-gcc-6-x86 4.9.6-3 amd64 Compiler for Linux on x86 (meta-package)
ii linux-headers-4.9.0-1-amd64 4.9.6-3 amd64 Header files for Linux 4.9.0-1-amd64
ii linux-headers-4.9.0-1-common 4.9.6-3 all Common header files for Linux 4.9.0-1
ii linux-image-4.9.0-1-amd64 4.9.6-3 amd64 Linux 4.9 for 64-bit PCs (signed)
ii linux-image-4.9.0-1-amd64-dbgsym 4.9.6-3 amd64 Debug symbols for linux-image-4.9.0-1-amd64
ii linux-image-amd64 4.9+78 amd64 Linux for 64-bit PCs (meta-package)
ii linux-kbuild-4.9 4.9.6-3 amd64 Kbuild infrastructure for Linux 4.9
ii linux-libc-dev:amd64 4.9.6-3 amd64 Linux support headers for userspace development
ii util-linux-locales 2.29.1-1 all locales files for util-linux


$ dpkg -l | grep gcc
ii gcc 4:6.3.0-1 amd64 GNU C compiler
ii gcc-6 6.3.0-6 amd64 GNU C compiler
ii gcc-6-base:amd64 6.3.0-6 amd64 GCC, the GNU Compiler Collection (base package)
ii libgcc-6-dev:amd64 6.3.0-6 amd64 GCC support library (development files)
ii libgcc1:amd64 1:6.3.0-6 amd64 GCC support library
ii linux-compiler-gcc-6-x86 4.9.6-3 amd64 Compiler for Linux on x86 (meta-package)

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Wed, 2017-03-01 at 18:58 +0530, Ritesh Raj Sarraf wrote:
> But in my logs, runtime_defines.h is properly detected. I'll try to run the
> command on a Debian kernel to verify if it is broken or not.

Forgot to add the log.

```
rrs@learner:/tmp$ sudo stap -v -e 'probe vfs.read {printf("read performed\n");
exit()}'
[sudo] password for rrs: 
Pass 1: parsed user script and 118 library scripts using
106660virt/39568res/6268shr/33552data kb, in 200usr/20sys/258real ms.
Pass 2: analyzed script: 1 probe, 1 function, 4 embeds, 0 globals using
246904virt/181448res/7688shr/173796data kb, in 1810usr/660sys/4018real ms.
Pass 3: translated to C into
"/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c" using
246904virt/181640res/7880shr/173796data kb, in 10usr/50sys/60real ms.
In file included from /usr/share/systemtap/runtime/linux/runtime.h:209:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h: In function
‘__access_process_vm_’:
/usr/share/systemtap/runtime/linux/access_process_vm.h:36:61: error: passing
argument 6 of ‘get_user_pages_remote’ makes pointer from integer without a cast
[-Werror=int-conversion]
       ret = get_user_pages_remote (tsk, mm, addr, 1, write, 1, &page, &vma);
                                                             ^
In file included from ./include/linux/pid_namespace.h:6:0,
                 from ./include/linux/ptrace.h:8,
                 from ./include/linux/ftrace.h:13,
                 from ./include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
./include/linux/mm.h:1267:6: note: expected ‘struct page **’ but argument is of
type ‘int’
 long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,
      ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/share/systemtap/runtime/linux/runtime.h:209:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h:36:64: error: passing
argument 7 of ‘get_user_pages_remote’ from incompatible pointer type [-
Werror=incompatible-pointer-types]
       ret = get_user_pages_remote (tsk, mm, addr, 1, write, 1, &page, &vma);
                                                                ^
In file included from ./include/linux/pid_namespace.h:6:0,
                 from ./include/linux/ptrace.h:8,
                 from ./include/linux/ftrace.h:13,
                 from ./include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
./include/linux/mm.h:1267:6: note: expected ‘struct vm_area_struct **’ but
argument is of type ‘struct page **’
 long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,
      ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/share/systemtap/runtime/linux/runtime.h:209:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h:36:71: error: passing
argument 8 of ‘get_user_pages_remote’ from incompatible pointer type [-
Werror=incompatible-pointer-types]
       ret = get_user_pages_remote (tsk, mm, addr, 1, write, 1, &page, &vma);
                                                                       ^
In file included from ./include/linux/pid_namespace.h:6:0,
                 from ./include/linux/ptrace.h:8,
                 from ./include/linux/ftrace.h:13,
                 from ./include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
./include/linux/mm.h:1267:6: note: expected ‘int *’ but argument is of type
‘struct vm_area_struct **’
 long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,
      ^~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
scripts/Makefile.build:294: recipe for target
'/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.o' failed
make[1]: *** [/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.o]
Error 1
Makefile:1490: recipe for target '_module_/tmp/stapJQUBQF' failed
make: *** [_module_/tmp/stapJQUBQF] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_8ef46d39a073456b4762098be7ff765b_1501.ko" in
13140usr/1330sys/15462real ms.
Pass 4: compilation failed.  [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
2017-03-01 / 18:52:38 ♒♒♒  ☹  => 1  
```

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEQCVDstmIVAB/Yn02pjpYo/LhdWkFAli2zMMACgkQpjpYo/Lh
dWk/Ng/7BQas1KV5g0w6WaxiwuLhWwWkgOsQ/h4EtWUSj9bLW3kHR7NTLZlWp3cn
HP9Q7lBdogh8PEYmLsARiWdj8IS7xuIIX5Wfg9D4CKyEWWFrnfJdl+UtepuwYg36
bz6IWCUTgZCEBTfOWCNhyWg0t/PSheybYhJRMWYk7AYDcZUHQNYRW8oN5oncoulQ
o61/fCby52ZHq/CZSXkXZUTV6mWkOePQeRxn3kYSinBfAkGcYc9HCZICgzptwj8D
Pp84QTCSpg0jyFMEzPbTnpJ18YCBPh6EW+7+O2qqQizd02mTc3Hnec29nKYMef7j
xnF4gUkvaJponKPTtK34s51xx2Xk8RnHx0s14N/N79MlGKYoM0AmiQSkVhgX6to9
exWYIiw4eI1ZnXENB4suVmgSUrQ6Sx2HaDYCe1xHGbnwUQcU6bIVaUSq1p85Tx7G
FN2FNnXLSO6wKBmPty3ZfczsUsJg1uZwCYGgYXHy/OdScVQSvva4rqooh+ZZeyVD
sEStkoJcLe+sUphMF57xXmJCvIfYS/pgb5KMddIm9ANVMMOMj0cS+aByPRTiXbsR
I45b5fGxXxGleFKWQC8wXETC6W+sfkyaG7AzAh0feBKhpquwKqd2bWuIsD2mfeAf
LD5O744A18STaE/qPLEsiP445cPbFgxAmgBEw9Te3M3T8bz7Ck4=
=JZa3
-----END PGP SIGNATURE-----

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Control: tag -1 +moreinfo

On Wed, 2017-03-01 at 13:53 +0100, Jack Henschel wrote:
> after I set up systemtap, I ran the example command provided in the
> documentation:
> https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.htm
> l#testing
> > stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
>
> GCC (6) complains upon compiling the C source code into the kernel module that
> it is missing C Headers:
> > /tmp/stapHaiI5M/stap_a31ee9d9da16ac912b90eb18969508b5_1547_src.c:10:29:
> fatal error: runtime_defines.h: No such file or directory
> > #include "runtime_defines.h"

That file is provided by the systemtap-common package, which is properly marked
as a Depends for systemtap.

Over here, with a custom built kernel (Linux 4.10), I am getting compilation
errors, which are not relevant to the issue you've reported.

But in my logs, runtime_defines.h is properly detected. I'll try to run the
command on a Debian kernel to verify if it is broken or not.

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEQCVDstmIVAB/Yn02pjpYo/LhdWkFAli2zIcACgkQpjpYo/Lh
dWl7Eg//U23jiXQGAbybxlpgfn6GJA9OGuyAKHEqehvE/NBWrvHS6sSE/bU7nLBc
YENuEyR/u5dtQCzrYleDZL+pLrWVw7RZVbrwXzcXMPPwEPP1MIEFrCw9aGfk+j1j
OtDqAaUNkEfCLO4+Y7QLKKVbCWecjExHkY2SOF+p//Om2xOv1caWswytj7nqWspT
tKfTWj6FREMDyg2uivQkyyWqgWhmCtZm8ngQfK+zh9lpkBSJBn9cxGGKOsQgG99c
hthkWROMT+LxQJCV9JgBgfQcXeQvTgWgkaWKXSYFXUPTF4SJjORWEbhbQJh0ER8A
wxVoFlY3LG2OTVb2Z2Qp9WoY/cAjEw+muQuUoWDKoE2E9zcVhFeA2b15h04lzugE
la83E0Eqd9VwlluDRTajl+58PUgyARkiz6Sh3eE3CJqbnhAPZv6FX2qsOdWdMQ0q
6c+3OBo9uf26Q4is79o0mraqb/KQ0uULyjR2ItHlUVs+cwSl/HylcWYt9QbVjcep
zLJzjXN03P8meKCp0TpHgjanLqyD/3ntSCXR2rAxBCHF8Qzxsimw2RXcv1DEAXhF
vkfoOqLEnej5yVxtRhSYsDth+QJALpBfqgNEgvCmJqw1L0UOagPAQ7uoqcOQ/w9s
ZhyhYAuVAdXZ2c0rP0PcadTDMaUsvxk5Pd1v7+ij5Vv9JyJL7d4=
=2SEl
-----END PGP SIGNATURE-----

[Jack Henschel]

> Ritesh Raj Sarraf <r...@debian.org> hat am 1. März 2017 um 14:28 geschrieben:

> That file is provided by the systemtap-common package, which is properly marked
> as a Depends for systemtap.

I know, the package is installed and the files are present:
$ find /usr/share/systemtap/ -iname *runtime_defines.h*
/usr/share/systemtap/runtime/runtime_defines.h
/usr/share/systemtap/runtime/linux/runtime_defines.h

I'm kind of guessing there is something wrong the include statements the compiler gets:
> gcc-6 -Wp,-MD,/tmp/stapcukYUi/.stap_a31ee9d9da16ac912b90eb18969508b5_1547_aux_0.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/6/include -I/usr/src/linux-headers-4.9.0-1-common/arch/x86/include -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated -I/usr/src/linux-headers-4.9.0-1-common/include -I./include -I/usr/src/linux-headers-4.9.0-1-common/arch/x86/include/uapi -I/usr/src/linux-headers-4.9.0-1-common/include/uapi -I./include/generated/uapi -include /usr/src/linux-headers-4.9.0-1-common/include/linux/kconfig.h -I/usr/src/linux-headers-4.9.0-1-common//tmp/stapcukYUi -I/tmp/stapcukYUi -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -DCC_HAVE_ASM_GOTO -I/usr/src/linux-headers-4.9.0-1-common/include2/asm/mach-default -I/lib/modules/4.9.0-1-amd64/build -include /tmp/stapcukYUi/stapconf_0763f53e6c686e80bdfa9e79a1380e9b_711.h -freorder-blocks -fasynchronous-unwind-tables -Wframe-larger-than=512 -fno-ipa-icf -Wno-unused -Werror -I/usr/src/linux-headers-4.9.0-1-common/"/usr/share/systemtap/runtime" -DMODULE -DKBUILD_BASENAME='"stap_a31ee9d9da16ac912b90eb18969508b5_1547_aux_0"' -DKBUILD_MODNAME='"stap_a31ee9d9da16ac912b90eb18969508b5_1547"' -c -o /tmp/stapcukYUi/.tmp_stap_a31ee9d9da16ac912b90eb18969508b5_1547_aux_0.o /tmp/stapcukYUi/stap_a31ee9d9da16ac912b90eb18969508b5_1547_aux_0.c
> /tmp/stapcukYUi/stap_a31ee9d9da16ac912b90eb18969508b5_1547_src.c:10:29: fatal error: runtime_defines.h: No such file or directory
> #include "runtime_defines.h"

Is the argument `-I/usr/src/linux-headers-4.9.0-1-common/"/usr/share/systemtap/runtime"` valid?

> Over here, with a custom built kernel (Linux 4.10), I am getting compilation
> errors, which are not relevant to the issue you've reported.

Indeed, those errors do not have anything in common.

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Control: tag -1 +confirmed -moreinfo

On Wed, 2017-03-01 at 18:58 +0530, Ritesh Raj Sarraf wrote:

> But in my logs, runtime_defines.h is properly detected. I'll try to run the
> command on a Debian kernel to verify if it is broken or not.

I have been able to reproduce this on a Debian kernel.

rrs@chutzpah:~$ stap-
prep                                                          
22:25 ♒♒♒   ☺    
rrs@chutzpah:~$ stap -v -e 'probe vfs.read {printf("read performed\n");
exit()}'   
Pass 1: parsed user script and 119 library scripts using
115056virt/39720res/6372shr/33400data kb, in 130usr/0sys/340real ms.

Pass 2: analyzed script: 1 probe, 1 function, 4 embeds, 0 globals using

256132virt/182128res/7752shr/174476data kb, in 1770usr/160sys/6312real ms.

Pass 3: translated to C into

"/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c" using
256132virt/182376res/8000shr/174476data kb, in 0usr/0sys/7real ms.
/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c:10:29: fatal

error: runtime_defines.h: No such file or directory
 #include "runtime_defines.h"

                             ^
compilation terminated.
/usr/src/linux-headers-4.9.0-2-common/scripts/Makefile.build:298: recipe for
target '/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.o' failed
make[3]: *** [/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.o]
Error 1
make[3]: *** Waiting for unfinished jobs....
/usr/src/linux-headers-4.9.0-2-common/Makefile:1507: recipe for target
'_module_/tmp/stapUHT1si' failed
make[2]: *** [_module_/tmp/stapUHT1si] Error 2
Makefile:150: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:8: recipe for target 'all' failed
make: *** [all] Error 2

WARNING: kbuild exited with status: 2

Pass 4: compiled C into "stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538.ko" in
7560usr/630sys/10583real ms.

Pass 4: compilation failed.  [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

22:26 ♒♒♒    ☹  => 1  

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEQCVDstmIVAB/Yn02pjpYo/LhdWkFAli2/c8ACgkQpjpYo/Lh
dWkUbA/+PAOJ6+j07mVcMbv/ums5LWSQSb4SBLoRC9BRunUOOozA8yfWpflHRXrS
raLamnkE26xpd6HFZ0VVT3dFUH12malwn2GH2uEzVQMjiShxAqXaEJfuaLbuUIjk
T7Wg/TusRUhRBjAgoIAMIa5CiKjW/mxn0JcBZHiiNoppaOEpJ8jl+FuEvhtN3HLK
SbFT+1VcCxMK27yhEjuTDhoeSoCqmXhyIxvK9sSyGDCFG/90OKB3ggggOg5VkPcg
KxskOy0dIClRxUO2nvmxfsMeDImtZedUgsWZrq0/1+bCkeoHpCArqWqVfeAG86fl
STWW/04sMXG2cFhSEmV0igH/UbTuOYdzVz2XpAjZLMo+awNYs+6RhRiy4Hj2JV+Z
0EXseGdtHTojYKM1wZ6PI9nLtYbYIYgxiGezAQdaI9SeTiMCKHe5Llpa8OsRHo0c
7ts5FvEWL+7EpKdp+zO3UN4dQiG8nLwmKk9OGLyBmZq1FGyHg5qPWrWcyLLFMJc5
zf97Jpt082tZ2s2WVPLJdMH8zWSjb3c98EhS4qRE5F53bQdxVNEBbh2wx/Aokv7l
htBguJ4D/czjh/Qd4RbMybcKStvKPxu2Elkq1dLqF0RypJoAXlIGrrJDqtY3xZX8
DCmQCxvA0tkAb67uvpSbISBX9mEPZ7HvetSX175S2YZg2CVnYdY=
=wssJ
-----END PGP SIGNATURE-----

[Frank Ch. Eigler]

Hi -

> I have been able to reproduce this on a Debian kernel.

> [...]

> /tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c:10:29: fatal
> error: runtime_defines.h: No such file or directory
>  #include "runtime_defines.h"
>                              ^
> compilation terminated.

> [...]

We've received intermittent reports of this sort of thing happening on
a few non-RH platforms. A common element seemed to be some
nonstandard kernel build tree environment - some sort of customized
out-of-tree module builder/processing script or makefile, which was
confused by quotations in the stap-synthesized Makefile. We have not
gotten to the bottom of it yet, but it didn't seem like a stap bug.

https://sourceware.org/ml/systemtap/2017-q1/msg00025.html

- FChE

[Frank Ch. Eigler]

Hi -

> [...] Running in verbose mode, it shows: [...] Those paths are
> provided by the linux-headers-xxx-amd64 (or equivalent arch)
> package. In Debian, build/ and source/ paths are pointing to
> separate locations, provided by separate package. [...]

Could you provide a make V=1 transcript of any nontrivial out-of-tree
module (plus its Makefile) that builds & works on your kernel?

- FChE

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Wed, 2017-03-01 at 12:07 -0500, Frank Ch. Eigler wrote:
> We've received intermittent reports of this sort of thing happening on
> a few non-RH platforms.  A common element seemed to be some
> nonstandard kernel build tree environment - some sort of customized
> out-of-tree module builder/processing script or makefile, which was
> confused by quotations in the stap-synthesized Makefile.  We have not
> gotten to the bottom of it yet, but it didn't seem like a stap bug.
>
> https://sourceware.org/ml/systemtap/2017-q1/msg00025.html

Thank you for replying quick. I realized your email after I sent another one.
That email has some details about how the Debian kernel is organized, especially
the linux-headers package split into 2 and installing files into 2 different
paths.

rrs@chutzpah:~$ ls /usr/src/
broadcom-sta-6.30.223.271/    linux-headers-4.9.0-2-common/
linux-headers-4.9.0-2-amd64/  linux-kbuild-4.9@
23:00 ♒♒♒   ☺    


IIRC, on the RH side, it used to be a single path. Maybe still is the same.

Similarly, I did not see this error on my custom kernel too.

rrs@learner:/var/tmp$ ls /usr/src/
linux-headers-4.10.1+/        linux-headers-4.9.0-2-common/  linux-kbuild-4.9@
linux-headers-4.9.0-2-amd64/  linux-headers-4.9.9+/
2017-03-01 / 23:00:52 ♒♒♒  ☺  
rrs@learner:/var/tmp$ uname -a
Linux learner 4.10.1+ #20 SMP Sun Feb 26 21:22:45 IST 2017 x86_64 GNU/Linux
2017-03-01 / 23:00:58 ♒♒♒  ☺  


I suspect the same, that this may be specific to Debian Linux kernel.
I have also had eBPF compiler failing on something similar (Debian Bug: 849841).

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEQCVDstmIVAB/Yn02pjpYo/LhdWkFAli3Be8ACgkQpjpYo/Lh
dWl1ehAAiYole5d4WCRL/LOKcvzNlJw+KWQVecMRgU44K1cZsm/UqWXmIxMwkU35
csFsbjS8XK5fpz64lzpQkMqmtJ3hJwuX61iI+SHLbiHk1iN2UOQZ6ZkGsKUBgr6c
HjXvpKKM3JaQCeMtkAoFanH1a8S0u6bemPD3inLk69YtVpG/1fNQT08FYZUGBwZP
yg/AOqfX5/h+aCA8y0P0oF2jEjYtAvxAT4iog7QnYCu/KpNbdD4RzvOIM5vAb7if
r45oId5k9OwJKn/DHKZj6rYj2v7+oSkEcPtTJdLOUkIsnqE88MAcYqY498HRsiHv
tCVAln5DmaZNJzE6DTB5efSXk86nFlVhFYHXj3+j3z7IJ6RHCW5aZ2rVniMpu4/E
73MT55louz/AGwSrib+MEfiodpu3GI3Dk7uhDs36QxAObGb94DjNzjb5j+ZwPZRO
KuFa1jVWv9puIyXhX3J6Q98oU8rJPdZuA0crtDJOueSdHh1GLs6jFIMmpsN4Vup0
DPcLt6xq0rcakiWUKCZ+hfYbRcjRPJsVcN2nfmR/+vBX7s+eElcXgRrc7re/A979
3sE3uSE7fgmA3U33g7gHCieRKedgyaNigWhA8LVxT0WbNh6YBwKh6M6JluJ7azHH
bRYmsTBv2cyoOTTz7ZejaVU+XayANuOfBDp0MRrYAO6aDcUEPtc=
=Qg77
-----END PGP SIGNATURE-----

[Ritesh Raj Sarraf]

Hello Frank,

We have this bug report in Debian, where systemtap is looking for a local header
file rather than using the system provided one (provided by the systemtap-common
package).

The translate.cxx file writes the code. This looks like deliberately done, under
certain conditions. I am not sure what condition triggers this code.

Can you please provide any help ?

Running in verbose mode, it shows:

test -e include/generated/autoconf.h -a -e include/config/auto.conf ||
(                \
echo >&2;-                                                       \
echo >&2 "  ERROR: Kernel configuration is invalid.";           \
echo >&2 "         include/generated/autoconf.h or include/config/auto.conf are
missing.";\
echo >&2 "         Run 'make oldconfig && make prepare' on kernel src to fix
it.";      \
echo >&2 ;                                                      \
/bin/false)

Those paths are provided by the linux-headers-xxx-amd64 (or equivalent arch)
package. In Debian, build/ and source/ paths are pointing to separate locations,
provided by separate package.

rrs@chutzpah:/lib/modules/4.9.0-2-amd64$ ls -l
total 4080
lrwxrwxrwx  1 root root      36 Feb 17 18:48 build -> /usr/src/linux-headers-
4.9.0-2-amd64/
drwxr-xr-x 12 root root    4096 Feb 20 22:57 kernel/
-rw-r--r--  1 root root 1005836 Mar  1 19:55 modules.alias
-rw-r--r--  1 root root  965121 Mar  1 19:55 modules.alias.bin
-rw-r--r--  1 root root    4027 Feb 27 21:28 modules.builtin
-rw-r--r--  1 root root    5485 Mar  1 19:55 modules.builtin.bin
-rw-r--r--  1 root root  393918 Mar  1 19:55 modules.dep
-rw-r--r--  1 root root  543473 Mar  1 19:55 modules.dep.bin
-rw-r--r--  1 root root     402 Mar  1 19:55 modules.devname
-rw-r--r--  1 root root  131711 Feb 27 21:28 modules.order
-rw-r--r--  1 root root     523 Mar  1 19:55 modules.softdep
-rw-r--r--  1 root root  489848 Mar  1 19:55 modules.symbols
-rw-r--r--  1 root root  605149 Mar  1 19:55 modules.symbols.bin
lrwxrwxrwx  1 root root      37 Mar  1 22:52 source -> /usr/src/linux-headers-
4.9.0-2-common/
drwxr-xr-x  3 root root    4096 Mar  1 19:55 updates/
22:52 ♒♒♒   ☺    


The full verbose log from stap is attached with this email.

I suspect it to be a Debian kernel problem but wanted to check with you first.


Thanks,
Ritesh

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Thanks to hints from Frank, I can see the culprit now.

- -Werror -I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime" -DMODULE


The full compile command is:

gcc-6 -Wp,-MD,/tmp/staptZYmtX/.stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/6/include -I/usr/src/linux-headers-4.9.0-2-common/arch/x86/include -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated -I/usr/src/linux-headers-4.9.0-2-common/include -I./include -I/usr/src/linux-headers-4.9.0-2-common/arch/x86/include/uapi -I/usr/src/linux-headers-4.9.0-2-common/include/uapi -I./include/generated/uapi -include /usr/src/linux-headers-4.9.0-2-common/include/linux/kconfig.h -I/usr/src/linux-headers-4.9.0-2-common//tmp/staptZYmtX -I/tmp/staptZYmtX -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -DCC_HAVE_ASM_GOTO -I/usr/src/linux-headers-4.9.0-2-common/include2/asm/mach-default -I/lib/modules/4.9.0-2-amd64/build -include /tmp/staptZYmtX/stapconf_d8a215ec355839da11c41ac20d4e7e1e_702.h -freorder-blocks -fasynchronous-unwind-tables -Wframe-larger-than=512 -fno-ipa-icf -Wno-unused -Werror -I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime" -DMODULE -DKBUILD_BASENAME='"stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0"' -DKBUILD_MODNAME='"stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538"' -c -o /tmp/staptZYmtX/.tmp_stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.o /tmp/staptZYmtX/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.c
/tmp/staptZYmtX/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c:10:29: fatal error: runtime_defines.h: No such file or directory

iQIzBAEBCgAdFiEEQCVDstmIVAB/Yn02pjpYo/LhdWkFAli3CJEACgkQpjpYo/Lh
dWmjwxAAmK4QUX/kzfTbfbjbUf+id4hGwJMyusbhYeaIS29ntEv4v/MqkpqzPiVt
rBNkImwm3VG8BQITzP8j0zbQhH2nVT2xE6eK3YCJ6CghtV/IVaZDU8QOodgZ2SP7
nsK3Z68q4/dfTql3NGo1YCJIxEcmQcEj2I0g8k6qBCjNezDlYfwfkNCK0u4UmcLb
/e0Ca2uPF+OOagtZmV8TdzEmxoVpSrAd/YOyHjlsLhMnetMwF+4XZbQVnof+2Iek
nblSATi7d7ds8LwfFO7vQu2towkqYJQXBEMQDhvgLJFAyH7AqiNdhLag7UWrvSlW
qycQpDpbNRcdNGuSYWbP577sHhGyQoK8Yii9r63Cp1YY2J83+nxYzQqwQFmiEmmb
u1yiYgPdfH3Vvsm6nVzsXVF23vCccs2kywcPTIS06OLJOYBqjmydQZqCbVFRBy07
sdyMD5jQGz2rzL2LQEQf62z3jep5svCmPtXBJ4ls7+FX00bh5Sh1GbuHH5QElcdS
c/SqhBAV47sro9IS6NwY9lCuFsTz59B9bBiFT8ir4zNKFMJ64Zbq9bOoTJtSHdpW
KkpFEa5xAdXi33OK/SBdXEU0TmSgMxX3W5Rrh9P6ZG3axZVzqdKZyiFqOfU/XbVf
dBeeGnF0lQ72U2o7wCyIbxpRscCKIigE7CpskPgQMlrC88Z0rYY=
=3ZSv
-----END PGP SIGNATURE-----

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

This is specific to the linux-headers package and has nothing to do with
systemtap. Please ignore this one.

On Wed, 2017-03-01 at 22:56 +0530, Ritesh Raj Sarraf wrote:
> Running in verbose mode, it shows:
>
> test -e include/generated/autoconf.h -a -e include/config/auto.conf ||
> (                \
> echo >&2;-                                                       \
> echo >&2 "  ERROR: Kernel configuration is invalid.";           \
> echo >&2 "         include/generated/autoconf.h or include/config/auto.conf
> are
> missing.";\
> echo >&2 "         Run 'make oldconfig && make prepare' on kernel src to fix
> it.";      \
> echo >&2 ;                                                      \
> /bin/false)

- --

Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEQCVDstmIVAB/Yn02pjpYo/LhdWkFAli3B44ACgkQpjpYo/Lh
dWm2Ng/8Dd1mJUiEFClEYVb2nrmqh7HvE9SvZ7fFT1wx7CHEA9bcwrTeprBhmFj7
i4WxNaREVraPPUtxply6nQvUP+Y3IFFouD6qfCyFCpm6Vt9VUVLsV9z1L4y2VMx1
zKbqKMqq4ogyOMFfXHe8R264/uZ+3fiOnXfL6Z5NkcgRaB3mbllECxXKccMx5V/D
4kWuvn340Nt5tbJIY8zF2o4GMvDhYJ3sWrlqC37/4agWBozxQgyqEnRCO7LldZev
T47DKn7GcTsFJAdlUWL88pPzgr5Qaz7BCvgcMK5Thc91Pm1CM5y9nZIMv+bfdpnG
FuhuClRFxac0rGDcyvx5tSY15fnw/rVbEWUdl5/ZGhAZZnL85G4Bn/dydosHu4Ss
imgMN46UzsI0OvmXgHB6wvHyusOeDYzVzDiQPv0gSA2HURk0MDq4BbdXMSRP+Ge7
DPqE6PVMKKVdRPD3nCQnEUuUiQuIEZdc94B/AYSQSbNKRdJK1mu8fJtLh2KOrZnn
tkdrZn2Q8erNMw+Cn51MWnCWT8pUGTu++1zpdKPdwApOdDNU3Xi8IZNuStDvLy2E
14KheXfO4R72pqu2kGyQLyh11gjctemBfGcXC+Rn5bnNGXJB8bb3UUsGrkQdSBM7
TEwwjBrcN9ehp3huJ9n3zn9YdcUCm8QFTX2cnig2Rv1JFA4EOng=
=m61F
-----END PGP SIGNATURE-----

[Ritesh Raj Sarraf]

Here's a log attached. It was generated with V=1 through dkms on out-of-tree
module for broadcom-sta driver.

[Jack Henschel]

On 03/01/2017 06:44 PM, Ritesh Raj Sarraf wrote:
> Thanks to hints from Frank, I can see the culprit now.
>

> -Werror -I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime" -DMODULE
>
>
> The full compile command is:
>
> gcc-6 -Wp,-MD,/tmp/staptZYmtX/.stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/6/include -I/usr/src/linux-headers-4.9.0-2-common/arch/x86/include -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated -I/usr/src/linux-headers-4.9.0-2-common/include -I./include -I/usr/src/linux-headers-4.9.0-2-common/arch/x86/include/uapi -I/usr/src/linux-headers-4.9.0-2-common/include/uapi -I./include/generated/uapi -include /usr/src/linux-headers-4.9.0-2-common/include/linux/kconfig.h -I/usr/src/linux-headers-4.9.0-2-common//tmp/staptZYmtX -I/tmp/staptZYmtX -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -DCC_HAVE_ASM_GOTO -I/usr/src/linux-headers-4.9.0-2-common/include2/asm/mach-default -I/lib/modules/4.9.0-2-amd64/build -include /tmp/staptZYmtX/stapconf_d8a215ec355839da11c41ac20d4e7e1e_702.h -freorder-blocks -fasynchronous-unwind-tables -Wframe-larger-than=512 -fno-ipa-icf -Wno-unused -Werror -I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime" -DMODULE -DKBUILD_BASENAME='"stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0"' -DKBUILD_MODNAME='"stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538"' -c -o /tmp/staptZYmtX/.tmp_stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.o /tmp/staptZYmtX/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.c
> /tmp/staptZYmtX/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c:10:29: fatal error: runtime_defines.h: No such file or directory
>

I mentioned that in one of my earlier mails (#22).
Also I can confirm this not an issue with the _packaging_ of systemtap in Debian,
I downloaded and compiled the latest release tarball (3.1) directly from upstream and still got the same error.

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Wed, 2017-03-01 at 19:48 +0100, Jack Henschel wrote:
> I mentioned that in one of my earlier mails (#22).

Sorry about that. I completely missed that one.

There's also another odd thing I noticed.

```
- -I/usr/src/linux-headers-4.9.0-1-common//tmp/stapcukYUi -I/tmp/stapcukYUi
```

> Also I can confirm this not an issue with the _packaging_ of systemtap in
> Debian,
> I downloaded and compiled the latest release tarball (3.1) directly from
> upstream and still got the same error.

Thanks for the information.

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEQCVDstmIVAB/Yn02pjpYo/LhdWkFAli34MYACgkQpjpYo/Lh
dWkwqg/+K+uzl+0XK+NBHtoPDXFtz1yjxz6Nc0IrrqP62ObeikX9EX5uTWcKnn/L
vx6a/jWBlTMnY6yzNGyF1UXPlzYs1NFo6X8seGN1FGs1KylWTOLyAl1z/ZUQdY87
og3TAtWZzLgG8G5wxmhqsrvY98FTAyWOZahZW3CGPg4DwNXjcTpAZZgf6+X4jcnv
d72evuQKbo1NlqWWClWaRsFDxfT7rQdlKxjgZpcyKtgtxtEaJMh6AsQNY2NRFVDw
zgtztkQeurOnVf3JknIflXrNrw4S4jRC/Rxt6cb6a4UWT9FWv+NmOKNLktsv7cm/
Qqr/B1FiW1WTLW9yMNNRsJ73MG9WcESTqRdskzfSVyfwdfrfkq0PlugROPsGQs68
aSVHUF4f4haQxtmzHZg5MYTfnK/fPNFhGf335LFjJdYUyT35yfwJkCuM1EOp6Sif
4AnxhK81p3p5wU8JwhseOgV7dO4OoQ4DCB9XJA9InQkg/WiVct+B9hAusBUGqSV0
s3juvdO095qyW6X8/SofhG8MfCNsHHXI9gUmHXMwmrowFXmL+NmBoiIVagAtlKIO
iSg6YnSBJElbg+9/37+WZjpGLhS+Bg4/mt7w5l0k6yzfS8RbJXXtLChkikZGxZc7
YS5jl4VDaXQnDOt3cTj74GJxJ0mx77HjX+RtaEba+rpdihmMH1M=
=9pKd
-----END PGP SIGNATURE-----

[Ritesh Raj Sarraf]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Control: severity -1 grave
Control: tag -1 +help

On Thu, 2017-03-02 at 14:37 +0530, Ritesh Raj Sarraf wrote:
>
> > Also I can confirm this not an issue with the _packaging_ of systemtap in
> > Debian,
> > I downloaded and compiled the latest release tarball (3.1) directly from
> > upstream and still got the same error.
>
> Thanks for the information.
>

I just pushed 3.1 to Experimental, the reason being 3.0 is going to be useless
for Debian Stretch. We need 3.1 to get it working proper with 4.9 kernel too.

This upload (3.1) still doesn't solve the runtime_defines.h issue with Debian
kernels. But, at least, we know now that the issue is something specific to
Debian kernels only. As is shown below, and also mentioned by Frank.

Given that this bug makes systemtap useless, I have marked the bug accordingly
to grave. I have also marked this for help because I don't really have that much
"free" time to investigate it right now.

@Jack: I would like to reassign this bug to the kernel team. But it'd be nice if
I could get more users confirm that the bug doesn't appear with non-Debian
kernels. Would you be in a position to pick systemtap from Experimental, and
build a pristine upstream kernel image (you need to clone Linus's tree and just
run the bindeb-pkg make target) and test the combination together ?


Here are my results.

rrs@learner:~$ stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 470 library scripts using
123812virt/48564res/6640shr/42124data kb, in 110usr/10sys/129real ms.
Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using
280416virt/206592res/7948shr/198728data kb, in 1400usr/190sys/1594real ms.
Pass 3: using cached
/home/rrs/.systemtap/cache/f8/stap_f83081804fadc6e30161178b5574195d_2502.c
Pass 4: using cached
/home/rrs/.systemtap/cache/f8/stap_f83081804fadc6e30161178b5574195d_2502.ko
Pass 5: starting run.
read performed
Pass 5: run completed in 10usr/30sys/530real ms.
2017-03-02 / 22:58:35 ♒♒♒  ☺  

rrs@learner:~$ uname -a

Linux learner 4.10.1+ #20 SMP Sun Feb 26 21:22:45 IST 2017 x86_64 GNU/Linux

2017-03-02 / 22:58:39 ♒♒♒  ☺  

vs

rrs@chutzpah:~/Community/Packaging/systemtap (master)$ stap -ve 'probe begin { log("hello world") exit () }'
Pass 1: parsed user script and 471 library scripts using 123528virt/48648res/6764shr/41840data kb, in 140usr/0sys/149real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 124452virt/49620res/6956shr/42764data kb, in 10usr/0sys/5real ms.
Pass 3: translated to C into "/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.c" using 124452virt/49620res/6956shr/42764data kb, in 0usr/0sys/1real ms.
/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.c:10:29: fatal error: runtime_defines.h: No such file or directory

#include "runtime_defines.h"
^
compilation terminated.

/usr/src/linux-headers-4.9.0-2-common/scripts/Makefile.build:298: recipe for target '/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.o' failed
make[3]: *** [/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.o] Error 1

make[3]: *** Waiting for unfinished jobs....

/usr/src/linux-headers-4.9.0-2-common/Makefile:1507: recipe for target '_module_/tmp/stapJNAO0g' failed
make[2]: *** [_module_/tmp/stapJNAO0g] Error 2

Makefile:150: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:8: recipe for target 'all' failed
make: *** [all] Error 2
WARNING: kbuild exited with status: 2

Pass 4: compiled C into "stap_eed6170377f19ad002c0d2b53e5e7956_1067.ko" in 7810usr/760sys/9195real ms.

Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

23:00 ♒♒♒ ☹ => 1

rrs@chutzpah:~/Community/Packaging/systemtap (master)$ uname -a
Linux chutzpah 4.9.0-2-amd64 #1 SMP Debian 4.9.13-1 (2017-02-27) x86_64 GNU/Linux
23:00 ♒♒♒ ☺

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEQCVDstmIVAB/Yn02pjpYo/LhdWkFAli4WRoACgkQpjpYo/Lh
dWmNDw/+N08GXR4COXzZh+PU57mp1uHuK/8x4Atj3xGOrfvpbSZvo+uH0fVi9sRx
Nu7TK5z5EJ45RrdMwI2evrP9lx+wpUUDGlFUTLzh+J1GoN3Ea/T+TsWc/3CvXB8L
BKSfxI5FNQNM3Goq4ouTQUsSnb1YRoBN6vMSDGKS+5d6/n/0N6GoEk4ZLY4xOsrv
p8XnIB6RxiS3oT2hmjTewdbxd1l8gpyQv5Nhgg8y7tL64oo5Cd2+VSAO+YKXI2JN
LsNwgPpllMXW7LkhKDrYpmyStFmlNViLCrSiWiKZyMajRuCieV+xsdl8oVpxybOd
X9ZkGTzT6/4dLQ6P+2yfEj5hJatrbWXLQiysu6Npej+durQX+n5FBuIAU0/fdOzY
3eCh2nT63inwr3GLfZPccylggRBnC5lZV0p4WAL733kiIfaRfA2biN7c92hm1PdA
zQOazT2i6atfA/wjtob/whVa/R/CDnuiztG3Le7NG3qv5oYKStTqF2S4CJPzszta
ftpzw3PxzrFrOoiWCvpPY5rVvCx5hnsOfce64uVe0+tjK9ikepnW0pe28HihsDAT
Eh0g3fLXfnS8NkpV/ZWptVZ5iT3rYR4lfhUmbrEgqtk78AKzi8FTXdtwhaSFE9mH
WbhK+v2om8TfJJUCE8U2cU4B55vXSv4nR+bhDW+6qrVojwcDdwg=
=lo3e
-----END PGP SIGNATURE-----

[Jack Henschel]

On 03/02/2017 06:40 PM, Ritesh Raj Sarraf wrote:
> I just pushed 3.1 to Experimental, the reason being 3.0 is going to be useless
> for Debian Stretch. We need 3.1 to get it working proper with 4.9 kernel too.

Did I get that right: you are planning to ship SystemTap 3.1 with Stretch?
(If so: great, I'm all for it!)

> @Jack: I would like to reassign this bug to the kernel team. But it'd be nice if
> I could get more users confirm that the bug doesn't appear with non-Debian
> kernels. Would you be in a position to pick systemtap from Experimental, and
> build a pristine upstream kernel image (you need to clone Linus's tree and just
> run the bindeb-pkg make target) and test the combination together ?

Sure, I always wanted to build my own kernel, I guess I finally have a good reason to do so :-)

[Frank Ch. Eigler]

Hi -

On Thu, Mar 02, 2017 at 01:25:03PM -0800, Gerald Turner wrote:
> FWIW, I attempted to kludge around the mangled include argument by
> running stap with "-B 'EXTRA_CFLAGS += -I/usr/share/systemtap/runtime'".
> This seems to work around the compilation failure of missing
> runtime_defines.h in include path, however compiliation then fails due
> to what appears to be API changes in Linux kernel 4.9.

Those errors all have systemtap-side porting logic associated with
them, a combination of compile-time (c #ifdefs) and build-time (kbuild
flags) conditionals. I suspect your workaround hid one problem and
created these others. We need to keep looking for the cause.

- FChE

[Gerald Turner]

I modified systemtap buildrun.cxx so that the Makefile it produces
during Pass 3 doesn't have quotes around the include path, i.e.:

Before:

EXTRA_CFLAGS += -I"/usr/share/systemtap/runtime"

After:

EXTRA_CFLAGS += -I/usr/share/systemtap/runtime

Then stap works fine against the Debian kernel, running 4.10-1~exp1 at
the moment.

I cannot fathom how Debian's linux-headers packages would affect this
argument, but for the record, here's systemtap running gcc in Pass 3,
elided:

gcc-6 ... -I/usr/src/linux-headers-4.10.0-trunk-common/"/usr/share/systemtap/runtime" ...

And with buildrun.cxx modified so that quotations are removed from the
Makefile:

gcc-6 ... -I/usr/share/systemtap/runtime ...

How is /usr/src/linux-headers-4.10.0-trunk-common/ getting interpolated
into that path argument? Attached a Makefile produced by stap - maybe
somebody could guess why that interpolation occurs at line 125?

Apologies if I'm adding more noise than signal to the bug report. Frank
Ch. Eigler already addressed the quotation issue earlier in this bug
report.

--
Gerald Turner <gtu...@unzane.com> Encrypted mail preferred!
OpenPGP: 4096R / CA89 B27A 30FA 66C5 1B80 3858 EC94 2276 FDB8 716D

[Ben Hutchings]

Control: tag -1 upstream patch

On Fri, 03 Mar 2017 23:10:56 +0530 Ritesh Raj Sarraf <r...@debian.org> wrote:
[...]
> Dear (Linux) Kernel Team:
> 
> We currently have systemtap 3.0 in Debian testing  where this problem was
> reported.
> 
> In Experimental, I have also pushed the 3.1 release. The 3.1 release is reported
> to work perfectly in our tests with the custom built kernel. The problem is only
> seen with the Debian built Linux kernels.
[...]

I investigated this and found that it occurs when the kernel source and
object trees are separate (an "out-of-tree" build, not to be confused
with out-of-tree modules). We separate them in Debian kernel header
packages to avoid duplicating source files for each flavour.

When this is the case, the compiler is called in the root of the
object tree, and the kernel build system adjusts -I options in the
compiler flags to refer to subdirectories of the source tree if
necessary. Any directory name beginning with /, ./ or ../ is excluded
from this adjustment.

systemtap uses -I"/usr/share/systemtap/runtime", which ought to be
excluded... but make has no understanding of shell quoting, so it is
wrongly adjusted to something like
-I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime"
and runtime_defines.h cannot be found.

So my fix is to delete quotes before checking for the exclusions:

--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -206,7 +206,7 @@ hdr-inst := -f $(srctree)/scripts/Makefile.headersinst obj
# Prefix -I with $(srctree) if it is not an absolute path.
# skip if -I has no parameter
addtree = $(if $(patsubst -I%,%,$(1)), \
-$(if $(filter-out -I/% -I./% -I../%,$(1)),$(patsubst -I%,-I$(srctree)/%,$(1)),$(1)))
+$(if $(filter-out -I/% -I./% -I../%,$(subst $(quote),,$(subst $(squote),,$(1)))),$(patsubst -I%,-I$(srctree)/%,$(1)),$(1)))

# Find all -I options and call addtree
flags = $(foreach o,$($(1)),$(if $(filter -I%,$(o)),$(call addtree,$(o)),$(o)))
--- END ---

Ben.

--
Ben Hutchings
All the simple programs have been written, and all the good names
taken.

[Ben Hutchings]

On Sat, 2017-03-04 at 01:39 +0000, Ben Hutchings wrote:
[...]
> I investigated this and found that it occurs when the kernel source and
> object trees are separate (an "out-of-tree" build, not to be confused
> with out-of-tree modules).  We separate them in Debian kernel header
> packages to avoid duplicating source files for each flavour.
>
> When this is the case, the compiler is called in the root of the
> object tree, and the kernel build system adjusts -I options in the
> compiler flags to refer to subdirectories of the source tree if
> necessary.  Any directory name beginning with /, ./ or ../ is excluded
> from this adjustment.
>
> systemtap uses -I"/usr/share/systemtap/runtime", which ought to be
> excluded... but make has no understanding of shell quoting, so it is
> wrongly adjusted to something like
> -I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime"
> and runtime_defines.h cannot be found.

By the way, kbuild has been doing this ever since it started supporting
out-of-tree builds (around Linux 2.6.0). However, both the adjusted
and original -I options were used. Since Linux 4.8 only the adjusted
option is used.

[Jack Henschel]

On Sat, 04 Mar 2017 01:39:58 +0000 Ben Hutchings <b...@decadent.org.uk> wrote:
> So my fix is to delete quotes before checking for the exclusions:

Wow Ben, that was a really quick fix!

I'll test it on Monday and report back.

[Ritesh Raj Sarraf]

Thank you very much, Ben. I tested with your fix and it resolves the issue.

[Jack Henschel]

On 03/04/2017 10:24 AM, Ritesh Raj Sarraf wrote:
> Thank you very much, Ben. I tested with your fix and it resolves the issue.

Can confirm, the patch fixes it.

Thank you Ben and Ritesh for your super quick actions! :-)

Greetings
Jack