make extract missing file errors with matching syzbot configs
Nigel Ploof
Hello,
I assume I am missing some simple configuration step so I apologize in advance, but any help would be appreciated.
Why am I getting missing file errors if I am on the right kernel tree with matching commits and configs to syzbot?
Following the instructions to update a descriptor here: https://github.com/google/syzkaller/blob/master/docs/syscall_descriptions.md I run the make extract command seen below. On running the command I get errors noted by the instructions "make extract complains about missing header files or constants undefined on all architectures, try to use the latest linux-next tree (or if it happens to be broken at the moment, try a slightly older linux-next tree)." but I am using the commits and configs directly from syzbot so by my understanding there is no chance for mismatch.
Specifically:
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git on at 53d061c19dc4 and syzkaller at d2ee9228
These commits/configs were pulled from syzbot on 14JUN23
ci2-upstream-usb now 22h21m 2325 62076 33 1693770 53d061c19dc4 .config 1d08h d2ee9228 1d00h
In the kernel source at that commit I can see that the folder drivers/crypto/qat is indeed missing.
Thanks again,
Nigel
$ make extract TARGETOS=linux SOURCEDIR=/home/dev/vm_share/fuzzing/usb
Makefile:32: run command via tools/syz-env for best compatibility, see:
Makefile:33: https://github.com/google/syzkaller/blob/master/docs/contributing.md#using-syz-env
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=d2ee9228555a1697ea8957f68403df8e4c55f42d+ -X 'github.com/google/syzkaller/prog.gitRevisionDate=20230613-18
1115'" -o bin/syz-extract ./sys/syz-extract
bin/syz-extract -build -os=linux -sourcedir=/home/dev/vm_share/fuzzing/usb
make mrproper ARCH=i386 generating linux/386... dev_qat_adf_ctl.txt: failed to run compiler: x86_64-linux-gnu-gcc [-nostdinc -w -fmessage-length=0 -O3 -I. -D__KERNEL__ -DKBUILD_MODNAME="-" -I/home/dev/vm_share/fuzzing/usb/arch/x86/include -I/tmp/syzkaller-kernel-build1826289015/arch/x86/include/generated/uapi -I/tmp/syzkaller-kernel-build1826289015/arch/x86/include/generated -I/home/dev/vm_share/fuzzing/usb/arch/x86/include/asm/mach-malta -I/home/dev/vm_share/fuzzing/usb/arch/x86/include/asm/mach-generic -I/tmp/syzkaller-kernel-build1826289015/include -I/home/dev/vm_share/fuzzing/usb/include -I/home/dev/vm_share/fuzzing/usb/arch/x86/include/uapi -I/
tmp/syzkaller-kernel-build1826289015/arch/x86/include/generated/uapi -I/home/dev/vm_share/fuzzing/usb/include/uapi -I/tmp/syzkaller-kernel-build1826289015/include/generated/uapi -I/home/dev/vm_share/fuzz
ing/usb -I/home/dev/vm_share/fuzzing/usb/include/linux -I/tmp/syzkaller-kernel-build1826289015/syzkaller -include /home/dev/vm_share/fuzzing/usb/include/linux/kconfig.h -m32 -O2 -pthread -Wall -Werror -W
parentheses -Wframe-larger-than=16384]
exit status 1
<stdin>:9:10: fatal error: drivers/crypto/qat/qat_common/adf_cfg_user.h: No such file or directory
compilation terminated.
...
sys.txt: failed to run compiler: s390x-linux-gnu-gcc [-nostdinc -w -fmessage-length=0 -O3 -I. -D__KERNEL__ -DKBUILD_MODNAME="-" -I/home/dev/vm_share/fuzzing/usb/arch/s390/include -I/tmp/syzkaller-kernel-build3801004866/arch/s390/include/generated/uapi -I/tmp/syzkaller-kernel-build3801004866/arch/s390/include/generated -I/home/dev/vm_share/fuzzing/usb/arch/s390/include/asm/mach-malta -I/home/dev/vm_share/fuzzing/usb/arch/s390/include/asm/mach-generic -I/tmp/syzkaller-kernel-build3801004866/include -I/home/dev/vm_share/fuzzing/usb/include -I/home/dev/vm_share/fuzzing/usb/arch/s390/include/uapi -I/tmp/syzkaller-kernel-build3801004866/arch/s390/include/generated/uapi -I/home/dev/vm_share/fuzzing/usb/include/uapi -I/tmp/syzkaller-kernel-build3801004866/include/generated/uapi -I/home/dev/vm_share/fuzzing/usb -I/home/dev/vm_share/fuzzing/usb/include/linux -I/tmp/syzkaller-kernel-build3801004866/syzkaller -include /home/dev/vm_share/fuzzing/usb/include/linux/kconfig.h -fPIE -O2 -pthread -Wall -Werror -Wparentheses -Wframe-larger-than=16384]
exit status 1
<stdin>:117:10: fatal error: fs/cifs/cifsglob.h: No such file or directory
compilation terminated.
make: *** [Makefile:239: extract] Error 1
Dmitry Vyukov
These headers were probably removed from the kernel.
If you find the latest commit that updated const files here:
https://github.com/google/syzkaller/commits/master/sys/linux
e.g. now it's:
https://github.com/google/syzkaller/commit/24040eec46931af5a73c2aad1ee0acd4052ddb25
It should specify what commit was used to extract these.
If you use any other commit, then you need to do some updates to
descriptions to match that kernel version.
Nigel Ploof
kernel tree is gregkh/usb.git which is at 4ee94d940789
Dmitry Vyukov
>
> Thank you for the quick reply!
>
> So if I understand your response correctly you are saying that if I use the exact commits listed in syzbot for both syzkaller and the kernel I should not expect that `make extract` will always work. And, that this is because there may have been updates to the descriptions or to the kernel which cause a mismatch between the two that needs to be accounted for, possibly by changing the descriptors.
>
> e.g.
>
> and
> syzkaller is at 90d4044e which means syzkaller is using descriptions generated from linux-next-20230331
> so these descriptions will not match gregkh/usb.git at 4ee94d940789 since it is not even the same tree where the consts were pulled
>
> Is this correct?
>
> If it is correct, my general question is how does syzbot account for this mismatch and is how syzbot is doing it how I should do it as well?
>
> More specifically, what adjustments are made in order to keep ci2-upstream-usb up to date?
> - Is there a patch that is applied to the syzkaller descriptions at some stage for ci2-upstream-usb?
>
> Does `make extract` just not factor into syzbot automation so this is only an issue for those running syzkaller locally?
If you just want to run stock syzkaller locally, then you don't need
to care about any of this.
syzkaller already comes with .const files, so you don't need to run
'make extract'.
If you are trying to extend syzkaller descriptions, then you 'make
extract' on syzkaller HEAD
and the exact commit that was used during the previous const update
(e.g. next-20230331).
In some cases you can get away with using kernel HEAD (mainline or
linux-next) as well,
if there were not significant changes in the kernel.
Periodically we/you also need to update descriptions/const to the
latest kernel HEAD,
to account for changes in the kernel (like the ones you hit in the first email).
If you are trying to reproduce a syzbot bug, then you take
kernel/syzkaller commits from
the dashboard. You don't need to run 'make extract' in this case
(again, syzkaller comes with .const files).
Yes, the kernel commit used to extract const files does not match the
syzbot tested kernel commit.
This is not a problem because the kernel is aways backwards compatible.
I.e. new interfaces can be added. Until we add descriptions/consts for
them, they are not tested.
But this requires not just re-running 'make extract', but also writing
descriptions for these
interfaces, so it's not something that can happen auto-magically anyway.
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/a856ca43-2795-4bf0-b2af-ed95f583070fn%40googlegroups.com.
Nigel Ploof
Dmitry Vyukov
>
> Thanks for asking Dmitry! I'm just extending descriptions for usb.
>
> Once I removed the descriptions that were causing errors when running 'make extract' against gregkh/usb.git on my local setup I had no problems.
>
> I was just trying to make sure that any descriptions I come up with are runnable on the usb instance (since it uses a different tree / config) should they be reasonable enough to submit, but as you say everything should be backwards compatible.
Any difference in interfaces would mean that a normal C program will
do different different things when run on these kernels.
The only case when it matters: when you are adding descriptions to a
very new feature that was just merged into one of the trees.
> Thanks for the help!
Well, you can repay the favor by sending a PR with the improvements :)