syzkaller on kvm

[Dmitry Vyukov]

On Wed, Oct 14, 2015 at 8:35 PM, Sasha Levin <sasha...@oracle.com> wrote:
> On 10/14/2015 11:20 AM, Dmitry Vyukov wrote:
>> On Wed, Oct 14, 2015 at 12:29 AM, Sasha Levin <sasha...@oracle.com> wrote:
>>> On 10/13/2015 04:55 AM, Dmitry Vyukov wrote:
>>>> On Tue, Oct 13, 2015 at 4:51 AM, Sasha Levin <sasha...@oracle.com> wrote:
>>>>>> On 10/12/2015 04:40 AM, Dmitry Vyukov wrote:
>>>>>>>> // autogenerated by syzkaller (http://github.com/google/syzkaller)
>>>>>>
>>>>>> Hey Dmitry,
>>>>>>
>>>>>> Is there an easy way to run just the fuzzer inside a given VM? I'm doing
>>>>>> my testing inside kvmtool guests rather than qemu, and would rather not
>>>>>> use additional infrastructure or carry patches on top of your tree.
>>>> +syzkaller group
>>>>
>>>> Hi Sasha,
>>>>
>>>> syzkaller is a stateful fuzzer as opposed to trinity/iknowthis. The
>>>> state is meant to amplify fuzzer efficiency over time (at least that's
>>>> the plan). So a deployment should include a persistent process that
>>>> collects/distributes the state across a set of VMs/physical machines.
>>>>
>>>> I would happily incorporate code needed to run syzkaller with
>>>> kvmtools-based VMs. I understand the unwillingness to carry patches,
>>>> and in the end your setup can well be better than mine.
>>>>
>>>> The code to manage qemu-based VMs is in:
>>>> https://github.com/google/syzkaller/blob/master/vm/qemu/qemu.go
>>>> I guess we just need to alter the commands to create/start kvm-tools
>>>> based VMs. Can you advise on what are these commands?
>>>
>>> It's mostly simple:
>>>
>>> ./vm sandbox -n mode=user -k [kernel bzimage] -d dir --no-dhcp -m [mem] -c [cpus] -p "kernel params" -- /runtrin.sh
>>>
>>> Which will utilize virtio-9p transport to boot a kernel using 'dir' as rootfs,
>>> and then run 'runtrin.sh' once the vm has booted (which is just my bootstrap
>>> to start trinity right now.
>>>
>>> The question here is what do you expect on the networking side? kvmtool has a
>>> pretty simple usermode networking, but it might need tap...
>>
>>
>> What is ./vm binary? Is it the same as lkvm binary described here
>> https://git.kernel.org/cgit/linux/kernel/git/will/kvmtool.git/tree/Documentation/kvm-run.txt
>> ?
>
> Yeah, same one.
>
>> On networking side I expect two-way connectivity. I use scp/ssh to
>> deploy and run necessary binaries into a VM. And the binary inside of
>> VM connects back to a binary running on host. I use the following net
>> setup for qemu "-net user,host=1.2.3.4,hostfwd=tcp::23456-:22".
>> Scp/ssh access is probably not mandatory, especially if it uses a host
>> dir as disk. I can just put the binaries into the dir, and substitute
>> /runtrin.sh to start syzkaller.
>> But VM->host connectivity is mandatory in the current design.
>
> Ah, that should easily work out of the box. usermode networking defaults
> the host to 192.168.33.1, so connecting from inside the vm to any port on
> the host using that ip should work.




I got lkvm up and running.

I run it as:

./lkvm run --disk linux-0.2.img --mem=2048 --cpus=4 --kernel
arch/x86/boot/bzImage -n mode=user

However, networking does not work. When I ping 192.168.33.1 from
within VM, it says "Destination Host Unreachable".

I've also tried to run with "-n mode=user,host_ip=192.168.33.1" flag.
No success.


I also don't get how sandbox command works.
I've tried to run:

$ ./lkvm ... -- /bin/ls

but nothing happened. VM just booted as with lkvm run.
Was that program executed, and then boot process continues as usual?
ps does not work within the VM (image is obtained as described in README):

sh-2.05b# ps
[ 2.757686] ps[886]: segfault at f7736000 ip 00000000f772f737 sp
00000000fff64f00 error 4 in libproc.so.2.0.11[f7728000+a000]
Segmentation fault

so I can't check whether sandbox actually starts something or not...
Also should sandbox refer to a program on host or in VM image?
/bin/ls is present in both locations, so I would expect that to work anyways.

[Sasha Levin]

It's a bit tricky: when you start a disk image as opposed to a virtio-9p
transport kvmtool won't run dhcp automatically. You can fix that by either
running 'dhcpcd eth0' after boot or manually adding "ip=dhcp" to the kernel's
command line.

>
> I also don't get how sandbox command works.
> I've tried to run:
>
> $ ./lkvm ... -- /bin/ls
>
> but nothing happened. VM just booted as with lkvm run.
> Was that program executed, and then boot process continues as usual?
> ps does not work within the VM (image is obtained as described in README):
>
> sh-2.05b# ps
> [ 2.757686] ps[886]: segfault at f7736000 ip 00000000f772f737 sp
> 00000000fff64f00 error 4 in libproc.so.2.0.11[f7728000+a000]
> Segmentation fault
>
> so I can't check whether sandbox actually starts something or not...
> Also should sandbox refer to a program on host or in VM image?
> /bin/ls is present in both locations, so I would expect that to work anyways.

Sandbox is also something that'll only work with virtio-9p transport only. Try
the following:


sasha@lappy:~/kvmtool$ ./vm setup sandbox-test
A new rootfs 'sandbox-test' has been created in '/home/sasha/.lkvm/sandbox-test'.

You can now start it by running the following command:

lkvm run -d sandbox-test

sasha@lappy:~/kvmtool$ ./vm sandbox -d sandbox-test -k ~/linux/arch/x86/boot/bzImage -- ls
[...]
[ 1.201160] IP-Config: Got DHCP answer from 192.168.33.1, my address is 192.168.33.15
[ 1.202408] IP-Config: Complete:
[ 1.202886] device=eth0, hwaddr=02:15:15:15:15:15, ipaddr=192.168.33.15, mask=255.255.255.0, gw=192.168.33.1
[ 1.204358] host=192.168.33.15, domain=, nis-domain=(none)
[ 1.205210] bootserver=192.168.33.1, rootserver=0.0.0.0, rootpath=
[ 1.206146] nameserver0=127.0.1.1
[ 1.206803] ALSA device list:
[ 1.207281] No soundcards found.
[ 1.208093] md: Waiting for all devices to be available before autodetect
[ 1.209064] md: If you don't use raid, use raid=noautodetect
[ 1.210345] md: Autodetecting RAID arrays.
[ 1.210933] md: Scanned 0 and added 0 devices.
[ 1.211610] md: autorun ...
[ 1.212042] md: ... autorun DONE.
[ 1.213162] VFS: Mounted root (9p filesystem) on device 0:17.
[ 1.215178] devtmpfs: mounted
[ 1.216067] Freeing unused kernel memory: 1144K (ffffffff81f34000 - ffffffff82052000)
[ 1.217160] Write protecting the kernel read-only data: 14336k
[ 1.219479] Freeing unused kernel memory: 1568K (ffff880001878000 - ffff880001a00000)
[ 1.221641] Freeing unused kernel memory: 1300K (ffff880001cbb000 - ffff880001e00000)
Mounting...
[ 1.232203] kworker/u10:2 (839) used greatest stack depth: 13856 bytes left
bin etc host lib64 root sys usr virt
dev home lib proc sbin tmp var
[ 1.283868] ls (841) used greatest stack depth: 13568 bytes left
[ 2.286214] Unregister pv shared memory for cpu 2
[ 2.286229] Unregister pv shared memory for cpu 0
[ 2.286229] Unregister pv shared memory for cpu 1
[ 2.288100] Unregister pv shared memory for cpu 3
[ 2.289889] reboot: Restarting system
[ 2.292222] reboot: machine restart

# KVM session ended normally.


Thanks,
Sasha

[Dmitry Vyukov]

Hi Sasha,

I've made some progress with lkvm setup. Everything almost works now.
However I see some unexplainable network hangs. See the other email
titled "Network hangs when communicating with host".

Also, why does not "lkvm setup" allow to specify full path to the
created dir? It is somewhat inconvenient that it dictates where I
should put files. Especially for automated testing. I would like to
keep all temp files within a single directory.

[Dmitry Vyukov]

I've committed preliminary support for lkvm:
https://github.com/google/syzkaller/commit/587db99baa54c04e23c207aaf6a5d05d0e1bb791

However, it does not work due to the aforementioned networking issue.

[Sasha Levin]

No real reason. It was written to fit with our usecases at the time,
if you're interested this feature can be easily added.


Thanks,
Sasha