OpenBSD + syzkaller

[Dmitry Vyukov]

Hi Anton,

Also +Greg who expressed potential interest helping with
OpenBSD/syzkaller/syzbot setup.

FTR, the previous discussion about OpenBSD+syzkaller is here:
https://groups.google.com/d/msg/syzkaller/QPc8FpLr_aI/CGLjMM2OAQAJ

Anton just submitted support for OpenBSD vmm/vmctl:
https://github.com/google/syzkaller/commit/de20bcbb68a06e9589f4ad2bb162aa1feed0c5c9
So now syzkaller should run locally for you, right?

The main things that's left for syzbot integration are:
1. OpenBSD support in pkg/vcs to teach syzkaller how to checkout sources.
We already have git support, so this will boil down to just writing
some glue code.
E.g. see akaros support:
https://github.com/google/syzkaller/blob/master/pkg/vcs/akaros.go

2. OpenBSD support in pkg/build to teach syzkaller how to build
OpenBSD (on OpenBSD host). It's like how to run make and what/where
are the output artifacts. You can see existing examples:
https://github.com/google/syzkaller/blob/master/pkg/build/fuchsia.go
https://github.com/google/syzkaller/blob/master/pkg/build/akaros.go
https://github.com/google/syzkaller/blob/master/pkg/build/linux.go

3. Then we will need support in pkg/report. That's the part that
analyzes kernel console output and detects crashes, parses them,
extracts crash identity string, etc.

4. Last but not least we need some docs with instructions that will
allow other people to run syzkaller on OpenBSD and me to setup a
syzbot instance on GCE.
Are there prebuilt OpenBSD images available on GCE? I would expect
some public project that should provide them. +Matthew spotted in
"OpenBSD and Compute Engine" thread:
https://marc.info/?l=openbsd-misc&m=138757967321855&w=2

[Matthew Dempsky]

1. Upstream OpenBSD still uses CVS. But for bisection support, you probably want to use one of the Git mirrors like https://github.com/openbsd/src.

2. How much of OpenBSD are you hoping to build from source?

The full release instructions are at https://man.openbsd.org/release. At a macroscopic level, there are three major build artifacts: the kernel (src/sys), the base system (src), and the X11 distribution (xenocara).

OpenBSD strives to keep userland code built for release N still working with the kernel for N+1 (bumping time_t from 32-bit to 64-bit being a notable exception). Beyond that, mixing versions should still mostly work, but isn't guaranteed.

3. Off hand I don't have any advice here. OpenBSD does have an integrated kernel debugger though: https://man.openbsd.org/ddb

4. I don't know of any prebuilt OpenBSD images for GCE. Here's the script that Go uses for building their OpenBSD GCE builder: https://github.com/golang/build/tree/master/env/openbsd-amd64

[Anton Lindqvist]

On Sat, Aug 18, 2018 at 01:27:54PM -0700, Dmitry Vyukov wrote:
> Hi Anton,
>
> Also +Greg who expressed potential interest helping with
> OpenBSD/syzkaller/syzbot setup.
>
> FTR, the previous discussion about OpenBSD+syzkaller is here:
> https://groups.google.com/d/msg/syzkaller/QPc8FpLr_aI/CGLjMM2OAQAJ
>
> Anton just submitted support for OpenBSD vmm/vmctl:
> https://github.com/google/syzkaller/commit/de20bcbb68a06e9589f4ad2bb162aa1feed0c5c9
> So now syzkaller should run locally for you, right?

Yes, I'm in the process of submitting the remaining parts to syzkaller
adding support OpenBSD.

> The main things that's left for syzbot integration are:
> 1. OpenBSD support in pkg/vcs to teach syzkaller how to checkout sources.
> We already have git support, so this will boil down to just writing
> some glue code.
> E.g. see akaros support:
> https://github.com/google/syzkaller/blob/master/pkg/vcs/akaros.go
>
> 2. OpenBSD support in pkg/build to teach syzkaller how to build
> OpenBSD (on OpenBSD host). It's like how to run make and what/where
> are the output artifacts. You can see existing examples:
> https://github.com/google/syzkaller/blob/master/pkg/build/fuchsia.go
> https://github.com/google/syzkaller/blob/master/pkg/build/akaros.go
> https://github.com/google/syzkaller/blob/master/pkg/build/linux.go
>
> 3. Then we will need support in pkg/report. That's the part that
> analyzes kernel console output and detects crashes, parses them,
> extracts crash identity string, etc.

pkg/report bits are already done.

> 4. Last but not least we need some docs with instructions that will
> allow other people to run syzkaller on OpenBSD and me to setup a
> syzbot instance on GCE.
> Are there prebuilt OpenBSD images available on GCE? I would expect
> some public project that should provide them. +Matthew spotted in
> "OpenBSD and Compute Engine" thread:
> https://marc.info/?l=openbsd-misc&m=138757967321855&w=2

In other news: I committed initial kcov support to OpenBSD recently.
There's still some things I want to address in regards to the
implementation. Once that part is done, I will start look into syzbot. I
know too little about GCE et al. and have to do some reading before I
can answer your questions. Thanks for the pointers!

[Anton Lindqvist]

On Mon, Aug 20, 2018 at 12:55:36PM -0700, Matthew Dempsky wrote:
> 1. Upstream OpenBSD still uses CVS. But for bisection support, you probably
> want to use one of the Git mirrors like https://github.com/openbsd/src.

The Git mirror should be fine.

> 2. How much of OpenBSD are you hoping to build from source?
>
> The full release instructions are at https://man.openbsd.org/release. At a
> macroscopic level, there are three major build artifacts: the kernel
> (src/sys), the base system (src), and the X11 distribution (xenocara).
>
> OpenBSD strives to keep userland code built for release N still working
> with the kernel for N+1 (bumping time_t from 32-bit to 64-bit being a
> notable exception). Beyond that, mixing versions should still mostly work,
> but isn't guaranteed.

Using the latest (called snapshots in OpenBSD-terminology) sets which
includes userland could be used, this would simplify the build process.
This could however on some rare occasions cause problems during bisect
if the kernel ABI is changed at some point.

The only thing that then needs to be compiled is a custom kernel with
kcov support. I already have instructions for that, expect a PR to be
submitted to syzkaller soon.

> 3. Off hand I don't have any advice here. OpenBSD does have an integrated
> kernel debugger though: https://man.openbsd.org/ddb

That part is already covered[1][2].

[1] https://github.com/google/syzkaller/blob/master/vm/vmm/vmm.go#L271-L280
[2] https://github.com/google/syzkaller/pull/689/files#diff-f25981b17472970b8b103c76e8e032c4

[Greg Steuck]

I see a ton of recent progress in https://github.com/google/syzkaller/commits?author=mptre. It looks like the local case of running OpenBSD in vmm is already working and finding bugs. Is the main remaining work centered around syzbot now?

Am I too pessimistic in my expectation that vmm won't work under OpenBSD host running on GCE? Not that it's really needed, just curious.

Thanks

Greg

[Klemens Nanni]

On Mon, Sep 03, 2018 at 11:45:14AM -0700, Greg Steuck wrote:
> I see a ton of recent progress in
> https://github.com/google/syzkaller/commits?author=mptre. It looks like the
> local case of running OpenBSD in vmm is already working and finding bugs.

Indeed.

> Is the main remaining work centered around syzbot now?

Not just syzbot, what's also missing are syscall descriptions.

Docs could be in better shape but I already prepared patches for that.

It hasn't been running for that long on OpenBSD so general testing is welcome, I'd say.

[Greg Steuck]

On Mon, Sep 3, 2018 at 12:03 PM Klemens Nanni <k...@openbsd.org> wrote:

On Mon, Sep 03, 2018 at 11:45:14AM -0700, Greg Steuck wrote:
> I see a ton of recent progress in
> https://github.com/google/syzkaller/commits?author=mptre. It looks like the
> local case of running OpenBSD in vmm is already working and finding bugs.
Indeed.

> Is the main remaining work centered around syzbot now?
Not just syzbot, what's also missing are syscall descriptions.

Any pointers for this?

 

Docs could be in better shape but I already prepared patches for that.

It hasn't been running for that long on OpenBSD so general testing is welcome, I'd say.

My only hardware OpenBSD machine probably is too old to run with vmm (to be verified). I don't think GCE will work either (dmesg from 6.3 is below).

I upgraded the recipe of Golang build bot a bit so we can start from there: https://github.com/golang/build/pull/12

Thanks

Greg

SeaBIOS (version 1.8.2-20180718_090847-google)

Total RAM Size = 0x00000000f0000000 = 3840 MiB

CPUs found: 1     Max CPUs supported: 256

found virtio-scsi at 0:3

virtio-scsi vendor='Google' product='PersistentDisk' rev='1' type=0 removable=0

virtio-scsi blksize=512 sectors=20971520 = 10240 MiB

drive 0x000f2280: PCHS=0/0/0 translation=lba LCHS=1024/255/63 s=20971520

Booting from Hard Disk 0...

>> OpenBSD/amd64 BOOT 3.34

boot> 

booting hd0a:/bsd: 8616123+2454536+262168+0+667648 [652721+98+712080+493101]=0xd39d20

entry point at 0x1000158

[ using 1858848 bytes of bsd ELF symbol table ]

Copyright (c) 1982, 1986, 1989, 1991, 1993

The Regents of the University of California.  All rights reserved.

Copyright (c) 1995-2018 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 6.3 (GENERIC.MP) #107: Sat Mar 24 14:21:59 MDT 2018

    der...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

real mem = 4009701376 (3823MB)

avail mem = 3881103360 (3701MB)

mpath0 at root

scsibus0 at mpath0: 256 targets

mainbus0 at root

bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xbfffd060 (268 entries)

bios0: vendor Google version "Google" date 01/01/2011

bios0: Google Google Compute Engine

acpi0 at bios0: rev 0

acpi0: sleep states S3 S4 S5

acpi0: tables DSDT FACP SSDT APIC WAET SRAT

acpi0: wakeup devices

acpitimer0 at acpi0: 3579545 Hz, 24 bits

acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat

cpu0 at mainbus0: apid 0 (boot processor)

cpu0: Intel(R) Xeon(R) CPU @ 2.00GHz, 1800.60 MHz

cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,MELTDOWN

cpu0: 256KB 64b/line 8-way L2 cache

cpu0: smt 0, core 0, package 0

mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges

cpu0: apic clock running at 950MHz

ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins

acpiprt0 at acpi0: bus 0 (PCI0)

acpicpu0 at acpi0: C1(@1 halt!)

"ACPI0006" at acpi0 not configured

"QEMU0001" at acpi0 not configured

"ACPI0007" at acpi0 not configured

pvbus0 at mainbus0: KVM

pci0 at mainbus0 bus 0

pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02

pcib0 at pci0 dev 1 function 0 "Intel 82371AB PIIX4 ISA" rev 0x03

piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: SMBus disabled

virtio0 at pci0 dev 3 function 0 "Qumranet Virtio SCSI" rev 0x00

vioscsi0 at virtio0: qsize 8192

scsibus1 at vioscsi0: 253 targets

sd0 at scsibus1 targ 1 lun 0: <Google, PersistentDisk, 1> SCSI4 0/direct fixed serial.Google_PersistentDisk_

sd0: 10240MB, 512 bytes/sector, 20971520 sectors, thin

virtio0: msix shared

virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00

vio0 at virtio1: address 42:01:0a:a8:00:03

virtio1: msix shared

isa0 at pcib0

isadma0 at isa0

com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo

com0: console

com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo

com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo

pckbc0 at isa0 port 0x60/5 irq 1 irq 12

pckbd0 at pckbc0 (kbd slot)

wskbd0 at pckbd0 mux 1

pms0 at pckbc0 (aux slot)

wsmouse0 at pms0 mux 0

pcppi0 at isa0 port 0x61

spkr0 at pcppi0

vscsi0 at root

scsibus2 at vscsi0: 256 targets

softraid0 at root

scsibus3 at softraid0: 256 targets

root on sd0a (65fe190562b40e94.a) swap on sd0b dump on sd0b

Automatic boot in progress: starting file system checks.

/dev/sd0a (65fe190562b40e94.a): file system is clean; not checking

setting tty flags

pf enabled

starting network

vio0: bound to 10.168.0.3 from 169.254.169.254 (42:01:0a:a8:00:01)

reordering libraries: done.

starting early daemons: syslogd pflogd ntpd.

starting RPC daemons:.

savecore: no core dump

checking quotas: done.

clearing /tmp

kern.securelevel: 0 -> 1

creating runtime link editor directory cache.

preserving editor files.

starting network daemons: sshd smtpd sndiod.

--

nest.cx is Gmail hosted, use PGP for anything private. Key: http://goo.gl/6dMsr
Fingerprint: 5E2B 2D0E 1E03 2046 BEC3  4D50 0B15 42BD 8DF5 A1B0

[Klemens Nanni]

On Mon, Sep 03, 2018 at 02:54:36PM -0700, Greg Steuck wrote:
> Any pointers for this?
https://github.com/google/syzkaller/blob/master/docs/syscall_descriptions.md

See sys/openbsd/ for what's already there.

> My only hardware OpenBSD machine probably is too old to run with vmm (to be
> verified).

Does the CPU support virtualization?

> I don't think GCE will work either (dmesg from 6.3 is below).

No. Your dmesg shows virtio drivers being used, the CPU does not provide any of
the required extensions, thus vmm(4)[0] does not attach.

0: http://man.openbsd.org/vmm.4

[Dmitry Vyukov]

Maybe enabling nested virtualization will help?
https://cloud.google.com/compute/docs/instances/enable-nested-virtualization-vm-instances

We can also use "gce" VM type which creates other GCE VMs with the
target kernel for testing. So if OpenBSD can run on GCE, this will
work too. Even worked for Windows.

[Greg Steuck]

I can start a nested OpenBSD VM on GCE.

Enabling nested virtualization resulted in dmesg getting a couple new things on top of the original dmesg I sent:

< cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,MELTDOWN

---

> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,MELTDOWN

73a73

> vmm0 at mainbus0: VMX/EPT

From this point on I can run the vmctl(8) example to start a nested VM. The basic example produces a crashing VM below, but booting bsd.rd works so I presume I can get past this.

# vmctl create disk.img -s 4.5G

vmctl: imagefile created

# vmctl start "myvm" -m 1G -i 1 -b /bsd -d disk.img 

vmctl: started vm 1 successfully, tty /dev/ttyp1

buildlet# vmctl console 1

Connected to /dev/ttyp1 (speed 115200)

ddb{0}> trace                

db_enter() at db_enter+0x5

panic() at panic+0x129    

dk_mountroot() at dk_mountroot+0xde

main(0) at main+0x5ee              

end trace frame: 0x0, count: -4

ddb{0}> boot reboot                                                 

vmmci0: powerdown  

rebooting...     

[ using 1856768 bytes of bsd ELF symbol table ]

Copyright (c) 1982, 1986, 1989, 1991, 1993     

The Regents of the University of California.  All rights reserved.

Copyright (c) 1995-2018 OpenBSD. All rights reserved.  https://www.OpenBSD.org

                                                                              

OpenBSD 6.3 (GENERIC.MP) #107: Sat Mar 24 14:21:59 MDT 2018

    der...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

real mem = 1056964608 (1008MB)                                          

avail mem = 1017913344 (970MB)

mpath0 at root                

scsibus0 at mpath0: 256 targets

mainbus0 at root               

bios0 at mainbus0

acpi at bios0 not configured

cpu0 at mainbus0: (uniprocessor)

cpu0: Intel(R) Xeon(R) CPU @ 2.00GHz, 2003.94 MHz

cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,MELTDOWN

cpu0: 256KB 64b/line 8-way L2 cache

cpu0: smt 0, core 0, package 0     

pvbus0 at mainbus0: OpenBSD   

pci0 at mainbus0 bus 0     

pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00

virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00

viornd0 at virtio0                                             

virtio0: irq 3    

virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Storage" rev 0x00

vioblk0 at virtio1                                                 

scsibus1 at vioblk0: 2 targets

sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed

sd0: 4608MB, 512 bytes/sector, 9437184 sectors                             

virtio1: irq 5                                

virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00

vio0 at virtio2: address fe:e1:bb:d1:85:8b                         

virtio2: irq 6                            

virtio3 at pci0 dev 4 function 0 "OpenBSD VMM Control" rev 0x00

vmmci0 at virtio3                                              

virtio3: irq 7   

isa0 at mainbus0

isadma0 at isa0 

com0 at isa0 port 0x3f8/8 irq 4: ns16450, no fifo

com0: console                                    

vscsi0 at root

scsibus2 at vscsi0: 256 targets

softraid0 at root              

scsibus3 at softraid0: 256 targets

root on sd0a swap on sd0b dump on sd0b

panic: root filesystem has size 0     

Stopped at      db_enter+0x5:   popq    %rbp

    TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND

*     0      0      0     0x10000      0x200    0K swapper

db_enter() at db_enter+0x5                                

panic() at panic+0x129    

dk_mountroot() at dk_mountroot+0xde

main(0) at main+0x5ee              

end trace frame: 0x0, count: 11

[Greg Steuck]

On Mon, Sep 3, 2018 at 7:58 PM Dmitry Vyukov <dvy...@google.com> wrote:

Maybe enabling nested virtualization will help?
https://cloud.google.com/compute/docs/instances/enable-nested-virtualization-vm-instances

To recoup, yes, OpenBSD VMM seems to work in on GCE and can host another instance of OpenBSD inside.

 

We can also use "gce" VM type which creates other GCE VMs with the
target kernel for testing. So if OpenBSD can run on GCE, this will
work too. Even worked for Windows.

As far as I can tell OpenBSD 6.3 is running swimmingly on GCE. It would appear that syzbot launching a minimal OpenBSD image on GCE is the least complicated and resource intensive approach.

[Anton Lindqvist]

On Mon, Sep 03, 2018 at 11:45:14AM -0700, Greg Steuck wrote:

> I see a ton of recent progress in
> https://github.com/google/syzkaller/commits?author=mptre. It looks like the
> local case of running OpenBSD in vmm is already working and finding bugs.
> Is the main remaining work centered around syzbot now?

Next task on my list is adding support for OpenBSD to pkg/build.

Thanks for taking the time to investigate running OpenBSD on GCE.

[Dmitry Vyukov]

On Thu, Sep 6, 2018 at 8:12 PM, Anton Lindqvist <an...@basename.se> wrote:
> On Mon, Sep 03, 2018 at 11:45:14AM -0700, Greg Steuck wrote:
>> I see a ton of recent progress in
>> https://github.com/google/syzkaller/commits?author=mptre. It looks like the
>> local case of running OpenBSD in vmm is already working and finding bugs.
>> Is the main remaining work centered around syzbot now?
>
> Next task on my list is adding support for OpenBSD to pkg/build.
>
> Thanks for taking the time to investigate running OpenBSD on GCE.

You guys need to synchronize :)

---------- Forwarded message ----------
From: Greg Steuck (Sh-toy-k)

Looks like email is a better medium for this discussion short term. Quoting you:

> http://go/syzbot-internals
> but it's so large that you can't see where the kernels are built
> kernels are built by syzbot by syz-ci program
> and it starts managers with the new kernel

Thanks.

> OpenBSD can't be built on linux? or not? always forget which one can be built on linux

Cross-compiling OpenBSD will be a real chore. I don't think anybody tried it.

> If it can't, then we will need to create a master VM on GCE running OpenBSD
> this machine will run syz-ci and will built new OpenBSD kernels for testing

Yes, that's approximately what I thought. So the first order of
business is to come up with a recipe for making buildlets. Those
will probalby just download the most recent snapshot from the CDN.

The buildlet, when started, can get the OpenBSD source repo from git at
the specified label and produce testlet images. What's the drop-off place
for the images? A "simple" way to do it is to have a GCE disk be
mounted into the buildlet which will become a testlet OpenBSD image
after the buildlet reports success (and unmounts the image plus
signals completion).

> if we will use vmm, then we will need OpenBSD anyway, right?

I think VMM will just get in the way. I propose we run OpenBSD
testlets from images produced by buildlets on GCE. We'll monitor
the serial console of the OpenBSD testlets on GCE (as usual?).

> but if we are on linux, we could use qemu as well...
> but in order to use GCE VMs for testing, we will need to teach syz-ci to build GCE-compatible images
> this may be non-trivial to do on openbsd host

I think it's easy. It's running already. If we arrange for disk
handover buildlet to testlet I can run the whole thing under OpenBSD
on GCE.

> this is what we use to build gce-compatible images on linux
> https://github.com/google/syzkaller/blob/master/tools/create-gce-image.sh

Cool. Will go through this. Is it what I called "buildlet" or "testlet" image?

Thanks
Greg

[Dmitry Vyukov]

On Thu, Sep 6, 2018 at 9:04 PM, Dmitry Vyukov <dvy...@google.com> wrote:
> On Thu, Sep 6, 2018 at 8:12 PM, Anton Lindqvist <an...@basename.se> wrote:
>> On Mon, Sep 03, 2018 at 11:45:14AM -0700, Greg Steuck wrote:
>>> I see a ton of recent progress in
>>> https://github.com/google/syzkaller/commits?author=mptre. It looks like the
>>> local case of running OpenBSD in vmm is already working and finding bugs.
>>> Is the main remaining work centered around syzbot now?
>>
>> Next task on my list is adding support for OpenBSD to pkg/build.
>>
>> Thanks for taking the time to investigate running OpenBSD on GCE.
>
>
> You guys need to synchronize :)
>
> ---------- Forwarded message ----------
> From: Greg Steuck (Sh-toy-k)
>
> Looks like email is a better medium for this discussion short term. Quoting you:
>
>> http://go/syzbot-internals
>> but it's so large that you can't see where the kernels are built
>> kernels are built by syzbot by syz-ci program
>> and it starts managers with the new kernel
>
> Thanks.
>
>> OpenBSD can't be built on linux? or not? always forget which one can be built on linux
>
> Cross-compiling OpenBSD will be a real chore. I don't think anybody tried it.

ack

>> If it can't, then we will need to create a master VM on GCE running OpenBSD
>> this machine will run syz-ci and will built new OpenBSD kernels for testing
>
> Yes, that's approximately what I thought. So the first order of
> business is to come up with a recipe for making buildlets. Those
> will probalby just download the most recent snapshot from the CDN.
>
> The buildlet, when started, can get the OpenBSD source repo from git at
> the specified label and produce testlet images. What's the drop-off place
> for the images? A "simple" way to do it is to have a GCE disk be
> mounted into the buildlet which will become a testlet OpenBSD image
> after the buildlet reports success (and unmounts the image plus
> signals completion).
>
>> if we will use vmm, then we will need OpenBSD anyway, right?
>
> I think VMM will just get in the way. I propose we run OpenBSD
> testlets from images produced by buildlets on GCE. We'll monitor
> the serial console of the OpenBSD testlets on GCE (as usual?).

Yes, that part is already implemented and works.

>> but if we are on linux, we could use qemu as well...
>> but in order to use GCE VMs for testing, we will need to teach syz-ci to build GCE-compatible images
>> this may be non-trivial to do on openbsd host
>
> I think it's easy. It's running already. If we arrange for disk
> handover buildlet to testlet I can run the whole thing under OpenBSD
> on GCE.
>
>> this is what we use to build gce-compatible images on linux
>> https://github.com/google/syzkaller/blob/master/tools/create-gce-image.sh
>
> Cool. Will go through this. Is it what I called "buildlet" or "testlet" image?

So, due to limited human resources, we keep things simpler.
We don't have buildlet's per se. Instead we just have a single
persistent "master" machine that is setup manually once (but we really
need to have scripts that capture the process).
Also the "drop-off place for the images" is simply master machine
disk. The role build procedure is to produce small set of known
artifacts (kernel, image, config). Simplest examples are fuchsia and
gvisor:
https://github.com/google/syzkaller/blob/master/pkg/build/fuchsia.go
https://github.com/google/syzkaller/blob/master/pkg/build/gvisor.go
linux and akaros are somewhat more involved, but the interface is the same.
If we use GCE VMs as testlets that build process needs to produce
actual GCE-compatible image. For vmm (qemu) we can cheat and do
injected kernel boot with fixed image.

[Greg Steuck]

Anton, I started putting issues on github so that I remember what I'm up to.
syz-ci: OpenBSD port: https://github.com/google/syzkaller/issues/712

This issue seems somewhat related to what you are planning to do with pkg/build
as syz-ci should eventually be doing the builds.  My first step there would be to
get an easy to use GCE image generator.

Thanks
Greg

P.S. I moved Matthew to Bcc, dunno if he cares about this level of detail.

[Dmitry Vyukov]

On Thu, Sep 6, 2018 at 10:31 PM, Greg Steuck <gr...@nest.cx> wrote:
> Anton, I started putting issues on github so that I remember what I'm up to.
> syz-ci: OpenBSD port: https://github.com/google/syzkaller/issues/712
>
> This issue seems somewhat related to what you are planning to do with
> pkg/build
> as syz-ci should eventually be doing the builds. My first step there would
> be to
> get an easy to use GCE image generator.

Since we seem to be close to deploying syzbot instance, we need to
decide on bug reporting.
For linux and akaros we use googlegroups-based mailing lists:
https://groups.google.com/forum/#!forum/syzkaller-bugs
https://groups.google.com/forum/#!searchin/akaros/syzbot%7Csort:date

We could do the same for OpenBSD, i.e. a new google group. For
googlegroups we know that they work, how to set them up and we also
have logic to extract web links for reports, e.g. here status link at
the top that points to email report:
https://syzkaller.appspot.com/bug?extid=0049bebbf3042dbd2e8f
We could also use bu...@openbsd.org or a new mailing list on
openbsd.org. That will probably mean that links won't work (at least
out of the box), and maybe some other set of yet unknown problems.

[Dmitry Vyukov]

Woohoo! We have first kernel panic on dashboard:
https://syzkaller.appspot.com/bug?id=8a072ffc26eb4f44df80c48a5c859205f0fa0b93

[Dmitry Vyukov]

We don't have reproducers working yet, but that was this program:

16:00:15 executing program 0:
r0 = semget(0x3, 0x4, 0x0)
semop(r0, &(0x7f0000000000)=[{0x0, 0x1}, {0x0, 0x0, 0x800}], 0x2)
semget$private(0x0, 0x7, 0x591)
semop(r0, &(0x7f0000000040)=[{0x3, 0x1, 0x1800}, {0x3, 0xffff,
0x1000}, {0x3, 0x9, 0x1800}], 0x3)
ioctl$TIOCMSET(0xffffffffffffffff, 0x8004746d, &(0x7f0000000080)=0x4)

N in "executing program N:" matches N in process name "COMMAND: syz-executorN".

[Anton Lindqvist]

Nice! That's a use-after-free but I cannot reproduce it. Such panics are
often caused by two or more programs, where the last one is a symptom of
a previous one that caused a dangling pointer to be left around. Tried
running all generated programs in sequence without any luck.

Looking at the panic message, it's clear that it relates to the un_cnt
field of `struct sem_undo' being modified after free.

Will investigate further.