External USB and SSH

[Darrion Ramos]

Hello, I have looked through your documentation and could not find information on these things so I am asking here for some guidance. I would like to use S2E simply to provide line coverage of the kernel while I perform some testing on it. My questions:

1. Is there some way for me to access the VM image so I can setup SSH on it? I cannot launch the VM from regular Qemu command line since it seems it is a custom format. Or is there a way for me to export an existing Debian image to be compatible with S2E?

2. I would like to connect an external USB device to the VM. I am familiar with how to do this with Qemu and where to add the arguments in the s2e launch script. But the Qemu binary in s2e does not support the USB passthrough. Is it possible for me to recompile your Qemu version with USB support or compile Qemu from source with s2e support?

Thanks, Darrion.

[Vitaly Chipounov]

Hi Darrion,

On Wed, Sep 18, 2024 at 10:39 PM Darrion Ramos <darrio...@gmail.com> wrote:

Hello, I have looked through your documentation and could not find information on these things so I am asking here for some guidance. I would like to use S2E simply to provide line coverage of the kernel while I perform some testing on it.

Do you need symbolic execution? If not, then USB/passthrough will be considerably easier to set up.

S2E comes with a single-path build with symbolic execution disabled. You can still get all the instrumentation/coverage capabilities. Use the --single-path switch when creating a new project.

 

My questions:

1. Is there some way for me to access the VM image so I can setup SSH on it? I cannot launch the VM from regular Qemu command line since it seems it is a custom format. Or is there a way for me to export an existing Debian image to be compatible with S2E?

You can install any packages you want (including SSH) in the guest by customizing launch.sh in the guest-images repo, then rebuilding the image.

There is no automated way to import an existing image. In general, building an image is a complex process. You need to install an S2E-compatible Linux kernel, a few guest tools and scripts to make the VM usable with S2E plugins. It would still run if you don't do that, but you'd lose instrumentation capabilities and code coverage.

Regarding network access, it is disabled because it does not work well in multi-path mode. Switching paths causes all sorts of weird things, especially with the user space network emulation. A tap interface might work better. You should be able to add the required options in launch-s2e.sh.

 

2. I would like to connect an external USB device to the VM. I am familiar with how to do this with Qemu and where to add the arguments in the s2e launch script. But the Qemu binary in s2e does not support the USB passthrough. Is it possible for me to recompile your Qemu version with USB support or compile Qemu from source with s2e support?

If the feature is not compiled, you could tweak the configure flags here to add it. The external USB device may get confused if you run the driver on multiple-paths though. We can discuss further options if you need symbolic execution.

Vitaly

[Darrion Ramos]

Hi Vitaly,

I do not need symbolic execution. However, when changing a project's mode from s2e to s2e_sp the project crashes although the same configuration works in s2e mode. Is s2e_sp mode required for network access to work? I thought that as long as I was not doing any symbolic execution it would work. If I need to get s2e_sp to work here is the crash output:

Starting libs2e...
Opening /dev/kvm
Initializing qemu64-s2e cpu
Using module /home/darrion/s2e/install/share/libs2e/op_helper_sp.bc.x86_64
S2E: output directory = "./s2e-out-14"
lua: attempting to load models.lua
KLEE: WARNING: unsupported intrinsic llvm.rint.f64
KLEE: WARNING: unsupported intrinsic llvm.fmuladd.f64
KLEE: WARNING: unsupported intrinsic llvm.fma.f32
KLEE: WARNING: unsupported intrinsic llvm.fma.f64
Using log level override 'debug'
Setting console level to 'debug'
Creating plugin CorePlugin
Creating plugin BaseInstructions
Creating plugin HostFiles
Creating plugin Vmi
Creating plugin MemUtils
Creating plugin WebServiceInterface
Creating plugin ExecutionTracer
Creating plugin ModuleTracer
Creating plugin KeyValueStore
Creating plugin StatsTracker
Creating plugin TranslationBlockCoverage
Creating plugin ModuleExecutionDetector
Creating plugin ForkLimiter
Creating plugin ProcessExecutionDetector
Creating plugin ModuleMap
Creating plugin MemoryMap
Creating plugin MultiSearcher
Creating plugin CUPASearcher
Creating plugin StaticFunctionModels
Creating plugin TestCaseGenerator
Creating plugin Screenshot
Creating plugin LinuxMonitor
Creating plugin LuaBindings
Creating plugin LuaCoreEvents
Initializing LuaBindings
Initializing LuaCoreEvents
LuaCoreEvents: Registering instrumentation for core signals
Initializing Screenshot
Initializing TestCaseGenerator
Initializing MultiSearcher
Initializing ForkLimiter
Initializing StatsTracker
Initializing KeyValueStore
Initializing ExecutionTracer
Initializing WebServiceInterface
WebServiceInterface: SeedSearcher not present, seed statistics will not be available
WebServiceInterface: Recipe plugin not present, recipe statistics will not be available
Initializing Vmi
Vmi: adding path /home/darrion/s2e/projects/linux-kernel
Vmi: adding path /home/darrion/s2e/images/ubuntu-22.04-x86_64/guestfs
Initializing HostFiles
Initializing BaseInstructions
Initializing LinuxMonitor
Initializing ModuleMap
Initializing ProcessExecutionDetector
ProcessExecutionDetector: no modules configured
Initializing MemoryMap
Initializing MemUtils
Initializing ModuleExecutionDetector
Initializing StaticFunctionModels
StaticFunctionModels: Model count: 0
Initializing CUPASearcher
MultiSearcher: Registering CUPASearcher
MultiSearcher: Switching to CUPASearcher
CUPASearcher: CUPASearcher is now active
Initializing TranslationBlockCoverage
Initializing ModuleTracer
Initializing CorePlugin
[Z3] Initializing
0 [State 0] Created initial state
Initializing periodic timer
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.40000001H:EAX.kvm-nopiodelay [bit 1]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.40000001H:EAX.kvmclock [bit 3]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.40000001H:EAX.kvm-asyncpf [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.40000001H:EAX.kvm-steal-time [bit 5]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.40000001H:EAX.kvm-pv-eoi [bit 6]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.40000001H:EAX.kvmclock-stable-bit [bit 24]
Adding CPU (addr = 0x78e0ac0087d0, size = 0x2cec0)
s2e-block: dirty sectors on close:0
s2e-block: dirty after restore: 16328 (ro=1)
s2e-block: wasted sectors: 0
./launch-s2e.sh: line 121: 17493 Aborted                 (core dumped) LD_PRELOAD=$LIBS2E $QEMU $QEMU_ARGS $*

CORE DUMP OUTPUT:
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=137181645506112) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=137181645506112) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=137181645506112, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007cc422a42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007cc422a287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007cc423e9ec15 in pabort (msg=<optimized out>) at /home/darrion/s2e/source/s2e/klee/include/klee/Common.h:36
#6  0x00007cc423e9ebfd in s2e::S2EExecutionStateMemory::read (this=0x378f <irq_stack_backing_store+6031>, address=6,
    width=581528060, addressType=(unknown: 0xf17ff450))
    at /home/darrion/s2e/source/s2e/libs2ecore/src/S2EExecutionStateMemory.cpp:113
#7  0x00007cc423d50d53 in s2e::plugins::BaseLinuxMonitor::verifyLinuxCommand (this=this@entry=0x5d3b104bea00,
    state=state@entry=0x5d3b104d7e70, guestDataPtr=guestDataPtr@entry=18446683600572153236,
    guestDataSize=guestDataSize@entry=84, cmd=cmd@entry=0x7cc3f17ff940 "")
    at /home/darrion/s2e/source/s2e/libs2eplugins/src/s2e/Plugins/OSMonitors/Linux/BaseLinuxMonitor.cpp:42
#8  0x00007cc423d535ec in s2e::plugins::BaseLinuxMonitor::handleOpcodeInvocation (this=0x5d3b104bea00,
    state=0x5d3b104d7e70, guestDataPtr=18446683600572153236, guestDataSize=84)
    at /home/darrion/s2e/source/s2e/libs2eplugins/src/s2e/Plugins/OSMonitors/Linux/BaseLinuxMonitor.h:182
#9  0x00007cc423d155cc in s2e::plugins::BaseInstructions::invokePlugin (this=0x5d3b0face350, state=0x5d3b104d7e70)
    at /home/darrion/s2e/source/s2e/libs2eplugins/src/s2e/Plugins/Core/BaseInstructions.cpp:559
#10 0x00007cc423eae373 in sigc::signal<void, s2e::S2EExecutionState*, unsigned long>::emit (this=<optimized out>,
    params=2816, params=2816) at /home/darrion/s2e/source/s2e/libfsigc++/include/fsigc++/fsigc++.h:280
#11 helper_s2e_tcg_custom_instruction_handler (arg=2816)
    at /home/darrion/s2e/source/s2e/libs2ecore/src/CorePluginInterface.cpp:86
#12 0x00007cc4080e8041 in code_gen_buffer ()
#13 0x00007cc423ebc021 in fetch_and_run_tb (prev_tb=0x0 <fixed_percpu_data>, tb_exit_code=0, env=0x7cc4040087d0)
    at /home/darrion/s2e/source/s2e/libcpu/src/cpu-exec.c:292
#14 execution_loop (env=0x7cc4040087d0) at /home/darrion/s2e/source/s2e/libcpu/src/cpu-exec.c:459
#15 cpu_x86_exec (env=0x7cc4040087d0) at /home/darrion/s2e/source/s2e/libcpu/src/cpu-exec.c:544
#16 0x00007cc423d0e3b3 in s2e::kvm::VCPU::coroutineFcn (opaque=0x7cc404000b70)
    at /home/darrion/s2e/source/s2e/libs2e/src/s2e-kvm-vcpu.cpp:281
#17 0x00007cc425a65826 in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>)
    at /home/darrion/s2e/source/s2e/libcoroutine/src/coroutine-ucontext.c:117
#18 0x00007cc422a5a130 in ?? () at ../sysdeps/unix/sysv/linux/x86_64/__start_context.S:90
   from /lib/x86_64-linux-gnu/libc.so.6

If I do not need to have s2e_sp working for networking I am not sure why I am not able to access the vm via SSH. Here are the steps I have taken:

1. Added openssh-server and openssh-client to the COMMON_PACKAGES in guest_image's launch.sh

2. Compiled a new image

3. Modified the guestfs of the image using a portion of syzkaller's image creation script:

#!/usr/bin/env bash
DIR=guestfs
RELEASE=s2e

# Set some defaults and enable promtless ssh to the machine for root.
sudo sed -i '/^root/ { s/:x:/::/ }' $DIR/etc/passwd
echo 'T0:23:respawn:/sbin/getty -L ttyS0 115200 vt100' | sudo tee -a $DIR/etc/inittab
printf '\nauto eth0\niface eth0 inet dhcp\n' | sudo tee -a $DIR/etc/network/interfaces
echo '/dev/root / ext4 defaults 0 0' | sudo tee -a $DIR/etc/fstab
echo 'debugfs /sys/kernel/debug debugfs defaults 0 0' | sudo tee -a $DIR/etc/fstab
echo 'securityfs /sys/kernel/security securityfs defaults 0 0' | sudo tee -a $DIR/etc/fstab
echo 'configfs /sys/kernel/config/ configfs defaults 0 0' | sudo tee -a $DIR/etc/fstab
echo 'binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc defaults 0 0' | sudo tee -a $DIR/etc/fstab
echo -en "127.0.0.1\tlocalhost\n" | sudo tee $DIR/etc/hosts
echo "nameserver 8.8.8.8" | sudo tee -a $DIR/etc/resolv.conf
echo "syzkaller" | sudo tee $DIR/etc/hostname
ssh-keygen -f $RELEASE.id_rsa -t rsa -N ''
sudo mkdir -p $DIR/root/.ssh/
cat $RELEASE.id_rsa.pub | sudo tee $DIR/root/.ssh/authorized_keys

4. Added the following flags to the QEMU_ARGS in the project's launch-s2e.sh: 

-net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 -net nic,model=e1000

5. Try to connect after launching project with: ssh -i s2e.id_rsa -p 10021 root@localhost

I am thinking that either the SSH service is not activated in the VM (not sure how to check or activate it from s2e) or the port forwarding I am doing is not supported by s2e. I also noticed that there are other files with qemu network options such as the image.json in the image's directory. But changing this results in: Could not set up host forwarding rule 'tcp:127.0.0.1:10021-:22' during launch. So its not clear to me where I need to set the networking argument for qemu. Is it in the project launch files or in the image's files?

For recompiling s2e's qemu I believe I was mistakenly trying to recompile it with the s2e build --recompile option. I am working on building s2e from source now and will see if that works.

Thanks, Darrion.

[Vitaly Chipounov]

Hi,

I fixed the crash here: https://github.com/S2E/s2e/pull/104
Other than that, SSH should already be installed in the guest and the network works for me. There is no need to specify the nic model.
Note that the nic may not be called eth0. In my guest, it's enp0s3.

$ GUI=1 ./launch-s2e.sh -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22

This will start S2E with GUI enabled. You may interrupt bootstrap.sh with ctrl+c to get to the prompt and figure out the commands.
You'll likely need to add sudo dhclient enp0s3 to bootstrap.sh.

Vitaly

--
You received this message because you are subscribed to the Google Groups "S2E Developer Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to s2e-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/3316f8c6-4a66-4769-a03f-c0a241e8c288n%40googlegroups.com.

[Darrion Ramos]

Hi Vitaly, thanks for fixing the crash and your suggestions.

I was able to get the SSH to work with the dhclient command. I was also able to add my other configurations in the bootstrap once I understood that you were booting from a snapshot so my modifications to the guestfs were not doing anything.

Also, regarding the USB pass-through. I was able to recompile the qemu binary with the proper support. But the passthrough is not working properly. lsusb outputs nothing on the VM, not even a hub. Is there some configuration needed in the image creation to allow this? When unplugging and reinserting the usb keyboard there is a udev error as you can see in the last line of this screenshot. I have read that this may be due to a dated libusb issue but this should be using the same version I used when building qemu from source. Unless s2e has its own isolated libusb version it is using. Do you know of any s2e configurations or anything else that could be affecting this? The device passes through fine with standalone qemu. 

Lastly, should I be concerned about the other warnings in the screenshot? It doesn't look like I need to but just double checking.

You received this message because you are subscribed to a topic in the Google Groups "S2E Developer Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/s2e-dev/x0j9sxEWVRU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to s2e-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CACjSjct8dc7U9bJMYdDEux%2BgC91bsU9PVv9hbjW71zsEo%2Butfw%40mail.gmail.com.

--

-Thanks, Darrion

[Vitaly Chipounov]

Hi,

The warnings in the screenshot are benign. Regarding USB, did you make it work with vanilla QEMU? S2E uses QEMU 3.0, so a first step would be to check that it works there without S2E.

I suppose you added the required USB hardware to the command line in bootstrap.sh (alongside your nic settings). You might need to trigger a pci bus rescan in the guest to detect it.

Vitaly

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CAE%3D33HX4jfPrxWqwg%2BWN14YKX8OWjDiF7fuHRgSZv5r1TQGrFg%40mail.gmail.com.

[Darrion Ramos]

I have not added anything in the bootstrap file for USB. I am just specifying the following within the launch-s2e.sh: -usb -device nec-usb-xhci -device usb-host,vendorid=xxxx,productid=xxxx. Is there something else I need to specify in the bootstrap?

Yes it works with vanilla qemu, but I was not aware s2e was using qemu 3.0. When I tried running the qemu 3.0 binary without s2e it similarly failed to pass through the USB device. Is there a reason s2e uses qemu 3.0? I do not know the extent of the modifications, but I am assuming it's not very feasible for me to add the s2e modifications to a newer qemu version. Perhaps you know of a different pass-through solution?

I also tried to refresh the pci bus in the vm but it did not affect the output of lsusb.

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CACjSjcs%2Bj86S61hM-sF5d%3DdJm0LWmBTKK_GJ9ucZhNJWryazXQ%40mail.gmail.com.

--

-Thanks, Darrion

[Vitaly Chipounov]

Hi,

On Tue, Oct 1, 2024 at 2:38 AM Darrion Ramos <darrio...@gmail.com> wrote:

I have not added anything in the bootstrap file for USB. I am just specifying the following within the launch-s2e.sh: -usb -device nec-usb-xhci -device usb-host,vendorid=xxxx,productid=xxxx. Is there something else I need to specify in the bootstrap?

That sounds right, there is nothing to put in bootstrap.sh besides the pci bus rescan. Since the snapshot was taken without USB, the guest needs to refresh the devices. Here's how it looks for me before and after rescan:

QEMU sees the host device:

Unfortunately, lsusb is still empty. I don't know why. I added the following commands to my bootstrap.sh:

sudo dhclient
sudo apt-get install usbutils
echo 1 | sudo tee /sys/bus/pci/rescan

 

Yes it works with vanilla qemu, but I was not aware s2e was using qemu 3.0. When I tried running the qemu 3.0 binary without s2e it similarly failed to pass through the USB device. Is there a reason s2e uses qemu 3.0? I do not know the extent of the modifications, but I am assuming it's not very feasible for me to add the s2e modifications to a newer qemu version. Perhaps you know of a different pass-through solution?

Then that means we'll need to upgrade QEMU. There is no particular reason it uses 3.0, I didn't get a chance to upgrade it. Doing so is conceptually straightforward [1], you need to apply the S2E commits on top of the current version (disk snapshots, kvm extensions mostly). Probably a month or two of full time work if you have no prior knowledge of QEMU internals. Alternatively, you could check what changed in the USB stack between 3.0 and the current version, then cherry-pick the changes. You could also try with QEMU 3.1, 4.0, 5.0, or whatever minimum version works. If 3.1 works, it may be easy to add S2E commits on top.

Vitaly

[1] http://s2e.systems/docs/DesignAndImplementation/KvmInterface.html

 

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CAE%3D33HV0Lp0%2Bk_U4rsd1nQref%2BRh9dgaPTyi_Tp6iDnHQTHAzg%40mail.gmail.com.

[Darrion Ramos]

Hi Vitaly, thanks again for taking the time to discuss this.

> Then that means we'll need to upgrade QEMU.

I previously made an error when I said I tested the QEMU 3.0 binary
without S2E. I ran the S2E’s QEMU with the QEMU command line interface
instead of from an S2E project. But the QEMU itself is modified in S2E
so that was not correct.

I have been trying to download and test a clean version of QEMU 3.0 to
properly test and see if USB passthrough works on that version.
Unfortunately, I have not been able to fix what I believe is some
dependency issue with X11. There are errors with an unknown type name
“Window” during the make process.

> You could also try with QEMU 3.1, 4.0, 5.0, or whatever minimum version works. If 3.1 works, it may be easy to add S2E commits on top.

I was able to build QEMU 4.0 without issues. There the USB device
passes through properly in lsusb but the udev error is still present,
so that may be benign. I could start looking into patching S2E commits
on top of QEMU 4.0 but not properly testing 3.0 makes me think the
problem still may be with S2E.

For what my group would like to test we need functional deep
suspension in the VM (our goal is to investigate some of the power
management bugs within drivers such as how interrupt handlers
interfere with suspend/sleep events). Currently, I see only s2idle is
supported in the VM and issuing a systemctl suspend does not work
properly and there is some workqueue call trace in the dmesg.

I understand that S2E was likely designed without considering suspend
since it normally will not be used in S2E. Would you have any ideas on
if these issues originate from the S2E VM image or the DBT-based KVM
Implementation? I am specifically wondering if the DBT emulation on
its own would be causing issues with suspend since I am not interested
in symbolic execution, just the coverage. If not, perhaps I could
somehow utilize just that portion of S2E as essentially a hypervisor
with LCOV.


On Tue, Oct 1, 2024 at 2:58 PM 'Vitaly Chipounov' via S2E Developer
Forum <s2e...@googlegroups.com> wrote:
>
> Hi,
>

> On Tue, Oct 1, 2024 at 2:38 AM Darrion Ramos <darrio...@gmail.com> wrote:
>>
>> I have not added anything in the bootstrap file for USB. I am just specifying the following within the launch-s2e.sh: -usb -device nec-usb-xhci -device usb-host,vendorid=xxxx,productid=xxxx. Is there something else I need to specify in the bootstrap?
>
>
> That sounds right, there is nothing to put in bootstrap.sh besides the pci bus rescan. Since the snapshot was taken without USB, the guest needs to refresh the devices. Here's how it looks for me before and after rescan:
>

> QEMU sees the host device:
>
>

> Unfortunately, lsusb is still empty. I don't know why. I added the following commands to my bootstrap.sh:
>
> sudo dhclient
> sudo apt-get install usbutils
> echo 1 | sudo tee /sys/bus/pci/rescan
>
>
>>
>>
>> Yes it works with vanilla qemu, but I was not aware s2e was using qemu 3.0. When I tried running the qemu 3.0 binary without s2e it similarly failed to pass through the USB device. Is there a reason s2e uses qemu 3.0? I do not know the extent of the modifications, but I am assuming it's not very feasible for me to add the s2e modifications to a newer qemu version. Perhaps you know of a different pass-through solution?
>
>
> Then that means we'll need to upgrade QEMU. There is no particular reason it uses 3.0, I didn't get a chance to upgrade it. Doing so is conceptually straightforward [1], you need to apply the S2E commits on top of the current version (disk snapshots, kvm extensions mostly). Probably a month or two of full time work if you have no prior knowledge of QEMU internals. Alternatively, you could check what changed in the USB stack between 3.0 and the current version, then cherry-pick the changes. You could also try with QEMU 3.1, 4.0, 5.0, or whatever minimum version works. If 3.1 works, it may be easy to add S2E commits on top.
>
> Vitaly
>
> [1] http://s2e.systems/docs/DesignAndImplementation/KvmInterface.html
>
>>
>>
>> On Mon, Sep 30, 2024 at 5:15 AM 'Vitaly Chipounov' via S2E Developer Forum <s2e...@googlegroups.com> wrote:
>>>
>>> Hi,
>>>
>>> The warnings in the screenshot are benign. Regarding USB, did you make it work with vanilla QEMU? S2E uses QEMU 3.0, so a first step would be to check that it works there without S2E.
>>> I suppose you added the required USB hardware to the command line in bootstrap.sh (alongside your nic settings). You might need to trigger a pci bus rescan in the guest to detect it.
>>>
>>> Vitaly
>>>
>>> On Sun, Sep 29, 2024 at 4:03 AM Darrion Ramos <darrio...@gmail.com> wrote:
>>>>
>>>> Hi Vitaly, thanks for fixing the crash and your suggestions.
>>>>
>>>> I was able to get the SSH to work with the dhclient command. I was also able to add my other configurations in the bootstrap once I understood that you were booting from a snapshot so my modifications to the guestfs were not doing anything.
>>>>
>>>> Also, regarding the USB pass-through. I was able to recompile the qemu binary with the proper support. But the passthrough is not working properly. lsusb outputs nothing on the VM, not even a hub. Is there some configuration needed in the image creation to allow this? When unplugging and reinserting the usb keyboard there is a udev error as you can see in the last line of this screenshot. I have read that this may be due to a dated libusb issue but this should be using the same version I used when building qemu from source. Unless s2e has its own isolated libusb version it is using. Do you know of any s2e configurations or anything else that could be affecting this? The device passes through fine with standalone qemu.
>>>>
>>>> Lastly, should I be concerned about the other warnings in the screenshot? It doesn't look like I need to but just double checking.
>>>>
>>>>
>>>>
>>>>

> To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CACjSjctdric%3DKDt-tDVQWnQjpQnRGyj%3DvDiHh8Ktn5V__2L-Qw%40mail.gmail.com.



--
-Thanks, Darrion

[Vitaly Chipounov]

Hi,

Please find my answers below.

On Mon, Oct 7, 2024 at 5:50 AM Darrion Ramos <darrio...@gmail.com> wrote:

Hi Vitaly, thanks again for taking the time to discuss this.

> Then that means we'll need to upgrade QEMU.
I previously made an error when I said I tested the QEMU 3.0 binary
without S2E. I ran the S2E’s QEMU with the QEMU command line interface
instead of from an S2E project. But the QEMU itself is modified in S2E
so that was not correct.

I have been trying to download and test a clean version of QEMU 3.0 to
properly test and see if USB passthrough works on that version.
Unfortunately, I have not been able to fix what I believe is some
dependency issue with X11. There are errors with an unknown type name
“Window” during the make process.

The S2E version of QEMU has this fixed, you may cherry pick the commit from there.

 

> You could also try with QEMU 3.1, 4.0, 5.0, or whatever minimum version works. If 3.1 works, it may be easy to add S2E commits on top.
I was able to build QEMU 4.0 without issues. There the USB device
passes through properly in lsusb but the udev error is still present,
so that may be benign. I could start looking into patching S2E commits
on top of QEMU 4.0 but not properly testing 3.0 makes me think the
problem still may be with S2E.

Depending on the complexity, it might be better to just upgrade to the latest version.

It would be a pity to redo all the work from scratch again. Try with QEMU 3.1 first, which should be much easier.

 

For what my group would like to test we need functional deep
suspension in the VM (our goal is to investigate some of the power
management bugs within drivers such as how interrupt handlers
interfere with suspend/sleep events). Currently, I see only s2idle is
supported in the VM and issuing a systemctl suspend does not work
properly and there is some workqueue call trace in the dmesg.

I understand that S2E was likely designed without considering suspend
since it normally will not be used in S2E. Would you have any ideas on
if these issues originate from the S2E VM image or the DBT-based KVM
Implementation? I am specifically wondering if the DBT emulation on
its own would be causing issues with suspend since I am not interested
in symbolic execution, just the coverage. If not, perhaps I could
somehow utilize just that portion of S2E as essentially a hypervisor
with LCOV.

I am not familiar enough with power management to answer this. I believe that it should all be mostly implemented in software (ACPI tables, etc.).

Perhaps some CPU support might be needed too (SMM, etc.). You should check with vanilla QEMU, see if suspend/resume works there.

In theory, S2E doesn't care, it just executes x86 instructions.

Vitaly

 

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CAE%3D33HXaNr0BbY0jwEN0e99gdv3R9uMq6wQtF6xg4YvQdJrZgQ%40mail.gmail.com.

[Darrion Ramos]

Hi,

The S2E version of QEMU has this fixed, you may cherry pick the commit from there.

Thanks, this worked. QEMU is not the problem. I should have looked earlier, but the S2E kernel is not configured with CONFIG_USB_SUPPORT so this is the source of the issue with USB. 

I have a custom kernel of 6.6.43 that already works with qemu. I would like to try and modify it to work with S2E but I am having difficulty following your instructions in the s2e-linux-kernel README. In step 2 for "Extending" it says to copy include/s2e/*/*_monitor.h from an existing kernel. But I only see this in the s2e-linux-kernel directory, not within a kernel so it is not clear where this should be copied to in my kernel. For step 5 is there not a plugin from an existing kernel I can use? Let's say I finish these steps, I haven't been able to find how I can configure s2e or a specific image to use that kernel. I read that the image somehow has the .deb files injected into it. So I guess I need to create a new image, but I am having a hard time figuring out how to modify the makefile to use a different kernel. 

Also, is there a way for me to create a new snapshot for the image to run from? If I have the vmlinux of a custom kernel, could I replace the one in the guestfs and create a new snapshot to have changes take effect?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CACjSjcsrgVUSJ%3DK-e5Ys%3DgU058Jhrtbnj5EkdJ8FboNfyHSBfQ%40mail.gmail.com.

--

-Thanks, Darrion

[Vitaly Chipounov]

Hi,

On Wed, Oct 9, 2024 at 9:25 PM Darrion Ramos <darrio...@gmail.com> wrote:

The S2E version of QEMU has this fixed, you may cherry pick the commit from there.

Thanks, this worked. QEMU is not the problem. I should have looked earlier, but the S2E kernel is not configured with CONFIG_USB_SUPPORT so this is the source of the issue with USB.

Oh nice fine, I didn't think of that. You may add any options you want to the kernel:
https://github.com/S2E/s2e-linux-kernel/blob/master/linux-6.8.2/config-x86_64
https://github.com/S2E/s2e-linux-kernel/blob/master/linux-6.8.2/config-i386

Then rebuild the images. It will be easier than integrating your own kernel.

You may also use make menuconfig inside the kernel folder, then copy the config file to config-i386 or config-x86_64.

 

I have a custom kernel of 6.6.43 that already works with qemu. I would like to try and modify it to work with S2E but I am having difficulty following your instructions in the s2e-linux-kernel README. In step 2 for "Extending" it says to copy include/s2e/*/*_monitor.h from an existing kernel. But I only see this in the s2e-linux-kernel directory, not within a kernel so it is not clear where this should be copied to in my kernel. For step 5 is there not a plugin from an existing kernel I can use? Let's say I finish these steps, I haven't been able to find how I can configure s2e or a specific image to use that kernel. I read that the image somehow has the .deb files injected into it. So I guess I need to create a new image, but I am having a hard time figuring out how to modify the makefile to use a different kernel.

You could have a look at the commit history in the repo when I migrated from 4.9.3 to 6.8.2 (in s2e-linux-kernel and guest-images).

 

Also, is there a way for me to create a new snapshot for the image to run from? If I have the vmlinux of a custom kernel, could I replace the one in the guestfs and create a new snapshot to have changes take effect?

It would be a nice feature to have. For this to happen, we should figure out a way to instrument the kernel without having to patch its source. Exploring ebpf or systemtap could be an option.
That would enable using arbitrary guest images with S2E.

The guestfs folder is a copy of the executable files inside the image. This copy may be used by plugins when they need to read binaries, which is more reliable than trying to read them from the guest's memory. Changing anything in that folder will have no effect on the guest image itself.

Vitaly

 

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CAE%3D33HWGKiypB7EMOwWi0D1tJxGB4ROq0xs8MAUjdtm0W15Lug%40mail.gmail.com.

[Darrion Ramos]

Hi Vitaly, I have been able to fix most of my issues with USB and PM support after changing the kernel config and rescanning the pci bus.

From what I have been reading and experimenting with, it seems like the line coverage provided by S2E only provides a 0 or 1 to determine if the line was used. I would like to be able to see if lines were accessed multiple times for certain code. In your documentation I see you are caching the translation blocks to avoid redundant translations. Do you know if it is feasible for me to modify the plug-ins to change this?

Now that I have configured the kernel with ACPI support and suspend-to-RAM, there is an s2e memory access issue when resuming from suspend-to-RAM. I would appreciate it if you are able to provide some insight on this. Below is a screenshot of the failure message. I tried to analyze the core dump file but it is truncated and I have not found a way to fix that yet (ulimit is set to unlimited). Perhaps the program is being shut down before it can finish writing the dump.

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CACjSjctrDu9wNM5jBh_x9sy7jBvmUGWCU%2BZDzTCFwyCH%2B2t%2BJQ%40mail.gmail.com.

--

-Thanks, Darrion

[Vitaly Chipounov]

Hi Darrion,

On Sat, Oct 19, 2024 at 8:59 PM Darrion Ramos <darrio...@gmail.com> wrote:

Hi Vitaly, I have been able to fix most of my issues with USB and PM support after changing the kernel config and rescanning the pci bus.

I am glad you made progress!

 

From what I have been reading and experimenting with, it seems like the line coverage provided by S2E only provides a 0 or 1 to determine if the line was used. I would like to be able to see if lines were accessed multiple times for certain code. In your documentation I see you are caching the translation blocks to avoid redundant translations. Do you know if it is feasible for me to modify the plug-ins to change this?

The engine translates and instruments blocks once, then caches them for performance reasons. You should not change this.

What you need is to instrument each instruction so that a counter is incremented every time that instruction is executed.

You may reuse the instruction counter tutorial [1] with minimal changes. Replace m_count with map<pc, count>, then dump it when the state terminates.

You may want to translate absolute program counters to module-relative ones in order to get source info with, e.g., addr2line.

 

Now that I have configured the kernel with ACPI support and suspend-to-RAM, there is an s2e memory access issue when resuming from suspend-to-RAM. I would appreciate it if you are able to provide some insight on this. Below is a screenshot of the failure message. I tried to analyze the core dump file but it is truncated and I have not found a way to fix that yet (ulimit is set to unlimited). Perhaps the program is being shut down before it can finish writing the dump.

I will need a lot more than this to debug it. Please export the S2E project. Ideally, I should be able to reproduce the crash by running ./launch-s2e.sh on top of master (i.e., no custom plugins or images should be required).

Vitaly

[1] https://s2e.systems/docs/Howtos/WritingPlugins.html

 

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CAE%3D33HXuW3tUnrC092cbSSjfv%2BL%3Ds3K-2KAesZ3btypCvnGTsQ%40mail.gmail.com.

[Darrion Ramos]

Hi, 

The engine translates and instruments blocks once, then caches them for performance reasons. You should not change this.

What you need is to instrument each instruction so that a counter is incremented every time that instruction is executed.

You may reuse the instruction counter tutorial [1] with minimal changes. Replace m_count with map<pc, count>, then dump it when the state terminates.

You may want to translate absolute program counters to module-relative ones in order to get source info with, e.g., addr2line.

Thanks for this explanation. I will look into this more and get back to you.

I will need a lot more than this to debug it. Please export the S2E project. Ideally, I should be able to reproduce the crash by running ./launch-s2e.sh on top of master (i.e., no custom plugins or images should be required).

Unfortunately, the existing s2e kernel is not configured with ACPI. Therefore it would be required to recompile the s2e kernel with the kernel config attached and create an image with that kernel to reproduce this error. 

To reproduce after reconfiguring the kernel:

1. Start launch-s2e with the GUI enabled.

2. Enter systemctl suspend

3. Wait for suspension to complete based on LinuxMonitor output

4. Ensure the VM's GUI window is selected and resume the VM with a keypress 

5. S2E should crash at this point

If it does not, it is possible the keypress did not resume the VM. (QEMU has some strange resume quirks. I resume from a laptop keyboard which may behave differently from a connected desktop peripheral). In this case, try resuming the VM from the qemu socket located in the project directory:

1. sudo socat - unix-connect:qemu-monitor-socket

2. issue <system_wakeup> from the (qemu) shell

Notes: 

- I am using an ubuntu-22.04-x86_64 host and image, but I don't see a reason for the issue to be OS dependent.

- I have been running the project with a modified s2e that supports qemu usb passthrough. I believe that the project attached should not have any dependency to this. But if there is, let me know and I can recreate the project from a clean s2e.

Thanks again for helping me out with this niche implementation.

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CACjSjcvehE8Gvn5pARaSFDQRbyz83G-B4WV73JqAir%2BGFRYdYQ%40mail.gmail.com.

--

-Thanks, Darrion

[Vitaly Chipounov]

Hi,

I reproduced and fixed the crash: https://github.com/S2E/s2e/pull/105
The guest still freezes after resume though.

Vitaly

To view this discussion on the web visit https://groups.google.com/d/msgid/s2e-dev/CAE%3D33HWtA%2BT5R5MM3nLQ9o8P%2BTc3uABT8Kw5X%2BYsf%3DPt1sv3vw%40mail.gmail.com.

[Darrion Ramos]

Hi Vitaly,

Thanks for fixing that. I have been working on getting a better understanding of what is going on after the wakeup signal is received by s2e's qemu. From what I understand, the guest is not freezing but s2e is infinitely looping an ioctl call to kvm. I have attached some of my gdb debugging logs that show the ioctl loop, then another showing why the ioctl is being repeated. I see that the ioctl is failing with an EINTR exit due to m_cpuBuffer having an exit status of KVM_EXIT_INTR. But I don't understand yet how that originates and why it is not being cleared. Do you know what may be happening here and if this is even something that should not be happening? 

To me it looks like it should not be stuck there but I also don't understand exactly what the ioctl is accomplishing, nor the entirety of the program. I believe thread 1 controlling the qemu main loop and thread 9 controlling the qemu kvm ioctls are the main things I should be concerned about for this, but please inform me if otherwise.

Since s2e utilizes its own interface to talk with KVM I think perhaps there is some communication that is missing after the wakeup signal that qemu normally delivers to KVM. But that is just speculation.

I appreciate any ideas you have on this or thoughts on how I should further debug it, thanks.

To view this discussion visit https://groups.google.com/d/msgid/s2e-dev/CACjSjcvfg38%2B6MH3nfaBcOPEZRUtG5CJ5VU-81vT3nJ40joPEQ%40mail.gmail.com.

--

-Thanks, Darrion

[Vitaly Chipounov]

Hi Darrion,

Thanks for investigating this, it will be of great help. I'll have a look in the next few days.

Vitaly

To view this discussion visit https://groups.google.com/d/msgid/s2e-dev/CAE%3D33HUy4h6iTjZ3dO3Ti0f1kbFjgUTXB%3D2nbxPkz%3D3tVJLb7w%40mail.gmail.com.

[Vitaly Chipounov]

Hi,

I fixed it, please check the PR.

Vitaly

[Darrion Ramos]

Thank you, I appreciate it!

To view this discussion visit https://groups.google.com/d/msgid/s2e-dev/CACjSjct_NwJELwnMmiAuZyu4JNSyeDRRauu3Wyfh_Zkew5LLpA%40mail.gmail.com.

--

-Thanks, Darrion