Packet Drill remote mode between host and qemu

[Mohamed Husain]

Hi,

Did anyone try testing packet drill in Linux running on qemu ? I know it will run only on local LAN, but it is any way test on a virtual machine.

Thanks.

[Neal Cardwell]

Hi,

I have not used packetdrill with qemu. However, packetdrill remote mode should work between virtual machines as long as the virtual machines "think" they are connected by Ethernet-like network devices on the same virtual L2 broadcast domain. In the past I have successfully used packetdrill remote mode between two vmware virtual machines on the same laptop.

And you should be able to use normal tun-based packetdrill tests on a virtual machine as well. That should work inside basically any virtual machine running Linux, FreeBSD, etc.

best,

neal

--
You received this message because you are subscribed to the Google Groups "packetdrill" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packetdrill...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/d96d5edb-b4f0-4e82-a1ce-683d44a98530n%40googlegroups.com.

[Mohamed Husain]

Hi Neal,

Thanks for your response.

I tried the remote mode in QEMU setup using TAP0 interface. But the script gets hanged at accept syscall in the GRO script.

From the source code I can see it is waiting at here.

This is the launch command.

qemu-system-x86_64 -M q35 -device qemu-xhci,id=xhci -m 4G -enable-kvm $IVSHMEM_OPTS -kernel  $KERNEL -append "root=/dev/sda ro console=ttyS0 net.ifnames=0" -hda $IMAGE -no-reboot -nographic \
    -net nic,model=e1000 \
    -net tap,ifname=tap0,script=no,downscript=no

This config inside VM.

root@syzkaller:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.2.0.100  netmask 255.255.255.0  broadcast 10.2.0.255
        inet6 2601:5c0:c200:4460:5054:ff:fe12:3456  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::5054:ff:fe12:3456  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:12:34:56  txqueuelen 1000  (Ethernet)
        RX packets 992  bytes 113453 (110.7 KiB)
        RX errors 343  dropped 0  overruns 0  frame 343
        TX packets 1170  bytes 92864 (90.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Packetdrill in Host:(IP 10.2.0.1)

sudo ./packetdrill --wire_server --wire_server_dev=tap0

Packetdrill in VM:(IP 10.2.0.100)

sudo ./packetdrill -v --wire_client --wire_server_ip=10.2.0.1  ../tcp/gro/gro-mss-option.pkt \
 --tolerance_usecs=9000  --local_ip=172.16.0.1 --gateway_ip=172.16.0.2 --netmask_ip=255.255.255.0 --remote_ip=172.16.0.2

Please let me know If am missing something.

Thanks

Mohamed

[Neal Cardwell]

It looks like you are trying to get packetdrill to work in a scenario where the client is inside a qemu virtualized environment, and the server is outside the virtualized environment, on the host. The packetdrill tool is not designed for that to work. You may be able to get that to work with enough coding and debugging. However, is there a reason to do that? Why add the complexity from the virtualization layer "boundary crossing"?

Why not use one of the supported scenarios, like:

(a) packetdrill local mode, inside a VM or on a physical machine

(b) packetdrill remote mode, where both server and client are either (a) inside the same L2 virtualized network, or (b) inside the same L2 physical network

Can you help me understand your motivation for trying to get packetdrill to work in a scenario where the client is inside a qemu virtualized environment, and the server is outside the virtualized environment, on the host?

thanks,

neal

To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/caf408f2-043a-4ac6-b639-dad42fc931dfn%40googlegroups.com.

[Mohamed Husain]

Hi Neal,

We are planning to use packetdrill for our kernel fuzzer to test the kernel's network stack. Since packetdrill scripts have many test cases, we are thinking that it will increase our code coverage. The reason we want a remote mode is that we will inject the fuzzed packet through the QEMU interface, so the packets will pass through the emulated NIC and its device driver.

I greped the tcpdump data during sendfile test I can see connection being reset during initial handshake. Any reason connection gets reset?

Setup details:

packetdrill server: Host

packetdrill client:  QEMU OS

Packetdrill in Host:(IP 10.2.0.1 , 172.16.0.2)

sudo ./packetdrill --wire_server --wire_server_dev=tap0

Packetdrill in VM:(IP 10.2.0.100, 172.16.0.1 )

sudo ./packetdrill -v --wire_client --wire_server_ip=10.2.0.1  ../tcp/gro/gro-mss-option.pkt \
 --tolerance_usecs=9000  --local_ip=172.16.0.1 --gateway_ip=172.16.0.2 --netmask_ip=255.255.255.0 --remote_ip=172.16.0.2

17:38:10.702393 IP 172.16.0.2.42245 > 172.16.0.1.8080: Flags [S], seq 0, win 32792, options [mss 1000,sackOK,nop,nop,nop,wscale 10], length 0
17:38:10.702622 IP 172.16.0.1.8080 > 172.16.0.2.42245: Flags [S.], seq 837981436, ack 1, win 65535, options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0
17:38:10.702803 IP 172.16.0.2.42245 > 172.16.0.1.8080: Flags [R], seq 1, win 0, length 0
17:38:10.717568 IP 172.16.0.2.42245 > 172.16.0.1.8080: Flags [.], ack 1, win 514, length 0

17:38:10.717606 IP 172.16.0.1.8080 > 172.16.0.2.42245: Flags [R], seq 837981437, win 0, length 0
17:38:11.931398 IP 172.16.0.2.42245 > 172.16.0.1.8080: Flags [R.], seq 1, ack 1, win 514, length 0

Thanks

Mohamed

[Neal Cardwell]

> 17:38:10.702803 IP 172.16.0.2.42245 > 172.16.0.1.8080: Flags [R], seq 1, win 0, length 0

In that scenario it looks like the RST above was sent by the host kernel, which does not know about this connection (only the packetdrill processes and the kernel under test know about the connection). 

Does your host machine have that IP address configured on some network interface (try running "ip addr list")? The packetdrill tool is meant to be used with test IPs (for --local_ip=, --gateway_ip=, -remote_ip=) that are not in use on NICs on the machines you are using.

best,

neal

To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/7022114c-1081-49f0-9830-1d242cf8fe77n%40googlegroups.com.

[Mohamed Husain]

Hi,

Here is the o/p of  `ip addr list`.

husain@firebolt:~/packetdrill/gtests/net/packetdrill$ ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether f4:8e:38:e9:0a:6b brd ff:ff:ff:ff:ff:ff
3: wlxd03745d04a8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d0:37:45:d0:4a:8b brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.14/24 brd 192.168.0.255 scope global dynamic noprefixroute wlxd03745d04a8b
       valid_lft 2456sec preferred_lft 2456sec
    inet6 2601:5c0:c200:4460:e022:5bcf:6c88:b121/64 scope global temporary dynamic
       valid_lft 308082sec preferred_lft 49248sec
    inet6 2601:5c0:c200:4460:71fb:a70c:e9fd:1232/64 scope global temporary deprecated dynamic
       valid_lft 308082sec preferred_lft 0sec
    inet6 2601:5c0:c200:4460:c13c:1072:a652:cea7/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 308082sec preferred_lft 308082sec
    inet6 fe80::8b9a:74be:9e73:a7d2/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:ff:16:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:ff:16:cb brd ff:ff:ff:ff:ff:ff
7: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether d6:f1:25:4d:f1:c5 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.1/24 scope global tap0
       valid_lft forever preferred_lft forever
    inet 172.16.0.2/24 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::d4f1:25ff:fe4d:f1c5/64 scope link
       valid_lft forever preferred_lft forever
8: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 0e:88:e8:c9:68:06 brd ff:ff:ff:ff:ff:ff
    inet 10.2.1.1/24 scope global tap1
       valid_lft forever preferred_lft forever
    inet6 fe80::c88:e8ff:fec9:6806/64 scope link

       valid_lft forever preferred_lft forever

Thanks,

Mohamed

[Neal Cardwell]

Yes, it seems like your tap0 device on your host machine has 172.16.0.2 configured. AFAICT at the very least you will need to switch to telling packetdrill to use IP address ranges that are not in use on any NICs on the guest kernel under test  or the host kernel doing the packet sniffing/injection.

After that is fixed, there may be other issues as well.

BTW, re:

We are planning to use packetdrill for our kernel fuzzer to test the kernel's network stack. Since packetdrill scripts have many test cases, we are thinking that it will increase our code coverage. The reason we want a remote mode is that we will inject the fuzzed packet through the QEMU interface, so the packets will pass through the emulated NIC and its device driver.

If that is your goal, AFAICT you can achieve that goal with packetdrill running between two qemu VMs on the same virtual network, rather than trying to run a test across the real/virtual boundary. You may want to try that if this approach of spanning the virtual/physical boundary does not work.

best,

neal

To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/bd238d25-35df-43f8-8938-94c926bea364n%40googlegroups.com.

[Mohamed Husain]

Hi Neal,

Thanks for your insights. I tried the remote mode between two VMs connected by bridged network with the following command.

sudo ./packetdrill -v --wire_client --wire_server_ip=10.2.0.1  ../tcp/gro/gro-mss-option.pkt \
 --tolerance_usecs=9000  --local_ip=172.16.0.1 --gateway_ip=172.16.0.2 --netmask_ip=255.255.255.0 --remote_ip=172.16.0.2

It got stuck at same accept syscall.

But when I changed the IP to --remote_ip=172.16.0.2 to --remote_ip=172.16.1.1, tests run without any failure.

My question is why 172.16.1.1 works and it is also in different sub-net compared to  local_ip. I am bit confused. Can you explain why the remote_ip in different sub-net works properly.

Once again thanks for your help.

Thanks,

Mohamed

[Neal Cardwell]

On Thu, May 12, 2022 at 12:00 PM Mohamed Husain <mohamedh...@gmail.com> wrote:

Hi Neal,

Thanks for your insights. I tried the remote mode between two VMs connected by bridged network with the following command.

sudo ./packetdrill -v --wire_client --wire_server_ip=10.2.0.1  ../tcp/gro/gro-mss-option.pkt \
 --tolerance_usecs=9000  --local_ip=172.16.0.1 --gateway_ip=172.16.0.2 --netmask_ip=255.255.255.0 --remote_ip=172.16.0.2

It got stuck at same accept syscall.

But when I changed the IP to --remote_ip=172.16.0.2 to --remote_ip=172.16.1.1, tests run without any failure.

My question is why 172.16.1.1 works and it is also in different sub-net compared to  local_ip. I am bit confused. Can you explain why the remote_ip in different sub-net works properly.

Once again thanks for your help.

Thanks,

Mohamed

I'm not sure what exact routing pathology you are running into when you were using the same value for --gateway_ip= and --remote_ip=. But note that when doing that packetdrill will end up executing route commands that boil down to "please route traffic for IP foo through gateway foo", which is apparently causing the test to not work. If you want to debug it and root-cause the failure, that would be welcome.

But please note that the default IPv4 and IPv6 test addresses used by packetdrill have the property that the remote_ip is in a different subnet than the --local_ip= and --gateway_ip=. So I would strongly recommend sticking with that model that you found works: the --remote_ip= is in a different subnet than the --local_ip= and --gateway_ip=.

best,

neal

 

To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/d202aa37-ab7e-47c9-b436-b6258eabba7an%40googlegroups.com.

[Maciej Jablonski]

I have been having the same problem for years. And had some really crazy workarounds. I have found the gateway fix pretty neat. I reckon it prevents the packets hitting kernel stack.

Let me share my full script that reproduces the issue when gateway_ip is not set (RST pkts) and does work with gateway_ip.
The script contains full config running client/server in separate namespaces.

# packetdrill binary
PD=packetdrill/gtests/net/packetdrill/packetdrill

# setup two namespaces from scratch
# each has a phys device port iface (they are looped back IA <==> IB)
sudo ip netns del n1
sudo ip netns add n1
sudo ip netns del n0
sudo ip netns add n0

# My usecase is phys devices, but veth pair good enough to demonstrate
IA=ens3f0np0v
IB=ens3f1np1v
sudo ip link add $IB netns n1 type veth peer name $IA netns n0
sudo ip netns exec n0 ifconfig $IA up
sudo ip netns exec n1 ifconfig $IB up

sudo ifconfig $IA up
sudo ip link set netns n1 $IB
sudo ip netns exec n1 ifconfig $IB up
sudo ip netns exec n1 ifconfig lo up
sudo ip link set netns n0 $IA
sudo ip netns exec n0 ifconfig $IA up
sudo ip netns exec n0 ifconfig lo up


# create veth device pair, wich each end in its namespace
# this is for control connection between packetdrill client and server
sudo ip link add veth1 netns n1 type veth peer name veth0 netns n0
sudo ip netns exec n0 ifconfig veth0 192.168.3.10/24 up
sudo ip netns exec n1 ifconfig veth1 192.168.3.11/24 up

sudo ip netns exec n1 ping -c2 192.168.3.10

sudo ip netns exec n0 ip addr flush dev $IA
sudo ip netns exec n1 ip addr flush dev $IB
sudo ip netns exec n1 $PD -vvv --wire_server --wire_server_dev=$IB &

# remove --gateway_ip=192.168.0.3  and client packets are replied to with RST
sudo ip netns exec n0  $PD --non_fatal=packet -vvv --wire_client --netmask=255.255.255.0 \
    --wire_server_ip=192.168.3.11 --local_ip=192.168.0.2 --remote_ip=192.168.0.1 --gateway_ip=192.168.0.3 --wire_client_dev=$IA src/tests/packetdrill/gtests/net/packetdrill/tests/connect/http-get-nonblocking-ts.pkt