More tcp fastopen tests failures

[Ravi Kerur]

Hello,

Tests are failing on kernel 5.19 and 6.1

I am running tests with

sudo ./packetdrill/run_all.py -S -s -v -L -l tcp/

Couldn't see any logs, how do I check the reasons for failure?

KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-connect-bypass-errno.pkt (ipv4-mapped-v6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-connect-bypass-errno.pkt (ipv6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-connect-bypass.pkt (ipv4)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-connect-bypass.pkt (ipv4-mapped-v6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-connect-bypass.pkt (ipv6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-sendmsg-multi-iov.pkt (ipv4)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-sendmsg-multi-iov.pkt (ipv4-mapped-v6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-sendmsg-multi-iov.pkt (ipv6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-sendto-errnos.pkt (ipv4)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-sendto-errnos.pkt (ipv4-mapped-v6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-sendto-errnos.pkt (ipv6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-sendto.pkt (ipv4)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-sendto.pkt (ipv4-mapped-v6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill-unmod/packetdrill/gtests/net/tcp/fastopen/client/blocking-sendto.pkt (ipv6)]

Thanks,

Ravi

[Neal Cardwell]

Thanks for the report!

The "KILL" there means that the test timed out and run_all.py killed it. I'm not sure why those tests are not working in your environment; they work for me on a GCP VM running  Ubuntu 22.04 with a recent kernel based on a net-next tree shortly after 6.4 (ba2274dcfda8).

You might try to debug by running the full packetdrill command line, with instrumentation from tcpdump and strace; perhaps something like:

pushd tcp/fastopen/client/

tcpdump -w /tmp/1.pcap -n -i any port 8080 & 

strace -ttt /root/packetdrill/gtests/net/packetdrill/packetdrill --send_omit_free --ip_version=ipv4 --local_ip=192.168.0.1 --gateway_ip=192.168.0.1 --netmask_ip=255.255.0.0 --remote_ip=192.0.2.1 -D TFO_COOKIE=3021b9d889017eeb -D CODE=host_unreachable -D CMSG_LEVEL_IP=SOL_IP -D CMSG_TYPE_RECVERR=IP_RECVERR cookie-req-timeout.pkt > /tmp/1.strace.txt 2>&1

popd

killall tcpdump

Looking at that command line, perhaps your environment is already using the RFC 1918 private ranges like 192.168/16 prefix for something, and so there are problems with routing or assigning addresses to interfaces? It might be useful to see the output of "ip route show" and "ip link show" on your machine, if you can share.

best regards,

neal

--
You received this message because you are subscribed to the Google Groups "packetdrill" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packetdrill...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/403407b6-9b72-43f7-a81d-ed4a9380a8fdn%40googlegroups.com.

[Ravi Kerur]

Thanks Neal.

I collected the logs as you requested and I don't see anything obvious in strace.txt. In my config I don't have 192.168.0.0 subnet. Attaching strace and pcap from the tests. Please let me know if anything looks suspicious and I need to modify my config.

It fails on both 5.19 and 6.1 kernel so it could be something config related.

sudo ip route show
default via 10.20.21.1 dev ens5
default via 10.20.20.1 dev ens4 proto dhcp src 10.20.20.2 metric 100
10.20.20.0/24 via 10.20.20.1 dev ens4 proto dhcp src 10.20.20.2 metric 100
10.20.20.1 dev ens4 proto dhcp scope link src 10.20.20.2 metric 100
10.20.21.0/24 via 10.20.21.1 dev ens5
10.20.21.1 dev ens5 scope link
35.191.0.0/16 via 10.20.20.1 dev ens4
35.235.240.0/20 via 10.20.20.1 dev ens4
130.211.0.0/22 via 10.20.20.1 dev ens4
sudo ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1343 qdisc mq state UP group default qlen 1000
    link/ether 42:01:0a:14:14:02 brd ff:ff:ff:ff:ff:ff
    inet 10.20.20.2/32 scope global dynamic ens4
       valid_lft 3573sec preferred_lft 3573sec
    inet 10.0.1.0/32 brd 10.255.255.255 scope global ens4:256
       valid_lft forever preferred_lft forever
    inet6 fe80::4001:aff:fe14:1402/64 scope link
       valid_lft forever preferred_lft forever
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1343 qdisc prio state UP group default qlen 1000
    link/ether 42:01:0a:14:15:02 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 10.20.21.2/32 brd 10.20.21.2 scope global dynamic ens5
       valid_lft 2659sec preferred_lft 2659sec
    inet6 fe80::4001:aff:fe14:1502/64 scope link
       valid_lft forever preferred_lft forever
5: tun0: <NO-CARRIER,POINTOPOINT,MULTICAST,NOARP,UP> mtu 1500 qdisc fq state DOWN group default qlen 500
    link/none
 sudo ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1343 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 42:01:0a:14:14:02 brd ff:ff:ff:ff:ff:ff
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1343 qdisc prio state UP mode DEFAULT group default qlen 1000
    link/ether 42:01:0a:14:15:02 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
5: tun0: <NO-CARRIER,POINTOPOINT,MULTICAST,NOARP,UP> mtu 1500 qdisc fq state DOWN mode DEFAULT group default qlen 500
    link/none

sudo sysctl -a | grep tcp | grep fast
net.ipv4.tcp_fastopen = 459779
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 0
net.ipv4.tcp_fastopen_key = a1a1a1a1-b2b2b2b2-c3c3c3c3-d4d4d4d4

PCAP

sudo tcpdump -r /tmp/1.pcap
reading from file /tmp/1.pcap, link-type LINUX_SLL (Linux cooked v1)
17:35:48.678997 IP 192.168.0.1.36580 > 192.0.2.1.http-alt: Flags [S], seq 3446072174, win 65535, options [mss 1460,nop,nop,sackOK,nop,wscale 8,tfo  cookiereq,nop,nop], length 0
17:35:49.712607 IP 192.168.0.1.36580 > 192.0.2.1.http-alt: Flags [S], seq 3446072174, win 65535, options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0
17:35:49.722968 IP 192.0.2.1.http-alt > 192.168.0.1.36580: Flags [S.], seq 123, ack 3446072175, win 5840, options [mss 1460,nop,wscale 6], length 0
17:35:49.722990 IP 192.168.0.1.36580 > 192.0.2.1.http-alt: Flags [.], ack 1, win 256, length 0
17:35:49.733419 IP 192.168.0.1.36580 > 192.0.2.1.http-alt: Flags [F.], seq 1, ack 1, win 256, length 0
17:35:49.743652 IP 192.0.2.1.http-alt > 192.168.0.1.36580: Flags [F.], seq 1, ack 2, win 92, length 0
17:35:49.743660 IP 192.168.0.1.36580 > 192.0.2.1.http-alt: Flags [.], ack 2, win 256, length 0
17:35:49.814073 IP 192.168.0.1.36582 > 192.0.2.1.http-alt: Flags [S], seq 2905271828, win 65535, options [mss 1460,nop,nop,sackOK,nop,wscale 8,tfo  cookiereq,nop,nop], length 0
17:35:50.864605 IP 192.168.0.1.36582 > 192.0.2.1.http-alt: Flags [S], seq 2905271828, win 65535, options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0
17:35:50.874897 IP 192.0.2.1.http-alt > 192.168.0.1.36582: Flags [S.], seq 123, ack 2905271829, win 60000, options [mss 1240,tfo  cookie aaaabbbbcccc], length 0
17:35:50.874910 IP 192.168.0.1.36582 > 192.0.2.1.http-alt: Flags [.], ack 1, win 65535, length 0
17:35:50.885331 IP 192.168.0.1.36582 > 192.0.2.1.http-alt: Flags [F.], seq 1, ack 1, win 65535, length 0
17:35:50.895552 IP 192.0.2.1.http-alt > 192.168.0.1.36582: Flags [F.], seq 1, ack 2, win 92, length 0
17:35:50.895559 IP 192.168.0.1.36582 > 192.0.2.1.http-alt: Flags [.], ack 2, win 65535, length 0
17:35:50.966017 IP 192.168.0.1.36596 > 192.0.2.1.http-alt: Flags [S], seq 3737035539:3737036739, win 65535, options [mss 1460,nop,nop,sackOK,nop,wscale 8,tfo  cookie aaaabbbbcccc], length 1200: HTTP
17:35:50.989264 IP 192.0.2.1.http-alt > 192.168.0.1.36596: Flags [R.], seq 0, ack 3737035540, win 0, length 0
17:35:50.989301 IP 192.0.2.1.http-alt > 192.168.0.1.36582: Flags [R.], seq 2, ack 2, win 92, length 0
17:35:50.989333 IP 192.0.2.1.http-alt > 192.168.0.1.36580: Flags [R.], seq 2, ack 2, win 92, length 0

Thanks,

Ravi