packetdrill hangs on recvfrom

[wang Jianjian]

Hi,

While I run ./tests/linux/fast_retransmit/fr-4pkt-sack-linux.pkt, it always hangs until I press Ctrl+C.

I use gdb to debug packetdrill, seems it hangs on recvfrom, waiting the packet from kernel.

But why doesn't kernel reponse?

Here is the gdb stack:

(gdb) set args -v ./tests/linux/fast_retransmit/fr-4pkt-sack-linux.pkt

(gdb) r

Starting program: /auto/home2/wangj85/git/packetdrill/gtests/net/packetdrill/./packetdrill -v ./tests/linux/fast_retransmit/fr-4pkt-sack-linux.pkt

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Detaching after fork from child process 31641.

Detaching after fork from child process 31645.

[New Thread 0x7ffff7ffc700 (LWP 31653)]

inbound injected packet:  0.000086 S 0:0(0) win 65535 <mss 1000,sackOK,nop,nop,nop,wscale 7>

^C

Program received signal SIGINT, Interrupt.

0x0000000000427257 in recvfrom ()

(gdb) bt

#0  0x0000000000427257 in recvfrom ()

#1  0x000000000040858b in packet_socket_receive (psock=0x71d800, direction=DIRECTION_OUTBOUND, packet=0x72f880, in_bytes=0x7fffffffdb78)

    at packet_socket_linux.c:233

#2  0x00000000004072f7 in netdev_receive_loop (psock=0x71d800, layer=PACKET_LAYER_3_IP, direction=DIRECTION_OUTBOUND, packet=0x7fffffffdc60,

    num_packets=0x7fffffffdbb0, error=0x7fffffffdcb0) at netdev.c:447

#3  0x000000000040725d in local_netdev_receive (a_netdev=0x717280, packet=0x7fffffffdc60, error=0x7fffffffdcb0) at netdev.c:423

#4  0x000000000040e8b6 in netdev_receive (netdev=0x717280, packet=0x7fffffffdc60, error=0x7fffffffdcb0) at netdev.h:81

#5  0x000000000041145e in sniff_outbound_live_packet (state=0x71ccc0, expected_socket=0x71c720, packet=0x7fffffffdc60, error=0x7fffffffdcb0)

    at run_packet.c:1123

#6  0x0000000000411810 in do_outbound_script_packet (state=0x71ccc0, packet=0x71c3f0, socket=0x71c720, error=0x7fffffffdcb0) at run_packet.c:1230

#7  0x0000000000411cab in run_packet_event (state=0x71ccc0, event=0x71bc80, packet=0x71c3f0, error=0x7fffffffdcf0) at run_packet.c:1344

#8  0x000000000040d9a6 in run_local_packet_event (state=0x71ccc0, event=0x71bc80, packet=0x71c3f0) at run.c:387

#9  0x000000000040dd5b in run_script (config=0x7fffffffdd90, script=0x7fffffffdd60) at run.c:550

#10 0x000000000040214a in main (argc=3, argv=0x7fffffffe138) at packetdrill.c:109

(gdb)

thanks,

Jan

[wang Jianjian]

Forget to say, my kernel is  centos 7 3.10.

[root@wj packetdrill]# uname -a

Linux wj.localdomain 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[Neal Cardwell]

I suspect a conflict between an IP address range already in use on
your system, and the IPv4 address ranges used by packetdrill by
default, which are:

* - local address: 192.168.0.0/16 private IP space (RFC 1918)
* - remote address: 192.0.2.0/24 TEST-NET-1 range (RFC 5737)

Can you please run the following sequence of commands to help isolate
what is going wrong?

ip addr list
nstat > /dev/null
tcpdump -n -ttt -i any port 8080 &
packetdrill ./tests/linux/fast_retransmit/fr-4pkt-sack-linux.pkt
# then press ctrl-C
kill %1
nstat

thanks,
neal

> --
> You received this message because you are subscribed to the Google Groups
> "packetdrill" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to packetdrill...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Neal Cardwell]

BTW, if there is an IP address conflict then you can try to resolve it
by adding something like the following to your packetdrill command
line, switching to a different private IP space for the test:

--local_ip=10.0.0.1 --gateway_ip=10.0.0.2 --netmask_ip=255.255.255.0
--remote_ip=10.1.0.3

hope that helps,
neal

[Jan Wang]

Thanks Neal, per your suggestion, below is the output:

[root@wj packetdrill]# ip add list

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:50:56:82:1e:6a brd ff:ff:ff:ff:ff:ff

    inet 10.62.97.126/23 brd 10.62.97.255 scope global dynamic ens32

       valid_lft 527057sec preferred_lft 527057sec

    inet6 fe80::250:56ff:fe82:1e6a/64 scope link

       valid_lft forever preferred_lft forever

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN

    link/ether 02:42:8e:ee:27:c8 brd ff:ff:ff:ff:ff:ff

    inet 172.17.42.1/16 scope global docker0

       valid_lft forever preferred_lft forever

[root@wj packetdrill]# nstat > /dev/null

[root@wj packetdrill]# tcpdump -n -ttt -i any port 8080 &

[1] 3798

[root@wj packetdrill]# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

[root@wj packetdrill]# ./packetdrill  tests/linux/fast_retransmit/fr-4pkt-sack-linux.pkt

00:00:00.000000 IP 192.0.2.1.33923 > 192.168.0.1.webcache: Flags [S], seq 0, win 65535, options [mss 1000,sackOK,nop,nop,nop,wscale 7], length 0

^C

[root@wj packetdrill]# kill %1

1 packet captured

24 packets received by filter

0 packets dropped by kernel

[root@wj packetdrill]# nstat

#kernel

IpInReceives                    382                0.0

IpInDelivers                    282                0.0

IpOutRequests                   219                0.0

IcmpOutErrors                   1                  0.0

IcmpOutTimeExcds                1                  0.0

IcmpMsgOutType3                 1                  0.0

TcpActiveOpens                  2                  0.0

TcpInSegs                       279                0.0

TcpOutSegs                      215                0.0

UdpInDatagrams                  3                  0.0

UdpOutDatagrams                 3                  0.0

Ip6InReceives                   1                  0.0

Ip6InDelivers                   1                  0.0

Ip6InMcastPkts                  1                  0.0

Ip6InOctets                     104                0.0

Ip6InMcastOctets                104                0.0

Icmp6InMsgs                     1                  0.0

Icmp6InRouterAdvertisements     1                  0.0

Icmp6InType134                  1                  0.0

TcpExtDelayedACKs               6                  0.0

TcpExtTCPHPHits                 55                 0.0

TcpExtTCPPureAcks               38                 0.0

TcpExtTCPHPAcks                 49                 0.0

IpExtInBcastPkts                89                 0.0

IpExtInOctets                   94650              0.0

IpExtOutOctets                  31140              0.0

IpExtInBcastOctets              26820              0.0

[1]+  Done                    tcpdump -n -ttt -i any port 8080

[Jan Wang]

And I am running packetdrill in a VM not on a physical machine.

[Neal Cardwell]

Hi,

It looks like there is something particular about your environment
that is causing packetdrill not to work for your tests. You might
check for firewall rules with "iptables -L". Or you could try
different IP ranges by adding packetdrill command line options like "
--local_ip=10.0.0.1 --gateway_ip=10.0.0.2 --netmask_ip=255.255.255.0
--remote_ip=10.1.0.3".

Other than that, I'm sorry, but I'm afraid I don't have time to dig
further. If this is blocking you, and you just want to run packetdrill
tests inside some VM environment, I can confirm that packetdrill works
with Ubuntu or Debian images on Google Cloud (and you can get a free
Google Cloud trial account - https://cloud.google.com/free/ ).

cheers,
neal

[Jan Wang]

Thanks Neal.

You are right. It works now after I turn off the firewall.

[Neal Cardwell]

That's great to hear! Thanks for the update.

neal

[Jan Wang]

Looks like many test cases cannot work due to some tso/pacing patch(I guess).

After change the script per tcpdump output, it can run.

packetdrill output:

[root@wj packetdrill]# ./packetdrill tests/linux/initial_window/iw10-base-case.pkt

tests/linux/initial_window/iw10-base-case.pkt:15: error handling packet: live packet field ipv4_total_length: expected: 14640 (0x3930) vs actual: 2960 (0xb90)

script packet:  0.200000 P. 1:14601(14600) ack 1

actual packet:  0.200169 . 1:2921(2920) ack 1 win 115

tcpdump output:

 [root@wj packetdrill]# tcpdump -n -i any port 8080

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

05:00:05.196436 IP 192.0.2.1.38628 > 192.168.0.1.webcache: Flags [S], seq 0, win 32792, options [mss 1460,sackOK,nop,nop,nop,wscale 7], length 0

05:00:05.196481 IP 192.168.0.1.webcache > 192.0.2.1.38628: Flags [S.], seq 686260032, ack 1, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

05:00:05.296450 IP 192.0.2.1.38628 > 192.168.0.1.webcache: Flags [.], ack 1, win 257, length 0

05:00:05.296537 IP 192.168.0.1.webcache > 192.0.2.1.38628: Flags [.], seq 1:2921, ack 1, win 115, length 2920

05:00:05.296550 IP 192.168.0.1.webcache > 192.0.2.1.38628: Flags [.], seq 2921:5841, ack 1, win 115, length 2920

05:00:05.296560 IP 192.168.0.1.webcache > 192.0.2.1.38628: Flags [.], seq 5841:8761, ack 1, win 115, length 2920

05:00:05.296569 IP 192.168.0.1.webcache > 192.0.2.1.38628: Flags [.], seq 8761:11681, ack 1, win 115, length 2920

05:00:05.296581 IP 192.168.0.1.webcache > 192.0.2.1.38628: Flags [P.], seq 11681:14601, ack 1, win 115, length 2920

[Neal Cardwell]

Yes, the Linux TCP kernel has changed a lot over the years, and I'm
afraid our team has not had the cycles to maintain the public versions
of this set of packetdrill tests for the Linux kernel.

neal