Q on local and remote tests

[Ravi Kerur]

Hello,

I started using packetdrill on Ubuntu 5.15 and running into issues on both local and remote tests.

Client:

sudo ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:dc:f1:1b brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    inet 10.4.0.34/16 metric 100 brd 10.4.255.255 scope global dynamic ens160
       valid_lft 24198sec preferred_lft 24198sec
    inet6 2620:130:800a:19:20c:29ff:fedc:f11b/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591959sec preferred_lft 604759sec
    inet6 fe80::20c:29ff:fedc:f11b/64 scope link
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:dc:f1:25 brd ff:ff:ff:ff:ff:ff
    altname enp11s0
    inet 10.4.99.105/24 brd 10.4.99.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fedc:f125/64 scope link
       valid_lft forever preferred_lft forever
4: tun0: <NO-CARRIER,POINTOPOINT,MULTICAST,NOARP,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 500
    link/none

Server:

sudo ip addr list
[sudo] password for rkerur:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:9d:b4:0f brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    inet 10.4.104.104/16 brd 10.4.255.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 2620:130:800a:19:250:56ff:fe9d:b40f/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591922sec preferred_lft 604722sec
    inet6 fe80::250:56ff:fe9d:b40f/64 scope link
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:9d:42:28 brd ff:ff:ff:ff:ff:ff
    altname enp11s0
    inet 10.4.99.103/24 brd 10.4.99.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe9d:4228/64 scope link
       valid_lft forever preferred_lft forever

I run local tests as follows

sudo ./packetdrill/run_all.py -S -v -L -l tcp/

...

KILL [/home/rkerur/packetdrill/gtests/net/tcp/zerocopy/small.pkt (ipv4)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill/gtests/net/tcp/zerocopy/small.pkt (ipv4-mapped-v6)]
stdout:
stderr:
KILL [/home/rkerur/packetdrill/gtests/net/tcp/zerocopy/small.pkt (ipv6)]
stdout:
stderr:
Ran  463 tests:    0 passing,    0 failing,  463 timed out (378.99 sec): tcp/

I run remote tests as follows

Server:

./packetdrill/packetdrill -v --wire_server --wire_server_dev=ens192

Client:

./packetdrill/packetdrill -v --wire_client --wire_client_dev=ens192 --wire_server_ip=10.4.99.103 --local_ip=192.168.0.1 --gateway_ip=192.168.0.1 --netmask_ip=255.255.0.0 --remote_ip=192.0.2.1 ./tcp/fast_retransmit/fr-4pkt-fack-last-byte.pkt

1. Is there a way to run all tests f.e. "*.pkt" ?

2. For the above test, error logs are as follows

Server:

./packetdrill/packetdrill -v --wire_server --wire_server_dev=ens192
please specify --wire_client_dev=<eth_dev_name>

Client:

/packetdrill/packetdrill -v --wire_client --wire_client_dev=ens192 --wire_server_ip=10.4.99.103 --local_ip=192.168.0.1 --gateway_ip=192.168.0.1 --netmask_ip=255.255.0.0 --remote_ip=192.0.2.1 ./tcp/fast_retransmit/fr-4pkt-fack-last-byte.pkt
remote side closed connection
error in TCP connection to wire server: error reading WIRE_SERVER_READY

Anything in the configuration I need to make changes for this to work?

Thanks,

Ravi

[Neal Cardwell]

Yes, *pkt should work, if you want to test all the scripts in a directory. If you want to run all the tests in a directory hierarchy you can try running your own shell script or modifying packetdrill/run_all.py in the packetdrill source distribution (right now IIC packetdrill/run_all.py only works for local-mode testing).

 

2. For the above test, error logs are as follows

Server:

./packetdrill/packetdrill -v --wire_server --wire_server_dev=ens192
please specify --wire_client_dev=<eth_dev_name>

Client:

/packetdrill/packetdrill -v --wire_client --wire_client_dev=ens192 --wire_server_ip=10.4.99.103 --local_ip=192.168.0.1 --gateway_ip=192.168.0.1 --netmask_ip=255.255.0.0 --remote_ip=192.0.2.1 ./tcp/fast_retransmit/fr-4pkt-fack-last-byte.pkt
remote side closed connection
error in TCP connection to wire server: error reading WIRE_SERVER_READY

Anything in the configuration I need to make changes for this to work?

Can you please fetch and build the latest packetdrill code ( https://github.com/google/packetdrill/blob/master/README.md#how-to-get-started-with-packetdrill ) and then try the remote mode recipe in the README (  https://github.com/google/packetdrill/blob/master/README.md#remote-mode ), and share the output? The remote mode usage has recently changed, and is much simpler, so hopefully that will simplify your test scenario.

Thanks!

neal

 

Thanks,

Ravi

--
You received this message because you are subscribed to the Google Groups "packetdrill" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packetdrill...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/9ca299f5-52e4-41f9-bfda-b128fcbf6339n%40googlegroups.com.

[Ravi Kerur]

I was using the latest packetdrill code I think. Still the same behavior when I use --wire_server_at=<ip>. Server side is closing the connection.

Attached client, server debug outputs and pcap on client side.

I am not sure whether the test actually succeeded or not.

Thanks,

Ravi

[Ravi Kerur]

For local tests I ran run_all.py with -s option and it starts showing the results.

sudo ./packetdrill/run_all.py -S -v -L -l -s tcp/

OK   [/home/rkerur/packetdrill-latest/packetdrill/gtests/net/tcp/splice/tcp_splice_loop_test.pkt (ipv4)]
stdout:
stderr:
OK   [/home/rkerur/packetdrill-latest/packetdrill/gtests/net/tcp/splice/tcp_splice_loop_test.pkt (ipv6)]
stdout:
stderr:
OK   [/home/rkerur/packetdrill-latest/packetdrill/gtests/net/tcp/splice/tcp_splice_loop_test.pkt (ipv4-mapped-v6)]
stdout:
stderr:
Ran    3 tests:    3 passing,    0 failing,    0 timed out (2.64 sec): tcp/splice

These test cases basically tests tcp rfc compliance?

[Ravi Kerur]

Please let me know if there is a workaround for remote tests. With latest packetdrill same issue is seen

Based on README If I run server

sudo ./packetdrill --wire_server --debug   I get following errors

please specify --wire_server_dev=<eth_dev_name>

Server:

sudo ./packetdrill --wire_server --wire_server_dev=ens192 --debug
config.c 424] finalize_config: config->live_local_ip_string: []
wire_conn.c 43] wire_conn_new
wire_conn.c 119] wire_conn_bind_listen
wire_conn.c 138] wire_conn_accept
wire_conn.c 144] accepted fd 5
wire_conn.c 43] wire_conn_new
wire_conn.c 72] set_default_tcp_options fd 5
link_layer.c 46] get_hw_address for device ens192
link_layer.c 60] ens192 HWaddr: 00:50:56:9d:42:28
wire_server.c 539] start_wire_server_thread
wire_conn.c 138] wire_conn_accept
wire_server.c 471] wire_server_thread
config.c 273] set_default_config
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_COMMAND_LINE_ARGS
wire_server.c 101] argc = 8
wire_server.c 117] argv[0] = './packetdrill'
wire_server.c 117] argv[1] = '--wire_server_at=10.4.99.103'
wire_server.c 117] argv[2] = '--debug'
wire_server.c 117] argv[3] = '../tcp/blocking/blocking-accept.pkt'
wire_server.c 117] argv[4] = '../tcp/blocking/blocking-connect.pkt'
wire_server.c 117] argv[5] = '../tcp/blocking/blocking-read.pkt'
wire_server.c 117] argv[6] = '../tcp/blocking/blocking-write.pkt'
wire_server.c 117] argv[7] = '--wire_server'
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT_PATH
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_HARDWARE_ADDR
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_IP_ADDR
wire_server.c 233] got WIRE_IP_ADDR: [127.0.1.1]
run.c 692] parse_and_run_script: ../tcp/blocking/blocking-accept.pkt
parser.y 157] copy_script
parser.y 164] copy_script: 472 bytes
config.c 703] parse_and_finalize_config
config.c 642] parse_command_line_options argc=8
config.c 644] argv[0] = './packetdrill'
config.c 644] argv[1] = '--wire_server_at=10.4.99.103'
config.c 644] argv[2] = '--debug'
config.c 644] argv[3] = '../tcp/blocking/blocking-accept.pkt'
config.c 644] argv[4] = '../tcp/blocking/blocking-connect.pkt'
config.c 644] argv[5] = '../tcp/blocking/blocking-read.pkt'
config.c 644] argv[6] = '../tcp/blocking/blocking-write.pkt'
config.c 644] argv[7] = '--wire_server'
config.c 477] process_option 275 ('') = 10.4.99.103
config.c 477] process_option 288 (' ') = (null)
config.c 477] process_option 273 ('') = (null)
please specify --wire_client_dev=<eth_dev_name>

Client:

sudo ./packetdrill --wire_server_at=10.4.99.103 ../tcp/blocking/*pkt --debug
config.c 273] set_default_config
run.c 692] parse_and_run_script: ../tcp/blocking/blocking-accept.pkt
parser.y 172] read_script(../tcp/blocking/blocking-accept.pkt)
parser.y 215] read_script: 472 bytes
config.c 703] parse_and_finalize_config
config.c 642] parse_command_line_options argc=7
config.c 644] argv[0] = './packetdrill'
config.c 644] argv[1] = '--wire_server_at=10.4.99.103'
config.c 644] argv[2] = '--debug'
config.c 644] argv[3] = '../tcp/blocking/blocking-accept.pkt'
config.c 644] argv[4] = '../tcp/blocking/blocking-connect.pkt'
config.c 644] argv[5] = '../tcp/blocking/blocking-read.pkt'
config.c 644] argv[6] = '../tcp/blocking/blocking-write.pkt'
config.c 477] process_option 275 ('') = 10.4.99.103
config.c 477] process_option 288 (' ') = (null)
net_utils.c 220] using primary device: ens192
config.c 256] defaulting live_local_ip_string to: 127.0.1.1
config.c 424] finalize_config: config->live_local_ip_string: [127.0.1.1]
run.c 563] run_script: running script
wire_client_netdev.c 110] wire_client_netdev_new
net_utils.c 163] net_setup_dev_address: found: 1
net_utils.c 166] net_setup_dev_address: found on correct device
wire_client.c 259] wire_client_init
link_layer.c 46] get_hw_address for device ens192
run_system_call.c 3501] syscall thread: starting and locking
link_layer.c 60] ens192 HWaddr: 00:0c:29:dc:f1:25
wire_conn.c 43] wire_conn_new
wire_conn.c 99] wire_conn_connect
wire_conn.c 72] set_default_tcp_options fd 4
wire_conn.c 180] wire_conn_write -> op: WIRE_COMMAND_LINE_ARGS
wire_conn.c 180] wire_conn_write -> op: WIRE_SCRIPT_PATH
wire_conn.c 180] wire_conn_write -> op: WIRE_SCRIPT
wire_conn.c 180] wire_conn_write -> op: WIRE_HARDWARE_ADDR
wire_conn.c 180] wire_conn_write -> op: WIRE_IP_ADDR
wire_conn.c 224] wire_conn_read

remote side closed connection
error in TCP connection to wire server: error reading WIRE_SERVER_READY

On Tuesday, August 1, 2023 at 2:00:57 PM UTC-7 ncar...@google.com wrote:

[Neal Cardwell]

Thanks for the detailed bug report! That was a big help. I think I see at least one bug that was causing problems, and I have pushed a fix to the packetdrill repo at https://github.com/google/packetdrill ... the fix is here:

  https://github.com/google/packetdrill/commit/40894edb1d3ce279ecda8914aaab66aa7000a634

Can you please do a "git fetch packetdrill" or "git fetch origin" (depending on what the remote packetdrill repo is called in your git client) to fetch the latest code,  then do a "cd gtests/net/packetdrill/; make clean ; make" to build the fix, and then re-test with the new packetdrill binary? If it fails, can you please post the client- and server-side debug output as before?

Thanks!

neal

To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/55774eaf-0584-4e05-8d2d-e828a03c1810n%40googlegroups.com.

[Ravi Kerur]

With the latest changes, server SIGSEGV's

sudo gdb ./packetdrill

(gdb) run --wire_server --wire_server_dev=ens192  --debug
Starting program: /home/rkerur/packetdrill-latest/packetdrill/gtests/net/packetdrill/packetdrill --wire_server --wire_server_dev=ens192  --debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

config.c 424] finalize_config: config->live_local_ip_string: []

[Detaching after vfork from child process 122800]

wire_conn.c 43] wire_conn_new
wire_conn.c 119] wire_conn_bind_listen
wire_conn.c 138] wire_conn_accept
wire_conn.c 144] accepted fd 5
wire_conn.c 43] wire_conn_new
wire_conn.c 72] set_default_tcp_options fd 5
link_layer.c 46] get_hw_address for device ens192
link_layer.c 60] ens192 HWaddr: 00:50:56:9d:42:28

wire_server.c 543] start_wire_server_thread
[New Thread 0x7ffff7ff8640 (LWP 122804)]
wire_conn.c 138] wire_conn_accept
wire_server.c 475] wire_server_thread

config.c 273] set_default_config
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_COMMAND_LINE_ARGS

wire_server.c 102] argc = 9
wire_server.c 121] argv[0] = './packetdrill'
wire_server.c 121] argv[1] = '(null)'
wire_server.c 121] argv[2] = '(null)'
wire_server.c 121] argv[3] = '(null)'
wire_server.c 121] argv[4] = '(null)'
wire_server.c 121] argv[5] = '(null)'
wire_server.c 121] argv[6] = '(null)'
wire_server.c 121] argv[7] = '(null)'
wire_server.c 121] argv[8] = '--wire_server'

wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT_PATH
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_HARDWARE_ADDR
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_IP_ADDR

wire_server.c 237] got WIRE_IP_ADDR: [127.0.1.1]

run.c 692] parse_and_run_script: ../tcp/blocking/blocking-accept.pkt
parser.y 157] copy_script
parser.y 164] copy_script: 472 bytes
config.c 703] parse_and_finalize_config

config.c 642] parse_command_line_options argc=9

config.c 644] argv[0] = './packetdrill'

config.c 644] argv[1] = '(null)'
config.c 644] argv[2] = '(null)'
config.c 644] argv[3] = '(null)'
config.c 644] argv[4] = '(null)'
config.c 644] argv[5] = '(null)'
config.c 644] argv[6] = '(null)'
config.c 644] argv[7] = '(null)'
config.c 644] argv[8] = '--wire_server'

Thread 2 "packetdrill" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7ff8640 (LWP 122804)]
0x00000000004a4900 in _getopt_internal_r ()
(gdb) bt
#0  0x00000000004a4900 in _getopt_internal_r ()
#1  0x00000000004a4c76 in _getopt_internal ()
#2  0x00000000004a4da2 in getopt_long ()
#3  0x0000000000405e62 in parse_command_line_options (argc=9, argv=0x7ffff0000d30, config=0x577c98) at config.c:657
#4  0x00000000004060cd in parse_and_finalize_config (invocation=0x7ffff7ff8160) at config.c:710
#5  0x000000000042aec9 in yyparse () at /home/rkerur/packetdrill-latest/packetdrill/gtests/net/packetdrill/parser.y:609
#6  0x0000000000429ec4 in parse_script (config=0x577c98, script=0x577f38, callback_invocation=0x7ffff7ff8160) at /home/rkerur/packetdrill-latest/packetdrill/gtests/net/packetdrill/parser.y:255
#7  0x00000000004137e1 in parse_script_and_set_config (argc=9, argv=0x7ffff0000d30, config=0x577c98, script=0x577f38, script_path=0x7ffff0000e40 "../tcp/blocking/blocking-accept.pkt",
    script_buffer=0x7ffff0001230 "// Test for blocking accept.\n\n`../common/defaults.sh`\n\n// Establish a connection.\n    0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n   +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n   +0 bind(3"...) at run.c:704
#8  0x00000000004314f0 in wire_server_thread (arg=0x577c80) at wire_server.c:494
#9  0x000000000045cd1b in start_thread ()
#10 0x00000000004ff3f0 in clone3 ()
(gdb)

Client:

sudo ./packetdrill --wire_server_at=10.4.99.103 ../tcp/blocking/*.pkt --debug

config.c 273] set_default_config
run.c 692] parse_and_run_script: ../tcp/blocking/blocking-accept.pkt
parser.y 172] read_script(../tcp/blocking/blocking-accept.pkt)
parser.y 215] read_script: 472 bytes
config.c 703] parse_and_finalize_config

config.c 642] parse_command_line_options argc=8
config.c 644] argv[0] = './packetdrill'
config.c 644] argv[1] = '--wire_server_at=10.4.99.103'

config.c 644] argv[2] = '--wire_client_dev=ens192'
config.c 644] argv[3] = '--debug'
config.c 644] argv[4] = '../tcp/blocking/blocking-accept.pkt'
config.c 644] argv[5] = '../tcp/blocking/blocking-connect.pkt'
config.c 644] argv[6] = '../tcp/blocking/blocking-read.pkt'
config.c 644] argv[7] = '../tcp/blocking/blocking-write.pkt'

config.c 477] process_option 275 ('') = 10.4.99.103

config.c 477] process_option 277 ('') = ens192

config.c 477] process_option 288 (' ') = (null)

config.c 256] defaulting live_local_ip_string to: 127.0.1.1
config.c 424] finalize_config: config->live_local_ip_string: [127.0.1.1]
run.c 563] run_script: running script
wire_client_netdev.c 110] wire_client_netdev_new
net_utils.c 163] net_setup_dev_address: found: 1
net_utils.c 166] net_setup_dev_address: found on correct device

wire_client.c 255] wire_client_init

link_layer.c 46] get_hw_address for device ens192
run_system_call.c 3501] syscall thread: starting and locking
link_layer.c 60] ens192 HWaddr: 00:0c:29:dc:f1:25
wire_conn.c 43] wire_conn_new
wire_conn.c 99] wire_conn_connect
wire_conn.c 72] set_default_tcp_options fd 4
wire_conn.c 180] wire_conn_write -> op: WIRE_COMMAND_LINE_ARGS
wire_conn.c 180] wire_conn_write -> op: WIRE_SCRIPT_PATH
wire_conn.c 180] wire_conn_write -> op: WIRE_SCRIPT
wire_conn.c 180] wire_conn_write -> op: WIRE_HARDWARE_ADDR
wire_conn.c 180] wire_conn_write -> op: WIRE_IP_ADDR
wire_conn.c 224] wire_conn_read

[Ravi Kerur]

Looks like remote mode is completely broken on latest. With the following diffs I managed to fix the crash. Now, client side complains about tun0 device not found. Why in remote mode tun0 device is needed?

Diffs:

diff --git a/gtests/net/packetdrill/wire_server.c b/gtests/net/packetdrill/wire_server.c
index 8377c4f..87bf702 100644
--- a/gtests/net/packetdrill/wire_server.c
+++ b/gtests/net/packetdrill/wire_server.c
@@ -89,7 +89,7 @@ static void wire_server_free(struct wire_server *wire_server)
 static void wire_server_unserialize_argv(struct wire_server *wire_server,
                                         const char *args, int args_len)
 {
-       int argc, i;
+       int argc, i, j;
        char **argv = NULL;
        const char *end = NULL;
 
@@ -108,19 +108,22 @@ static void wire_server_unserialize_argv(struct wire_server *wire_server,
        argv = calloc(argc + 1, sizeof(char *));
 
        end = args;
-       for (i = 0; i < argc; ++i) {
+       for (i = 0, j = 0; i < argc; ++i) {
                if (strstr(end, "-wire_client") ||
-                   strstr(end, "-wire_server_at"))
+                   strstr(end, "-wire_server_at")) {
+                       end += strlen(end) + 1; /* + 1 for '\0' */
                        continue;       /* don't use client-only arguments */
-               argv[i] = strdup(end);
+               }
+               argv[j++] = strdup(end);
                end += strlen(end) + 1; /* + 1 for '\0' */
        }
-       asprintf(&argv[argc-1], "--wire_server");
+       asprintf(&argv[j], "--wire_server");
 
-       for (i = 0; i < argc; ++i)
+       for (i = 0; i < j; ++i)
                DEBUGP("argv[%d] = '%s'\n", i, argv[i]);
 
-       wire_server->argc = argc;
+       //wire_server->argc = argc;
+       wire_server->argc = j;
        wire_server->argv = argv;
 }

 

Client debug logs:

sudo ./packetdrill --wire_server_at=10.4.99.103 --wire_client_dev=ens192 ../tcp/blocking/blocking-accept.pkt --debug

config.c 273] set_default_config
run.c 692] parse_and_run_script: ../tcp/blocking/blocking-accept.pkt
parser.y 172] read_script(../tcp/blocking/blocking-accept.pkt)
parser.y 215] read_script: 472 bytes
config.c 703] parse_and_finalize_config

config.c 642] parse_command_line_options argc=5

config.c 644] argv[0] = './packetdrill'
config.c 644] argv[1] = '--wire_server_at=10.4.99.103'
config.c 644] argv[2] = '--wire_client_dev=ens192'
config.c 644] argv[3] = '--debug'
config.c 644] argv[4] = '../tcp/blocking/blocking-accept.pkt'

config.c 477] process_option 275 ('') = 10.4.99.103
config.c 477] process_option 277 ('') = ens192
config.c 477] process_option 288 (' ') = (null)
config.c 256] defaulting live_local_ip_string to: 127.0.1.1
config.c 424] finalize_config: config->live_local_ip_string: [127.0.1.1]
run.c 563] run_script: running script
wire_client_netdev.c 110] wire_client_netdev_new
net_utils.c 163] net_setup_dev_address: found: 1
net_utils.c 166] net_setup_dev_address: found on correct device

wire_client.c 259] wire_client_init
link_layer.c 46] get_hw_address for device ens192
run_system_call.c 3501] syscall thread: starting and locking
link_layer.c 60] ens192 HWaddr: 00:0c:29:dc:f1:25
wire_conn.c 43] wire_conn_new
wire_conn.c 99] wire_conn_connect
wire_conn.c 72] set_default_tcp_options fd 4
wire_conn.c 180] wire_conn_write -> op: WIRE_COMMAND_LINE_ARGS
wire_conn.c 180] wire_conn_write -> op: WIRE_SCRIPT_PATH
wire_conn.c 180] wire_conn_write -> op: WIRE_SCRIPT
wire_conn.c 180] wire_conn_write -> op: WIRE_HARDWARE_ADDR
wire_conn.c 180] wire_conn_write -> op: WIRE_IP_ADDR
wire_conn.c 224] wire_conn_read

wire_conn.c 233] wire_conn_read -> op: WIRE_SERVER_READY
Cannot find device "tun0"
../tcp/blocking/blocking-accept.pkt: error executing init command: non-zero status 1

Server debug logs:

sudo ./packetdrill --wire_server --wire_server_dev=ens192 --debug
config.c 424] finalize_config: config->live_local_ip_string: []
wire_conn.c 43] wire_conn_new
wire_conn.c 119] wire_conn_bind_listen
wire_conn.c 138] wire_conn_accept
wire_conn.c 144] accepted fd 5
wire_conn.c 43] wire_conn_new
wire_conn.c 72] set_default_tcp_options fd 5
link_layer.c 46] get_hw_address for device ens192
link_layer.c 60] ens192 HWaddr: 00:50:56:9d:42:28

wire_server.c 546] start_wire_server_thread
wire_conn.c 138] wire_conn_accept
wire_server.c 478] wire_server_thread

config.c 273] set_default_config
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_COMMAND_LINE_ARGS

wire_server.c 102] argc = 5
wire_server.c 123] argv[0] = './packetdrill'
wire_server.c 123] argv[1] = '--debug'
wire_server.c 123] argv[2] = '../tcp/blocking/blocking-accept.pkt'
wire_server.c 123] argv[3] = ''

wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT_PATH
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_HARDWARE_ADDR
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_IP_ADDR

wire_server.c 240] got WIRE_IP_ADDR: [127.0.1.1]

run.c 692] parse_and_run_script: ../tcp/blocking/blocking-accept.pkt
parser.y 157] copy_script
parser.y 164] copy_script: 472 bytes
config.c 703] parse_and_finalize_config

config.c 642] parse_command_line_options argc=4

config.c 644] argv[0] = './packetdrill'

config.c 644] argv[1] = '--debug'
config.c 644] argv[2] = '../tcp/blocking/blocking-accept.pkt'
config.c 644] argv[3] = ''

config.c 477] process_option 288 (' ') = (null)

config.c 424] finalize_config: config->live_local_ip_string: [127.0.1.1]

wire_server.c 505] wire_server_thread: live_local_ip_string: [127.0.1.1]
wire_server_netdev.c 192] wire_server_netdev_new
net_utils.c 163] net_setup_dev_address: found: 0
net_utils.c 38] running: 'ip addr add 192.168.0.1/16 dev ens192 > /dev/null 2>&1'
net_utils.c 40] result: 0
packet_socket_linux.c 93] device index: ens192 -> 3
packet_socket_linux.c 181] packet_socket_set_filter: ip: 127.0.1.1 src: 8080 dst: 8080
packet_socket_linux.c 213] filter constants:
packet_socket_linux.c 215] { 0x28,   0,   0, 0x0000000c },
packet_socket_linux.c 215] { 0x15,   0,  15, 0x00000800 },
packet_socket_linux.c 215] { 0x20,   0,   0, 0x0000001a },
packet_socket_linux.c 215] { 0x15,   0,  13, 0x7f000101 },
packet_socket_linux.c 215] { 0x30,   0,   0, 0x00000017 },
packet_socket_linux.c 215] { 0x15,   2,   0, 0x00000084 },
packet_socket_linux.c 215] { 0x15,   1,   0, 0x00000006 },
packet_socket_linux.c 215] { 0x15,   0,   7, 0x00000011 },
packet_socket_linux.c 215] { 0x28,   0,   0, 0x00000014 },
packet_socket_linux.c 215] { 0x45,   7,   0, 0x00001fff },
packet_socket_linux.c 215] { 0xb1,   0,   0, 0x0000000e },
packet_socket_linux.c 215] { 0x48,   0,   0, 0x0000000e },
packet_socket_linux.c 215] { 0x15,   3,   0, 0x00001f90 },
packet_socket_linux.c 215] { 0x48,   0,   0, 0x00000010 },
packet_socket_linux.c 215] { 0x15,   1,   2, 0x00001f90 },
packet_socket_linux.c 215] { 0x15,   0,   1, 0x00000001 },
packet_socket_linux.c 215] { 0x06,   0,   0, 0x00040000 },
packet_socket_linux.c 215] { 0x06,   0,   0, 0x00000000 },
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        
DROP       udp  --  127.0.1.1            0.0.0.0/0            udp spt:8080
DROP       udp  --  127.0.1.1            0.0.0.0/0            udp dpt:8080
DROP       tcp  --  127.0.1.1            0.0.0.0/0            tcp spt:8080
DROP       tcp  --  127.0.1.1            0.0.0.0/0            tcp dpt:8080

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        
wire_conn.c 180] wire_conn_write -> op: WIRE_SERVER_READY

run_system_call.c 3501] syscall thread: starting and locking

wire_conn.c 224] wire_conn_read
remote side closed connection

run_system_call.c 3621] main thread: signaling syscall thread to exit
run_system_call.c 3628] main thread: unlocking, waiting for syscall thread exit
run_system_call.c 3509] syscall thread: in state 4
run_system_call.c 3584] syscall thread: unlocking and exiting
run_system_call.c 3632] main thread: joined syscall thread; relocking
wire_server_netdev.c 250] wire_server_netdev_free
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        
net_utils.c 38] running: 'ip addr del 192.168.0.1/16 dev ens192 > /dev/null 2>&1'
net_utils.c 40] result: 0
wire_server.c 539] wire_server_thread: connection is done
^C

[Ravi Kerur]

Any workaround for remote tests failure?

Thanks.

[Neal Cardwell]

Hi,

Sorry about the issues. I have reverted the buggy 40894edb1d3c commit.

 I have pushed a second attempt at a bug fix to the packetdrill repo at: https://github.com/google/packetdrill 

Can you please do a "git fetch packetdrill" or "git fetch origin" (depending on what the remote packetdrill repo is called in your git client) to fetch the latest code,  then do a "cd gtests/net/packetdrill/; make clean ; make" to build the fix, and then re-test with the new packetdrill binary? If it fails, can you please post the client- and server-side debug output as before?

I would bet that the  complaint about the tun0 device not being found is because you are running scripts that are designed to be run in local mode, and the initialization script that those tests run (  gtests/net/common/defaults.sh ) features a line that references tun0, which is present in local mode:

  tc qdisc add dev tun0 root pfifo

best regards,

neal

To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/94f3d36c-6629-44d0-b73f-473179dea763n%40googlegroups.com.

[Ravi Kerur]

Yes, basically I created a tun0 to make some progress. It still doesn't work.

Errors shown below, it is with the latest code

*client*

sudo ./packetdrill --wire_server_at=10.20.21.6 --wire_client_dev=ens5 ../tcp/blocking/blocking-accept.pkt --debug

config.c 273] set_default_config
run.c 692] parse_and_run_script: ../tcp/blocking/blocking-accept.pkt
parser.y 172] read_script(../tcp/blocking/blocking-accept.pkt)
parser.y 215] read_script: 472 bytes

config.c 705] parse_and_finalize_config
config.c 644] parse_command_line_options argc=5
config.c 646] argv[0] = './packetdrill'
config.c 646] argv[1] = '--wire_server_at=10.20.21.6'
config.c 646] argv[2] = '--wire_client_dev=ens5'
config.c 646] argv[3] = '--debug'
config.c 646] argv[4] = '../tcp/blocking/blocking-accept.pkt'
config.c 477] process_option 275 ('') = 10.20.21.6
config.c 477] process_option 277 ('') = ens5

config.c 477] process_option 288 (' ') = (null)

config.c 256] defaulting live_local_ip_string to: 10.20.21.2
config.c 424] finalize_config: config->live_local_ip_string: [10.20.21.2]

run.c 563] run_script: running script
wire_client_netdev.c 110] wire_client_netdev_new
net_utils.c 163] net_setup_dev_address: found: 1
net_utils.c 166] net_setup_dev_address: found on correct device
wire_client.c 259] wire_client_init

link_layer.c 46] get_hw_address for device ens5
link_layer.c 60] ens5 HWaddr: 42:01:0a:14:15:02

wire_conn.c 43] wire_conn_new
wire_conn.c 99] wire_conn_connect
wire_conn.c 72] set_default_tcp_options fd 4

run_system_call.c 3501] syscall thread: starting and locking

wire_conn.c 180] wire_conn_write -> op: WIRE_COMMAND_LINE_ARGS
wire_conn.c 180] wire_conn_write -> op: WIRE_SCRIPT_PATH
wire_conn.c 180] wire_conn_write -> op: WIRE_SCRIPT
wire_conn.c 180] wire_conn_write -> op: WIRE_HARDWARE_ADDR
wire_conn.c 180] wire_conn_write -> op: WIRE_IP_ADDR
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SERVER_READY

Error: Exclusivity flag on, cannot modify.
../tcp/blocking/blocking-accept.pkt: error executing init command: non-zero status 2

*Server*

sudo ./packetdrill --wire_server --wire_server_dev=ens5 --debug

config.c 424] finalize_config: config->live_local_ip_string: []
wire_conn.c 43] wire_conn_new
wire_conn.c 119] wire_conn_bind_listen
wire_conn.c 138] wire_conn_accept
wire_conn.c 144] accepted fd 5
wire_conn.c 43] wire_conn_new
wire_conn.c 72] set_default_tcp_options fd 5

link_layer.c 46] get_hw_address for device ens5
link_layer.c 60] ens5 HWaddr: 42:01:0a:14:15:06

wire_server.c 539] start_wire_server_thread
wire_conn.c 138] wire_conn_accept
wire_server.c 471] wire_server_thread
config.c 273] set_default_config
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_COMMAND_LINE_ARGS

wire_server.c 101] argc = 5

wire_server.c 117] argv[0] = './packetdrill'

wire_server.c 117] argv[1] = '--wire_server_at=10.20.21.6'

wire_server.c 117] argv[2] = '--debug'
wire_server.c 117] argv[3] = '../tcp/blocking/blocking-accept.pkt'

wire_server.c 117] argv[4] = '--wire_server'

wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT_PATH
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_HARDWARE_ADDR
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_IP_ADDR

wire_server.c 233] got WIRE_IP_ADDR: [10.20.21.2]

run.c 692] parse_and_run_script: ../tcp/blocking/blocking-accept.pkt
parser.y 157] copy_script
parser.y 164] copy_script: 472 bytes

config.c 705] parse_and_finalize_config
config.c 644] parse_command_line_options argc=5
config.c 646] argv[0] = './packetdrill'
config.c 646] argv[1] = '--wire_server_at=10.20.21.6'
config.c 646] argv[2] = '--debug'
config.c 646] argv[3] = '../tcp/blocking/blocking-accept.pkt'
config.c 646] argv[4] = '--wire_server'
config.c 477] process_option 275 ('') = 10.20.21.6

config.c 477] process_option 288 (' ') = (null)
config.c 477] process_option 273 ('') = (null)

net_utils.c 220] using primary device: ens5
config.c 424] finalize_config: config->live_local_ip_string: [10.20.21.2]
wire_server.c 498] wire_server_thread: live_local_ip_string: [10.20.21.2]

wire_server_netdev.c 192] wire_server_netdev_new
net_utils.c 163] net_setup_dev_address: found: 0

net_utils.c 38] running: 'ip addr add 192.168.0.1/16 dev ens5 > /dev/null 2>&1'
net_utils.c 40] result: 0
packet_socket_linux.c 93] device index: ens5 -> 3
packet_socket_linux.c 181] packet_socket_set_filter: ip: 10.20.21.2 src: 8080 dst: 8080

packet_socket_linux.c 213] filter constants:
packet_socket_linux.c 215] { 0x28,   0,   0, 0x0000000c },
packet_socket_linux.c 215] { 0x15,   0,  15, 0x00000800 },
packet_socket_linux.c 215] { 0x20,   0,   0, 0x0000001a },

packet_socket_linux.c 215] { 0x15,   0,  13, 0x0a141502 },

packet_socket_linux.c 215] { 0x30,   0,   0, 0x00000017 },
packet_socket_linux.c 215] { 0x15,   2,   0, 0x00000084 },
packet_socket_linux.c 215] { 0x15,   1,   0, 0x00000006 },
packet_socket_linux.c 215] { 0x15,   0,   7, 0x00000011 },
packet_socket_linux.c 215] { 0x28,   0,   0, 0x00000014 },
packet_socket_linux.c 215] { 0x45,   7,   0, 0x00001fff },
packet_socket_linux.c 215] { 0xb1,   0,   0, 0x0000000e },
packet_socket_linux.c 215] { 0x48,   0,   0, 0x0000000e },
packet_socket_linux.c 215] { 0x15,   3,   0, 0x00001f90 },
packet_socket_linux.c 215] { 0x48,   0,   0, 0x00000010 },
packet_socket_linux.c 215] { 0x15,   1,   2, 0x00001f90 },
packet_socket_linux.c 215] { 0x15,   0,   1, 0x00000001 },
packet_socket_linux.c 215] { 0x06,   0,   0, 0x00040000 },
packet_socket_linux.c 215] { 0x06,   0,   0, 0x00000000 },
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

DROP       udp  --  10.20.21.2           0.0.0.0/0            udp spt:8080
DROP       udp  --  10.20.21.2           0.0.0.0/0            udp dpt:8080
DROP       tcp  --  10.20.21.2           0.0.0.0/0            tcp spt:8080
DROP       tcp  --  10.20.21.2           0.0.0.0/0            tcp dpt:8080

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        
wire_conn.c 180] wire_conn_write -> op: WIRE_SERVER_READY
run_system_call.c 3501] syscall thread: starting and locking
wire_conn.c 224] wire_conn_read
remote side closed connection
run_system_call.c 3621] main thread: signaling syscall thread to exit
run_system_call.c 3628] main thread: unlocking, waiting for syscall thread exit
run_system_call.c 3509] syscall thread: in state 4
run_system_call.c 3584] syscall thread: unlocking and exiting
run_system_call.c 3632] main thread: joined syscall thread; relocking
wire_server_netdev.c 250] wire_server_netdev_free
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        

net_utils.c 38] running: 'ip addr del 192.168.0.1/16 dev ens5 > /dev/null 2>&1'
net_utils.c 40] result: 0
wire_server.c 532] wire_server_thread: connection is done

[Ravi Kerur]

Figured out why I am seeing

Error: Exclusivity flag on, cannot modify.

Still it isn't clear to me why tun0 interface is needed for remote testing. With pfifo it's still timing out, debug logs shown below

Client logs

run.c 610] live_start_time_usecs is 1691711053009468
wire_conn.c 180] wire_conn_write -> op: WIRE_CLIENT_STARTING
run.c 390] gettimeofday: 1691711053.009265
run_system_call.c 3480] 6: system call: socket
run_system_call.c 3337] 6: invoke call: socket
run.c 350] waiting until 1691711053009468 -- now is 1691711053009278
run_system_call.c 1392] socket() creating new socket: script_fd: 3 live_fd: 5
run.c 390] gettimeofday: 1691711053.009493
run_system_call.c 3480] 7: system call: setsockopt
run_system_call.c 3337] 7: invoke call: setsockopt
run.c 350] waiting until 1691711053009496 -- now is 1691711053009499
run.c 186] expected: 0.000 actual: 0.000  (secs)
run.c 390] gettimeofday: 1691711053.009509
run_system_call.c 3480] 8: system call: bind
run_system_call.c 3337] 8: invoke call: bind
run.c 350] waiting until 1691711053009512 -- now is 1691711053009515
run.c 186] expected: 0.000 actual: 0.000  (secs)
run_system_call.c 1426] run_syscall_bind
run.c 390] gettimeofday: 1691711053.009526
run_system_call.c 3480] 9: system call: listen
run_system_call.c 3337] 9: invoke call: listen
run.c 350] waiting until 1691711053009529 -- now is 1691711053009531
run.c 186] expected: 0.000 actual: 0.000  (secs)
run.c 390] gettimeofday: 1691711053.009538
run_system_call.c 3480] 10: system call: accept
run_system_call.c 3431] main thread: signal enqueued
run_system_call.c 3438] main thread: waiting for dequeued signal; state: 1
run_system_call.c 3509] syscall thread: in state 1
run_system_call.c 3527] syscall thread: invoking syscall
run_system_call.c 3337] 10: invoke call: accept
run.c 350] waiting until 1691711053009540 -- now is 1691711053009568
run.c 186] expected: 0.000 actual: 0.000  (secs)
run_system_call.c 1148] syscall thread: begin_syscall signals dequeued
run_system_call.c 3451] main thread: unlocking and yielding
run_system_call.c 3457] main thread: checking syscall thread state
run_system_call.c 3462] main thread: locking and reading state
run_system_call.c 3467] main thread: continuing after syscall
run.c 390] gettimeofday: 1691711053.009722
wire_conn.c 180] wire_conn_write -> op: WIRE_PACKETS_START
run.c 390] gettimeofday: 1691711053.009736
run.c 390] gettimeofday: 1691711053.009739
run.c 390] gettimeofday: 1691711053.009741
wire_client.c 206] wire_client_receive_packets_done
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_PACKETS_DONE
../tcp/blocking/blocking-accept.pkt:13: error handling packet: Timed out waiting for packet

Server logs

wire_conn.c 144] accepted fd 6
wire_conn.c 43] wire_conn_new
wire_conn.c 72] set_default_tcp_options fd 6

wire_conn.c 233] wire_conn_read -> op: WIRE_CLIENT_STARTING
wire_server.c 411] wire_server_run_script
wire_server.c 414] live_start_time_usecs is 1691711053009381
run.c 390] gettimeofday: 1691711053.009383
wire_server.c 440] SYSCALL_EVENT happens on client side...
run.c 390] gettimeofday: 1691711053.009396
wire_server.c 440] SYSCALL_EVENT happens on client side...
run.c 390] gettimeofday: 1691711053.009407
wire_server.c 440] SYSCALL_EVENT happens on client side...
run.c 390] gettimeofday: 1691711053.009417
wire_server.c 440] SYSCALL_EVENT happens on client side...
run.c 390] gettimeofday: 1691711053.009426
wire_server.c 440] SYSCALL_EVENT happens on client side...
run.c 390] gettimeofday: 1691711053.009430
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_PACKETS_START
run_packet.c 2000] 12: packet
run_packet.c 1683] find_or_create_socket_for_script_packet
run_packet.c 241] live: local: 10.20.21.2.8080
run_packet.c 244] live: remote: 192.0.2.1.50803
run_packet.c 247] live: ISN: 0
run.c 350] waiting until 1691711053109818 -- now is 1691711053009829
run_packet.c 1941] do_inbound_script_packet
run_packet.c 581] map_inbound_packet
wire_server_netdev.c 281] wire_server_netdev_send
run.c 390] gettimeofday: 1691711053.109870
run_packet.c 2000] 13: packet
run_packet.c 1683] find_or_create_socket_for_script_packet
run_packet.c 1739] do_outbound_script_packet
run_packet.c 1752] SYNACK script.local_isn: 0
run_packet.c 1756] Expecting packet with payload 0 bytes
run_packet.c 1614] sniff_outbound_live_packet
wire_server_netdev.c 307] wire_server_netdev_receive
run_packet.c 1622] sniff_outbound_live_packet: received packet
wire_server.c 382] wire_server_run_packet_event: error!
wire_conn.c 180] wire_conn_write -> op: WIRE_PACKETS_DONE
../tcp/blocking/blocking-accept.pkt:13: error handling packet: Timed out waiting for packet

run_system_call.c 3621] main thread: signaling syscall thread to exit
run_system_call.c 3628] main thread: unlocking, waiting for syscall thread exit
run_system_call.c 3509] syscall thread: in state 4
run_system_call.c 3584] syscall thread: unlocking and exiting
run_system_call.c 3632] main thread: joined syscall thread; relocking

wire_server_netdev.c 281] wire_server_netdev_send

wire_server_netdev.c 250] wire_server_netdev_free
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        
net_utils.c 38] running: 'ip addr del 192.168.0.1/16 dev ens5 > /dev/null 2>&1'
net_utils.c 40] result: 0
wire_server.c 532] wire_server_thread: connection is done

[Neal Cardwell]

Hi,

I've pushed to https://github.com/google/packetdrill some fixes for remote ("wire server") mode. Hopefully this will improve the behavior. Can you please pull the latest code (up through and including c0b9f43a289e320e05b20140eea144efbb2e3c8f), rebuild, and re-run your tests, and let us know what the symptoms are (output with --debug) if things are not working?

Thanks!
neal

To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/f5678dfe-c5d4-4bc6-99aa-c19e9f60c250n%40googlegroups.com.