Re: Running remote tests

[Neal Cardwell]

Thanks for the report. Interestingly, I noticed yesterday that recent versions of tcpdump crash in a similar way ("error 4 in libnss_files") in some cases. 

Would you be able to start the server process from within gdb and print a stack backtrace with "backtrace" to help us see where exactly the crash is happening?

thanks,

neal

On Tue, Nov 14, 2023 at 10:48 AM Kalai Bakthavathsalam <kalai...@gmail.com> wrote:

Hello,

I started using packetdrill on Ubuntu 20.04/ Kernel 5.15.0-88-generic and seeing segmentation fault on the server side.

Test bed:

 <Server>----192.168.1.0/24------<Client>

Server:

4: ens5f0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

    link/ether 10:70:fd:1e:53:ba brd ff:ff:ff:ff:ff:ff
    altname enp81s0f0np0
    inet 192.168.1.1/24 scope global ens5f0
       valid_lft forever preferred_lft forever
    inet6 2022::1/64 scope global
       valid_lft forever preferred_lft forever

./packetdrill -v --wire_server --wire_server_dev=ens5f0 --debug

Client:

4: ens5f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:70:fd:1e:55:22 brd ff:ff:ff:ff:ff:ff
    altname enp81s0f0np0
    inet 192.168.1.2/24 scope global ens5f0
       valid_lft forever preferred_lft forever
    inet6 2022::2/64 scope global
       valid_lft forever preferred_lft forever

./packetdrill --ip_version=ipv4 --wire_server_at=192.168.1.1 --wire_client_dev=ens5f0 -v --local_ip=10.0.0.1 --gateway_ip=10.0.0.1 --netmask_ip=255.255.0.0 --remote_ip=192.0.2.1 ~packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt

When the client session starts session fails with following messages:

Client:

remote side closed connection
error in TCP connection to wire server: error reading WIRE_SERVER_READY

Server:

config.c 424] finalize_config: config->live_local_ip_string: []
wire_conn.c 43] wire_conn_new
wire_conn.c 119] wire_conn_bind_listen
wire_conn.c 138] wire_conn_accept
wire_conn.c 144] accepted fd 5
wire_conn.c 43] wire_conn_new
wire_conn.c 72] set_default_tcp_options fd 5
link_layer.c 46] get_hw_address for device ens5f0
link_layer.c 60] ens5f0 HWaddr: 10:70:fd:1e:53:ba
wire_server.c 539] start_wire_server_thread
wire_conn.c 138] wire_conn_accept
wire_server.c 471] wire_server_thread
config.c 273] set_default_config
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_COMMAND_LINE_ARGS
wire_server.c 101] argc = 10
wire_server.c 117] argv[0] = './packetdrill'
wire_server.c 117] argv[1] = '--ip_version=ipv4'
wire_server.c 117] argv[2] = '--wire_server_at=192.168.1.1'
wire_server.c 117] argv[3] = '-v'
wire_server.c 117] argv[4] = '--local_ip=10.0.0.1'
wire_server.c 117] argv[5] = '--gateway_ip=10.0.0.1'
wire_server.c 117] argv[6] = '--netmask_ip=255.255.0.0'
wire_server.c 117] argv[7] = '--remote_ip=192.0.2.1'
wire_server.c 117] argv[8] = '~packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt'
wire_server.c 117] argv[9] = '--wire_server'
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT_PATH
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_HARDWARE_ADDR
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_IP_ADDR
wire_server.c 233] got WIRE_IP_ADDR: [10.0.0.1]
run.c 692] parse_and_run_script: ~packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt
parser.y 157] copy_script
parser.y 164] copy_script: 1290 bytes
config.c 705] parse_and_finalize_config
config.c 644] parse_command_line_options argc=10
config.c 646] argv[0] = './packetdrill'
config.c 646] argv[1] = '--ip_version=ipv4'
config.c 646] argv[2] = '--wire_server_at=192.168.1.1'
config.c 646] argv[3] = '-v'
config.c 646] argv[4] = '--local_ip=10.0.0.1'
config.c 646] argv[5] = '--gateway_ip=10.0.0.1'
config.c 646] argv[6] = '--netmask_ip=255.255.0.0'
config.c 646] argv[7] = '--remote_ip=192.0.2.1'
config.c 646] argv[8] = '~/packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt'
config.c 646] argv[9] = '--wire_server'
config.c 477] process_option 256 ('') = ipv4
config.c 477] process_option 275 ('') = 192.168.1.1
config.c 477] process_option 118 ('v') = (null)
config.c 477] process_option 263 ('') = 10.0.0.1
config.c 517] process_option setting live_local_ip_string to: 10.0.0.1
config.c 477] process_option 264 (') = 10.0.0.1
config.c 477] process_option 265 ('     ') = 255.255.0.0
config.c 477] process_option 262 ('') = 192.0.2.1
config.c 477] process_option 273 ('') = (null)
Segmentation fault

Sever dmesg:

[1146176.657829] packetdrill[3484715]: segfault at 63 ip 00007f60f351f5ca sp 00007f60f31f5450 error 4 in libnss_files-2.31.so[7f60f351e000+7000]
[1146176.657859] Code: 48 89 44 24 10 48 85 c0 0f 88 22 04 00 00 0f 84 52 04 00 00 e8 f7 ef ff ff 48 0f be 55 20 4d 89 ec 48 8b 08 49 89 c7 48 89 d0 <f6> 44 51 01 20 74 1b 0f 1f 80 00 00 00 00 49 0f be 54 24 01 49 83

Let me know if I need to change anything in the configuration of Server/Client.

Thanks....Kalai

--
You received this message because you are subscribed to the Google Groups "packetdrill" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packetdrill...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/f9b74f96-8542-4429-befb-bb0eeca09aaan%40googlegroups.com.

[Kalai Bakthavathsalam]

Hi Neal,

Here is the backtrace:

kalaivanan@nc-gpu-07:~/packetdrill/gtests/net/packetdrill$ sudo gdb ./packetdrill
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./packetdrill...
(gdb) run --wire_server --wire_server_dev=ens5f0 --debug
Starting program: /home/kalaivanan/packetdrill/gtests/net/packetdrill/packetdrill --wire_server --wire_server_dev=ens5f0 --debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

config.c 424] finalize_config: config->live_local_ip_string: []

[Detaching after vfork from child process 3549046]

wire_conn.c 43] wire_conn_new
wire_conn.c 119] wire_conn_bind_listen
wire_conn.c 138] wire_conn_accept
wire_conn.c 144] accepted fd 5
wire_conn.c 43] wire_conn_new
wire_conn.c 72] set_default_tcp_options fd 5
link_layer.c 46] get_hw_address for device ens5f0
link_layer.c 60] ens5f0 HWaddr: 10:70:fd:1e:53:ba
wire_server.c 539] start_wire_server_thread

[New Thread 0x7ffff7cb3700 (LWP 3549151)]

wire_conn.c 138] wire_conn_accept
wire_server.c 471] wire_server_thread
config.c 273] set_default_config
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_COMMAND_LINE_ARGS
wire_server.c 101] argc = 10
wire_server.c 117] argv[0] = './packetdrill'
wire_server.c 117] argv[1] = '--ip_version=ipv4'
wire_server.c 117] argv[2] = '--wire_server_at=192.168.1.1'
wire_server.c 117] argv[3] = '-v'
wire_server.c 117] argv[4] = '--local_ip=10.0.0.1'
wire_server.c 117] argv[5] = '--gateway_ip=10.0.0.1'
wire_server.c 117] argv[6] = '--netmask_ip=255.255.0.0'
wire_server.c 117] argv[7] = '--remote_ip=192.0.2.1'

wire_server.c 117] argv[8] = '/home/kalaivanan/packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt'

wire_server.c 117] argv[9] = '--wire_server'
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT_PATH
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_HARDWARE_ADDR
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_IP_ADDR
wire_server.c 233] got WIRE_IP_ADDR: [10.0.0.1]

run.c 692] parse_and_run_script: /home/kalaivanan/packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt

parser.y 157] copy_script
parser.y 164] copy_script: 1290 bytes
config.c 705] parse_and_finalize_config
config.c 644] parse_command_line_options argc=10
config.c 646] argv[0] = './packetdrill'
config.c 646] argv[1] = '--ip_version=ipv4'
config.c 646] argv[2] = '--wire_server_at=192.168.1.1'
config.c 646] argv[3] = '-v'
config.c 646] argv[4] = '--local_ip=10.0.0.1'
config.c 646] argv[5] = '--gateway_ip=10.0.0.1'
config.c 646] argv[6] = '--netmask_ip=255.255.0.0'
config.c 646] argv[7] = '--remote_ip=192.0.2.1'

config.c 646] argv[8] = '/home/kalaivanan/packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt'

config.c 646] argv[9] = '--wire_server'
config.c 477] process_option 256 ('') = ipv4
config.c 477] process_option 275 ('') = 192.168.1.1
config.c 477] process_option 118 ('v') = (null)
config.c 477] process_option 263 ('') = 10.0.0.1
config.c 517] process_option setting live_local_ip_string to: 10.0.0.1
config.c 477] process_option 264 (') = 10.0.0.1
config.c 477] process_option 265 ('     ') = 255.255.0.0
config.c 477] process_option 262 ('') = 192.0.2.1
config.c 477] process_option 273 ('') = (null)

Thread 2 "packetdrill" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7cb3700 (LWP 3549151)]
0x00007ffff7fda5ca in internal_getent (stream=stream@entry=0x7fffe8000b60, result=result@entry=0x7ffff7cb0520, buffer=buffer@entry=0x7ffff7cb09d0 "", buflen=buflen@entry=1024, errnop=errnop@entry=0x7ffff7cb36b0, herrnop=herrnop@entry=0x7ffff7cb36e8,
    af=0) at nss_files/files-XXX.c:173
173     nss_files/files-XXX.c: No such file or directory.
(gdb) bt
#0  0x00007ffff7fda5ca in internal_getent (stream=stream@entry=0x7fffe8000b60, result=result@entry=0x7ffff7cb0520, buffer=buffer@entry=0x7ffff7cb09d0 "", buflen=buflen@entry=1024, errnop=errnop@entry=0x7ffff7cb36b0,
    herrnop=herrnop@entry=0x7ffff7cb36e8, af=0) at nss_files/files-XXX.c:173
#1  0x00007ffff7fdb9f4 in _nss_files_gethostbyname4_r (name=0x7ffff0009020 "nc-gpu-07", pat=0x7ffff7cb06c8, buffer=0x7ffff7cb09d0 "", buflen=1024, errnop=0x7ffff7cb36b0, herrnop=0x7ffff7cb36e8, ttlp=0x0) at nss_files/files-hosts.c:400
#2  0x00000000004aef8f in gaih_inet.constprop ()
#3  0x00000000004b0ad9 in getaddrinfo ()
#4  0x00000000004079de in string_to_ip (host=0x7ffff0009020 "nc-gpu-07", ip=0x7ffff7cb0eb0, error=0x7ffff7cb0ea8) at ip_address.c:159
#5  0x0000000000409c11 in get_primary_ip (ip=0x7ffff7cb0eb0, error=0x7ffff7cb0ea8) at net_utils.c:198
#6  0x0000000000409c66 in get_primary_nic () at net_utils.c:212
#7  0x000000000040507e in finalize_config (config=0x561d68) at config.c:417
#8  0x0000000000405efe in parse_and_finalize_config (invocation=0x7ffff7cb2d40) at config.c:716
#9  0x0000000000429feb in yyparse () at parser.y:609
#10 0x000000000042906b in parse_script (config=0x561d68, script=0x562008, callback_invocation=0x7ffff7cb2d40) at parser.y:255
#11 0x0000000000413006 in parse_script_and_set_config (argc=10, argv=0x7ffff0000d40, config=0x561d68, script=0x562008, script_path=0x7ffff0000fd0 "/home/kalaivanan/packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt",
    script_buffer=0x7ffff0001a50 "// Verify that setsockopt calls that force a route refresh do not\n// cause problems matching SACKs with packets in the write queue.\n// This variant tests IP_TOS.\n\n`../common/defaults.sh`\n\n// Establish"...)
    at run.c:704
#12 0x0000000000430473 in wire_server_thread (arg=0x561d50) at wire_server.c:490
#13 0x0000000000433089 in start_thread (arg=<optimized out>) at pthread_create.c:477
#14 0x00000000004b5d63 in clone ()

Thanks....Kalai

[Kalai Bakthavathsalam]

Hi Neal,

Here is the backtrace:

kalaivanan@nc-gpu-07:~/packetdrill/gtests/net/packetdrill$ sudo gdb ./packetdrill
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./packetdrill...
(gdb) run --wire_server --wire_server_dev=ens5f0 --debug
Starting program: /home/kalaivanan/packetdrill/gtests/net/packetdrill/packetdrill --wire_server --wire_server_dev=ens5f0 --debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

config.c 424] finalize_config: config->live_local_ip_string: []

[Detaching after vfork from child process 3549046]

wire_conn.c 43] wire_conn_new
wire_conn.c 119] wire_conn_bind_listen
wire_conn.c 138] wire_conn_accept
wire_conn.c 144] accepted fd 5
wire_conn.c 43] wire_conn_new
wire_conn.c 72] set_default_tcp_options fd 5
link_layer.c 46] get_hw_address for device ens5f0
link_layer.c 60] ens5f0 HWaddr: 10:70:fd:1e:53:ba
wire_server.c 539] start_wire_server_thread

[New Thread 0x7ffff7cb3700 (LWP 3549151)]

wire_conn.c 138] wire_conn_accept
wire_server.c 471] wire_server_thread
config.c 273] set_default_config
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_COMMAND_LINE_ARGS
wire_server.c 101] argc = 10
wire_server.c 117] argv[0] = './packetdrill'
wire_server.c 117] argv[1] = '--ip_version=ipv4'
wire_server.c 117] argv[2] = '--wire_server_at=192.168.1.1'
wire_server.c 117] argv[3] = '-v'
wire_server.c 117] argv[4] = '--local_ip=10.0.0.1'
wire_server.c 117] argv[5] = '--gateway_ip=10.0.0.1'
wire_server.c 117] argv[6] = '--netmask_ip=255.255.0.0'
wire_server.c 117] argv[7] = '--remote_ip=192.0.2.1'

wire_server.c 117] argv[8] = '/home/kalaivanan/packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt'

wire_server.c 117] argv[9] = '--wire_server'
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT_PATH
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_SCRIPT
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_HARDWARE_ADDR
wire_conn.c 224] wire_conn_read
wire_conn.c 233] wire_conn_read -> op: WIRE_IP_ADDR
wire_server.c 233] got WIRE_IP_ADDR: [10.0.0.1]

run.c 692] parse_and_run_script: /home/kalaivanan/packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt

parser.y 157] copy_script
parser.y 164] copy_script: 1290 bytes
config.c 705] parse_and_finalize_config
config.c 644] parse_command_line_options argc=10
config.c 646] argv[0] = './packetdrill'
config.c 646] argv[1] = '--ip_version=ipv4'
config.c 646] argv[2] = '--wire_server_at=192.168.1.1'
config.c 646] argv[3] = '-v'
config.c 646] argv[4] = '--local_ip=10.0.0.1'
config.c 646] argv[5] = '--gateway_ip=10.0.0.1'
config.c 646] argv[6] = '--netmask_ip=255.255.0.0'
config.c 646] argv[7] = '--remote_ip=192.0.2.1'

config.c 646] argv[8] = '/home/kalaivanan/packetdrill/gtests/net/tcp/sack/sack-route-refresh-ip-tos.pkt'

config.c 646] argv[9] = '--wire_server'
config.c 477] process_option 256 ('') = ipv4
config.c 477] process_option 275 ('') = 192.168.1.1
config.c 477] process_option 118 ('v') = (null)
config.c 477] process_option 263 ('') = 10.0.0.1
config.c 517] process_option setting live_local_ip_string to: 10.0.0.1
config.c 477] process_option 264 (') = 10.0.0.1
config.c 477] process_option 265 ('     ') = 255.255.0.0
config.c 477] process_option 262 ('') = 192.0.2.1
config.c 477] process_option 273 ('') = (null)

Thanks....Kalai

[Neal Cardwell]

Hi,

I've pushed to https://github.com/google/packetdrill some fixes for remote ("wire server") mode. Hopefully this will improve the behavior. Can you please pull the latest code (up through and including c0b9f43a289e320e05b20140eea144efbb2e3c8f), rebuild, and re-run your tests, and let us know what the symptoms are (output with --debug) if things are not working?

Thanks!
neal

To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/0a72b7cb-90c7-4743-b2dc-c319f6347c20n%40googlegroups.com.