Packetdrill throws segfault for pthread_create in shared library

[Ibtisam Tariq]

Hi 

I am trying to test a shared library using packetdrill, in which a thread is created in packetdrill_interface_init using pthread_create. At pthread_create(within the library) packetdrill gives segfault but the same library runs fine in the test application. I have created a test application to replicate the issue. The test application has the same flow as used in packetdrill, create a side thread then in the main thread use dlopen to open the library then call the packetdrill_interface_init function. I have added a sample code and commands to run them. Please have a look.

Library code:

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
void* thread_func(void* arg)
{
        printf("In library pthread\n");
        sleep(1);
        printf("Exiting the library pthread\n");
        pthread_exit(NULL);
        return NULL;
}
void* packetdrill_interface_init(const char *flags,void * d)
{
        printf("In library packetdrill_interface_init function\n");
        pthread_t ptid;
       
        // Creating a new thread
        pthread_create(&ptid, NULL, &thread_func, NULL);
        printf("created a thread in library\n");
        pthread_exit(NULL);
        pthread_join(ptid, NULL);
        return NULL;
}

Command using to build library: 

gcc -shared -fPIC -o libtest_pthread.so <file_name>  -lpthread

Test application code (where library is running fine):

#include <assert.h>
#include <dlfcn.h>
#include <pthread.h>
#include <stdint.h>
#include <stdio.h>
void func_for_so_handling()
{
void *handler;
void (*func_init)(const char *,void*);
printf("In app function for dlopen\n");
handler=dlopen("./libtest_pthread.so",RTLD_NOW | RTLD_LOCAL
| RTLD_NODELETE | RTLD_DEEPBIND| RTLD_GLOBAL);
if (!handler)
{
printf("Unable to open library\n");
printf("%s\n", dlerror());
return;
}
dlerror();
/* clear any existing error */
printf("After calling dlopen on library in app\n");
func_init = dlsym(handler, "packetdrill_interface_init");
char *error = dlerror();
if (error) {
printf("%s\n", error);
return;
}
func_init(NULL,NULL);
dlclose(handler);
printf("Handling of shared library completed\n");
}
//Mock fun
void mock_fun()
{
printf("In mock_function of application\n");
sleep(10);
printf("Exiting mock function thread\n");
//something else for liefetime of program
}
int main()
{
pthread_t pt2;
pthread_create(&pt2, NULL, &mock_fun, NULL);
func_for_so_handling();
pthread_join(pt2,NULL);
return 0;
}

Command to build test_application

gcc app.c -o a.out -ldl -lpthread

Output of test application

$ ./a.out 

In app function for dlopen
In mock_function of application
After calling dlopen on library in app
In library packetdrill_interface_init function
created a thread in library
In library pthread
Exiting the library pthread
Exiting mock function thread

Output of packetdrill

~/tmp/packetdrill/gtests/net/tcp/blocking$ sudo ../../packetdrill/packetdrill  --so_filename=<path_to_lib>/libtest_pthread.so --local_ip=172.16.0.1 --gateway_ip=172.16.0.2 --netmask_ip=255.255.255.0 --remote_ip=172.16.1.1/24 --wire_client --wire_client_dev="eno8" --wire_server_ip=<server_ip> ./blocking-write.pkt

In library packetdrill_interface_init function

Segmentation fault

On server side the command is: sudo ./packetdrill --wire_server --wire_server_dev=enp1s0f0

--

Regards,  

- Ibtisam

[Neal Cardwell]

Thanks for the post.

Can you run the crashing packetdrill process in gdb and use a "backtrace" command to see which thread is crashing and what the stack looks like at that point? Even better might be something like "thread apply all bt" to get a backtrace for all threads.

neal

--
You received this message because you are subscribed to the Google Groups "packetdrill" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packetdrill...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/CA%2B8bGBstm3FqBwx1LgYkaQDSoX47C%3D71WQDnVStMZT4HTEzxwg%40mail.gmail.com.

[Ibtisam Tariq]

Hi 

Thank you for the reply.

At client side, I have run the command:

~/tmp/packetdrill/gtests/net/tcp/blocking$ sudo gdb --args ../../packetdrill/packetdrill  --so_filename=/home/ibtisam/dpdk-main/app/libtest_pthread.so --local_ip=172.16.0.1 --gateway_ip=172.16.0.2 --netmask_ip=255.255.255.0 --remote_ip=172.16.1.1/24 --wire_client --wire_client_dev="eno8" --wire_server_ip=192.168.3.120 ./blocking-write.pkt

Output

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after vfork from child process 3209928]
[New Thread 0x7ffff7ff8700 (LWP 3209932)]
In library packetdrill_interface_init function

Thread 1 "packetdrill" received signal SIGSEGV, Segmentation fault.
__GI__dl_allocate_tls_init (result=0x7ffff747f700) at ../elf/dl-tls.c:465
465 ../elf/dl-tls.c: No such file or directory.

(gdb) bt
#0  __GI__dl_allocate_tls_init (result=0x7ffff747f700) at ../elf/dl-tls.c:465
#1  0x00007ffff77ac323 in allocate_stack (stack=<synthetic pointer>, pdp=<synthetic pointer>,
    attr=0x7fffffffdd90) at allocatestack.c:622
#2  __pthread_create_2_1 (newthread=<optimized out>, attr=<optimized out>,
    start_routine=<optimized out>, arg=<optimized out>) at pthread_create.c:660
#3  0x00007ffff77f4206 in packetdrill_interface_init ()
   from /home/ibtisam/dpdk-main/app/libtest_pthread.so
#4  0x000000000042cbd5 in so_instance_init (instance=0x552340, config=0x7fffffffdf20,
    script=0x7fffffffdef0, state=0x551f10) at so_testing.c:146
#5  0x0000000000410fcf in run_script (config=0x7fffffffdf20, script=0x7fffffffdef0) at run.c:581
#6  0x0000000000402124 in main (argc=10, argv=0x7fffffffe2f8) at packetdrill.c:109
(gdb) thread apply all bt
Thread 2 (Thread 0x7ffff7ff8700 (LWP 3209932)):
#0  __lll_lock_wait (futex=futex@entry=0x551f10, private=0) at lowlevellock.c:52
#1  0x0000000000430453 in __pthread_mutex_lock (mutex=0x551f10) at ../nptl/pthread_mutex_lock.c:80
#2  0x0000000000416e6c in run_lock (state=0x551f10) at run.h:125
#3  0x000000000041f83c in system_call_thread (arg=0x551f10) at run_system_call.c:3470
#4  0x000000000042e3a9 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00000000004ac6a3 in clone ()

Thread 1 (Thread 0x54a880 (LWP 3209927)):
#0  __GI__dl_allocate_tls_init (result=0x7ffff747f700) at ../elf/dl-tls.c:465
#1  0x00007ffff77ac323 in allocate_stack (stack=<synthetic pointer>, pdp=<synthetic pointer>, attr=0x7fffffffdd90) at allocatestack.c:622
#2  __pthread_create_2_1 (newthread=<optimized out>, attr=<optimized out>, start_routine=<optimized out>, arg=<optimized out>) at pthread_create.c:660
#3  0x00007ffff77f4206 in packetdrill_interface_init () from /home/ibtisam/dpdk-main/app/libtest_pthread.so
#4  0x000000000042cbd5 in so_instance_init (instance=0x552340, config=0x7fffffffdf20, script=0x7fffffffdef0, state=0x551f10) at so_testing.c:146
#5  0x0000000000410fcf in run_script (config=0x7fffffffdf20, script=0x7fffffffdef0) at run.c:581
#6  0x0000000000402124 in main (argc=10, argv=0x7fffffffe2f8) at packetdrill.c:109

Regards,

- Ibtisam

[Neal Cardwell]

This seems unrelated to packetdrill per se, and instead seems to be some interaction between the pthreads library and dynamically loading shared libraries. Perhaps it is related to the options that are passed to gcc for compiling or ld for linking. You may want to audit the gcc and ld options used for your test program and packetdrill. You may want to double-check that the shared library and packetdrill are picking up the same version of libpthread (perhaps checking both with "ldd").

I'm afraid I don't have time to dig into it more, but perhaps others on the list will have ideas.

best regards,

neal

To view this discussion on the web visit https://groups.google.com/d/msgid/packetdrill/3ed482a8-7478-46cf-b8f3-578ff8b12c2an%40googlegroups.com.