segmentation fault in queue (pop)

[sothy shan]

Hello,

When I run http server in a single core, I modifed in seastar code to run seastar::thread.

In the thread, I wrote a function for processing msg from main http server using  seastar queue


future<> tm::process_Msg( )
{

      keep_doing([this]{

      std::cout << "Loop"<< std::endl;
      return  _queueTM->pop_eventually().then ([this] (temporary_buffer<char> fr){
               return  this->read_queue_msg(std::move(fr));
              });
                  //return make_ready_future<>();
      });
     return make_ready_future<>();

}


++++++++++++++++++++++++++++++
Process_Msg is made to get the msg from queue all the time. I used keep_doing abstraction.
The code compiled. Pop_eventually() got a first message. When second time, it throws segmentation fault.

The place the execption is thrown here:

Program received signal SIGSEGV, Segmentation fault.
pop (this=0x600000088bd0) at /home/sothy/netbricks/seastar/core/queue.hh:143

143        T data = std::move(_q.front());

(gdb) bt
#0  pop (this=0x600000088bd0) at /home/sothy/netbricks/seastar/core/queue.hh:143
#1  pop_eventually (this=0x600000088bd0) at /home/sothy/netbricks/seastar/core/queue.hh:160
#2  operator() (__closure=0x600000120638) at topology/topologymanager.cc:47


Do you have any idea why second time, it throws segmentaitob fault?

Thanks and I can provide more details. I am using bit old seastar code (nearly one year ago, I downloaded and playing when I am getting error).

Best regards
Sothy

[Tomasz Grabiec]

On Sun, Apr 9, 2017 at 12:06 PM, sothy shan <soth...@gmail.com> wrote:

Hello,

When I run http server in a single core, I modifed in seastar code to run seastar::thread.

In the thread, I wrote a function for processing msg from main http server using  seastar queue


future<> tm::process_Msg( )
{

      keep_doing([this]{

      std::cout << "Loop"<< std::endl;
      return  _queueTM->pop_eventually().then ([this] (temporary_buffer<char> fr){
               return  this->read_queue_msg(std::move(fr));
              });
                  //return make_ready_future<>();
      });
     return make_ready_future<>();

Are you sure that the tm object remains alive for as long as keep_doing() runs? Note that process_Msg() returns and its resulting future resolves immediately.

 

}


++++++++++++++++++++++++++++++
Process_Msg is made to get the msg from queue all the time. I used keep_doing abstraction.
The code compiled. Pop_eventually() got a first message. When second time, it throws segmentation fault.

The place the execption is thrown here:

Program received signal SIGSEGV, Segmentation fault.
pop (this=0x600000088bd0) at /home/sothy/netbricks/seastar/core/queue.hh:143

143        T data = std::move(_q.front());

(gdb) bt
#0  pop (this=0x600000088bd0) at /home/sothy/netbricks/seastar/core/queue.hh:143
#1  pop_eventually (this=0x600000088bd0) at /home/sothy/netbricks/seastar/core/queue.hh:160
#2  operator() (__closure=0x600000120638) at topology/topologymanager.cc:47


Do you have any idea why second time, it throws segmentaitob fault?

Thanks and I can provide more details. I am using bit old seastar code (nearly one year ago, I downloaded and playing when I am getting error).

Best regards
Sothy

--
You received this message because you are subscribed to the Google Groups "seastar-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev+unsubscribe@googlegroups.com.
To post to this group, send email to seast...@googlegroups.com.
Visit this group at https://groups.google.com/group/seastar-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/1a7d08a1-daf6-4d36-8348-b431c5472f6d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sothy shan]

On Mon, Apr 10, 2017 at 1:02 PM, Tomasz Grabiec <tgra...@scylladb.com> wrote:

On Sun, Apr 9, 2017 at 12:06 PM, sothy shan <soth...@gmail.com> wrote:

Hello,

When I run http server in a single core, I modifed in seastar code to run seastar::thread.

In the thread, I wrote a function for processing msg from main http server using  seastar queue


future<> tm::process_Msg( )
{

      keep_doing([this]{

      std::cout << "Loop"<< std::endl;
      return  _queueTM->pop_eventually().then ([this] (temporary_buffer<char> fr){
               return  this->read_queue_msg(std::move(fr));
              });
                  //return make_ready_future<>();
      });
     return make_ready_future<>();

Are you sure that the tm object remains alive for as long as keep_doing() runs? Note that process_Msg() returns and its resulting future resolves immediately.

Thanks, I observed also. same. My question is that keep_doing will be running forever.So created thread should be lived?

In my case, once resulting future resolves, spawned thread is also dead. why?

I provide you how I created thread.

auto tm=new seastar::thread([this]{
         //uint8_t x=200;
         //uint8_t *data=&x;
         //topology::topology_manager

         top_man topman(this->_queue);
         topman.process_Msg();
         std::cout << "Inside " << std::endl;
       });
       std::cout << "After" << std::endl;


       tm->join().then([tm]{
           std::cout<< "finished thread" << std::endl;
           delete t1;
       });

[Tomasz Grabiec]

Here, process_Msg() returns without waiting on keep_doing(), so execution continues in this thread and soon reaches the end of the thread function. When you exit that function, "topman" goes out of scope. In order to fix this you could change two things:

1) Change process_Msg() to resolve after keep_doing():

  future<> tm::process_Msg() {

        return keep_doing(...);

  }

Now the contract between tm and its callers is that the caller must ensure that tm stays alive until the returned future resolves (process_Msg is an "async method" of tm).

2) ensure that topman stays in scope until process_Msg() resolves.

When in a seastar thread, like in your code, you can call get() and it will block the current thread until the future resolves, preventing topman from going out of scope before that:

  

   top_man topman(this->_queue);

   topman.process_Msg().get();

When not in a seastar thread, you would need to ensure that by other means. You cannot block the reactor thread like that, get() would abort. To ensure something lives around a single async operation you can use do_with():

 future<> main() {

   return do_with(top_man(this->_queue), [] (top_man& topman) {

       return topman.process_Msg();

   });

 }

 

[sothy shan]

My understanding is that, instead of creating thread, you ask me to create the a single async operation. I took ur main method into my main function. I run the code.

My process _Msg() has this code:

 return _queueTM->pop_eventually().then ([this] (temporary_buffer<char> fr){
                   return  this->read_queue_msg(std::move(fr));
                     });

Now I got the segmentation fault error again.

Program received signal SIGSEGV, Segmentation fault.

memory::small_pool::allocate (this=0x7ffff7fcce48) at core/memory.cc:852
852         _free = _free->next;
(gdb) bt
#0  memory::small_pool::allocate (this=0x7ffff7fcce48) at core/memory.cc:852
#1  0x00000000004a67be in memory::allocate (size=36) at core/memory.cc:970
#2  0x00000000004a68c5 in operator new[] (size=<optimized out>) at core/memory.cc:1282

*********************************************************

#7  0x0000000000611363 in operator() (fr=<error reading variable: access outside bounds of object referenced via synthetic pointer>, __closure=0x6000001e34a8)
    at topology/topologymanager.cc:59

Here, I feel queue is active. All the time it produce size of queue is zero. Anyhow the  problem is happening second time content reading during all the test. 

Thanks again for your help.

 

 

[Tomasz Grabiec]

That's not enough context for me to determine if the code is correct. For instance, did you change process_Msg() and its caller as suggested? 

Now I got the segmentation fault error again.

Program received signal SIGSEGV, Segmentation fault.
memory::small_pool::allocate (this=0x7ffff7fcce48) at core/memory.cc:852
852         _free = _free->next;
(gdb) bt
#0  memory::small_pool::allocate (this=0x7ffff7fcce48) at core/memory.cc:852
#1  0x00000000004a67be in memory::allocate (size=36) at core/memory.cc:970
#2  0x00000000004a68c5 in operator new[] (size=<optimized out>) at core/memory.cc:1282

Looks like a memory corruption, which could be caused by use-after-free errors.

In such cases it's often productive to run a debug-mode build with the sanitizers enabled, they should catch such errors earlier and give more details about the cause.

[sothy shan]

YES.

Now I got the segmentation fault error again.

Program received signal SIGSEGV, Segmentation fault.
memory::small_pool::allocate (this=0x7ffff7fcce48) at core/memory.cc:852
852         _free = _free->next;
(gdb) bt
#0  memory::small_pool::allocate (this=0x7ffff7fcce48) at core/memory.cc:852
#1  0x00000000004a67be in memory::allocate (size=36) at core/memory.cc:970
#2  0x00000000004a68c5 in operator new[] (size=<optimized out>) at core/memory.cc:1282

Looks like a memory corruption, which could be caused by use-after-free errors.

In such cases it's often productive to run a debug-mode build with the sanitizers enabled, they should catch such errors earlier and give more details about the cause.

I will check. 

[sothy shan]

Actually, I compiling a code using
g++-5 `pkg-config --cflags --libs /home/sothy/netbricks/seastar/build/release/seastar.pc` testn9.cc -lstdc++  -fsanitize=address -fsanitize=leak -fuse-ld=gold

+++++++++++++++++++++++++++++++++

I couldnt use memory sanitize.

I got another error. I used gdb to see the error

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.

0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff6365c44 in __once_proxy () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007ffff315ca80 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#3  0x00000000004a5c3a in __gthread_once (__func=<optimized out>, __once=0x7673d0 <memory::mem_base()::flag>)
    at /usr/include/x86_64-linux-gnu/c++/5/bits/gthr-default.h:699
#4  call_once<memory::mem_base()::<lambda()> > (__once=..., __f=<optimized out>) at /usr/include/c++/5/mutex:729
#5  mem_base () at core/memory.cc:144
#6  memory::cpu_pages::initialize (this=this@entry=0x7ffff7fcc980) at core/memory.cc:639
#7  0x00000000004a5e80 in initialize (this=0x7ffff7fcc980) at core/memory.cc:423
#8  memory::cpu_pages::find_and_unlink_span (this=this@entry=0x7ffff7fcc980, n_pages=n_pages@entry=1) at core/memory.cc:433
#9  0x00000000004a652b in memory::cpu_pages::find_and_unlink_span_reclaiming (this=this@entry=0x7ffff7fcc980, n_pages=1) at core/memory.cc:454
#10 0x00000000004a6b33 in memory::cpu_pages::allocate_large_and_trim<memory::cpu_pages::allocate_large(unsigned int)::<lambda(unsigned int, unsigned int)> >(unsigned int, memory::cpu_pages::<lambda(unsigned int, unsigned int)>) (this=this@entry=0x7ffff7fcc980, n_pages=n_pages@entry=1, trimmer=..., trimmer@entry=...)
    at core/memory.cc:472
#11 0x00000000004a6dfa in allocate_large (n_pages=1, this=<optimized out>) at core/memory.cc:509
#12 memory::small_pool::add_more_objects (this=0x7ffff7fcce98) at core/memory.cc:884
#13 0x00000000004a6fc5 in memory::small_pool::allocate (this=0x7ffff7fcce98) at core/memory.cc:846
#14 0x00000000004a7cae in memory::allocate (size=size@entry=32) at core/memory.cc:970
#15 0x00000000004a7e43 in malloc (n=n@entry=32) at core/memory.cc:1117
#16 0x00000000004a7ec4 in calloc (nmemb=nmemb@entry=1, size=size@entry=32) at core/memory.cc:1149
#17 0x00007ffff2dc66d2 in __cxa_thread_atexit_impl (func=0x4a7420 <memory::cpu_pages::~cpu_pages()>, obj=0x7ffff7fcc980, dso_symbol=0x75afe8)
    at cxa_thread_atexit_impl.c:46
#18 0x00000000004a7c97 in memory::allocate (size=size@entry=32) at core/memory.cc:361
#19 0x00000000004a7e43 in malloc (n=n@entry=32) at core/memory.cc:1117
#20 0x00000000004a7ec4 in calloc (nmemb=nmemb@entry=1, size=size@entry=32) at core/memory.cc:1149
#21 0x00007ffff491f690 in _dlerror_run (operate=operate@entry=0x7ffff491f130 <dlsym_doit>, args=args@entry=0x7fffffffe510) at dlerror.c:141
#22 0x00007ffff491f198 in __dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:70
#23 0x00007ffff6f259fc in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
#24 0x00007ffff6efb20f in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
#25 0x00007ffff6f0eef8 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
#26 0x00007ffff7dea25a in _dl_init (main_map=0x7ffff7ffe1c8, argc=2, argv=0x7fffffffe608, env=0x7fffffffe620) at dl-init.c:111
#27 0x00007ffff7ddb30a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#28 0x0000000000000002 in ?? ()
#29 0x00007fffffffe858 in ?? ()
#30 0x00007fffffffe885 in ?? ()
#31 0x0000000000000000 in ?? ()


May be I have to compile from scratch ?