Segmentation faults

[Matt C]

I have 0.14.2 working on three Ubuntu 16.04 sensors now, but I'm getting segfaults.  They happen most on the most heavily loaded box.  Here are a few of the crashes:

1. with packetThreads=8

Jun 17 15:01:19 parsers.c:160 moloch_parsers_asn_get_sequence(): 1 0x7fffa8d9e8f8 1 1 32 0
Jun 17 15:01:19 parsers.c:160 moloch_parsers_asn_get_sequence(): 0 0x7fffa8d9e8fc 0 27 4 0
Jun 17 15:01:19 parsers.c:160 moloch_parsers_asn_get_sequence(): 1 0x7fffa8d9e902 0 27 22 0

Thread 9 "moloch-pkt7" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffeb7fe700 (LWP 107942)]
__GI___libc_realloc (oldmem=0x3b040206a75a3530, bytes=3475234937) at malloc.c:3000
3000    malloc.c: No such file or directory.

2. with packetThreads=1, just to see if that would help

Jun 17 15:03:53 parsers.c:160 moloch_parsers_asn_get_sequence(): 1 0x7fff9f0d002f 1 1 22 0
Jun 17 15:03:53 parsers.c:160 moloch_parsers_asn_get_sequence(): 0 0x7fff9f0d0033 0 27 4 0
Jun 17 15:03:53 parsers.c:160 moloch_parsers_asn_get_sequence(): 1 0x7fff9f0d0039 0 27 12 0

Thread 2 "moloch-pkt0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff363a700 (LWP 108067)]
__memmove_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:356
356     ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory.

3. packetThreads=1

Jun 17 15:03:53 parsers.c:160 moloch_parsers_asn_get_sequence(): 1 0x7fff9f0d002f 1 1 22 0
Jun 17 15:03:53 parsers.c:160 moloch_parsers_asn_get_sequence(): 0 0x7fff9f0d0033 0 27 4 0
Jun 17 15:03:53 parsers.c:160 moloch_parsers_asn_get_sequence(): 1 0x7fff9f0d0039 0 27 12 0

Thread 2 "moloch-pkt0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff363a700 (LWP 108067)]
__memmove_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:356
356     ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory.

4. packetThreads=1

Jun 17 15:31:52 parsers.c:160 moloch_parsers_asn_get_sequence(): 1 0x7fffcb4d090d 1 1 40 0
Jun 17 15:31:52 parsers.c:160 moloch_parsers_asn_get_sequence(): 0 0x7fffcb4d0911 0 27 8 0
Jun 17 15:31:52 parsers.c:160 moloch_parsers_asn_get_sequence(): 1 0x7fffcb4d091b 0 27 26 0

Thread 5 "moloch-pcap0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe3079700 (LWP 181645)]
magazine_chain_pop_head (magazine_chunks=<optimized out>) at gslice.c:539
539     gslice.c: No such file or directory.

Any ideas or guidance on how I can help?

- Matt

[Andy]

If you could send a back trace that would be useful.

https://github.com/aol/moloch/wiki/FAQ#Moloch_capture_crashes

[Matt C]

Sure, here are two:

Program terminated with signal SIGSEGV, Segmentation fault.

#0  __memmove_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:356

356     ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory.

[Current thread is 1 (Thread 0x7ffff363a700 (LWP 108067))]

(gdb) bt

#0  __memmove_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:356

#1  0x00007ffff1b256b1 in memmove (__len=<optimized out>, __src=0x7fff9f0d0109, __dest=0x7fff9f0cf730)

    at /usr/include/x86_64-linux-gnu/bits/string3.h:59

#2  krb5_tcp_parser (session=0x7fff9f033450, uw=<optimized out>, data=<optimized out>, remaining=<optimized out>,

    which=<optimized out>) at krb5.c:255

#3  0x0000000000465423 in moloch_packet_process_data (session=0x7fff9f033450,

    data=0x7fffc51b13c6 "\\è´H\357\336\f,\202\347\024\327k@\321$&\252\362\362\375yE\240\243\064\\\374\230\375\365\353D\220\346I\314\377\240Ç\376\232b\277\062\226\267ѵݴ\340\210\r\033r\216\331`bg-|zTm\253Q\342\063#II9\240\063\210aA\263|i+\377w\017\326<<\b\354\276\342M8%\361\206\320\001\276\237?Sh\211X\232\260:\340=\333\354\377zÝ\222\030\311p\vz\277\v\234\271J\017\265\256\237GÉ°\005\026\211;X \326\325Í£E\036\374&\377~n\334\032\355\242\021\365\264\277\064B\360\252!}(>\v\003\352\362\034\203Kp\r0E+\263(\232\b=\255\306:]\245t`l5\220\030\235"..., len=1260, which=0) at packet.c:122

#4  0x0000000000465500 in moloch_packet_tcp_finish (session=0x7fff9f033450) at packet.c:167

#5  0x00000000004663e6 in moloch_packet_thread (threadp=0x0) at packet.c:612

#6  0x00000000004bab0f in g_thread_proxy (data=0x168dc00) at gthread.c:776

#7  0x00007ffff715f6fa in start_thread (arg=0x7ffff363a700) at pthread_create.c:333

#8  0x00007ffff6194b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Program terminated with signal SIGSEGV, Segmentation fault.

#0  __GI___libc_realloc (oldmem=0x3b040206a75a3530, bytes=3475234937) at malloc.c:3000

3000    malloc.c: No such file or directory.

[Current thread is 1 (Thread 0x7fffeb7fe700 (LWP 107942))]

(gdb) bt

#0  __GI___libc_realloc (oldmem=0x3b040206a75a3530, bytes=3475234937) at malloc.c:3000

#1  0x00000000004a3238 in g_realloc (mem=0x3b040206a75a3530, n_bytes=3475234937) at gmem.c:174

#2  0x00000000cf23e079 in ?? ()

#3  0x00000000004880b9 in g_nearest_pow (num=<optimized out>) at garray.c:764

#4  g_array_maybe_expand (array=0x3b040206a75a3530, len=<optimized out>) at garray.c:776

#5  0x0000000000488418 in g_array_append_vals (farray=0x3b040206a75a3530, data=0xcf23e079, len=1) at garray.c:416

#6  0x00000000004660f5 in moloch_packet_thread (threadp=0x7) at packet.c:549

#7  0x00000000004bab0f in g_thread_proxy (data=0x168de30) at gthread.c:776

#8  0x00007ffff715f6fa in start_thread (arg=0x7fffeb7fe700) at pthread_create.c:333

#9  0x00007ffff6194b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

- Matt

[Andy]

I commit a fix to the new krb5 stuff, can you see if the cores go away.

Thanks,

Andy

[Matt C]

That sounds right.  I noticed the crashes went away when I ran it with no parsers, so I started adding them back in.  The crashes came back when I got up to the letter "k." :)

- Matt

[Matt C]

Ok, we've been crash free for several hours now since the update.  The patch looks good.

On a related note, the krb5 parser is awesome.  This is really helpful.

- Matt

On Sunday, June 19, 2016 at 8:07:06 AM UTC-4, Andy wrote:

[Andy]

Great!

If you want more fields or have suggestions feel free to open up some github issues

[erik clark]

Is it possible for me to build just the new (ldap/krb5) parser shared objects without downloading and rebuilding the full moloch capture package? 

I am trying to identify how I can push partial build requests for  (specifically) new parsers pushed, without rebuilding the whole deployment. Sneakernetting a very small handful of parser c and jade files is a lot more feasible than doing a complete rebuild.

[Matt C]

Here's what I did this morning:

cd moloch/capture/parsers
git pull
make
scp <new parser files> <where they need to go>

- Matt

[Andy]

Sometimes, in this case no because there were new routines added to core.  There are checks when the plugins/parsers load that it will tell you if its compatible or not.

Andy

On Monday, June 20, 2016 at 2:28:13 PM UTC-4, erik clark wrote:

[Andy]

On Monday, June 20, 2016 at 2:33:41 PM UTC-4, Matt C wrote:

Here's what I did this morning:

cd moloch/capture/parsers
git pull
make
scp <new parser files> <where they need to go>

Yep, if you are just going from one commit to the next your luck is much higher on it working. :)

I definitely think setting up a jenkins (or whatever) script that produces an rpm (or whatever) and moving that around is the easiest, and what we do.  Its about 15M

https://github.com/aol/moloch/wiki/jenkins 

[erik clark]

Hm, ok. Thanks guys! Slowly ramping up our distribution buildout. Looks like we are going to struggle with package management and our ES cluster. :/ I think package management I  might be able to deal with, but I just don't have the hardware to handle our throughput at the moment to our ES cluster. 

I saw you mentioned Jenkins a few threads ago, but didnt have time to dig into it. I will probably spend the next week or so trying to sort that out, see if it works. We are still determining if we are going to go with Ansible or Puppet. If we go with Ansible, will have to figure out what kind of playbooks we will need to handle this.