dbslayer dying unexpectedly

[Devin Venable]

My dbslayer daemons seem to be dying an untimely death after running for at least 24 hours.  This doesn't always happen.  Yesterday I found that one of the three daemons started the previous day was still going.  This morning I found the three I started yesterday had given up the ghost.

I'm running dbslayer on Ubuntu 8.04 servers.

Does dbslayer recycle itself by design, or does this indicate a bug?  If the former, what strategy should be used to keep it up and running?

[Mike Welles]

Devon:

I don't think it's any sort of intended feature. Does it leave any
core files behind? I'm sending this from my phone, so I can't check
the source to be sure, but I think it changes it's working directory
to /tmp when it starts. If so, they should be there. If not you
might try running it debug mode through gdb tto see if you can catch
the error. The command would be along the lines of:

gdb /path/to/dbslayer
run [your usual args] -d

When it dies, you can get the stack trace via 'w'.

- Mike

[Devin Venable]

Mike,

No core dump file, but that's because Ubuntu suppresses core dumps by default.  I'll run it under gdb as suggested and let you know if I catch it next time around.

Devin

[Devin Venable]

gdb reported:

Program received signal SIGPIPE, Broken pipe.
[Switching to Thread -144725104 (LWP 8453)]
0xffffe402 in __kernel_vsyscall ()

Call stack:

#0  0xffffe402 in __kernel_vsyscall ()
#1  0xf7ec417b in ?? () from /lib/tls/i686/cmov/libpthread.so.0
#2  0xf7f245cc in apr_socket_send () from /usr/lib/libapr-1.so.0
#3  0x0804a524 in handle_response (td=0xffbdb880, qd=0x80567f0,
    equery=0x8086a38 "%7B%22SQL%22%3A%20%22use%20nexgenshard%3Bselect%20SQL_CALC_FOUND_ROWS%20id%2C%20source%2C%20object%2C%20title%2C%20format%2C%20searchtext%2C%20inserted%2C%20alt_pkval%2C%20match%28searchtext%29%20agai"...,
    http_request=0x8086880 "GET /db/?%7B%22SQL%22%3A%20%22use%20nexgenshard%3Bselect%20SQL_CALC_FOUND_ROWS%20id%2C%20source%2C%20object%2C%20title%2C%20format%2C%20searchtext%2C%20inserted%2C%20alt_pkval%2C%20match%28searchtext%"...,
    header_response=0x804f9eb "200 OK", response_code=200,
    message=0x820bb80 "{\"SERVER\" : \"shard\" , \"RESULT\" : [{\"SUCCESS\" : true} , {\"TYPES\" : [\"MYSQL_TYPE_LONG\" , \"MYSQL_TYPE_LONG\" , \"MYSQL_TYPE_LONG\" , \"MYSQL_TYPE_VAR_STRING\" , \"MYSQL_TYPE_VAR_STRING\" , \"MYSQL_TYPE_BLOB\" , \""...) at dbslayer.c:78
#4  0x0804a6f8 in handle_db (td=0xffbdb880, qd=0x80567f0, dbhandle=0x8058658,
    equery=0x8086a38 "%7B%22SQL%22%3A%20%22use%20nexgenshard%3Bselect%20SQL_CALC_FOUND_ROWS%20id%2C%20source%2C%20object%2C%20title%2C%20format%2C%20searchtext%2C%20inserted%2C%20alt_pkval%2C%20match%28searchtext%29%20agai"...,
    http_request=0x8086880 "GET /db/?%7B%22SQL%22%3A%20%22use%20nexgenshard%3Bselect%20SQL_CALC_FOUND_ROWS%20id%2C%20source%2C%20object%2C%20title%2C%20format%2C%20searchtext%2C%20inserted%2C%20alt_pkval%2C%20match%28searchtext%"...)
    at dbslayer.c:185

Any insights?

Devin

[Derek Gottfrid]

devin

Yes, that makes perfect sense. It should be ignoring SIGPIPE and it
has in the past but for some odd reason - I commented it out.

http://code.google.com/p/dbslayer/source/browse/trunk/common/slayer_http_server.c#82

I will have to fix that but if you feel brave please comment out that
line recompile and see if this solves your issues.

d

[Devin Venable]

Yeah, I know how it goes with temporary commented lines sneaking into source commits.  I'll make the change and recompile.  On another note, I intend to create an init script for bringing the daemon up and down.  Would you consider making your init script available (putting it under svn), assuming that you have and use one?


Devin

[Derek Gottfrid]

devin - we would happily include your script if suitable into the svn
repo and tarballs.

d