pinba was crashed with mysqld

[Evgeny Kokovikhin]

Hello.

I have installed pinba.

Versions:
dovg@pinba:~$ uname -a
Linux pinba 2.6.32-5-openvz-amd64 #1 SMP Tue Jun 14 10:46:15 UTC 2011
x86_64 GNU/Linux
dovg@pinba:~$ /usr/sbin/mysqld --version
/usr/sbin/mysqld Ver 5.1.49-3 for debian-linux-gnu on x86_64
((Debian)
Pinba's version is 0.0.6. We made debian package from sources.

Yesterday pinba server was crashed with the log:
Sep 22 18:24:31 pinba mysqld: 110922 18:24:31 - mysqld got signal 11 ;
Sep 22 18:24:31 pinba mysqld: This could be because you hit a bug. It
is also possible that this binary
Sep 22 18:24:31 pinba mysqld: or one of the libraries it was linked
against is corrupt, improperly built,
Sep 22 18:24:31 pinba mysqld: or misconfigured. This error can also be
caused by malfunctioning hardware.
Sep 22 18:24:31 pinba mysqld: We will try our best to scrape up some
info that will hopefully help diagnose
Sep 22 18:24:31 pinba mysqld: the problem, but since we have already
crashed, something is definitely wrong
Sep 22 18:24:31 pinba mysqld: and this may fail.
Sep 22 18:24:31 pinba mysqld:
Sep 22 18:24:31 pinba mysqld: key_buffer_size=16777216
Sep 22 18:24:31 pinba mysqld: read_buffer_size=131072
Sep 22 18:24:31 pinba mysqld: max_used_connections=2
Sep 22 18:24:31 pinba mysqld: max_threads=151
Sep 22 18:24:31 pinba mysqld: threads_connected=1
Sep 22 18:24:31 pinba mysqld: It is possible that mysqld could use up
to
Sep 22 18:24:31 pinba mysqld: key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 346514 K
Sep 22 18:24:31 pinba mysqld: bytes of memory
Sep 22 18:24:31 pinba mysqld: Hope that's ok; if not, decrease some
variables in the equation.
Sep 22 18:24:31 pinba mysqld:
Sep 22 18:24:31 pinba mysqld: thd: 0x0
Sep 22 18:24:31 pinba mysqld: Attempting backtrace. You can use the
following information to find out
Sep 22 18:24:31 pinba mysqld: where mysqld died. If you see no
messages after this, something went
Sep 22 18:24:31 pinba mysqld: terribly wrong...
Sep 22 18:24:31 pinba mysqld: stack_bottom = (nil) thread_stack
0x30000
Sep 22 18:24:31 pinba mysqld:
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7fd508c8d829]
Sep 22 18:24:31 pinba mysqld:
/usr/sbin/mysqld(handle_segfault+0x404)[0x7fd508997b14]
Sep 22 18:24:31 pinba mysqld: /lib/libpthread.so.0(+0xef60)
[0x7fd5081f9f60]
Sep 22 18:24:31 pinba mysqld:
/usr/lib/mysql/plugin/libpinba_engine.so(_Z17pinba_merge_poolsv+0x5a5)
[0x7fd503c43475]
Sep 22 18:24:31 pinba mysqld:
/usr/lib/mysql/plugin/libpinba_engine.so(_Z16pinba_stats_mainPv+0x204)
[0x7fd503c42b04]
Sep 22 18:24:31 pinba mysqld: /lib/libpthread.so.0(+0x68ba)
[0x7fd5081f18ba]
Sep 22 18:24:31 pinba mysqld: /lib/libc.so.6(clone+0x6d)
[0x7fd506d3a02d]
Sep 22 18:24:31 pinba mysqld: The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains
Sep 22 18:24:31 pinba mysqld: information that should help you find
out what is causing the crash.

It isn't a big problem for me but I should send a bug report I think ;)

[Antony Dovgal]

On 09/23/2011 11:13 AM, Evgeny Kokovikhin wrote:
> Hello.
>
> I have installed pinba.
>
> Versions:
> dovg@pinba:~$ uname -a
> Linux pinba 2.6.32-5-openvz-amd64 #1 SMP Tue Jun 14 10:46:15 UTC 2011
> x86_64 GNU/Linux
> dovg@pinba:~$ /usr/sbin/mysqld --version
> /usr/sbin/mysqld Ver 5.1.49-3 for debian-linux-gnu on x86_64
> ((Debian)
> Pinba's version is 0.0.6. We made debian package from sources.

Please make sure the MySQL sources used to compile the plugin were configured _exactly the same way_ as the sources used to compile mysqld.
That's critical and any wrong or missing configure option may lead to unpredictable crashes.

If that doesn't help you can also try with a fresh Git snapshot:
http://pinba.org/files/snapshots/pinba_engine-latest.tar.gz

And if that doesn't help, you can compile both MySQL and Pinba with debug and get a gdb backtrace of that crash.
But we'll come to that part when you're done with the other options.

--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime profiling for PHP