pinba engine - mysqld: malloc(): memory corruption
airo
i have small problem =)
today mysql with pinba engine twice has written to the console this
text and hovered
*** glibc detected *** /usr/sbin/mysqld: malloc(): memory corruption:
0x000000000dd881a0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3ad58724ac]
/lib64/libc.so.6(__libc_malloc+0x7a)[0x3ad587402a]
/usr/sbin/mysqld(my_malloc+0x32)[0x838e82]
/usr/sbin/mysqld(handle_connections_sockets+0x369)[0x5be319]
/usr/sbin/mysqld(main+0xc24)[0x5c22f4]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3ad581d974]
/usr/sbin/mysqld[0x50cb59]
======= Memory map: ========
00400000-00b0f000 r-xp 00000000 08:05
576578 /usr/sbin/mysqld
00d0e000-00d70000 rw-p 0070e000 08:05
576578 /usr/sbin/mysqld
00d70000-00d87000 rw-p 00d70000 00:00 0
00f6f000-00ff6000 rw-p 0076f000 08:05
576578 /usr/sbin/mysqld
0d5f3000-0de1a000 rw-p 0d5f3000 00:00 0
401bc000-401bd000 ---p 401bc000 00:00 0
401bd000-40bbd000 rw-p 401bd000 00:00 0
40bbd000-40bbe000 ---p 40bbd000 00:00 0
40bbe000-40bfe000 rw-p 40bbe000 00:00 0
40bfe000-40bff000 ---p 40bfe000 00:00 0
40bff000-40c3f000 rw-p 40bff000 00:00 0
4136f000-41370000 ---p 4136f000 00:00 0
41370000-41d70000 rw-p 41370000 00:00 0
41d70000-41d71000 ---p 41d70000 00:00 0
41d71000-42771000 rw-p 41d71000 00:00 0
42771000-42772000 ---p 42771000 00:00 0
42772000-43172000 rw-p 42772000 00:00 0
43172000-43173000 ---p 43172000 00:00 0
43173000-43b73000 rw-p 43173000 00:00 0
43b73000-43b74000 ---p 43b73000 00:00 0
43b74000-44574000 rw-p 43b74000 00:00 0
44574000-44575000 ---p 44574000 00:00 0
44575000-44f75000 rw-p 44575000 00:00 0
44f75000-44f76000 ---p 44f75000 00:00 0
44f76000-45976000 rw-p 44f76000 00:00 0
45976000-45977000 ---p 45976000 00:00 0
45977000-46377000 rw-p 45977000 00:00 0
46377000-46378000 ---p 46377000 00:00 0
46378000-46d78000 rw-p 46378000 00:00 0
46d78000-46d79000 ---p 46d78000 00:00 0
46d79000-46db9000 rw-p 46d79000 00:00 0
3ad5400000-3ad541c000 r-xp 00000000 08:01
191627 /lib64/ld-2.5.so
3ad561b000-3ad561c000 r--p 0001b000 08:01
191627 /lib64/ld-2.5.so
3ad561c000-3ad561d000 rw-p 0001c000 08:01
191627 /lib64/ld-2.5.so
3ad5800000-3ad594c000 r-xp 00000000 08:01
193322 /lib64/libc-2.5.so
3ad594c000-3ad5b4c000 ---p 0014c000 08:01
193322 /lib64/libc-2.5.so
3ad5b4c000-3ad5b50000 r--p 0014c000 08:01
193322 /lib64/libc-2.5.so
3ad5b50000-3ad5b51000 rw-p 00150000 08:01
193322 /lib64/libc-2.5.so
3ad5b51000-3ad5b56000 rw-p 3ad5b51000 00:00 0
3ad5c00000-3ad5c02000 r-xp 00000000 08:01
193323 /lib64/libdl-2.5.so
3ad5c02000-3ad5e02000 ---p 00002000 08:01
193323 /lib64/libdl-2.5.so
3ad5e02000-3ad5e03000 r--p 00002000 08:01
193323 /lib64/libdl-2.5.so
3ad5e03000-3ad5e04000 rw-p 00003000 08:01
193323 /lib64/libdl-2.5.so
3ad6000000-3ad6082000 r-xp 00000000 08:01
193328 /lib64/libm-2.5.so
3ad6082000-3ad6281000 ---p 00082000 08:01
193328 /lib64/libm-2.5.so
3ad6281000-3ad6282000 r--p 00081000 08:01
193328 /lib64/libm-2.5.so
3ad6282000-3ad6283000 rw-p 00082000 08:01
193328 /lib64/libm-2.5.so
3ad6400000-3ad6416000 r-xp 00000000 08:01
193327 /lib64/libpthread-2.5.so
3ad6416000-3ad6615000 ---p 00016000 08:01
193327 /lib64/libpthread-2.5.so
3ad6615000-3ad6616000 r--p 00015000 08:01
193327 /lib64/libpthread-2.5.so
3ad6616000-3ad6617000 rw-p 00016000 08:01
193327 /lib64/libpthread-2.5.so
3ad6617000-3ad661b000 rw-p 3ad6617000 00:00 0
3ad6800000-3ad6815000 r-xp 00000000 08:01
193337 /lib64/libselinux.so.1
3ad6815000-3ad6a15000 ---p 00015000 08:01
193337 /lib64/libselinux.so.1
3ad6a15000-3ad6a17000 rw-p 00015000 08:01
193337 /lib64/libselinux.so.1
3ad6a17000-3ad6a18000 rw-p 3ad6a17000 00:00 0
3ad6c00000-3ad6c3b000 r-xp 00000000 08:01
193336 /lib64/libsepol.so.1
3ad6c3b000-3ad6e3b000 ---p 0003b000 08:01
193336 /lib64/libsepol.so.1
3ad6e3b000-3ad6e3c000 rw-p 0003b000 08:01 193336
version mysql 5.1.37-community MySQL Community Server - this a last
stable version
pinba engine version pinba_engine-200908271955
any idea?
Antony Dovgal
> Dear Developers,
>
> i have small problem =)
> today mysql with pinba engine twice has written to the console this
> text and hovered
>
> *** glibc detected *** /usr/sbin/mysqld: malloc(): memory corruption:
> 0x000000000dd881a0 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3ad58724ac]
> /lib64/libc.so.6(__libc_malloc+0x7a)[0x3ad587402a]
> /usr/sbin/mysqld(my_malloc+0x32)[0x838e82]
> /usr/sbin/mysqld(handle_connections_sockets+0x369)[0x5be319]
> /usr/sbin/mysqld(main+0xc24)[0x5c22f4]
> /lib64/libc.so.6(__libc_start_main+0xf4)[0x3ad581d974]
> /usr/sbin/mysqld[0x50cb59]
Well, not much data here.
I can only see that something in MySQL causes a memory-related problem.
You can try to debug it this way: build both MySQL and Pinba with --enable-debug (it's --with-debug for MySQL)
and run MySQL with valgrind:
valgrind --tool=memcheck --leak-check=yes --num-callers=30 --show-reachable=yes --log-file=/tmp/mysql.log /usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/var --user=mysql --log-error=/usr/local/mysql/var/mysqld.log --pid-file=/usr/local/mysql/var/thinkpad.pid --socket=/usr/local/mysql/var/mysql.sock --port=3306
(you may need to change the paths)
Send me that /tmp/mysql.log after the crash.
--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime statistics for PHP
Aleksandr Khomyakov
I took the time to make mysql with debug.
Unfortunately the data coming from the php-cgi is not recorded in the table, but module is loaded.
mysql> show plugins;
+------------+--------+----------------+--------------------+---------+
| Name | Status | Type | Library | License |
+------------+--------+----------------+--------------------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MRG_MYISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| PINBA | ACTIVE | STORAGE ENGINE | libpinba_engine.so | GPL |
+------------+--------+----------------+--------------------+---------+
mysql after some time hangs still
I started it with valgrind and last lines of log file By joining letter
ihanick
In my environment there is about 200-500 pinba packets per second, and
very looong script names, for small script names it's works cool!
This is a part off backtrace.
#7 0x00007fab367ac276 in free () from /lib/libc.so.6
No symbol table info available.
#8 0x00007fab33b80c14 in j__udyLFreeJLW (Pjlw=0x5a78, Pop1=<value
optimized out>, Pjpm=0x6) at JudyLMallocIF.c:738
Words = 3
#9 0x00007fab33b80969 in JudyLIns (PPArray=0x233b008, Index=<value
optimized out>, PJError=<value optimized out>) at JudyLIns.c:1802
Pjlwnew = <value optimized out>
pop1 = 1
Pjv = (Pjv_t) 0x238eeb0
Pjvnew = (Pjv_t) 0x233ccc8
Pjpm = <value optimized out>
offset = 0
#10 0x00007fab33bb0a0f in JudySLIns (PPArray=0x233b008, Index=<value
optimized out>, PJError=0x0) at JudySL.c:521
pos = (const uint8_t *) 0x7fab24974ec8 "x.php?
level=48&order=&m_act%5Bgenre%5D%5B%5D=1750&m_act%5Bgenre%5D%5B
%5D=3&m_act%5Bgenre%5D%5B%5D=19&m_act%5Bgenre%5D%5B%5D=xxxxxxxxxxxxxx"
pos2 = (const uint8_t *) 0x236ce28 "x.phtml"
len = 139
len2 = 8
scl2 = 2
indexword = 8659982727342156901
indexword2 = <value optimized out>
PPValue = <value optimized out>
PPValue2 = (PPvoid_t) 0x238eeb0
Pscl = (Pscl_t) 0x0
#11 0x00007fab346fa184 in pinba_update_report5_add (report=0x2311e88,
record=0x7fab24b82fe0) at data.cc:388
index = "a4:/index.php?level=48&order=&m_act%5Bgenre%5D%5B
%5D=1750&m_act%5Bgenre%5D%5B%5D=3&m_act%5Bgenre%5D%5B%5D=19&m_act
%5Bgenre%5D%5B%5D=xxxxxxxxxxxxxx"
data = (pinba_report5_data *) 0x6f6e696b2e777777
ppvalue = (PPvoid_t) 0x0
index_len = 146
dummy = 1936289648
#12 0x00007fab346fb8b8 in pinba_update_reports_add
(record=0x7fab24b82fe0) at data.cc:1250
No locals.
#13 0x00007fab346ff500 in pinba_merge_pools () at pool.cc:450
timer_id = 32683
k = 0
tag_id = 0
temp_pool = (pinba_pool *) 0x2311b60
request_pool = (pinba_pool *) 0x2311b90
request = (class Pinba::Request *) 0x7fab337fa010
tmp_record = (pinba_tmp_stats_record *) 0x7fab337fa010
record = (pinba_stats_record *) 0x7fab24b82fe0
timer = (pinba_timer_record *) 0x0
pos = {request_id = 0, position = 0}
word_ptr = (pinba_word *) 0x0
timer_value = 0
i = 36772704
j = 0
timers_cnt = 0
timer_tag_cnt = 0
timer_hit_cnt = 0
dict_size = 0
tag_value = 914201903
tag_name = 32683
ti = 0
tt = 0
ppvalue = (PPvoid_t) 0x0
word_id = 0
str = (string *) 0x0
tag = (pinba_tag *) 0x0
res = 1075832145
#14 0x00007fab346fe15e in pinba_stats_main (arg=0x0) at pool.cc:498
tv1 = {tv_sec = 0, tv_usec = 9932}
launch = {tv_sec = 1255805534, tv_usec = 886415}
#15 0x00007fab37cce3ba in start_thread () from /lib/libpthread.so.0
and a part of valgrind memcheck log
==23535== Invalid write of size 1
==23535== at 0x4C28504: memcpy (mc_replace_strmem.c:402)
==23535== by 0x8540B05: pinba_update_report4_add(_pinba_report*,
_pinba_stats_record const*) (data.cc:311)
==23535== by 0x8542871: pinba_update_reports_add
(_pinba_stats_record const*) (data.cc:1246)
==23535== by 0x85464FF: pinba_merge_pools() (pool.cc:450)
==23535== by 0x854515D: pinba_stats_main(void*) (pool.cc:498)
==23535== by 0x4E333B9: start_thread (in /lib/libpthread-2.9.so)
==23535== by 0x6350FCC: clone (in /lib/libc-2.9.so)
==23535== Address 0x18540b95 is not stack'd, malloc'd or (recently)
free'd
Antony Dovgal
> Maybe my debug information is more usefull?
> In my environment there is about 200-500 pinba packets per second, and
> very looong script names, for small script names it's works cool!
> This is a part off backtrace.
> ==23535== Invalid write of size 1
> ==23535== at 0x4C28504: memcpy (mc_replace_strmem.c:402)
> ==23535== by 0x8540B05: pinba_update_report4_add(_pinba_report*,
> _pinba_stats_record const*) (data.cc:311)
> ==23535== by 0x8542871: pinba_update_reports_add
> (_pinba_stats_record const*) (data.cc:1246)
> ==23535== by 0x85464FF: pinba_merge_pools() (pool.cc:450)
> ==23535== by 0x854515D: pinba_stats_main(void*) (pool.cc:498)
> ==23535== by 0x4E333B9: start_thread (in /lib/libpthread-2.9.so)
> ==23535== by 0x6350FCC: clone (in /lib/libc-2.9.so)
> ==23535== Address 0x18540b95 is not stack'd, malloc'd or (recently)
> free'd
Unfortunately, this part doesn't make much sense (to me) either..
There is a check right before memcpy() to make sure the data fits the buffer
(and if it doesn't, we truncate it), so it doesn't matter how long your script names are,
Pinba stores only 128 first symbols.
Could you plz send me the full log?
Also what's your version of Judy, your OS and you architecture?
Can you reproduce the crash reliably?
Antony Dovgal
> I have a bug with pinba-engine -git and 0.0.4 revision on amd64, and
> pinba-php-extension 0.0.4
> mysql thread_stack is 131072 bytes.
> backtrace for -git version
Could you check if the following patch fixes it for you?
http://dev.daylessday.org/diff/pinba_memcpy_static.diff
Ihalainen Nickolay
Also, I can see some compilation warnings:
ha_pinba.cc:404: warning: comparison between signed and unsigned
integer expressions
Antony Dovgal
> Thanks, Antony! it completely works.
> Also, I can see some compilation warnings:
> ha_pinba.cc:404: warning: comparison between signed and unsigned
> integer expressions
Yes, I've taken care of those, too.
Thanks again for the report.
Evgeniy Potapov
Just a quick nerd note.
hostname record size, which is defined in pinba_types.h in pinba engine
#define PINBA_HOSTNAME_SIZE 17
is kinda low for some cases
of course it's easy to change the value there (and in extension part
if needed we more than 32 symbols in hostname)
but it's not an everyday thing regular user do.
probably it would be better to increase this limit?
what do you think?
Best Regards,
Evgeniy Potapov
eapo...@gmail.com
Antony Dovgal
> Hi there!
>
> Just a quick nerd note.
>
> hostname record size, which is defined in pinba_types.h in pinba engine
> #define PINBA_HOSTNAME_SIZE 17
> is kinda low for some cases
>
> of course it's easy to change the value there (and in extension part
> if needed we more than 32 symbols in hostname)
IIRC there are no hard-coded limits in the extension, what do you mean?
> but it's not an everyday thing regular user do.
>
> probably it would be better to increase this limit?
> what do you think?
Maybe.
What makes me uncertain it's that you're the first to ask about it,
therefore it doesn't seem to be a problem for others.
But keeping this number low helps to keep memory consumption lower.
Can you explain why did you need to raise this limit?
Evgeniy Potapov
"apps.somethingwithlongname.com"
and for these servers we were needed to collect req_per_sec stats for
each hostname located on them.
while we tried to create few basic reports we've found that we have
"apps.somethingwi"-like values in hostname field because of 16-chars-
limit
after that we've increased the limit in the engine and I wrote this
message.
for me the problem is that some pinba users could meet the same
trouble, and and they could worry about necessity to change the source
code to fix the problem
that's not very serious problem, and there are no real cases now, I
agree, that's why I noted it's a nerd note ;)
Alexey A. Rybak
> therefore it doesn't seem to be a problem for others.
> But keeping this number low helps to keep memory consumption lower.
correct me if I'm wrong but I know the only one reason when it really
matters - if you have static buffer for this ;)
>
> Can you explain why did you need to raise this limit?
From the user point of view 64 should be quite OK for the hostname. To
be limited with 17 is a very big surprise. Why not 15? or 13?
--
wbr,
fisher
Antony Dovgal
>> What makes me uncertain it's that you're the first to ask about it,
>> therefore it doesn't seem to be a problem for others.
>> But keeping this number low helps to keep memory consumption lower.
>
> correct me if I'm wrong but I know the only one reason when it really
> matters - if you have static buffer for this ;)
Sure, it's static and it's used for each & every record, so +15 bytes to this buffer
means increasing overall memory consumption on pinba_request_pool_size*15 bytes (18Mb in our particular case).
>> Can you explain why did you need to raise this limit?
>
>>From the user point of view 64 should be quite OK for the hostname. To
> be limited with 17 is a very big surprise. Why not 15? or 13?
It's as easy as 16 (a good round number, yes) + 1 for the null terminator =)
Nikolay M. Didenko
>> Just a quick nerd note.
>>
>> hostname record size, which is defined in pinba_types.h in pinba engine
>> #define PINBA_HOSTNAME_SIZE 17
>> is kinda low for some cases
>>
>> of course it's easy to change the value there (and in extension part
>> if needed we more than 32 symbols in hostname)
>
> IIRC there are no hard-coded limits in the extension, what do you mean?
>
>> but it's not an everyday thing regular user do.
>>
>> probably it would be better to increase this limit?
>> what do you think?
>
> Maybe.
> What makes me uncertain it's that you're the first to ask about it,
> therefore it doesn't seem to be a problem for others.
> But keeping this number low helps to keep memory consumption lower.
>
> Can you explain why did you need to raise this limit?
>
It would be great to have a method to change the hostname, like this:
pinba_hostname_set('bla-bla-bla').
This method suppose to change the hostname on the right one or on the
readable alias.
--
Nikolay Didenko
Antony Dovgal
> It would be great to have a method to change the hostname, like this:
> pinba_hostname_set('bla-bla-bla').
> This method suppose to change the hostname on the right one or on the
> readable alias.
Added, feel free to test the snapshot.
Diff: http://github.com/tony2001/pinba_extension/commit/20a01074f4046e27a902b0bfd7800210debf5063
Snapshot: http://pinba.org/files/snapshots/pinba_extension-latest.tar.gz
The hostname is now reset to the result of gethostname() on each request startup,
so that custom hostname set in the current request won't affect the following ones.
Nikolay M. Didenko
> On 13.11.2009 16:59, Nikolay M. Didenko wrote:
>> It would be great to have a method to change the hostname, like this:
>> pinba_hostname_set('bla-bla-bla').
>> This method suppose to change the hostname on the right one or on the
>> readable alias.
>
> Added, feel free to test the snapshot.
>
> Diff: http://github.com/tony2001/pinba_extension/commit/20a01074f4046e27a902b0bfd7800210debf5063
> Snapshot: http://pinba.org/files/snapshots/pinba_extension-latest.tar.gz
>
> The hostname is now reset to the result of gethostname() on each request startup,
> so that custom hostname set in the current request won't affect the following ones.
Thank you!
--
Best regards,
Nikolay Didenko.