Can Percona Server have thread memory usage statistics?

[Lixun Peng]

Hi Guys,

I always found my MySQL use many memory, even though Buffer Pool is small. So I think Server used many memory.

I developed a demo patch (see picture), it can display how many memory used of threads. 

Expect this, I think how many sort_buffer/join_buffer/net_buffer memory used of threads are also need display, it will let DBAs know the memory cost detail.

Do you think this function is very useful?

--
Best Wishes

Lixun Peng (P.Linux)

Mobile Phone:
+86 18658156856 (Hangzhou)
Gtalk: penglixun(at)gmail.com
Twitter: http://www.twitter.com/plinux
Blog: http://www.penglixun.com

[Lixun Peng]

by the way, the demo patch is based on MySQL 5.6.6-labs.

[Laurynas Biveinis]

Lixun -

The feature seems to be interesting at least to me personally. The
patch is based on 5.6 however and currently we don't have a code base
to integrate to. Feel free to open a wishlist bug in Launchpad for
Percona Server with the patch attached.

Thank you,
Laurynas

> --
> You received this message because you are subscribed to the Google Groups
> "Percona Discussion" group.
> To post to this group, send email to percona-d...@googlegroups.com.
> To unsubscribe from this group, send email to
> percona-discuss...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/percona-discussion?hl=en.



--
Laurynas
www.percona.com

[Eric Bergen]

That is very cool. Do you have a way to roll up all the memory usage
and track a high water mark?

Eric Bergen
eric....@gmail.com
http://www.ebergen.net

[ser...@pisem.net]

Hi, Laurynas!

On Apr 27, Laurynas Biveinis wrote:
> Lixun -
>
> The feature seems to be interesting at least to me personally. The
> patch is based on 5.6 however and currently we don't have a code base
> to integrate to. Feel free to open a wishlist bug in Launchpad for
> Percona Server with the patch attached.

The patch is small and can be trivially applied to any version, even to
5.1.

But how portable this malloc_usable_size() is?
I've never heard about it before. It doesn't have a manpage (on a couple
of linux boxes that I've tried, although google shows quite a few
manpages for it). But google also shows discussion mails about removing
malloc_usable_size() from somewhere and "don't use" elsewhere.

Regards,
Sergei

[Lixun Peng]

Hi Sergei,

Because I don't know how to get a pointor own how many memory, I remember have a function "_msize" at Windows, then I found the  “malloc_usable_size” at Linux, the usage  like “_msize”.

If have any other simple way to know a pointor own how many memory, I will change this code.

Thanks,

Lixun

2012/4/27 <ser...@pisem.net>

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To post to this group, send email to percona-d...@googlegroups.com.
To unsubscribe from this group, send email to percona-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/percona-discussion?hl=en.

[ser...@pisem.net]

Hi, Lixun!

On Apr 27, Lixun Peng wrote:
> Hi Sergei,
>
> Because I don't know how to get a pointor own how many memory, I
> remember have a function "_msize" at Windows, then I found the
> “malloc_usable_size” at Linux, the usage like “_msize”.
> If have any other simple way to know a pointor own how many memory, I
> will change this code.

I don't :)

I've only found malloc_usable_size and _msize - and only after
seeing your patch.

We (*) certainly want to track memory usage per connection and globally
in the server. I thought that the only way of doing it would be to store
the size directly in the allocated block.

But now I'll check whether malloc_usable_size/_msize are good enough.
So, I'm really interested to know how portable they are.
FreeBSD ? Mac OS X? Solaris?

Regards,
Sergei

(*) We means MariaDB, not Percona :)

[Lixun Peng]

Hi Sergei,

I only have Linux and Mac OS X environment.

I test on GCC 4.1 and GCC 4.2, GCC 4.1 have  malloc_usable_size   and GCC 4.3 have  malloc_size.

Which  malloc_usable_size can run on Linux, and  malloc_size can run on Mac OS X.

I think we can use macro  to select  malloc_usable_size or  malloc_size.

Before I found  malloc_usable_size/ malloc_size, I tested a method that by allocate space to store size.

Like "point=malloc(size+sizeof(size_t)); memcpy(point, &size, sizeof(size_t)); return print+sizeof(size_t);" and "memcpy(&size, point, sizeof(size_t)); free(point-sizeof(size_t))",

but it run test case fail, so I give up this way.

Because all of my OS are Linux, so I use "malloc_usable_size/ malloc_size " way.

I think you can try the way above. I have no enough time to DEBUG, because I'm a DBA, I only can try above way if I have enough  time. 

2012/4/27 <ser...@pisem.net>

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To post to this group, send email to percona-d...@googlegroups.com.
To unsubscribe from this group, send email to percona-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/percona-discussion?hl=en.

[ser...@pisem.net]

Hi, Lixun!

On Apr 27, Lixun Peng wrote:
> Hi Sergei,
>

> I only have Linux and Mac OS X environment. I test on GCC 4.1 and GCC
> 4.2, GCC 4.1 have malloc_usable_size and GCC 4.3 have malloc_size.
> Which malloc_usable_size can run on Linux, and malloc_size can run
> on Mac OS X.
> I think we can use macro to select malloc_usable_size or malloc_size.

Certainly.

> Before I found malloc_usable_size/ malloc_size, I tested a method
> that by allocate space to store size.
> Like "point=malloc(size+sizeof(size_t)); memcpy(point, &size,
> sizeof(size_t)); return print+sizeof(size_t);" and "memcpy(&size, point,
> sizeof(size_t)); free(point-sizeof(size_t))",
> but it run test case fail, so I give up this way.

Yes, that's what I meant. I wonder why the tests fail for you.
But never mind, it's not really important, if malloc_usable_size is
sufficient.

> Because all of my OS are Linux, so I use "malloc_usable_size/
> malloc_size " way.
> I think you can try the way above. I have no enough time to DEBUG,
> because I'm a DBA, I only can try above way if I have enough time.

Well, I think you've done more than enough already :)
I'll see if we can get it into MariaDB.
But I don't know what are Percona's criteria for including your patch.

Regards,
Sergei

[Lixun Peng]

Hi Sergei,

I found this macro can run on most of OS.

#if defined(__APPLE__)

#define _msize(p)      malloc_size(p)

#elif defined(__sun)

#define _msize(p)      (*((size_t*)(p)-1))

#elif !defined(_WIN32)

#define _msize(p)      malloc_usable_size(p) 

#endif

I test on OSX/Linux 2.6/Linux 3.0/FreeBSD/Solaris, use "_msize" instead of "malloc_usable_size", all OS can run well.

I update the patch yesterday, add monitor function on realloc, and only init callback function once.

2012/4/27 <ser...@pisem.net>

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To post to this group, send email to percona-d...@googlegroups.com.
To unsubscribe from this group, send email to percona-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/percona-discussion?hl=en.

[Vadim Tkachenko]

Lixun,

It definitely looks interesting.
Most interesting is this is a short patch.

We will take look if we can include it in Percona Server 5.5, or it is
going to be 5.6

Thanks,
Vadim

Vadim Tkachenko, CTO, Percona Inc.
Phone +1-925-400-7377,  Skype: vadimtk153
Schedule meeting: http://tungle.me/VadimTkachenko

Looking for Replication with Data Consistency?
Try Percona XtraDB Cluster!

[Lixun Peng]

Hi Vadim,

I have tried to merge to Percona 5.5.22-25.2, it's ok.

it test by myself, on OSX/Linux.

2012/4/28 Vadim Tkachenko <va...@percona.com>

[Laurynas Biveinis]

Lixun -

Thanks. I have created a request at
https://bugs.launchpad.net/percona-server/+bug/991651
to track this.

Laurynas
www.percona.com