See memory of table with attributes

[Mike McLeod]

Is there a way to determine how much memory a table with attributes occupies?

So here's an example of what I'm talking about:

I start off with the following memory:

q).Q.w[]

used| 119680

heap| 67108864

peak| 67108864

wmax| 0

mmap| 0

mphy| 9419714560

syms| 574

symw| 18675

I then create a table with the partition attribute:

q)t:([]a:`p#til 1000000;b:1000000?0 1 2)

I then have the following memory profile:

q).Q.w[]

used| 75617312

heap| 201326592

peak| 201326592

wmax| 0

mmap| 0

mphy| 9419714560

syms| 574

symw| 18675

So there's been an increase of 75497632 bytes. However the function -22! returns a much smaller amount:

q)-22!t

16000039

Is there a function similar to -22! that would return the full amount of memory t (and overhead related to t )takes up in memory? I know that I could delete t from the global namespace and check .Q.w before and after, but I'm hoping for a way to see the memory without destroying t. 

Any help is greatly appreciated!

Thanks!

[RAHUL ASATI]

Hi Mike,

-22! just gives the memory used by values of t and does not include memory usage by attributes. For your table you can calculate the same value as below:

total integer values=2000000

total memory=8*2000000=16000000

And there will be some overhead to store table info.

Not sure if there is any function to get full memory info but one way is to store the table on disk as splayed table and check the memory usage of it either directly from disk or by mapping the table to service (mmap value in .Q.w[]). That will give you total memory usage including memory used by attributes data structure.

For this, you can create your own function which will save table on disk, get the memory usage and deletes the table from disk.

[Jack Andrews]

Would count[t]#t also copy the attributes?

If so, monitoring memory usage before and after a # would give you some insight. 

On Wednesday, November 25, 2015, Matthew McAuley <matthewmc...@gmail.com> wrote:

Hi Mike, 

It is a good suggestion of saving the table to disk, recording the memory usage, then deleting from disk. However a fundamental problem with this approach is that it doesn't take into account the memory overhead of holding a kdb object on memory, so normally you end up with under-estimated result. 

My initial thoughts were to define a function to note the used memory before, copy the table to a new tTemp variable, calculate the difference in memory, then delete the temporary variable (tTemp deletes after function execution, so no need to do this step explicitly):

f: { memBefore: .Q.w[][`used] ; tTemp:t; memChange: .Q.w[][`used] - memBefore ; memChange }

f[] returns 6

Unfortunately that isn't a working solution, but does give a little clarity on what kdb+ does when a table is copied. We see little change in the memory usage because instead of storing the copied table in memory, the tTemp simply points to your original table.


One way to force a new instance of the table in memory is to just recreate your original table creation: t:([]a:`p#til 1000000;b:1000000?0 1 2)

f: { memBefore: .Q.w[][`used] ; t2::([]a:`p#til 1000000;b:1000000?0 1 2); memChange: .Q.w[][`used] - memBefore; memChange }

f[] returns 75497568

This requires you to know the original table creation function so that it can be recreated and stored as t2. If you are working with a manipulated table and cannot simply recreate it, then this solution isn't appropriate.


Thanks,
Matthew

KDB+ Developer, AquaQ Analytics Ltd

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To post to this group, send email to personal...@googlegroups.com.
Visit this group at http://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

[Matthew McAuley]

Thinking about it again, it requires a slightly more involved solution. A better function is:

f: { memBefore: .Q.w[][`used] ; t1:{@[x;y;#[z]]}/[1 _ 0!x;key attributes;attributes:exec last a by c from meta x where not null a]; memChange: .Q.w[][`used] - memBefore; memChange }

t:([]a:`p#til 1000000;b:1000000?0 1 2)

f t

75497664

This function will create a full, temporary copy of the table in memory.


Memory is allocated in blocks of 2^, increasing the table size to 1200000 will dramatically increase the size in memory. Comparing here on a table with and without attributes.

q)f:{memBefore: .Q.w[][`used] ; t1:{@[x;y;#[z]]}/[1 _ 0!x;key attributes;attributes:exec last a by c from meta x where not null a]; memChange: .Q.w[][`used] - memBefore; memChange }

q)t:([]a:til 1200000;b:1200000?0 1 2)

q)-22!t

19200039

q)f t

33554592

q)update `p#a from `t

`t

q)-22!t

19200039

q)f t

150995136

Thanks,
Matthew

To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbplus+unsubscribe@googlegroups.com.
To post to this group, send email to personal-kdbplus@googlegroups.com.

[Charles Skelton]

for a vector with no attr, vector capacity is power of 2
`long$2 xexp ceiling 2 xlog 16+elementSize*count

e.g. til 100000000
q)`long$2 xexp ceiling 2 xlog 16+8*100000000
1073741824

see section 42 of http://kx.com/q/d/q.htm for info on attr space usage.

42 Attributes

example overhead
`s#2 2 3 sorted 0
`u#2 4 5 unique 16*u
`p#2 2 1 parted (4*u;16*u;4*u+1)
`g#2 1 2 grouped (4*u;16*u;4*u+1;4*n)
The byte overheads use n(number of elements) and u(number of uniques).

Those attr usage numbers need updating for v3.x. I think they would be
`u#2 4 5 unique 32*u
`p#2 2 1 parted (8*u;32*u;8*u+1)
`g#2 1 2 grouped (8*u;32*u;8*u+1;8*n)

So for your example
q){`long$2 xexp ceiling 2 xlog 16+(8*count x)+sum 8 32 8*count distinct x}t`a
67108864
q){`long$2 xexp ceiling 2 xlog 16+8*count x)}t`b
8388608
q)67108864+8388608
75497472

if you're targeting the general case, ref counting allows same object to be present in multiple containers; observe ref count with -16!x.

This should be sufficient information as a basis for someone to cook up a function that works for any object. If you do, please share it with the community.

--

You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbpl...@googlegroups.com.
To post to this group, send email to personal...@googlegroups.com.

[WooiKent Lee@AquaQ]

Hi all,

I've been working on this functionality for our TorQ framework and wrote a blog about this problem:

http://www.aquaq.co.uk/q/adventure-in-retrieving-memory-size-of-kdb-object/

It has been quite enjoyable working on this and I'll be excited to see what you guys think about it :)

Best,

WooiKent Lee

Financial Software Developer

AQUAQ Analytics

"This email, its contents and any files attached are a confidential communication and are intended only for the named addressees indicated in the message. 

If you are not the named addressee or if you have received this email in error, you may not, without the consent of AquaQ Analytics, copy, use or rely on any information or attachments in any way. Please notify the sender by return email and delete it from your email system. 

Unless separately agreed, AquaQ Analytics does not accept any responsibility for the accuracy or completeness of the contents of this email or its attachments. Please note that any views, opinion or advice contained in this communication are those of the sending individual and not those of AquaQ Analytics and AquaQ Analytics shall have no liability whatsoever in relation to this communication (or its content) unless separately agreed"