How can I know which objects are moved from young generation to old generation?

[Jie Song]

We are developing a time-series database in Java, which has a big heap (tens of GB). We are trying to leak short-life objects from young gen to old gen as less as possible (short-life in my context means objects whose life is about several minutes - long enough to escape the minor GC and will be accumulated in old gen, and eventually trigger a full gc). So far, the leaking rate is about 140K/minute. 

Since the heap is huge, so a typical approach (e.g. create 2 heap dumps, load them into yourkit to do comparison) seems not work for us. Also, to compare histograms generated by jmap is very tedious.

I'm wondering if there is a tool allowing me to know which objects are promoted from young gen to old gen?

best regards

-Jie

[Aleksey Shipilev]

On 12/18/2014 09:01 PM, 'Jie Song' via mechanical-sympathy wrote:
> We are developing a time-series database in Java, which has a big heap
> (tens of GB). We are trying to leak short-life objects from young gen to
> old gen as less as possible (short-life in my context means objects
> whose life is about several minutes - long enough to escape the minor GC
> and will be accumulated in old gen, and eventually trigger a full gc).

> So far, the leaking rate is about 50K/sec.

>
> Since the heap is huge, so a typical approach (e.g. create 2 heap dumps,
> load them into yourkit to do comparison) seems not work for us. Also, to
> compare histograms generated by jmap is very tedious.
>
> I'm wondering if there is a tool allowing me to know which objects are
> promoted from young gen to old gen?

Well, if you know the generation boundaries, you can probably use that
knowledge. Demangle the object references, and see in which part of the
memory they belong. -XX:+PrintGCDetails prints the generation boundaries
at the end of the run.

Or, you may track the suspicious objects and see when their addresses
stop changing after young GCs (this applies for
young-scavenge/old-mark-sweep collectors). JOL [1] has an example [2]
for tracking a single object. I know this thing might not scale for the
entire heap, as it produces garbage on its own.

Or, you can "just" hack the promotion method in a relevant OpenJDK GC,
and print the bastards out. I think for ParallelGC,
PSPromotionManager::copy_to_survivor_space after "// Otherwise try
allocating obj tenured" is the place.

Thanks,
-Aleksey.

[1] http://openjdk.java.net/projects/code-tools/jol/
[2]
http://hg.openjdk.java.net/code-tools/jol/file/8f820e6eb250/jol-samples/src/main/java/org/openjdk/jol/samples/JOLSample_19_Promotion.java

[Aleksey Shipilev]

On 12/19/2014 01:30 AM, Aleksey Shipilev wrote:
> Or, you can "just" hack the promotion method in a relevant OpenJDK GC,
> and print the bastards out. I think for ParallelGC,
> PSPromotionManager::copy_to_survivor_space after "// Otherwise try
> allocating obj tenured" is the place.

Yeah, having fun with it:

diff -r fb4ba04c587b
src/share/vm/gc_implementation/parallelScavenge/psPromotionManager.inline.hpp
---
a/src/share/vm/gc_implementation/parallelScavenge/psPromotionManager.inline.hpp
Thu Dec 11 23:06:14 2014 -0800
+++
b/src/share/vm/gc_implementation/parallelScavenge/psPromotionManager.inline.hpp
Fri Dec 19 02:30:49 2014 +0300
@@ -117,6 +117,8 @@

// Otherwise try allocating obj tenured

if (new_obj == NULL) {
+ tty->print("%d, %s, promoted\n", o->size(),
o->klass()->name()->as_klass_external_name());
+
#ifndef PRODUCT
if (Universe::heap()->promotion_should_fail()) {
return oop_promotion_failed(o, test_mark);

...and then:

$ java ... | grep promoted | awk '{ arr[$2]+=$1 } END { for (key in arr)
printf("%s\t%s\n", arr[key], key) }' | sort -n
...
187230 org.openjdk.jol.info.ClassData,
192100 org.openjdk.jol.info.FieldData,
220359 java.util.ArrayList,
371910 [Ljava.lang.Object;,
720671 java.util.HashMap$TreeNode,
863172 [Ljava.util.HashMap$Node;,
968862 java.lang.String,
2122970 [B,
2252487 java.lang.Long,
2948636 java.util.HashMap$Node,
4582206 [C,

More introspection is possible, especially if the promotion rate is low.

-Aleksey.

[Jie Song]

Hi Aleksey

I'll try your approach and report back the result.

thanks

-Jie

[Gil Tene]

I may be missing the point. 

1. If you want to reduce "leakage" into oldgen of objects that will die soon after, the object type doesn't matter. You just need to keep them in newgen until the die. You can increase the time they have in newgen (for a given application throughput) by increasing the newgen size or the promotion threshold (or both). It's trivial to double or triple the time you have before promotion.

2. If you want to reduce "leakage" into oldgen of objects that will stay there for a very long time (which would seem common in a time-series DB if it's keeping in-memory stuff), you are probably out of luck. But if you want to know what those objects are, just profile the heap after you've been running for a long time. The vast majority of live objects will be those in the oldgen, and their types will be evident.

3. The exception to #1 is arrays large enough to be directly allocated into oldgen. But you'd know if you had those. And they wouldn't be appearing at only 140KB/min.

<Gil-bragging-warning>

Of course, you could just switch to a collector where this sort wasted engineering brainpower isn't required, and enter a much nicer world. A world where "slow leaks" into oldgen are about as painful as blinking. Where time-series things, cache things, and moving window in-memory things aren't shunned, told they are doing something wrong, or preached about their business problem being a "strange use case". Where allocating at 4GB/sec and promoting at 200MB/sec is not a big deal, doesn't hurt anyone, and is nothing to worry about or waste any time on. In that world you can even tell what the velocities of the various types through the heap are, for free. But people who already live there generally don't care enough to look into the promoted objects type mix. They like spending quality time in their hammocks and thinking deep thoughts instead.

</Gil-bragging-warning>

On Thursday, December 18, 2014 10:01:30 AM UTC-8, Jie Song wrote:

We are developing a time-series database in Java, which has a big heap (tens of GB). We are trying to leak short-life objects from young gen to old gen as less as possible (short-life in my context means objects whose life is about several minutes - long enough to escape the minor GC and will be accumulated in old gen, and eventually trigger a full gc). So far, the leaking rate is about 140K/minute. 

Since the heap is huge, so a typical approach (e.g. create 2 heap dumps, load them into yourkit to do comparison) seems not work for us. Also, to compare histograms generated by jmap is very tedious.

I'm wondering if there is a tool allowing me to know which objects are promoted from young gen to old gen?

best regards

-Jie

[Jie Song]

Hi Gil

Thanks for your response. 

For your point 1, I'm wondering there are objects whose life is about tens of minutes (not very long, but long enough to escape from a really high tenuring threshold settings).  we already tune the young gen tenuring threshold and increase its size so no object leakage to oldGen because of no-enough-youngGen-space. But it is a two-edged sword. A bigger tenuring threshold and young gen size will punish us with longer stop-the-world pause (it's about 150ms every minute right now). 

For point 2, if i have 100GB heap, are there tools that can just give me histograms of all objects (garbage and live) in oldGen? Jmap profiles all objects youngGen and oldGen. so if snapshot1 created by jmap tell you there are 100 Foo objects, and snapshot2 tell you there are 250 Foo objects, you can't conclude that 150 Foo objects leaked to oldGen because those 150 object might be short-lived objects in youngGen. 

Another reason I want to figure this out is curiosity - what other techniques I can use to answer this question besides comparing two jmap snapshots.

best regards

-Jie

[Kirk Pepperdine]

Hi Jie,

Gil’s answer is spot on…. even the shameless self promotion ;-).. but if still can’t make things work then the best thing to do is mark your application more memory efficient… Memory profile to reduce creation rates even more and improve information density in your data structures.

Regards,

Kirk

On Dec 19, 2014, at 7:01 AM, 'Jie Song' via mechanical-sympathy <mechanica...@googlegroups.com> wrote:

Hi Gil

Thanks for your response. 

For your point 1, I'm wondering there are objects whose life is about tens of minutes (not very long, but long enough to escape from a really high tenuring threshold settings).  we already tune the young gen tenuring threshold and increase its size so no object leakage to oldGen because of no-enough-youngGen-space. But it is a two-edged sword. A bigger tenuring threshold and young gen size will punish us with longer stop-the-world pause (it's about 150ms every minute right now). 

Another reason I want to figure this out is curiosity - what other techniques I can use to answer this question besides comparing two jmap snapshots.

best regards

-Jie

On Thursday, December 18, 2014 8:54:38 PM UTC-8, Gil Tene wrote:

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.