[Wala-wala] GC overhead limit exceeded when frequently invoking getCallsiteSignature

[Parjanya Vyas]

Hi all,

I apologize in advance if this question has been already addressed or is too naive. When using class hierarchy analysis with a large code base, I encounter the 'GC overhead limit exceeded' exception that stems from MethodReference::getCallsiteSignature method when it is invoked frequently.

The exception only occurs when analyzing a large analysis scope that consists of multiple large dex files. There is no fixed point that I could observe when this exception is triggered.

Can someone please help me out with this? Any guidance will be highly appreciated! I can provide more details on the analysis data/processing that I am attempting, if needed.

Thanks,

Parjanya Vyas

[Raffi T Khatchadourian]

Parjanya,

What is the value that you are giving to -Xmx when you run java?

_______________________________________________

Wala-wala mailing list

Wala...@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/wala-wala

[Parjanya Vyas]

Hi Raffi,

Thanks very much for your response.

I use '12000m' with -Xmx right now.

Thanks,

Parjanya

---

From: Raffi T Khatchadourian <raffi.khat...@hunter.cuny.edu>
Sent: Thursday, September 1, 2022 10:44 AM
To: wala...@lists.sourceforge.net <wala...@lists.sourceforge.net>
Subject: Re: [Wala-wala] GC overhead limit exceeded when frequently invoking getCallsiteSignature

 

[Manu Sridharan]

Hi Parjanya, are you saying that WALA is throwing an exception, or that there is an error from the JVM? If it’s the former can you link to the line of code throwing the exception and paste a stack trace excerpt?

Thanks,

Manu 

--

Sent from my phone. Please pardon brevity

[Manu Sridharan]

After Googling I see the error message is likely due to an OutOfMemoryError.  WALA class hierarchy construction has been battle-tested with large classpaths of jars, but I haven’t done much with large dex files.  It’s possible there is a memory leak somewhere.  I’d say the simplest thing to try for you is to increase the max heap size, running on a machine with more RAM if possible; do you still get an OOM with 24G of heap?

If you do still get an OOM with more heap, we can try to dig deeper.  Out of curiosity, what is the total size of the dex files in your class hierarchy?  And are you just trying to build a class hierarchy, or also a call graph?

--Manu 

[Parjanya Vyas]

Hi Manu,

Thanks a lot for your response.

It is the JVM that throws the 'GC overhead limit exceeded' exception that stems from a WALA method.

Here is the stack trace:

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

      at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68)

      at java.lang.StringBuilder.<init>(StringBuilder.java:89)

      at com.ibm.wala.types.TypeName$TypeNameKey.toUnicodeString(TypeName.java:319)

      at com.ibm.wala.types.TypeName.toUnicodeString(TypeName.java:123)

      at com.ibm.wala.types.Descriptor$Key.appendSemicolonIfNeeded(Descriptor.java:227)

      at com.ibm.wala.types.Descriptor$Key.toString(Descriptor.java:200)

      at com.ibm.wala.types.Descriptor.toString(Descriptor.java:109)

      at java.lang.String.valueOf(String.java:2994)

      at java.lang.StringBuilder.append(StringBuilder.java:131)

      at com.ibm.wala.types.MethodReference.getSignature(MethodReference.java:223)

My guess is that the issue is due to the method 'UTF8Convert::fromUTF8' allocating a new Array every time it needs to construct the string. Rapid repeated invocation of this method can probably result in a frequent allocation/deallocation of the array resulting in the GC being invoked very frequently. However, I might be wrong.

For your queries:

1. The total size of all the dex files included is 87.6 MB
2. No, I am not constructing a callgraph, only the class hierarchy.

Note that creating the class hierarchy is not an issue. The exception occurs only when I am analyzing the constructed class hierarchy during which I invoke the 'CallSiteReference.getDeclaredTarget.getSignature' repeatedly for different methods.

Thanks,
Parjanya

---

From: Manu Sridharan <ma...@sridharan.net>
Sent: Friday, September 2, 2022 11:25 AM
To: WALA discussion and Q&A <wala...@lists.sourceforge.net>

[Manu Sridharan]

Hi Parjanya,

An OutOfMemoryError occurs when the GC is unable to collect enough memory to allow for future allocations.  This does not occur due to frequent allocation; it happens because too much memory is being retained by the program.  Also, the stack trace associated with an OutOfMemoryError usually does not give good guidance as to what is causing the problem.

Based on what you wrote below, I suspect the issue here is within your own code.  Do you have a large List or Map for caching analysis results that keeps growing?  That’s often the root cause of a leak.  When I want to debug this sort of issue I use the Eclipse MAT tool to analyze a heap snapshot:

https://www.eclipse.org/mat/

To collect a heap snapshot when the program runs out of memory, pass the -XX:+HeapDumpOnOutOfMemoryError parameter to the JVM.  I would test this with a much smaller heap than 12000m, since the heap dump will be roughly the size of the heap.  Once you open the heap snapshot in MAT, you should be able to see what data structure is consuming a lot of memory.

Thanks,

Manu 

[Parjanya Vyas]

Hi Manu,

Thank you very much for your response.

Sure, I will try using that particular tool to check if I can find a memory leak within my code.

However, during multiple runs of the code the stack trace always points to the getSignature method whenever this exception is thrown.

Moreover, the below StackOverflow answer states that this exception with 'GC overhead limit exceeded' message is thrown when GC is running excessively without being able to claim enough memory:

https://stackoverflow.com/questions/1393486/error-java-lang-outofmemoryerror-gc-overhead-limit-exceeded

Do you think there might be a chance the issue is indeed within WALA code?

Thanks,
Parjanya

---

From: Manu Sridharan <ma...@sridharan.net>
Sent: Friday, September 2, 2022 2:33:36 PM

[Manu Sridharan]

It’s possible something is wrong in WALA, but I would want to see more evidence from a tool like MAT before digging more.

--Manu

[Parjanya Vyas]

Sure, that makes sense.

I will get back with the memory trace.

Thanks a lot!

Parjanya

---

From: Manu Sridharan <ma...@sridharan.net>
Sent: Friday, September 2, 2022 3:35:36 PM