Instrumenting error caused by: java.lang.ArrayIndexOutOfBoundsException: -1

[Tool Ply]

I tried to instrument a jar which was translated from an apk by using dex2jar. Then the error arised.

There are some details about command and exception.

java -jar /home/ren/MyProject/tcg-android/jacoco-android/org.jacoco.cli/target/org.jacoco.cli-0.8.1-SNAPSHOT-nodeps.jar instrument mobileqq_android-dex2jar.jar —dest operation_pool/

Exception in thread "main" java.io.IOException: Error while instrumenting /home/ren/.local/share/Trash/files/jacoco-android/instrument/mobileqq_andro...@abap.class.

  at org.jacoco.cli.internal.core.instr.Instrumenter.instrumentError(Instrumenter.java:174)

  at org.jacoco.cli.internal.core.instr.Instrumenter.instrument(Instrumenter.java:123)

  at org.jacoco.cli.internal.core.instr.Instrumenter.instrument(Instrumenter.java:148)

  at org.jacoco.cli.internal.core.instr.Instrumenter.instrument(Instrumenter.java:168)

  at org.jacoco.cli.internal.core.instr.Instrumenter.instrumentAll(Instrumenter.java:207)

  at org.jacoco.cli.internal.core.instr.Instrumenter.instrumentZip(Instrumenter.java:235)

  at org.jacoco.cli.internal.core.instr.Instrumenter.instrumentAll(Instrumenter.java:210)

  at org.jacoco.cli.internal.commands.Instrument.instrument(Instrument.java:88)

  at org.jacoco.cli.internal.commands.Instrument.execute(Instrument.java:58)

  at org.jacoco.cli.internal.Main.execute(Main.java:89)

  at org.jacoco.cli.internal.Main.main(Main.java:104)

Caused by: java.lang.ArrayIndexOutOfBoundsException: -1

  at org.jacoco.cli.internal.asm.Frame.merge(Frame.java:1415)

  at org.jacoco.cli.internal.asm.CurrentFrame.execute(CurrentFrame.java:52)

  at org.jacoco.cli.internal.asm.MethodWriter.visitVarInsn(MethodWriter.java:781)

  at org.jacoco.cli.internal.asm.ClassReader.readCode(ClassReader.java:1472)

  at org.jacoco.cli.internal.asm.ClassReader.readMethod(ClassReader.java:1126)

  at org.jacoco.cli.internal.asm.ClassReader.accept(ClassReader.java:698)

  at org.jacoco.cli.internal.asm.ClassReader.accept(ClassReader.java:500)

  at org.jacoco.cli.internal.asm.ClassWriter.toByteArray(ClassWriter.java:1042)

  at org.jacoco.cli.internal.core.instr.Instrumenter.instrument(Instrumenter.java:102)

  at org.jacoco.cli.internal.core.instr.Instrumenter.instrument(Instrumenter.java:121)

  ... 9 more

ps: 

    both the newest code from github and release version 0.8.0 arise this exception.

[Evgeny Mandrikov]

Please provide "abap.class".

On Wednesday, March 21, 2018 at 6:03:53 PM UTC+1, Tool Ply wrote:

I tried to instrument a jar which was translated from an apk by using dex2jar. Then the error arised.

There are some details about command and exception.

java -jar /home/ren/MyProject/tcg-android/jacoco-android/org.jacoco.cli/target/org.jacoco.cli-0.8.1-SNAPSHOT-nodeps.jar instrument mobileqq_android-dex2jar.jar —dest operation_pool/

Exception in thread "main" java.io.IOException: Error while instrumenting /home/ren/.local/share/Trash/files/jacoco-android/instrument/mobileqq_android-dex2j...@abap.class.

[Tool Ply]

在 2018年3月22日星期四 UTC+8上午1:56:59，Evgeny Mandrikov写道：

[Evgeny Mandrikov]

$ javap -v -p aaqc.class | grep "major version"

  major version: 50

i.e. contains Java 1.6 bytecode.

http://www.jacoco.org/jacoco/trunk/doc/faq.html clearly states : class files version 1.6 and above have to contain valid stackmap frames

$ javap -v -p aaqc.class | grep StackMapTable | wc -l

0

i.e. there is no stackmap frames in aaqc.class 

If other tools create invalid class files JaCoCo cannot be used on those files.

[Tool Ply]

Ok，thank you

在 2018年3月23日星期五 UTC+8上午1:21:33，Evgeny Mandrikov写道：

[Tool Ply]

There is another .class file that failed jacoco 0.8.0. The interesting is that the major version is 49 which means Java 1.5 bytecode. So I tried following things.

1. Is this a legal .class?

    being unfamiliar with .class. I can just decompile UiApiPlugin.class by javap. Though verbose information I got, I am still not sure whether it is legal class.

2. Does Jacoco support Java 1.5 bytecode?

    It is correct. JaCoCo requires java 1.5

So, how to confirm whether the class is legal

在 2018年3月23日星期五 UTC+8上午1:21:33，Evgeny Mandrikov写道：

$ javap -v -p aaqc.class | grep "major version"

  major version: 50

[Marc Hoffmann]

Hi,

looks like dex2jar is creating invalid class files. I did a quick sanity check with ASMs CheckClassAdapter and get the following result:

Exception in thread "main" java.lang.RuntimeException: Execution can fall off end of the code <clinit>()V
  at org.objectweb.asm.util.CheckMethodAdapter$1.visitEnd(CheckMethodAdapter.java:467)
  at org.objectweb.asm.MethodVisitor.visitEnd(MethodVisitor.java:878)
  at org.objectweb.asm.util.CheckMethodAdapter.visitEnd(CheckMethodAdapter.java:1038)
  at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1130)
  at org.objectweb.asm.ClassReader.accept(ClassReader.java:698)
  at org.objectweb.asm.ClassReader.accept(ClassReader.java:500)

As mentioned before: JaCoCo requires valid class files as an input.

Regards,
-marc

--
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/548cedbb-7def-42b1-a355-f27f8d87a522%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

 

[Evgeny Mandrikov]

Marc, how you execute CheckClassAdapter ?

My attempt shows a lot of "ClassNotFoundException" because of missing classes that this class depends on, but no "Execution can fall off end of the code".

Also looking at "<clinit>" in output of "javap -v -p UiApiPlugin.class", I don't see how it can fall off end of the code.

Even if class is potentially is invalid, I have strong feeling of something fishy here - an attempt to add frame into class that originally has no frames.

Seems that for instructions of method "a(Lcom/tencent/biz/webviewplugin/PubAccountUIPlugin;)Ljava/lang/String" , I'm currently investigating and will keep you posted.

To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+unsubscribe@googlegroups.com.

[Evgeny Mandrikov]

It is actually pretty easy to demonstrate that we add frames near probes into classes that have no frames initially - see attachment.

Maybe this shocks ASM during expansion of wide jumps in absence of other frames, causing AIOOBE? There is definitely such expansion during instrumentation of "UiApiPlugin.class" because stack trace contains call of "ClassReader.accept" from "ClassWriter.toByteArray". But so far didn't succeeded with creation/reduction of reproducer for AIOOBE.

[Evgeny Mandrikov]

On Wednesday, March 28, 2018 at 9:02:57 PM UTC+2, Evgeny Mandrikov wrote:

It is actually pretty easy to demonstrate that we add frames near probes into classes that have no frames initially - see attachment.

Given that changelog for JaCoCo 0.5.6 contains "Don't insert stackmap frames for class files before version 1.6" - either this was overlooked, or regression.

 

Maybe this shocks ASM during expansion of wide jumps in absence of other frames, causing AIOOBE? There is definitely such expansion during instrumentation of "UiApiPlugin.class" because stack trace contains call of "ClassReader.accept" from "ClassWriter.toByteArray". But so far didn't succeeded with creation/reduction of reproducer for AIOOBE.

And here is reproducer of AIOBE in attachment.

So, Marc, I think that we should congratulate Qian Ren about uncovering all this! And I'm going to create ticket. WDYT?

[Tool Ply]

Thank you very much. Evgeny and Marc

I am supposed to add some details to perfect jacoco. the first two classes, abap.classs and aaqc.class, are created by dex2jar 2.x, while the last one, UiApiPlugin.class, was created by enjarify which is developed by Google and shows the superiority.

So the conclusion is jacoco has a little bug with handling frame right? 

I notice that jacoco could exclude some classes in execution analysis, However, it could not exclude the classes in instrumentation so that I have to delete the .class manually before instrumentation to avoid this issues. Is there any other more elegant way.

在 2018年3月29日星期四 UTC+8上午5:34:54，Evgeny Mandrikov写道：

[Marc Hoffmann]

Hi Evgeny,

I was as bit confused at the beginning as we have ClassFileVersionsTest which tests for existence of frames.

But after a closer look I realized that this test only checks whether our generated runtime access methods insert frames. Probe insertion may also cause frames. The test case contains no code which causes such additional frames. That's why the test case is probably missing it.

But still confused as we run our integration test suite on Java 5...

I can work on a fix.

Cheers,
-marc

--

You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/fad9941c-eca0-457e-a5b3-dc11cc789db4%40googlegroups.com.

[Marc Hoffmann]

> Marc, how you execute CheckClassAdapter ?

I did it completely wrong: Created a small Java main() but accedentially used SKIP_CODE for the accept method().

Sorry for the noise...

I'll now try to improve ClassFileVersionsTest to demonstrate the problem.

Regards,
-marc

To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/21b73862-10db-4c82-a5f5-47483694a3be%40googlegroups.com.

[Evgeny Mandrikov]

On Thu, Mar 29, 2018 at 7:30 AM Marc Hoffmann <hoff...@mountainminds.com> wrote:

Hi Evgeny,

I was as bit confused at the beginning as we have ClassFileVersionsTest which tests for existence of frames.

But after a closer look I realized that this test only checks whether our generated runtime access methods insert frames. Probe insertion may also cause frames. The test case contains no code which causes such additional frames. That's why the test case is probably missing it.

 

Yep and looking at https://github.com/jacoco/jacoco/commit/4cfef884b41ffca31f15bb6e53e7ef62366c5602 this was overlooked since then and not a regression.

 

But still confused as we run our integration test suite on Java 5...

StackMapTable attribute was introduced in Java 6, from https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7-110 :

The StackMapTable attribute must be recognized and correctly read by a class file reader if the class file's version number is 50.0 or above and the Java Virtual Machine implementation recognizes class files whose version number is 50.0 or above.

Our validation tests do not perform explicit verification of absence of attribute when compiling into 49 bytecode, nor reading of instrumented classes back, only their loading into JVM and JVM 5 simply ignores this attribute.

Also you might remember that loading of class into JVM doesn't guarantee good bytecode verification - https://github.com/jacoco/jacoco/commit/cdd0ec17b976892ad8c8805e90831d0baa470b1a , https://github.com/jacoco/jacoco/issues/626 , etc

However this might bother other tools that read classes after our instrumentation and use attribute even if class file version is lower than 50. This AIOBE is kind of example of this: we shoot into own foot - in case of resizing of wide jumps ASM for writing of class performs reading (which is not the case in absence of such resizing) and during this operates with frames regardless of class file version solely based on presence of frames. BTW presence of wide jumps (big methods) is quite rare - just quite recently we fixed https://github.com/jacoco/jacoco/pull/177 and https://github.com/jacoco/jacoco/pull/462 connected with this resizing.

On Thu, Mar 29, 2018 at 11:17 AM Marc Hoffmann <hoff...@mountainminds.com> wrote:

I'll now try to improve ClassFileVersionsTest to demonstrate the problem.

Maybe better / simpler to add explicit check of absence of frames into all validation tests - e.g. into org.jacoco.core.test.InstrumentingLoader ?

[Marc Hoffmann]

> Maybe better / simpler to add explicit check of absence of frames into all validation tests - e.g. into org.jacoco.core.test.InstrumentingLoader ?

Good idea, I will extract a "FrameCheckAdapter" or so which can be used in both places.

--

You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/CAEPFu696MvK0qA4rqXmOUjTccX4PhkOXNFR3uktgyur5-7%3Dqmg%40mail.gmail.com.

[Marc Hoffmann]

Here is a proposed fix:

https://github.com/jacoco/jacoco/pull/667

I didn't yet expand it to other validation tests as they depend on the toolchain JDK.

Cheers,
-marc

On 2018-03-29 12:54, Evgeny Mandrikov wrote:

--

You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/CAEPFu696MvK0qA4rqXmOUjTccX4PhkOXNFR3uktgyur5-7%3Dqmg%40mail.gmail.com.

[Evgeny Mandrikov]

On Thu, Mar 29, 2018 at 6:58 PM Marc Hoffmann <hoff...@mountainminds.com> wrote:

Here is a proposed fix:

https://github.com/jacoco/jacoco/pull/667

I didn't yet expand it to other validation tests as they depend on the toolchain JDK.

Not sure I understand what do you mean by  "depends on the toolchain JDK". Isn't it as simple as making assertion on classfiles that have version less than 1.5 - see "checkAbsenceOfFrames" in "AIOOBETest" in attachment to one of previous emails.

Ok, we can add this later, but IMO we should have such check in validation tests, because I have feeling that this might actually be a regression - see comments in PR.

[Evgeny Mandrikov]

On Thursday, March 29, 2018 at 4:52:36 AM UTC+2, Tool Ply wrote:

So the conclusion is jacoco has a little bug with handling frame right? 

I notice that jacoco could exclude some classes in execution analysis, However, it could not exclude the classes in instrumentation so that I have to delete the .class manually before instrumentation to avoid this issues. Is there any other more elegant way.

Ability to specify exclusions is not implemented for Command Line Interface, there are several aspects that complicate implementation and so far we had no time to address them. So for the time being your workaround is good one.

In the meantime fix for problem ( https://github.com/jacoco/jacoco/pull/667 ) was committed to the master branch, so will be available in next release. Meanwhile you can use snapshot version - http://www.jacoco.org/jacoco/trunk/doc/repo.html