java.lang.VerifyError running Daikon with Apache Hadoop

[Juan Jimenez]

Good Evening,

     My name is Juan Jimenez and i am trying to use Daikon in conjunction with Apache Hadoop distributed computing. I have recently installed a fresh copy of
both Daikon and Hadoop to my Ubuntu 64-bit 15.04 machine. Individually they both run fine. I have been able to start a distributed Hadoop MapReduce job to
count the words in a sample text file. I also have been able to separately run the Daikon StackAr example which produced the expected outcome flawlessly. My
intentions are now to run the Daikon Dynamic Invariant Detector on the previously working Hadoop wordcount job. The following is the command that is executed:

"$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS daikon.Chicory --daikon $CLASS "$@"

which successfully resolves to the following:

java -cp (insert extremely long classpath here) -ea -Xmx1000m -javaagent:/usr/local/daikon-5.2.12/java/ChicoryPremain.jar=--daikon --dtrace-file=RunJar.dtrace.gz org.apache.hadoop.util.RunJar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /user/user1/book.txt /user/user1/output

Once this command is executed i receive the following output:

Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
entered daikon.chicory.Runtime.setDtrace(./RunJar.dtrace.gz, false)...
Exception in thread "main" java.lang.VerifyError: Instruction type does not match stack map
Exception Details:
  Location:
    org/apache/hadoop/fs/FileSystem.processDeleteOnExit()V @10: aload_0
  Reason:
    Type top (current frame, locals[1]) is not assignable to integer (stack map, locals[1])
  Current Frame:
    bci: @10
    flags: { }
    locals: { 'org/apache/hadoop/fs/FileSystem', top, integer }
    stack: { }
  Stackmap Frame:
    bci: @10
    flags: { }
    locals: { 'org/apache/hadoop/fs/FileSystem', integer }
    stack: { }
  Bytecode:
    0x0000000: b205 7a59 0460 b305 7a3d 2a1c 1103 1503
    0x0000010: bd00 4eb8 057e 2ab4 0071 594c c22a b400
    0x0000020: 71b9 00cc 0100 4e2d b900 cd01 0099 0049
    0x0000030: 2db9 00ce 0100 c000 cf3a 042a 1904 b600
    0x0000040: 9c99 000b 2a19 0404 b600 c357 a700 213a
    0x0000050: 05b2 0040 bb00 2159 b700 2212 d0b6 0024
    0x0000060: 1904 b600 37b6 0027 b900 d102 002d b900
    0x0000070: d201 00a7 ffb4 2bc3 a700 0a3a 062b c319
    0x0000080: 06bf 2a1c 1103 1503 bd00 4e01 1105 89b8
    0x0000090: 0582 b1                               
  Exception Handler Table:
    bci [59, 76] => handler: 79
    bci [29, 120] => handler: 123
    bci [123, 127] => handler: 123
  Stackmap Table:
    append_frame(@10,Integer)
    same_frame(@22)
    append_frame(@39,Object[#634],Object[#641])
    append_frame(@76,Object[#534])
    same_locals_1_stack_item_frame(@79,Object[#628])
    same_frame(@109)
    chop_frame(@118,2)
    same_locals_1_stack_item_frame(@123,Object[#635])
    chop_frame(@130,1)

    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.getDeclaredMethod(Class.java:2128)
    at daikon.chicory.MethodInfo.initViaReflection(MethodInfo.java:151)
    at daikon.chicory.ClassInfo.initViaReflection(ClassInfo.java:87)
    at daikon.chicory.Runtime.getClassInfoFromClass(Runtime.java:757)
    at daikon.chicory.DaikonVariableInfo.addDeclVar(DaikonVariableInfo.java:825)
    at daikon.chicory.DaikonVariableInfo.addClassVars(DaikonVariableInfo.java:570)
    at daikon.chicory.DaikonVariableInfo.addChildNodes(DaikonVariableInfo.java:1296)
    at daikon.chicory.DaikonVariableInfo.addParameters(DaikonVariableInfo.java:449)
    at daikon.chicory.RootInfo.enter_process(RootInfo.java:57)
    at daikon.chicory.Runtime.process_new_classes(Runtime.java:464)
    at daikon.chicory.Runtime.enter(Runtime.java:261)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:83)
    at org.apache.hadoop.util.RunJar.unJar(RunJar.java:107)
    at org.apache.hadoop.util.RunJar.unJar(RunJar.java:81)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:209)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Exception in thread "Thread-1" java.lang.VerifyError: Instruction type does not match stack map
Exception Details:
  Location:
    org/apache/hadoop/fs/FileSystem.processDeleteOnExit()V @10: aload_0
  Reason:
    Type top (current frame, locals[1]) is not assignable to integer (stack map, locals[1])
  Current Frame:
    bci: @10
    flags: { }
    locals: { 'org/apache/hadoop/fs/FileSystem', top, integer }
    stack: { }
  Stackmap Frame:
    bci: @10
    flags: { }
    locals: { 'org/apache/hadoop/fs/FileSystem', integer }
    stack: { }
  Bytecode:
    0x0000000: b205 7a59 0460 b305 7a3d 2a1c 1103 1503
    0x0000010: bd00 4eb8 057e 2ab4 0071 594c c22a b400
    0x0000020: 71b9 00cc 0100 4e2d b900 cd01 0099 0049
    0x0000030: 2db9 00ce 0100 c000 cf3a 042a 1904 b600
    0x0000040: 9c99 000b 2a19 0404 b600 c357 a700 213a
    0x0000050: 05b2 0040 bb00 2159 b700 2212 d0b6 0024
    0x0000060: 1904 b600 37b6 0027 b900 d102 002d b900
    0x0000070: d201 00a7 ffb4 2bc3 a700 0a3a 062b c319
    0x0000080: 06bf 2a1c 1103 1503 bd00 4e01 1105 89b8
    0x0000090: 0582 b1                               
  Exception Handler Table:
    bci [59, 76] => handler: 79
    bci [29, 120] => handler: 123
    bci [123, 127] => handler: 123
  Stackmap Table:
    append_frame(@10,Integer)
    same_frame(@22)
    append_frame(@39,Object[#634],Object[#641])
    append_frame(@76,Object[#534])
    same_locals_1_stack_item_frame(@79,Object[#628])
    same_frame(@109)
    chop_frame(@118,2)
    same_locals_1_stack_item_frame(@123,Object[#635])
    chop_frame(@130,1)

    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.getDeclaredMethod(Class.java:2128)
    at daikon.chicory.MethodInfo.initViaReflection(MethodInfo.java:151)
    at daikon.chicory.ClassInfo.initViaReflection(ClassInfo.java:87)
    at daikon.chicory.Runtime.process_new_classes(Runtime.java:458)
    at daikon.chicory.Runtime.enter(Runtime.java:261)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:51)
Warning: Did not run Daikon because target exited with 1 status
juandi@ubuntu:/usr/local/hadoop-2.7.1$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /user/user1/book.txt /user/user1/output > ~/Desktop/out.txt
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
Exception in thread "main" java.lang.ClassFormatError: StackMapTable format error: bad offset for Uninitialized in method org.apache.hadoop.fs.FileSystem.checkAccessPermissions(Lorg/apache/hadoop/fs/FileStatus;Lorg/apache/hadoop/fs/permission/FsAction;)V
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.getDeclaredMethod(Class.java:2128)
    at daikon.chicory.MethodInfo.initViaReflection(MethodInfo.java:151)
    at daikon.chicory.ClassInfo.initViaReflection(ClassInfo.java:87)
    at daikon.chicory.Runtime.getClassInfoFromClass(Runtime.java:757)
    at daikon.chicory.DaikonVariableInfo.addDeclVar(DaikonVariableInfo.java:825)
    at daikon.chicory.DaikonVariableInfo.addClassVars(DaikonVariableInfo.java:570)
    at daikon.chicory.DaikonVariableInfo.addChildNodes(DaikonVariableInfo.java:1296)
    at daikon.chicory.DaikonVariableInfo.addParameters(DaikonVariableInfo.java:449)
    at daikon.chicory.RootInfo.enter_process(RootInfo.java:57)
    at daikon.chicory.Runtime.process_new_classes(Runtime.java:464)
    at daikon.chicory.Runtime.enter(Runtime.java:261)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:83)
    at org.apache.hadoop.util.RunJar.unJar(RunJar.java:107)
    at org.apache.hadoop.util.RunJar.unJar(RunJar.java:81)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:209)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Exception in thread "Thread-1" java.lang.ClassFormatError: StackMapTable format error: bad offset for Uninitialized in method org.apache.hadoop.fs.FileSystem.checkAccessPermissions(Lorg/apache/hadoop/fs/FileStatus;Lorg/apache/hadoop/fs/permission/FsAction;)V
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.getDeclaredMethod(Class.java:2128)
    at daikon.chicory.MethodInfo.initViaReflection(MethodInfo.java:151)
    at daikon.chicory.ClassInfo.initViaReflection(ClassInfo.java:87)
    at daikon.chicory.Runtime.process_new_classes(Runtime.java:458)
    at daikon.chicory.Runtime.enter(Runtime.java:261)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:51)


It seems like Daikon is having an issue performing dynamic analysis through the Hadoop distributed system but i do not fully understand what the exception is trying to tell me.
Can anyone shed some light on this, or have any experience working with Daikon and Hadoop? I am running java version "1.8.0_60". Any help would be appreciated.

- Juan Jimenez

[Mark Roberts]

We’re sorry your’re having trouble with Daikon.  It would be most helpful if you could provide a reproducible test case.  This would include which version of Hadoop and all the steps used to set it up and run your test.  Please include all commands and environment variables in addition to the test case source.

 

Thank you,

Mark Roberts

 

 

--
You received this message because you are subscribed to the Google Groups "Daikon discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to daikon-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Juan Jimenez]

Thank you for responding to my post. Attached to this message are 2 files:

   1. process.txt describes everything i did to obtain the error received

   2. book.txt is a file required to reproduce what i did (explained in process.txt)

If there is anything else that can help troubleshoot this problem, please let me know!

Sincerely,

Juan D. Jimenez

jua...@ubuntu:/usr/local/hadoop-2.7.1$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /user/user1/book.txt /user/user1/output > ~/Desktop/out.txt

[Mark Roberts]

Thank you for your very detailed problem report.  Using your process.txt file as a roadmap, I was able to reproduce your exception.  I am looking into the details and will let you know what I find.

 

Thanks,

Mark

[Mark Roberts]

To my knowledge, this is the first reported attempt to run Hadoop with Daikon.  It turned out to be quite a stress test and exposed three problems in Chicory, the Daikon front end for Java code.  I have corrected those issues and pushed the changes to the Daikon repository.

 

Some additional comments:

 

I made some further changes to the bin/Hadoop script – I have attached my version which makes additional changes to CLASSPATH and allocates more memory for Chicory.

 

Hadoop is a very large body of code – you will not be able to run Chicory on the entire process.  When I tried, it ran for 8 hours, produced almost 100 terabytes of output and still died with garbage collection problems prior to completion.  For large programs like this, the Chicory options ‘—ppt-select-pattern=regexp’ and ‘—ppt-omit-pattern=rexexp’ are almost always required to make the run time and data output manageable.  See:

http://plse.cs.washington.edu/daikon/download/doc/daikon.html#Program-points-in-Chicory-output

for more details.  You will see the combination of these arguments I used in my attached Hadoop script.

 

Good luck and let us know how it goes.

 

Mark Roberts