interpretation of testresults.xml

[Juliano Iyoda]

Hello,

I'm working with fixId = 47318.

When I diff testresults.xml from both pre and post-fix I got:

jmi@juliano:~/tmp/ibugs_aspectj-1.3$ diff versions/47318/pre-fix/
testresults.xml versions/47318/post-fix/testresults.xml
1c1
< <testsuite FAILING="85" PASSING="1098" SIZE="1183">
---
> <testsuite FAILING="83" PASSING="1100" SIZE="1183">
3070,3072c3070
< NAME="VerifyError if nested sync returning result">
< <error />
< </test>
---
> NAME="VerifyError if nested sync returning result" />
3581,3584c3579
< NAME="incrementally change only string literal, still expect
advice">
<
< <error />
< </test>
---
> NAME="incrementally change only string literal, still expect advice" />

By looking at these differences, I assumed that both tests
"VerifyError if nested sync returning result" and "incrementally
change only string literal, still expect advice" fail in the pre-fix
version and pass in the post-fix version.

Is that correct?

I'm asking that because I ran "VerifyError if nested sync returning
result" on the pre-fix and it seems to have passed.

jmi@juliano:~/tmp/ibugs_aspectj-1.3$ ant -f versions/47318/pre-fix/
org.aspectj/modules/tests/runtest.xml
Buildfile: versions/47318/pre-fix/org.aspectj/modules/tests/
runtest.xml

runtest:
[java] ------------ PASS: VerifyError if nested sync returning
result()
[java] --- result: true
[java] [ 0] ------------ PASS: CompilerRun(CompilerRun.Spec
AjdtCommand(1 paths))
[java] [ 0] --- result: true
[java] [ 0] --- messages
[java] [ 0] MessageHolder: (6 info)
[java] [ 0] [info 0]: info
org.aspectj.ajdt.ajc.AjdtCommand@b0bad7([-d, /tmp/SandboxWm2YUh/
classes, -classpath, ../lib/test/aspectjrt.jar:../lib/test/testing-
client.jar, /home/jmi/tmp/ibugs_aspectj-1.3/versions/47318/pre-fix/
org.aspectj/modules/tests/new/NestedSyncWithResult.java])
[java] [ 0] [info 1]: info zipfile classpath entry does not
exist: /usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/sunrsasign.jar
[java] [ 0] [info 2]: info directory classpath entry does not
exist: /usr/lib/jvm/java-6-sun-1.6.0.06/jre/classes
[java] [ 0] [info 3]: info compiling /home/jmi/tmp/
ibugs_aspectj-1.3/versions/47318/pre-fix/org.aspectj/modules/tests/new/
NestedSyncWithResult.java
[java] [ 0] [info 4]: info weaving
[java] [ 0] [info 5]: info might need to weave
[UnwovenClassFile(/tmp/SandboxWm2YUh/classes/Bug.class, Bug),
UnwovenClassFile(/tmp/SandboxWm2YUh/classes/NestedSyncWithResult
$1.class, NestedSyncWithResult$1), UnwovenClassFile(/tmp/SandboxWm2YUh/
classes/NestedSyncWithResult$2.class, NestedSyncWithResult$2),
UnwovenClassFile(/tmp/SandboxWm2YUh/classes/
NestedSyncWithResult.class, NestedSyncWithResult)](world=false)
[java] [ 1] ------------ PASS: JavaRun(JavaRun
(NestedSyncWithResult))
[java] [ 1] --- result: true
[java] [ 1] --- messages
[java] [ 1] MessageHolder: (2 info)
[java] [ 1] [info 0]: info JavaRun(NestedSyncWithResult)
[JavaRun$Spec]
[java] [ 1] [info 1]: info sandbox: Sandbox(SandboxWm2YUh,
new, classes)
[java]
[java] PASS Suite.Spec(/home/jmi/tmp/ibugs_aspectj-1.3/versions/
47318/pre-fix/org.aspectj/modules/tests) 1 tests (1 passed) 1 seconds

BUILD SUCCESSFUL
Total time: 2 seconds

Any help?

Juliano

[Valentin Dallmeier]

Hello Juliano!

Thank you for your interest in iBUGS, I hope that the repository can
help you. There is a slight issue with the outcome of test runs in
AspectJ, namely that the outcome of some tests depends on your
environment, e.g. the correct setting of the DISPLAY variable. You
should therefore always execute tests you're interested in and take a
look at the output to make sure that a test actually fails or passes.
I guess you're interested in runs that reproduce a bug. I recommend
you extract information about associated tests from the repository.xml
file (the tag is <testsforfix>), and then run all those tests to see
if they work in your environment. I've attached a sample of bug
identifiers for AspectJ where this was always the case in my
environment.

I hope this helps. If you have any more questions, please let me know!

Regards,

Valentin