Running tests

[cyrainfish]

Hi all,

 

iBugs is really cool, but I also have some questions.

 

I'm trying to use dynamic methods and collecting traces of failing test cases.

 

I found that some test cases failed in "pre-fix/testresults.xml" but passed in  "post-fix/testresults.xml", I also found many "<testsforfix>" in "repository.xml". So which kind should I choose?Could you please tell me the differences of the two kinds?

 

 

If I can only get few faiing test cases, then I need to generate test cases with tools like Randoop. I  tried to run your junit tests in eclpise, I found

1)I should edit some classpath value( like JRE14_LIB),  

2)  source code like "public static final String TESTDATA_PATH = "../weaver/testdata";".

 

Then I checked your "runtest.xml", like this:

 

<project name="runtest" default="runtest" basedir=".">

<target name="runtest">

<property name="moduledir" location="${basedir}/.."/>

<java classname="org.aspectj.weaver.loadtime.test.DocumentParserTestProxy" fork="true">

<classpath id="runtimeclasspath">

<pathelement location="${moduledir}/aj-build/jars/run-all-junit-tests-test-all.jar"/>

<pathelement location="${moduledir}/aj-build/jars/testing-drivers-test-all.jar"/>

<pathelement location="${moduledir}/aj-build/jars/testing-drivers-all.jar"/>

<pathelement location="${moduledir}/tests/realbugs"/>

<fileset dir="${moduledir}/lib">

<include name="**/*.jar"/>

</fileset>

</classpath>

<jvmarg value="-Xmx512m"/>

</java>

</target>

</project>

I really want to know how you deal with 1) classpath value 2)"TESTDATA_PATH = "../weaver/testdata" .

And how could I define my own command to run my own junit tests?

I'm not sure if you could understand my questions because my English is not good. Please let me know.

Thank you very much.

 

 

[Valentin Dallmeier]

Hi!

Thanks for your interest in iBugs.

The test failure lists in pre-fix/testresults.xml and post-fix/testresults.xml contain test outcomes as observed on one of our machines. These lists should give you a hint as to which tests usually fail. On the other hand <testsforfix> contains a lists of tests that were committed together with the fix. To make sure that a particular test actually fixes a defect, you should run the test on your machine in both versions and make sure that it fails in the pre-fix version and passes in the post-fix version. A good set of candidate test cases would be those in <testsforfix>

The class path for target runtest contains the testing jars which in turn contain all the classes that were built by the particular sub projects. That way we can make sure that all classes are actually loaded. If you want to add custom classes I suggest you prepend them to the class path. Please note that it's important to execute the tests in the correct directory so that access to resources like ../weaver/testdata is possible. Running the generated tests from Eclipse could be difficult. Wouldn't it be easier for you to run the tests using the commands in build.xml?

Regards,

Valentin

 

 

--
You received this message because you are subscribed to the Google Groups "iBugs" group.
To view this discussion on the web visit https://groups.google.com/d/msg/ibugs/-/3XBq8pCjNWMJ.
To post to this group, send email to ib...@googlegroups.com.
To unsubscribe from this group, send email to ibugs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ibugs?hl=en.

[cyrainfish]

Dear Valentin,

So I have to run the tests in pre-fix and post-fix version, but ignore the testresults.xml? I also found the number of tests in s in pre-fix/testresults.xml and post-fix/testresults.xml may be different. Then should I run all the tests in pre-fix or post-fix?

 

I chose one <testforfix> to run,like this:

<testsforfix ID="132591" type="new">
    <file LOCATION="org.aspectj/modules/ajde/testsrc/org/aspectj/ajde/OutxmlTest.java">
      <test NAME="testOutxmlToOutjarWithAop_xml"/>
    </file>
  </testsforfix>

It passed in post-fix version. Then I copyed runtest.xml and realbugs folder to pre-fix and ran, I got errors.

 

runtest:
     [java] .E
     [java] Time: 1.649
     [java] There was 1 error:
     [java] 1) testOutxmlToOutjarWithAop_xml(org.aspectj.ajde.OutxmlTestProxy)ja
va.lang.NoSuchMethodError: org.aspectj.ajde.OutxmlTest.testOutxmlToOutjarWithAop
_xml()V
     [java]     at org.aspectj.ajde.OutxmlTestProxy.testOutxmlToOutjarWithAop_xm
l(org/aspectj/ajde/OutxmlTest.java)
     [java]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcces
sorImpl.java:57)
     [java]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMet
hodAccessorImpl.java:43)
     [java]     at org.aspectj.ajde.OutxmlTestProxy.main(org/aspectj/ajde/Outxml
Test.java:53)

     [java] FAILURES!!!
     [java] Tests run: 1,  Failures: 0,  Errors: 1

 

I've known that's because the OutxmlTest.java files must be different in pre-fix and post-fix. But how to solve this?

 

In fact I use eclipse because I want to generate my own test cases. Then how could I use commands in build.xml to run my own test cases?

 

For your command "runjunittests" in build.xml, where could I find the results if I use it?

 

Thank you very much.

 

Cao Yu
在 2013年1月7日星期一UTC+8下午10时16分45秒，Valentin Dallmeier写道：

[Valentin Dallmeier]

Hi!

Sorry for the delay in getting back to you. There's a couple of questions in your email that are explained in detail in the paper on ibugs at http://www.st.cs.uni-saarland.de/publications/files/dallmeier-ase-2007.pdf . Please take a look at that paper, it also contains a small how-to.

For each bug in the repository, there is a pre-fix and a post-fix version of AspectJ. The difference between these two versions is the fix for the bug. If you want to have a failing test, you need to copy the test from the post-fix version to the the pre-fix version where it should fail. That's your failing test. The number of tests in testresults.xml is usually different between pre-fix and post-fix versions because the fix often adds new tests (which are the tests marked by <testsforfix>).

I hope these comments help.

Regards,

Valentin

To view this discussion on the web visit https://groups.google.com/d/msg/ibugs/-/6AZFMx8iWcoJ.