failing test runs on post-fix
Elton Alves
ibugs looks really cool!
i am having some problems. the build process seems ok. but i am running system tests (with target runharnesstests) on pre and pos-fix versions and **would expect** to see some failing runs on pre-fix and none on post-fix. however, to my surprise, the output looks very similar. i'm guessing that i don't understand the output or i have some possible configuration problem in my environment.
for example, i run the script below twice (passing "pre-fix" and "post-fix", respectively) and obtain the following output:
==================================================
################# RUNHARNESSTESTS ON 31423, PRE-FIX
## FAIL Suite.Spec(/home/elton/tmp/repository_tests/ibugs_aspectj-1.3/versions/31423/pre-fix/org.aspectj/modules/tests) 732 tests (5 skipped, 23 failed, 704 passed) 71 seconds
################# RUNHARNESSTESTS ON 31423, POST-FIX
## FAIL Suite.Spec(/home/elton/tmp/repository_tests/ibugs_aspectj-1.3/versions/31423/post-fix/org.aspectj/modules/tests) 733 tests (5 skipped, 24 failed, 704 passed) 60 seconds
==================================================
the script i use:
==================================================
#!/bin/bash
VERSION=31423
TAG=$1 ## "pre-fix" "post-fix"
ant checkoutversion -DfixId=$VERSION
## clean
ant cleanversion -DfixId=$VERSION -Dtag=$TAG
## build app
ant buildversion -DfixId=$VERSION -Dtag=$TAG
## build tests
ant buildtests -DfixId=$VERSION -Dtag=$TAG
## run system tests
ant runharnesstests -DfixId=$VERSION -Dtag=$TAG
==================================================
important to say that I only changed property.xml in the following line:
==================================================
<!--property name="JAVA_HOME" value="/System/Library/Frameworks/JavaVM.framework/Versions/1.4.2/Home"/-->
<property name="JAVA_HOME" value="/opt/j2sdk1.4.2_19/"/>
==================================================
Please, let me know how to proceed.
- is there a configuration check test?
- would it help if i send some logs? if yes, which?
thanks in advance,
Elton Alves
Valentin Dallmeier
Thank you for your interest in iBUGS. I took a look at your scripts
and it seems like you've done nothing wrong. The thing is that some of
the tests in AspectJ fail if you don't have a correct XWindow setup.
Can you try re-running the tests with a running XServer? If you're
working remotely on another machine, vnc might be a solution.
Nevertheless, even then there is usually only very few tests that fail
in pre-fix and not in post-fix. If you're trying to reproduce a
failure, you need to look for tests specified in <testsforfix> in file
repository.xml . Not all bugs have this tag. To make sure that a test
for a fix actually fixes the problem, execute the test in the post-fix
version and make sure it does not fail. Afterwards, copy the files
that are relevant for the test from post-fix to pre-fix and run the
test again. It should now fail.
I hope this information helps. Let me know if you have more questions.
And once again, thank you for your interest in iBUGS.
Regards,
Valentin
> --
> You received this message because you are subscribed to the Google Groups
> "iBugs" group.
> To post to this group, send email to ib...@googlegroups.com.
> To unsubscribe from this group, send email to
> ibugs+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/ibugs?hl=en.
>
Francisco Servant
Is there a simple (easily automated) way to copy the tests from post-
fix to pre-fix?
If I copy all committed files, then I am also copying the fix.
For JUnit tests, if I copy the .java files where the <testsforfix>
tests are located, I should be fine.
But, for Harness tests, is it enough to copy the .xml files where the
<testsforfix> are located? Should I copy additional .java files?
For Elton:
I think that you cannot expect to have 0 failures in post-fix tests. I
think that there are two reasons for this:
1. Developers may create the test without fixing the bug, because
fixing other bugs has higher priority.
2. Developers introduce code that breaks test cases that passed in
older versions, but they don't have time to fix them right now, or
fixing them is low priority.
Therefore, any single version can have any number of passing or
failing test cases in pre-fix and post-fix.
Again, this is my guess :)
Thanks!
Francisco
On Jul 21, 8:56 am, Valentin Dallmeier <valentin.dallme...@gmail.com>
wrote:
Valentin Dallmeier
> Is there a simple (easily automated) way to copy the tests from post-
> fix to pre-fix?
I do have tools for that, but they are undocumented and hence I don't
include them. For AspectJ the heuristic is to copy all files in
"org.aspectj/modules/tests/" .
Does that help?
Regards,
Valentin
Francisco Servant
On Jul 21, 10:33 am, Valentin Dallmeier <valentin.dallme...@gmail.com>
wrote:
Elton Alves
I am confused.
I use the sequence below for each version (I run several).
(* using proper parameter such as -DfixId=... -Dtag=pre-fix*)
ant checkoutversion
ant buildversion
ant buildversion (some versions build tests only after the second
buildversion)
ant buildtests
ant runharnesstests
The results don't match with the results from repository.xml. I noted
that the size of the test suite are different (as showed in the <pre/
post-fix-testcases> tag in repository.xml). Is the result fro
repository.xml counting harness and unit tests? Note I am only
considering harness tests.
As for Valentin reply: I am running the experiments in a machine
running Ubuntu 9.04 with gnome. I ran the command top and Xorg shows
up running (guess this is the window manager). I try to run startx
but I obtain the message below.
=======================
X: warning; process set to priority -1 instead of requested priority 0
Fatal server error:
Server is already active for display 0
If this server is no longer running, remove /tmp/.X0-lock
and start again.
…
=======================
The value of variable DISPLAY is “:0.0”. I am guessing that this may
not be the problem. Are there any other configuration that I need to
be aware.
Thanks a lot,
Elton
On Jul 21, 12:56 pm, Valentin Dallmeier <valentin.dallme...@gmail.com>
wrote:
>
> Thank you for your interest in iBUGS. I took a look at your scripts
> and it seems like you've done nothing wrong. The thing is that some of
> the tests in AspectJ fail if you don't have a correct XWindow setup.
> Can you try re-running the tests with a running XServer? If you're
> working remotely on another machine, vnc might be a solution.
> Nevertheless, even then there is usually only very few tests that fail
> in pre-fix and not in post-fix. If you're trying to reproduce a
> failure, you need to look for tests specified in <testsforfix> in file
> repository.xml . Not all bugs have this tag. To make sure that a test
> for a fix actually fixes the problem, execute the test in the post-fix
> version and make sure it does not fail. Afterwards, copy the files
> that are relevant for the test from post-fix to pre-fix and run the
> test again. It should now fail.
>
> I hope this information helps. Let me know if you have more questions.
> And once again, thank you for your interest in iBUGS.
>
> Regards,
>
> Valentin
>
Valentin Dallmeier
> I am confused.
Let's see if I can help you.
>
> I use the sequence below for each version (I run several).
>
> (* using proper parameter such as -DfixId=... -Dtag=pre-fix*)
> ant checkoutversion
> ant buildversion
> ant buildversion (some versions build tests only after the second
> buildversion)
> ant buildtests
> ant runharnesstests
Looks good.
>
> The results don't match with the results from repository.xml. I noted
> that the size of the test suite are different (as showed in the <pre/
> post-fix-testcases> tag in repository.xml). Is the result fro
> repository.xml counting harness and unit tests? Note I am only
> considering harness tests.
Yes, the results from repository.xml also include unit tests.
>
> As for Valentin reply: I am running the experiments in a machine
> running Ubuntu 9.04 with gnome. I ran the command top and Xorg shows
> up running (guess this is the window manager). I try to run startx
> but I obtain the message below.
>
> =======================
> X: warning; process set to priority -1 instead of requested priority 0
>
>
>
> Fatal server error:
>
> Server is already active for display 0
>
> If this server is no longer running, remove /tmp/.X0-lock
>
> and start again.
>
> …
>
> =======================
>
> The value of variable DISPLAY is “:0.0”. I am guessing that this may
> not be the problem. Are there any other configuration that I need to
> be aware.
Uhm, it looks like something's wrong with your X setup. This has
nothing to do with executing AspectJ tests. Can you execute other
tools that use X from the shell that you're using to execute the
tests? It is usually not possible to do this if you're logged in using
ssh.
Regards,
Valentin
Elton Alves
> Uhm, it looks like something's wrong with your X setup. This has
> nothing to do with executing AspectJ tests. Can you execute other
> tools that use X from the shell that you're using to execute the
> tests? It is usually not possible to do this if you're logged in using
> ssh.
of commands sent, ten times for each version and noted that three of
out them changed the results across this executions (124808, 59895,
46280).
Is that ok ?
thanks in advance,
Elton
On Jul 29, 6:20 am, Valentin Dallmeier <valentin.dallme...@gmail.com>
wrote:
Valentin Dallmeier
Sorry for the delay, but I was on a holiday.
> I ran the harness tests with xterm now. I executed , the last sequence
> of commands sent, ten times for each version and noted that three of
> out them changed the results across this executions (124808, 59895,
> 46280).
> Is that ok ?
Hmm, I've never witnessed this behavior. Could you let me know which
tests are changing their outcome?
Valentin
Elton Alves
I tried to identify the junit test with this behavior (harness in
these versions seem ok), but the results in 59895/pre|post-fix/
org.aspectj/modules/tests/bin/junitXmlOutput/html (for example) was
not completely generated. I can only analyse the junit results,
consistently, across the console output.
As you may expect these non-deterministic results are very bad for
us. The results are deterministic only in three of these versions,
namely 28974 82134 49657 (out of all revisions that showed less
failing tests in post-fix versions).
Considering these problems and the fact that I've already tried to run
the experiments in different machines (also asked different people), I
am wondering whether it is possible for you to take some action on
your side. For example:
- try to reproduce the problem in different environments*. say a
colleague's machine with no access to department's file system.
- prepare a script that reports the current configuration of your
system (running under the ibugs-aspectj dir). we can use that to
compare our outputs.
* my configuration:
- i have ubuntu 9.04,
- apache ant version 1.6.5.,
- java version "1.4.2_19" installed.
thanks,
Elton
On Aug 9, 7:31 am, Valentin Dallmeier <valentin.dallme...@gmail.com>
wrote:
Valentin Dallmeier
Hi!
This looks really weird. Before I reproduce this, can you try to further narrow down the tests that fail? One way might be to generate individual runs for all junit tests and compare console output.
Valentin
> > I ran the harness tests with xterm now....
--
You received this message because you are subscribed to the Google Groups "iBugs" group.
To post...