Testing on different platforms

[Ori Peleg]

The makefile already supports testing with different Python versions on *nix ("make testall"). I used to run it on Linux and FreeBSD (today only Linux).

A while ago I included Testoob in the pybots project (see here). I'm not sure if it's still running.

There's infrastructure for running an isolated test, I think it's still working. We can have our own buildbots if we like and have resources.

Windows was always the odd circumstance (batch file to run the tests, one version at a time). Today most of my development is on Windows so there are more batch files, but they aren't organized.

Running the tests with IronPython and Jython is also not streamlined.

Misha Seltzer wrote:

Also - Orip: We need to think of a feature for cross testing (different python versions).

We can (probably easily) make it to test on different python versions of the same system, but we might want to do some cross-system or over-the-network tests as well.

[Ronnie]

When I run the unittests on Unix, I get 61 failing testcases.
They all fail with the same message:
Executable 'testoob' not found in the path
This is of course logical since Unix doesn't recognize testoob.bat
I couldn't find a shell script with the same purpose. Is that correct?

Ronnie

---
$ python tests/alltests.py suite

ERROR: testXMLReporting
(large.test_commandline.CommandLineTestCase.testXMLReporting)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/large/test_commandline.py", line 221, in
testXMLReporting
root = ET.XML( self._get_file_report("xml") )
File "tests/large/test_commandline.py", line 195, in
_get_file_report
args = _testoob_args(options=["--%s=%s" % (option_name,
output_file)], tests=["CaseMixed"])
File "tests/large/test_commandline.py", line 28, in _testoob_args
result = [sys.executable, helpers.executable_path(), suite_file]
File "tests/helpers.py", line 26, in executable_path
return which("testoob")
File "tests/helpers.py", line 17, in which
raise RuntimeError("Executable %r not found in the path" %
executable)
RuntimeError: Executable 'testoob' not found in the path

----
Ran 156 tests in 7.556s
FAILED (errors=61)

On 27 jan, 13:18, Ori Peleg <ori...@gmail.com> wrote:
> The makefile already supports testing with different Python versions on *nix
> ("make testall"). I used to run it on Linux and FreeBSD (today only Linux).
> A while ago I included Testoob in the

> pybots<http://code.google.com/p/pybots/>project (see
> here<http://code.google.com/p/pybots/source/browse/trunk/slave/gheorghiu-2...>).

[Misha Seltzer]

There is a shell script in src/testoob/testoob

Generally, if you run "python ./setup.py install" it should install this script into some bin directory.

If you want to run the test in development mode (means it'll look for the script in src/testoob/testoob instead of looking for it in the path), you can define an environment variable "TESTOOB_DEVEL_TEST" (with any value) and run tests again.

--
ɐɥsıɯ

[Ronnie]

How stupid of me. setting TESTOOB_DEVEL_TEST indeed solved it.
I did find one remaining failing testcase:

FAIL: testTimeOut
(large.test_commandline.CommandLineTestCase.testTimeOut)

----------------------------------------------------------------------
Traceback (most recent call last):

File "tests/large/test_commandline.py", line 387, in testTimeOut
testoob.testing.command_line(args=args,
expected_error_regex=regex, expected_rc=1)
File "/home/rwestein/testoob/testoob/src/testoob/testing.py", line
152, in command_line
assert_matches(expected_error_regex, error,
filter=_normalize_newlines)
File "/home/rwestein/testoob/testoob/src/testoob/testing.py", line
67, in assert_matches
raise TestoobAssertionError(msg, description="assert_matches
failed")
TestoobAssertionError: [assert_matches failed]
'FF
======================================================================
FAIL: testBuisy (suites.CaseSlow.testBuisy)

----------------------------------------------------------------------
Traceback (most recent call last):

File "/home/rwestein/testoob/testoob/tests/suites.py", line 71, in
testBuisy
time.sleep(1)
File "/home/rwestein/testoob/testoob/src/testoob/main.py", line 175,
in alarm
raise AssertionError("Timeout")
AssertionError: Timeout

======================================================================
FAIL: testSleep (suites.CaseSlow.testSleep)

----------------------------------------------------------------------
Traceback (most recent call last):

File "/home/rwestein/testoob/testoob/tests/suites.py", line 65, in
testSleep
time.sleep(2)
File "/home/rwestein/testoob/testoob/src/testoob/main.py", line 175,
in alarm
raise AssertionError("Timeout")
AssertionError: Timeout

Failed 2 tests
- testBuisy (suites.CaseSlow)
- testSleep (suites.CaseSlow)
----------------------------------------------------------------------
Ran 2 tests in 1.995s
FAILED (failures=2)
' doesn't match regular expression 'FAIL: testBuisy \(suites\.CaseSlow
\.testBuisy\)
.*
AssertionError: Timeout.*
FAIL: testSleep \(suites\.CaseSlow\.testSleep\)
.*
AssertionError: Timeout.*Ran 2 tests in 2\.\d+s'

This testcase is always skipped on windows. I could easily solve it
by adding ".*" at the beginning of _timeout_regex_base (line 375 in
tests/large/test_commandline.py)
Did we get this testcase to pass at some point? If the fix I'm
describing
is indeed necessary, I'll add it in a branch.

Ronnie

On 27 jan, 14:45, Misha Seltzer <misha.selt...@gmail.com> wrote:
> There is a shell script in src/testoob/testoob
> Generally, if you run "python ./setup.py install" it should install this
> script into some bin directory.
>
> If you want to run the test in development mode (means it'll look for the
> script in src/testoob/testoob instead of looking for it in the path), you
> can define an environment variable "TESTOOB_DEVEL_TEST" (with any value) and
> run tests again.
>

[Misha Seltzer]

hmm.. there seem to be another error. it ran under 2 seconds (though test assumes it'll take more...)

But, generally, checking against regex is a hard-to-do-right thing, so we might have forgotten something there, and you're welcome to add anything you think will work right :)

--
ɐɥsıɯ

[Ronnie]

You're right again.
I ran the tests several times, and this testcase almost always runs in
1.99x seconds
Adding this to the regular expression also solves it. Adding .*
probably was
an incident (or matching to .* takes more than 0.00x seconds, which
would
also explain why that works)

I have no idea why it takes slightly less than 2 seconds.

Ronnie

On 27 jan, 15:38, Misha Seltzer <misha.selt...@gmail.com> wrote:
> hmm.. there seem to be another error. it ran under 2 seconds (though test
> assumes it'll take more...)
> But, generally, checking against regex is a hard-to-do-right thing, so we
> might have forgotten something there, and you're welcome to add anything you
> think will work right :)
>

[Ori Peleg]

On *nix I run the tests with "make test", this sets the necessary environment.

Regardless I would like to change this so that the "testoob" executable would be runnable from the development directory without requiring environment changes or installation, similar to what the Bazaar developers did - if you check out the Bazaar source code, you can run "./bzr" with no setup and it'll work. That would be cool.

--
Check out my blog: http://orip.org

[Ori Peleg]

I agree with Misha - our system tests parse the output, and many are brittle: they depend on too much from the output and break too easily. Making the regexps less sensitive is a good idea.