Should fast test suites be run from spkg-install?
Dr. David Kirkby
that slows the build time - in some cases very significantly. I think with Pari,
it takes about 10x as long to run the test suite as it does to just build the code.
But for some programs, the time need to run the test suite is very small indeed.
(Many take less than 10 seconds on my Sun Ultra 27).
Would it not be sensible to always run the test suites on packages where they
run quickly and exit if there are errors?
I'd suggest as a cut-off, using 30 seconds on sage.math. In other words, if a
test suite can be run in under 30 seconds on sage.math, then run it every time
Sage is built. If it takes longer than that, then only run it from spkg-check.
I doubt this would have much impact on the time to build Sage, but it may
improve the quality. The reasons I believe it will have minimal impact is
* Only a small number of packages have test suites. (About 20% to date).
* Of those that do have test suites, about 50% take less than 10 seconds to
run on my Sun Ultra 27 (probably a bit longer on sage.math)
* Several of the test suites would fall outside the 30 second cutoff, so would
not get run anyway.
Despite the tile of the ticket, if you ignore anything other than the first two
columns here, you will get a list of what packages have spkg-check files.
http://trac.sagemath.org/sage_trac/ticket/9281
Dave
Robert Bradshaw
+1 to running all short test suites on build if they're small enough
(e.g. less than 30 seconds, or, say, 25% of the build time). That is
assuming they're expected to pass, unlike some (scipy?).
- Robert
Simon King
On 23 Aug., 18:26, Robert Bradshaw <rober...@math.washington.edu>
wrote:
> (e.g. less than 30 seconds, or, say, 25% of the build time). That is
> assuming they're expected to pass, unlike some (scipy?).
spkg-install, as this thread's title suggests? I could imagine that
typically the tests of one standard spkg will depend on other standard
spkgs. So, tests will only be reasonable *after* the spkg-installs of
all standard spkgs are done.
But apart from that, I am +1
Cheers,
Simon
David Kirkby
> +1 to running all short test suites on build if they're small enough
> (e.g. less than 30 seconds, or, say, 25% of the build time). That is
> assuming they're expected to pass, unlike some (scipy?).
>
> - Robert
Robert,
Are you suggesting (1) or (2) below?
1) Run if tests take < 30 seconds AND <25% of the build time.
2) If tests take <30 seconds OR < 25% of the build time.
I can see a problem with the latter.
1) Some tests take <30 s to run, but that is probably more than 25% of
the build time. The simple packages take very little time to build,
but very little time to test too.
2) Some packages take a long time to build. ATLAS especially, can take
hours depending on whether it is aware of the tuning parameters for
your hardware. On t2.math, ATLAS takes about 8 hours to build. I think
it would be inappropriate to up to an additional 2 hours in that case
if the tests took 2 hours.
So personally I think a 30 second cutoff is sufficient, without
additional criteria.
Dave
David Kirkby
No, that should not be a problem. If spkg-check can run a test,
spkg-install should be able to too.
There is the odd package where Python is needed to run the tests, but
Python is not needed to build the package. They are rare. In this
case, Python is already marked as a prerequisite in
spkg/standard/deps, so no changes are needed.
This should be as simple as just running the tests from spkg-install,
unless I have overlooked something.
Dave
Dave.
David Kirkby
> Robert,
>
> Are you suggesting (1) or (2) below?
>
> 1) Run if tests take < 30 seconds AND <25% of the build time.
>
> 2) If tests take <30 seconds OR < 25% of the build time.
>
> I can see a problem with the latter.
I mean I can see a problem with both!
I think the 30 s on sage.math is a reasonable criteria alone. One
might argue for one minute, but I don't think the criteria needs to be
any more complex than just a simple time.
You could make it more complex if you wanted, but then you would need
to consider exceptions like ATLAS, for the reasons I stated.
Dave
Robert Bradshaw
I wasn't proposing a rule, just a rule of thumb. And I did mean OR. If
a package took 5 seconds to build, then 30 seconds to test, that would
not be an issue for me, as relative to the total build time it's still
negligible. OTOH, I wouldn't want to rule out a more expensive test
for a more significant chunk spkg just because it broke the 30 second
limit. Essentially what I'm saying is that, within reason, I'd be
willing to give slightly more time to larger packages than smaller
ones. ATLAS is a clear exception to the 25% idea, and I'm sure not the
only one.
- Robert
David Kirkby
>> You could make it more complex if you wanted, but then you would need
>> to consider exceptions like ATLAS, for the reasons I stated.
>
> I wasn't proposing a rule, just a rule of thumb. And I did mean OR. If
> a package took 5 seconds to build, then 30 seconds to test, that would
> not be an issue for me, as relative to the total build time it's still
> negligible. OTOH, I wouldn't want to rule out a more expensive test
> for a more significant chunk spkg just because it broke the 30 second
> limit. Essentially what I'm saying is that, within reason, I'd be
> willing to give slightly more time to larger packages than smaller
> ones. ATLAS is a clear exception to the 25% idea, and I'm sure not the
> only one.
>
> - Robert
Fair enough.
What you are proposing seems perfectly logical.
Dave