Racket Testing

[Paulo Matos]

Hi,

I was looking at exploding the Racket CI configurations to make them
more fine-grained.

For non x86_64 architectures with default flags / racketcs etc, there
are many failures and I would like to make the testing more fine-grained
so that I can lock a configuration /to success/ as soon as possible.
What I mean by this is... most our jobs at the moment have:
'allow_failure: true'. I want to reduce these.

Instead of having a testing job that does:
- raco test -l tests/racket/test 2>&1 | tee logs/test.log
- racket -l tests/racket/contract/all 2>&1 | tee logs/contract-test.log
- raco test -l tests/json/json 2>&1 | tee logs/json-test.log
- raco test -l tests/file/main 2>&1 | tee logs/file-test.log
- raco test -l tests/net/head 2>&1 | tee logs/net-head-test.log
- raco test -l tests/net/uri-codec 2>&1 | tee
logs/net-uri-codec-test.log
- raco test -l tests/net/url 2>&1 | tee logs/net-url-test.log
- raco test -l tests/net/url-port 2>&1 | tee logs/net-url-port-test.log
- raco test -l tests/net/encoders 2>&1 | tee logs/net-encoders-test.log
- raco test -l tests/openssl/basic 2>&1 | tee
logs/openssl-basic-test.log
- raco test -l tests/openssl/https 2>&1 | tee
logs/openssl-https-test.log
- raco test -l tests/match/main 2>&1 | tee logs/match-main-test.log
- raco test -l tests/zo-path 2>&1 | tee logs/zo-path-test.log
- raco test -l tests/xml/test 2>&1 | tee logs/xml-test.log
- raco test -l tests/db/all-tests 2>&1 | tee logs/db-test.log
- raco test -c tests/stxparse 2>&1 | tee logs/stxparse-test.log

When one of them fails, we silence the failure with 'allow_failure:
true'. The problem with this is that if a problem shows up somewhere
else, it will go unnoticed because we allow failure by default. I want
to make each of these a single job and allow failure only on those which
really fail and once they pass, lock them onto success by not allowing
failure anymore.

I initially copied the above lines without the redirections from travis
yml. Is this list of tests up-to-date? Is this all we have and are they
being run correctly?

I should point out I find it slightly strange that the tests seem to be
implemented in different ways since the contract tests are ran with
`racket -l`, and stxparse are ran with `raco test -c`. What's the reason
for this difference?

Kind regards,
--
Paulo Matos

[Sam Tobin-Hochstadt]

There are a few things here:

1. The set of tests is selected mostly based on what doesn't have a
lot of dependencies, tests things in the `racket/racket` repository,
and completes in the Travis CI time limit.
2. Some tests are run under `racket` vs `raco test` for historical
reasons, probably that could be fixed. I originally took the command
lines from the release checklist.
3. The tests are all together because building racket takes most of
the time, and therefore splitting the tests didn't make sense. For a
system with better pipeline support, splitting things up could work.

One important background piece of information is that because
drdr.racket-lang.org exists, typically other CI systems have focused
on either system coverage or configuration coverage, and mostly on the
core tests. Obviously that isn't ideal, but even before it tested
RacketCS, DrDr took 8+ CPU hours to run a single full test run, and so
doing that in other CI systems wasn't possible.

Sam

> --
> You received this message because you are subscribed to the Google Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.
> To post to this group, send email to racke...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/a010e885-4f63-44d4-e73e-5457a0159fdc%40linki.tools.
> For more options, visit https://groups.google.com/d/optout.

[Paulo Matos]

On 05/06/2019 22:11, Sam Tobin-Hochstadt wrote:
> There are a few things here:
>

Thanks for the clarification.

> 1. The set of tests is selected mostly based on what doesn't have a
> lot of dependencies, tests things in the `racket/racket` repository,
> and completes in the Travis CI time limit.

If there are more tests I should run please let me know. :) From what
you say the lists do not seem to be complete. Even if we do not run all
at every commit we could do it overnight - as we do for emulation
testing. Also, with GitlabCI we don't have a time limit anymore.

> 2. Some tests are run under `racket` vs `raco test` for historical
> reasons, probably that could be fixed. I originally took the command
> lines from the release checklist.

OK.

> 3. The tests are all together because building racket takes most of
> the time, and therefore splitting the tests didn't make sense. For a
> system with better pipeline support, splitting things up could work.
>

I now own a large server with 40cores and bucket loads of memory. In
total I now have about 70cores at home in servers for Racket CI.
Building racket3m takes about 3mins while building racketcs takes about
8mins with gcc -O3. I have the ability now to just run several tiny jobs
simultaneously which means I can have finer grained control over what's
passing and failing and mark it accordingly.

> One important background piece of information is that because
> drdr.racket-lang.org exists, typically other CI systems have focused
> on either system coverage or configuration coverage, and mostly on the
> core tests. Obviously that isn't ideal, but even before it tested
> RacketCS, DrDr took 8+ CPU hours to run a single full test run, and so
> doing that in other CI systems wasn't possible.
>

Now it is... :) What do you mean by a single full test run? All the
tests below, or running even further tests?

--
Paulo Matos