Board-independent tests

[Philip Levis]

We had a discussion on the core call today about board independent tests. The basic concern is that we want to be able to write kernel-level tests that are easy to run. The current approach (of commented-out lines in main.rs) isn’t great. You have to load the entire board configuration even if you’re only doing a small test and modify source code to run a test.

One approach using our current build system would be to make lots of new boards. E.g., there’s an imix-alarm-test board, an imix-uart-test board, etc. We could do this, but it means that tests are board specific and changes to kernel APIs mean changes to tons of tests (because they have the boot sequence).

We’d like a design that:
- Allows you to write very short tests: a test should be just the test code and not lots of boilerplate/extra dependencies,
- Allows you to run a hardware-independent (e.g., capsule) test on any platform (they aren’t board-specific unless the test is),
- Allows you to run tests without modifying any code: they should be invocable from the command line through make/installation/tockloader

The basic problem we have is the board/chip namespace. We have HILs that chips implement, but the implementations have hardware-specific names. I.e., there’s no way to say “the Alarm HIL implementation for whatever chip I’m on”. Instead, we say something like:

let ast = &sam4l::ast::AST;
let mux_alarm = AlarmMuxComponent::new(ast)

sam4l::ast::AST implements a HIL, so can be passed to a hardware-independent component (AlarmMuxComponent), but actually naming it require board-specific code. This line looks different on an nRF.

Here’s a straw man approach:

We specify a set of names for HIL implementations. E.g., alarm is ALARM.

The test harness for a given board presents these in a common namespace. E.g., board::ALARM.

The test harness for a board does all of the architectural setup: pins, etc. It’s a shell of a current board. It then calls a well-named function, such as “run_test”.

A test is in a directory. You compile it for a given board by typing the board name (e.g. make imix). It pulls in the test harness for that board and implements run_test. The implementation of run_test accesses any HIL implementations it needs through board:: and does everything it needs.

For starters, we can just have each test describe the expected output, which is manually verified. It could be “pass” at the console or something more complex. We can later discuss ways to automate some tests. This is intended to be about making compiling and running tests easier, automating will be the next step.

This is just a straw man — feedback appreciated.

Phil






———————
Philip Levis (he/him)
Associate Professor, Computer Science and Electrical Engineering
Faculty Director, lab64 Maker Space
Stanford University
http://csl.stanford.edu/~pal