Regression testing?

[Clifford Heath]

Hi folk.

As one of the new potential maintainers for HL2’s RTL gateware, I’m concerned that I don’t know enough about how to test any changes. I’d like to start a discussion about how we can automate some testing. Hopefully this message will invoke a response from some of the SDR program authors.

The biggest win I envisage is if we could add a loopback mode, so that Tx data runs through the whole Tx chain but just before being sent too the AD9866 chip, it gets fed back in to the receive chain. This way we could have pre-recorded or synthesised Tx signals and could look for an appropriate response from the receiver. Of course if we get nothing (or the wrong thing) we don’t know whether the receiver or transmitter path is broken, but at least it would provide a simple go/no-go test that could be scripted from Python to test almost any aspect of the signal path.

If I can add a loopback mode (and this requires knowing whether Metis has a place for such a thing) can I get a show of hands for folk willing to help create test cases?

Clifford Heath.

[ron.ni...@gmail.com]

Loopback testing is already a part of the standard HL2v9 bring-up tests.  A cable with a 30 dB attenuator between the two connectors on the N2ADR filter board allow the Tx Rx path, including the FPGA, AS9866, and filter board, but not the amp, to be tested.  See https://github.com/softerhardware/Hermes-Lite2/wiki/Raspberry-Pi-Test-and-Program for photos of the loopback cable, and the hl2_setup code here: https://github.com/softerhardware/Hermes-Lite2/tree/master/software/hl2setup which runs the loopback tests.

73, Ron, n6ywu

[Clifford Heath]

Hi Ron,

Thanks, that’s a great start. I had a quick scan of the code and I’m not sure how many of the possible modes it uses. It seems to have a static test it runs to verify operation, rather than a signal test that could e.g. probe for spurious signals originating in numeric overflows of filters, etc… It would be good to have a driver that says “transmit this, and expect to receive that”, for any pair of Tx/Rx files, and with the hardware previously set into any possible mode.

I had started for example to remove compile warnings about numeric truncation. So if there’s a 12-bit variable in a register somewhere, and the code adds 1 to it, well “1” is a 32-bit literal, so the result is 32 bits, so when you assign it back to the 12-bit register you get a truncation warning. To avoid that, you can use 12’1, a 12-bit literal “1”. But if there is a mistake in expressions like this, they could cause mysterious failures in specific code paths (not this one, which is probably safe). So if I want to be sure that my “warning reduction” change hasn’t broken anything, I need a test that actually exercises that code path.

We don’t have a way to measure the coverage of even the limited tests that we have, let alone determine if a given expression ever gets exercised by some test.

I envisage in the future being able to get (most of) the Hermes gateware running in a Verilator simulation, so you don’t even need hardware to run a regression test (obviously this won’t test the AD9866 or FX2 drivers, but could go close). If we make that possible, we could also get full coverage metrics, and work towards 100% coverage so we can deploy new code with the certainty that (barring device driver bugs) it will work as expected.

Clifford Heath.

--
You received this message because you are subscribed to the Google Groups "Hermes-Lite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/8ff51f5d-a78f-49dc-88cd-729b3f920e26n%40googlegroups.com.

[James Ahlstrom]

Perhaps you could use Quisk for Tx/Rx regression testing. In FDX (full duplex) mode Quisk receives its own transmission. There is enough leakage at 5 watts so you don't need a loopback cable and attenuator; just a dummy load. It might be sufficient to transmit on each mode and look for a clean spectrum. Integer overflows should show up as spurs in the FFT display. If you want something more automated, Quisk is open source and has a Python interface that could be used.

Jim

N2ADR