SR2014 Comp Software Fail

[Peter Law]

Hi,

I've cross-posted this to the srobo-devel list as much of this
information will be technical in nature, and relate to the internals
of the SRComp software.

On the srobo@ thread "SR2014 Comp Awards Fail" [0], I wrote:
> we gave the third place trophy to the wrong team. In a
> follow-up post I'll also detail some of the contributing factors.

While the error itself was human, there were a number of contributing
factors both in the software and the environment. I strongly believe
that these can be resolved, and the chances of future errors reduced.

# Some SRComp background

SRComp is also created such that its score loading mechanism will
simply reject everything if any of the scores don't make sense (eg: a
robot has too many tokens). It also separately has validation scripts
which attempt to check that the data being loaded makes sense (eg:
only the teams expected in a match are actually present in the
scores).

There was also a web-based score entry system which provided a form
that deliberately looked almost identical to the scoring sheets.
Unfortunately, it didn't allow the user to override the system with
regards to which teams were present in a given match [2488].

> By the time we'd run the final, we'd hit a pretty serious error in the
> knockouts portion of the competition software (the one Sam announced
> on the day). The effects of this, in combination with some of the
> self-checking that the system does to try to ensure the data it has is
> 'correct', left us unsure about the use of even the scoring input
> system.

# Effects

This bug was that it had no mechanism to cope with scoring ties when
working out who to promote to the next stage [2434].

This resulted first in some teams not being promoted when they should
have been, and later in the displayed schedule being wrong as we moved
to running the match entrants manually.

Starting with the second pair of quarter finals, the scores were
calculated by manually running the game scoring scripts against the
input data. The notes I made at the time are still in the competition
state repo [3], though they're admittedly somewhat cryptic [4].

# End result

It was the combination of:
* not being able to force the scoring system to accept that the teams
actually in the final were not those scheduled,
* a lack of clarity over exactly which invalid-state conditions would
cause the system to just reject the data
which resulted in the YAML file being edited rather than going through the UI.

It's also worth noting that the above issues were present _despite_
the software being operated by those who had written them.

# Improvements

Clearly the linked tickets are a good start towards avoiding these
issues in the future. I'd also highlight that there are a total of
three people who have done anything with the SRComp software, and that
its tests don't cover substantial portions of it.

We also need to have a clearer understanding of what sorts of invalid
state data the system will outright reject, and what is merely
expected to be caught by "validation". This may mean that all of the
error modes need to be documented explicitly somewhere.

Alternatively or even additionally, we might want to work towards
making SRComp able to load all and any scores, and just ignore (and
log) those which are invalid, rather than falling over. Given that it
is used both inside a self-reloading web service and various command
line tools, some effort may need to be spent to ensure that it works
well wherever it is consumed.

Peter

[0] https://groups.google.com/d/topic/srobo/cbcFKXUxLh8/discussion
[2434] https://www.studentrobotics.org/trac/ticket/2434
[2488] https://www.studentrobotics.org/trac/ticket/2488
[3] https://www.studentrobotics.org/cgit/comp/sr2014-comp.git/tree/finals-matches
[4] On lines starting with hyphens: the numbers in brackets are league
positions, the first column of bare numbers are the game score for the
team in their semi final, the second column is the scores in the
final.
On lines which don't start with a hyphen, the numbers are just the
league positions for the team.