Competition Software

[Peter Law]

Hi everyone,

As I've noted elsewhere, following this year's competition, I think
we've actually got a somewhat reasonable base for comp software, in
the form of poltergeist [0]. However, it can be improved, and should
be. With that in mind, I'd like to hear input from everyone to help
create a vision of what we want [1].

I've therefore started to create a broad user-spec [2] for what we
want out of competition software as a whole. The intention of this
page is to list the things that we want to achieve at the competition,
which can probably be solved in the software domain. In this context,
I'm deliberately not trying to pin down an implementation -- that can
be done separately [3], instead I'd like to create a prioritised list
of things we want to achieve.

At the moment, the feature list [2] contains just the stuff that I was
directly involved with, and I'm aware that it's lacking things like
the team-layout-colour-coded schedule, homepage mods, knockout render
& zone displays. Hopefully those who worked on these functionalities
can add to the page with the things they prioritised.

Once we know what we want, it should be much easier to create some
implementation specs, and then implement things against those. I was
hoping at this point to refer to Rob's sys-eng [4] stuff, possibly
with suggestions on how we could tweak that to adopt it better, but it
appears to have disappeared.

Ideally, this should allow us to create a rock solid suite that avoids
the last-minute rush we've had in the past.

All input is welcome.

Thanks,
Peter

[0] https://www.studentrobotics.org/cgit/comp/poltergeist.git
[1] I promise that's the last of the management speech
[2] https://www.studentrobotics.org/trac/wiki/Competition/Software
[3] I fully expect we'll prototype stuff, even while trying to pin
down the requirements -- creating demos to thrash out ideas can be
very helpful.
[4] https://groups.google.com/d/msg/srobo/DUhAPPECOTc/7OMHaKdkU2MJ

[Jeremy Morse]

Hi,

On 20/04/13 12:04, Peter Law wrote:
> I've therefore started to create a broad user-spec [2] for what we
> want out of competition software as a whole. The intention of this
> page is to list the things that we want to achieve at the competition,
> which can probably be solved in the software domain. In this context,
> I'm deliberately not trying to pin down an implementation -- that can
> be done separately [3], instead I'd like to create a prioritised list
> of things we want to achieve.

Looking good; My comments with no particular theme:
* I think the 'goodness' requirements, things like having two-match gap
should be a little vaguer. Our actual aim is to increase that metric
to as high as possible, I'd have said two matches was too low this
year if we weren't pressed for time.
* I wouldn't care so much about an output format, just that it meets
some kind of standard (TLAs, comma separated, one line per match for
example) that computers can read and from which a human visualisation
can be produced. We can then all write tools separately and compare
them.
* I also wouldn't say having a reproducible schedule was a requirement;
it's a good feature, but I would happily trade it for better goodness
metrics. I'm currently speculating on using some SAT solvers [0] to
spit out schedules, which may or may not be reproducible.
* All the data described should be published on the website first and
elsewhere later; everyone can get at the website, always (TM).
* An ability to view the schedule just for one team would be intensely
useful, a lot of teams would have benefited from that this year would
have been good.
* I vastly agree with publishing the game points; we are going to end
up making data entry errors inevitably (I certainly did), and
allowing teams to check this later will reduce suspicion and ease
complaints (CLF identified an error made for example, but it took up
quite a bit of several peoples time to check).

> At the moment, the feature list [2] contains just the stuff that I was
> directly involved with, and I'm aware that it's lacking things like
> the team-layout-colour-coded schedule, homepage mods, knockout render
> & zone displays. Hopefully those who worked on these functionalities
> can add to the page with the things they prioritised.

> Once we know what we want,

IMO we should spec out what information people need according to their
role and go from there; we may be able to explore better ways of
visualising or delivering information, and prioritising which will make
the most difference is good. Aiding the shepherds will be more useful
than the knockout data for spectators, IMO for example.

> Ideally, this should allow us to create a rock solid suite that avoids
> the last-minute rush we've had in the past.

I agree with this at full force.

[0] Which are designed to solving general NP complete problems.

--
Thanks,
Jeremy

[Peter Law]

Jeremy wrote:
> Looking good; My comments with no particular theme:
> * I think the 'goodness' requirements, things like having two-match gap
> should be a little vaguer. Our actual aim is to increase that metric
> to as high as possible, I'd have said two matches was too low this
> year if we weren't pressed for time.

Yup, changed.

> * I wouldn't care so much about an output format, just that it meets
> some kind of standard (TLAs, comma separated, one line per match for
> example) that computers can read and from which a human visualisation
> can be produced. We can then all write tools separately and compare
> them.

Yup, changed. Currently things use TLAs, pipe separation & one line
per match, so I've noted this & suggest it continue being used.

> * I also wouldn't say having a reproducible schedule was a requirement;
> it's a good feature, but I would happily trade it for better goodness
> metrics. I'm currently speculating on using some SAT solvers [0] to
> spit out schedules, which may or may not be reproducible.

Yup, now just a 'may'.

> * All the data described should be published on the website first and
> elsewhere later; everyone can get at the website, always (TM).

Yup, I've added a definitions section, noting that 'publicly visible'
effectively means this.

> * An ability to view the schedule just for one team would be intensely
> useful, a lot of teams would have benefited from that this year would
> have been good.

Added.

> * I vastly agree with publishing the game points; we are going to end
> up making data entry errors inevitably (I certainly did), and
> allowing teams to check this later will reduce suspicion and ease
> complaints (CLF identified an error made for example, but it took up
> quite a bit of several peoples time to check).

Indeed, added, I've also added stuff about the other check stuff I
ended up creating.

> IMO we should spec out what information people need according to their
> role and go from there; we may be able to explore better ways of
> visualising or delivering information, and prioritising which will make
> the most difference is good. Aiding the shepherds will be more useful
> than the knockout data for spectators, IMO for example.

I've added some notes about who certain features are aimed at, but a
discussion about what data each role needs would also be useful.
Perhaps the srobo-roles [1] document is a good place to start for
this?

Peter

[1] https://github.com/prophile/srobo-roles

[Rob Spanton]

Hey,

Peter Law wrote:
> I've therefore started to create a broad user-spec [2] for what we
> want out of competition software as a whole.

This generally looks sensible to me. I'm a bit concerned that people
are going to read it and believe that we're going to attempt to
implement all of it at once in one go. Hopefully we can avoid this.

> At the moment, the feature list [2] contains just the stuff that I was
> directly involved with, and I'm aware that it's lacking things like
> the team-layout-colour-coded schedule, homepage mods, knockout render
> & zone displays. Hopefully those who worked on these functionalities
> can add to the page with the things they prioritised.

I'll add this to my to-do list...

> Once we know what we want, it should be much easier to create some
> implementation specs, and then implement things against those. I was
> hoping at this point to refer to Rob's sys-eng [4] stuff, possibly
> with suggestions on how we could tweak that to adopt it better, but it
> appears to have disappeared.

Disappeared, but reappeared here:
https://xgoat.com/~rob/sr2013/syseng/process.html

( I relocated all my srobo.org/~rspanton stuff to xgoat.com/~rob )

R

[Sam Phippen]

Yo

Matches must be scheduled in ​rounds. A round is a block of matches that contains all teams exactly once

this requirement just got added. I think the must is needlessly strong, it sort of implies that all schedulers that don't do this are broken. I'm not sure I agree that that is the case. I think this should be a "should" or even "may find it helpful to". 

[Rob Spanton]

On Mon, 2013-04-22 at 01:20 +0100, Sam Phippen wrote:
> Matches must be scheduled in rounds. A round is a block of matches
> that contains all teams exactly once

> this requirement just got added. I think the must is needlessly
> strong, it sort of implies that all schedulers that don't do this are
> broken.

They are!

We want the schedule to be in rounds because:

* It makes the problem simpler for *people* and machines
* Ensures that team matches are adequately spaced throughout the
comp.

If you don't schedule in rounds, you end up with a schedule like the one
that we had in SR2012 where some teams were really bunched up.

Cheers,

R

[Alistair Lynn]

Hi Rob-

> * Ensures that team matches are adequately spaced throughout the
> comp.
>
> If you don't schedule in rounds, you end up with a schedule like the one
> that we had in SR2012 where some teams were really bunched up.

I would dispute that. Maximisation of spacing between a team's games as an optimisation problem would also solve this.

Alistair

[Sam Phippen]

Exactly. Rounds are a way to get the properties we want, it should be
up to the implementer to decide if they want to solve the problem
using rounds or some other technique.

Also rounds were not used for this years schedule and we were able to
optimise for two of the criteria by hand!

I definitely think this is too much of an implementation detail for
this comp software spec

Sent from my phone please excuse my brevity.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "Student Robotics" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to srobo+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Rob Spanton]

On Mon, 2013-04-22 at 01:29 +0100, Alistair Lynn wrote:
> I would dispute that. Maximisation of spacing between a team's games
> as an optimisation problem would also solve this.

Both match schedulers that we have seen so far (SR2012's and SR2013's)
have both neglected to space teams out sufficently well. Neither of
these used rounds. They both made the optimisation problem
unnecessarily hard for themselves by not using rounds.

Splitting into rounds make the optimisation more straightforward as it
reduces the search space significantly.

It seems pretty obvious to me that the scheduler needs to use rounds.
Why would we want one that did not?

Cheers,

R

[Rob Spanton]

On Mon, 2013-04-22 at 01:33 +0100, Sam Phippen wrote:
> Exactly. Rounds are a way to get the properties we want, it should be
> up to the implementer to decide if they want to solve the problem
> using rounds or some other technique.

I'm not convinced. We've left this decision to the implementer for the
last couple of years and it's gone wrong both times!

R

[Jeremy Morse]

Hi,

Without seeking to comment on the optimality of round scheduling, it has
other good properties too. If we lose power, network, gravity, or
anything to stop the league from running for a period of the
competition, then a round can be cut off the end of the day and the
league remains fair in terms of number of matches played.

--
Thanks,
Jeremy

[Alistair Lynn]

Hi Jeremy-

It's a good point, although it does mean that our rounds cannot start half way through a match.

Alistair

[Alistair Lynn]

Hi Rob-

> Splitting into rounds make the optimisation more straightforward as it
> reduces the search space significantly.

What I think Sam is saying is that given there is more than one way to approach the problem, a spec for requirements should document requirements rather than solutions.

Alistair

[Rob Spanton]

This is a matter of perspective. You can cast pretty much any item in a
spec as a requirement or a solution.

For example, you could say that naming teams with TLAs was a solution to
the team naming problem -- therefore the spec document should not
require that teams are named with TLAs.

The same applies to the rounds requirement I believe. As I've already
said, but you two are ignoring: we've had two match schedulers that have
both failed to implement rounds correctly (the first had no rounds and
so some teams had all their matches really bunched up, and the second
had some kind of buggy round situation where some teams didn't even
appear!).

I'll repeat my question: Why would we want a match scheduler that did
not have rounds?

R

[Alistair Lynn]

Hi Rob-

> but you two are ignoring

ಠ_ರೃ

> we've had two match schedulers that have both failed to implement rounds correctly (the first had no rounds and so some teams had all their matches really bunched up, and the second had some kind of buggy round situation where some teams didn't even appear!).

I am aware of that, and I'm also entirely aware that there may be other solutions we haven't even thought of to the problem. Again, rounds may be solution to the problem - but the actual /problem/ is bunching up of teams, which to me seems to be clearly what we should spec a scheduler must avoid.

Alistair

[Rob Spanton]

This is essentially the requirement that we've had for the last two
years. It was reasonably clear originally that teams needed their
matches spread out. After SR2012, it was even more clear. Yet the
SR2013 scheduler did not achieve this aim either. We need to find a way
of ensuring that our spec does not get misinterpreted by the
implementers in the ways that they did in SR2012 and SR2013.

So we learn from our past mistake. We know that the solution to the
bunching problem is to schedule in rounds. So we write it into the spec
to ensure that this mess-up never happens again. Sure, it means that
the implementer now has less freedom to try out different approaches
[1], but it ensures that we end up with something that will actually
work. We've tried giving them that freedom, but it's ended badly.

If the spec just had a line like "teams should be bunched up", then I
would fully expect us to end up with a suboptimal scheduler again.

Cheers,

Rob

[Alistair Lynn]

Hi Rob-

So we learn from our past mistake.  We know that the solution to the
bunching problem is to schedule in rounds.  So we write it into the spec
to ensure that this mess-up never happens again.  Sure, it means that
the implementer now has less freedom to try out different approaches
[1], but it ensures that we end up with something that will actually
work.  We've tried giving them that freedom, but it's ended badly.

Rigidity in the spec seems a poor approach here. I would much prefer to see a separately-developed system of tests which can be run against match schedules codifying any guarantees we want to make (match spacing being one) and using that to validate a scheduler than trying to force an approach by restricting the spec to a single solution.

 

If the spec just had a line like "teams should be bunched up", then I
would fully expect us to end up with a suboptimal scheduler again.

Yes, I'd imagine we would - I suspect you've negated the condition there :)

Alistair

[Alistair Lynn]

Hi Rob-

Rigidity in the spec seems a poor approach here. I would much prefer to see a separately-developed system of tests which can be run against match schedules codifying any guarantees we want to make (match spacing being one) and using that to validate a scheduler than trying to force an approach by restricting the spec to a single solution.

 

Actually, no, I misspeak - it's simpler than that. What we need is the spec to be concrete and testable on this point.

Alistair

[Rob Spanton]

On Mon, 2013-04-22 at 02:37 +0100, Alistair Lynn wrote:
> Rigidity in the spec seems a poor approach here. I would much prefer to see
> a separately-developed system of tests which can be run against match
> schedules codifying any guarantees we want to make

The spec is the thing that defines what those tests are. I think that
one of those tests would be that the matches are scheduled in rounds!

> (match spacing being
> one) and using that to validate a scheduler than trying to force an
> approach by restricting the spec to a single solution.

Adding the requirement for rounds is not forcing the spec to a single
solution. There will still be a plethora of solutions to the problem
that the spec outlines -- however, all of those solutions will involve
rounds.

So far all I've heard is speculation that there might be another way of
achieving the same thing that rounds achieve. No-one has actually said
what this is. I am inclined to believe that there are no other
reasonable ways.

Cheers,

Rob

[Alistair Lynn]

Hi Rob-

The spec is the thing that defines what those tests are.  I think that
one of those tests would be that the matches are scheduled in rounds!

What I was actually saying, as I clarified in an email shortly afterwards, is that the requirements need to be concrete. From what I understand of what you've said so far, your problem with vending a requirement of spacing out the matches is that it will be ignored. That, I suspect, is because it's vague.

If it can be put, perhaps with a statistical test, in a manner in which the answer to the question "is this schedule in spec?" is always "yes" or "no", this problem goes away. A scheduler which bunches up matches will unambiguously be out of spec.

 

Adding the requirement for rounds is not forcing the spec to a single
solution.  There will still be a plethora of solutions to the problem
that the spec outlines -- however, all of those solutions will involve
rounds.

So far all I've heard is speculation that there might be another way of
achieving the same thing that rounds achieve.  No-one has actually said
what this is.  I am inclined to believe that there are no other
reasonable ways.

Might I be so bold as to suggest that the statement "x may exist" is not identical to the statement "x is immediately apparent to the subscribers of the srobo mailing list". This particular problem is something we, as a group, have been investigating for all of 12 hours since Sam queried the rounds item in the spec last night on-list -- hardly enough time to conclude that there is only one possible solution to a problem!

As a matter of fact, there have even been mutterings on-list about SAT solvers[1] being used as a way of solving constraints in the schedule, which could well include the naive distribution problem.

Alistair

[1] http://thread.gmane.org/gmane.science.robotics.srobo.general/3573/focus=3574

[Rob Spanton]

Alistair wrote:
> What I was actually saying, as I clarified in an email shortly afterwards,
> is that the requirements need to be concrete. From what I understand of
> what you've said so far, your problem with vending a requirement of spacing
> out the matches is that it will be ignored. That, I suspect, is because
> it's vague.

And by saying that matches need to be scheduled in rounds (where each
round contains all teams exactly once), the spec is closer to being
concrete.

I wrote:
> So far all I've heard is speculation that there might be another way of
> achieving the same thing that rounds achieve. No-one has actually said
> what this is. I am inclined to believe that there are no other
> reasonable ways.

Alistair wrote:
> Might I be so bold as to suggest that the statement "x may exist" is not
> identical to the statement "x is immediately apparent to the subscribers of
> the srobo mailing list".

Of course. However, we're not working in some theoretical space here.
We're trying to engineer something and move on. We don't need to be
concerned with whether there are any other approaches because we have a
solution that completely fits with all of our constraints. There is no
point in wasting our time trying to ensure that there is no other way of
doing it. We have a solution that fits our needs. We should use it.

May I suggest that we set a time limit for your investigations into
whether there are other alternatives to rounds, because we want to move
on. How about 2 weeks?

Cheers,

Rob

[Sam Phippen]

Hi,

Regarding your question of "why shouldn't we use rounds?": We have no evidence that rounds are the only way to generate a schedule in the way we want. As such we shouldn't force people to use it, but I think we should recommend people investigate

So far nobody has actually built a round based scheduler that solves all our conditions, in 2012 we had a scheduler that didn't use rounds that failed, in 2013 we had a scheduler that did use rounds that failed. The reason for both failures was not the presence or lack of presence of rounds, but rather a distinct lack of testing. The conditions that we've asserted on the trac page minus the rounds condition still allow for a good schedule to be created. By requiring 2 matches out of the arena or more we solve the main problem that rounds try to solve. These conditions essentially state the same thing, but one is more open to different solutions.

I took a look at peter's blue flame and it doesn't appear to use rounds, yet it meets most of the conditions that we want. I have seen no strong evidence that suggests that rounds automatically imply a schedule that we want, and that software testing is far more important. Mister Rob: would you like to evaluate blue flame and see if you think its output is good, and if it isn't good enough I implore you to send patches that use rounds and see if it makes it any better. Even better than trying to argue about whether rounds actually help would be to have some kind of round based scheduler to compare against a non round based scheduler. As it is I see no strong reason why we should include this as a "must" condition on the spec.

[Rob Spanton]

On Mon, 2013-04-22 at 14:16 +0100, Sam Phippen wrote:
> Regarding your question of "why shouldn't we use rounds?": We have no
> evidence that rounds are the only way to generate a schedule in the
> way we want. As such we shouldn't force people to use it, but I think
> we should recommend people investigate

This is like saying to your plumber "I recommend that you use pipes to
move the water around my house" -- having just had two plumbers come
into your house who've made a mess of the plumbing.

> So far nobody has actually built a round based scheduler that solves
> all our conditions, in 2012 we had a scheduler that didn't use rounds
> that failed, in 2013 we had a scheduler that did use rounds that
> failed.

The SR2013 scheduler did not use rounds. A round is a set of matches
that include each team once. The SR2013 scheduler had a concept in it
called "rounds", but these did not include each team once -- a fact that
you, Sam, reported to me yourself!

> The reason for both failures was not the presence or lack of presence
> of rounds, but rather a distinct lack of testing.

I think it was a lack of both.

> I took a look at peter's blue flame and it doesn't appear to use
> rounds, yet it meets most of the conditions that we want. I have seen
> no strong evidence that suggests that rounds automatically imply a
> schedule that we want, and that software testing is far more
> important.

You just have to look at the logical implications of rounds to see that
they result in a schedule that is what we want!

Blueflame does not result in a schedule that can be easily truncated by
us -- which is the thing that Jeremy brought up in this thread earlier.
It's also a requirement in the spec as it stands.

> Mister Rob: would you like to evaluate blue flame and see if you think
> its output is good, and if it isn't good enough I implore you to send
> patches that use rounds and see if it makes it any better. Even better
> than trying to argue about whether rounds actually help would be to
> have some kind of round based scheduler to compare against a non round
> based scheduler.

I can link to a scheduler here that does exactly that here:
http://www.idleloop.com/matchmaker/
There is a download button at the bottom for evaluation purposes.

> As it is I see no strong reason why we should include this as a "must"
> condition on the spec.

By including the round requirement in the spec, the spec becomes
simpler. The following items could be removed:

* Teams must have their matches spread throughout the league.
* All teams (that make it to the competition) must be scheduled
matches.
* The number of matches that teams have must have a maximum
variance of 1 match.
* The schedule may be balanced such that a chunk of matches can be
dropped due to a time constraint without biasing the league.

Cheers,

Rob

[Alistair Lynn]

Hi Rob-

> * The number of matches that teams have must have a maximum
> variance of 1 match.

As an aside, I think this should be modified to require that the number of /non-surrogate appearances/ that teams have be exactly equal, and that teams have 0 or 1 surrogate appearances.

Alistair

[Peter Law]

Hi,

> For example, you could say that naming teams with TLAs was a solution to
> the team naming problem -- therefore the spec document should not
> require that teams are named with TLAs.

It doesn't -- the mention of TLAs in this section comes from a comment
about the current output format, rather than an actual requirement.
(Other sections do refer to TLAs because these are are the current
standard short team ids, and it's necessary to use them for display.)

> We know that the solution to the
> bunching problem is to schedule in rounds.

Correction, we know that *a* solution to the bunching problem exists
that involves the use of rounds. It is entirely possible that a
rounded scheduler could still produce bunched output. Consider the
following sample:

Round 1:
ABC | DEF | GHI
JKL | MNO | PQR
RST | UVW | XYZ

Round 2:
ABC | JKL | RST
DEF | MNO | UVW
GHI | PQR XYZ

Note that RST has adjacent matches, and that the presence of rounds
has done nothing to resolve this. I'm not saying that rounds can't
help, just that they're not the magic bullet they're being presented
as.

Having a bunching related requirement *does*, however, resolve this issue.

<*discussion of previous schedulers*>

I remain unconvinced that the _main_ issue with the previous
schedulers was the lack of rounds. A lack of testing, (both of the
logic, and of the output), as well as a lack of tooling with which to
perform said testing was the main issue.

SR2013 is the second competition where I've written schedule checking
scripts at high speed due to a gut feeling that things were wrong.
(I'm thankful this year it occurred before the competition, with
[admittedly barely] time to fix it). I do *not* believe that adding
rounds will make the schedules suddenly massively easier for humans to
check the schedules (I do expect some improvement, though only if the
rounds are marked, e.g. as above).

> So far all I've heard is speculation that there might be another way of
> achieving the same thing that rounds achieve. No-one has actually said
> what this is. I am inclined to believe that there are no other
> reasonable ways.

Reading "the same thing that rounds achieve" as being mostly about
bunching (lack of context in the original..):

This _is_ precisely what BlueFlame does. It avoids bunching by
scheduling matches for teams that haven't had a match recently. While
there isn't anything to actually prevent the bunching in the internal
logic, IIRC tests on the output put a minimum match proximity around 5
[1] before we've even started looking at improving it. I think this is
pretty solid evidence that this is another viable route.

> Blueflame does not result in a schedule that can be easily truncated by
> us -- which is the thing that Jeremy brought up in this thread earlier.
> It's also a requirement in the spec as it stands.

BlueFlame was not written with this requirement in mind -- it was
intended as an experiment to resolve the spacing issue, with some
handling of the even-opponents constraint.

> By including the round requirement in the spec, the spec becomes
> simpler. The following items could be removed:
>
> * Teams must have their matches spread throughout the league.
> * All teams (that make it to the competition) must be scheduled
> matches.
> * The number of matches that teams have must have a maximum
> variance of 1 match.
> * The schedule may be balanced such that a chunk of matches can be
> dropped due to a time constraint without biasing the league.

I don't like the idea of removing a large portion of the spec, and
replacing it with something that is clearly implementation-related,
and could be misunderstood. I also like that at the moment, each of
the feature points is understandable without having to add definitions
of things (we'd need to explain what rounds were, and why they're
good, which is basically a copy/paste of these anyway), and is
independently testable.

Furthermore, as noted above, I don't believe that rounds actually
ensure that all of these are included.

Thanks,
Peter

[1] using https://www.studentrobotics.org/cgit/comp/match-scheduler2.git/tree/checks/close.py

[Rob Spanton]

On Mon, 2013-04-22 at 21:45 +0100, Peter Law wrote:
> Round 1:
> ABC | DEF | GHI
> JKL | MNO | PQR
> RST | UVW | XYZ
>
> Round 2:
> ABC | JKL | RST
> DEF | MNO | UVW
> GHI | PQR XYZ
>
> Note that RST has adjacent matches, and that the presence of rounds
> has done nothing to resolve this. I'm not saying that rounds can't
> help, just that they're not the magic bullet they're being presented
> as.

I'm not suggesting that the "Teams must have gaps between any arena
appearances." line of the spec goes away, nor have I suggested that it
should. If you consider the next round of the schedule you've written
above, then RST would most definitely have a match that was non-adjacent
to its last appearance.

R

[Rob Spanton]

On Mon, 2013-04-22 at 21:45 +0100, Peter Law wrote:
> I remain unconvinced that the _main_ issue with the previous
> schedulers was the lack of rounds. A lack of testing, (both of the
> logic, and of the output), as well as a lack of tooling with which to
> perform said testing was the main issue.

I'm not saying that it was the main issue. It was a significant issue:
the people who wrote the scheduler did not develop the appreciation that
rounds significantly reduce the search space that the scheduler has to
deal with. It changes the match scheduler from searching for a needle
in a haystack to searching for a stick in a haystack.

We're not looking to write a spec that is entirely devoid of details
about a solution. I don't think any spec should ever be like that.
We're looking to write a spec that fits our needs, and will end up with
something that we can actually use. We know that scheduling in rounds
results in something usable (as demonstrated by the MatchMaker
scheduler), and we can see that it significantly reduces the search
space whilst bringing in useful constraints that we want. What we've
also seen is that the developers of our last two schedulers did not
absorb how much the inclusion of rounds would make the entire situation
easier. The best way of getting the implementers of the SR2014
scheduler to understand this is to put it in the spec.

All we've heard so far is the spreading of some FUD about whether this
might overly constrain the development of the match scheduler resulting
in some kind of badness. Yet there are no examples of how it would.
(Also note that the whole point of a spec is to constrain.)

How about this alternative approach: let's make rounds an intrinsic
feature of the match *schedule* and not the match *scheduler*. That way
you guys can have a vaguer match scheduler spec, and the invisible
monsters that you say might exist will no longer haunt us.

Cheers,

Rob

[Rob Spanton]

On Mon, 2013-04-22 at 21:45 +0100, Peter Law wrote:
> I don't like the idea of removing a large portion of the spec, and
> replacing it with something that is clearly implementation-related,
> and could be misunderstood.

As I said earlier: whether something is "implementation related" or not
is a matter of perspective. Things can be juggled back and forth
between those two classifications with relative ease. Therefore, it's
not useful to consider whether something is an implementation detail or
not when writing it in a spec.

What is more useful, is considering what needs to go into the spec to
ensure that we end up with a functional end-product. A spec is not some
kind of mathematical statement. It is a thing we design to ensure that
the *people* (i.e. non-mathematical, real things) who work towards it
actually end up with something that works.

When we perform that consideration, we can look at our experience (the
previous two schedulers, as well as the various bits of optimisation
code that many of us have written in the past) and other work that is
out there (the MatchMaker scheduler). This is what I am doing.

I've been involved in a project that did not learn from its mistakes
before. It was called Glacsweb. That project repeats the same mistakes
year-on-year: carving up a glacier to inject broken electronics into it.
They've done that something like 10 times now. That was terrible. They
don't learn from their mistakes. Let's not be terrible!

Cheers,

Rob

[Peter Law]

I wrote:
> I've therefore started to create a broad user-spec [2] for what we

> want out of competition software as a whole. The intention of this
> page is to list the things that we want to achieve at the competition,
> which can probably be solved in the software domain. In this context,
> I'm deliberately not trying to pin down an implementation -- that can
> be done separately [3], instead I'd like to create a prioritised list
> of things we want to achieve.

Rob wrote:
> We're not looking to write a spec that is entirely devoid of details
> about a solution. I don't think any spec should ever be like that.

Rob also wrote:
> What is more useful, is considering what needs to go into the spec to
> ensure that we end up with a functional end-product.

Indeed, this is pretty close to what I set out to achieve -- a list of
functional bits that we want, *without* any consideration for how we
get there. I don't see this as being devoid of information about the
solution, but at the same time, it intentionally doesn't care about
what the solution is (under the covers). Apologies if I didn't
communicate this separation clearly enough.

In this case, users don't care about whether it uses rounds, only that
it generates a fair schedule (as defined by the various things listed
on the page [0]).

I'm sure you're wondering why I'd thought there should be this
separation. I was mainly aiming to get a quick list of things up, that
anyone could contribute to, without needing to examine the problem in
detail. Also something that could be done in the half-dazed-about-SR
state we all were in the week following the competition, and something
that would avoid a repeat of the high-speed hacking that now seems to
happen at the competitions.

Jeremy also suggested working out what the various roles needed, and
going from there. This seemed to fit well, and is something I still
think needs doing.

Hindsight suggests that another reason might have been to avoid this
long discussion about something that, frankly, requires a greater
understanding of the problem than I was hoping would be required; but
this wasn't something I'd considered.

> A spec is not some
> kind of mathematical statement. It is a thing we design to ensure that
> the *people* (i.e. non-mathematical, real things) who work towards it
> actually end up with something that works.

I wasn't aiming to build a guide for the implementers, I'd
characterise it more like a user-feature-request-list.

While this topic has rather veered from that intention, I do think
that the discussion that has occurred regarding scheduling specifics
are probably quite useful, and can be used to build the software-spec,
as well as hinting at the final implementation.

Rob wrote:
*Let's learn from our mistakes*

I agree entirely.
I'd hoped that's what I was doing, by a) generating a list of stuff we
want, and b) publicising it, while c) examining a variety of
alternatives while there's plenty of time to play with.

Peter

[0] https://www.studentrobotics.org/trac/wiki/Competition/Software#MatchScheduler

[Richard Barlow]

On Tue, 2013-04-23 at 21:39 +0100, Peter Law wrote:
> Rob also wrote:
> > What is more useful, is considering what needs to go into the spec to
> > ensure that we end up with a functional end-product.
>
> Indeed, this is pretty close to what I set out to achieve -- a list of
> functional bits that we want, *without* any consideration for how we
> get there. I don't see this as being devoid of information about the
> solution, but at the same time, it intentionally doesn't care about
> what the solution is (under the covers). Apologies if I didn't
> communicate this separation clearly enough.
>
> In this case, users don't care about whether it uses rounds, only that
> it generates a fair schedule (as defined by the various things listed
> on the page [0]).

I'm going to be a user of this schedule, as is every blueshirt actually
orchestrating the competition. I do care and would like it to have
rounds. The spec covers all aspects of the final product and in this
case there is more than just the mathematical correctness to consider
and specify. Another equally important point to consider is the human
aspect of actually applying this schedule. I would much prefer a
schedule where it is absolutely trivial to modify its overall run time
(by dropping a round) at the drop of a hat. This should not require any
software to be re-run as it WILL happen when everyone is ablaze, there
can be nothing simpler than just not running the final round.

Rich

[Alistair Lynn]

Hi Rich-

> I'm going to be a user of this schedule, as is every blueshirt actually
> orchestrating the competition. I do care and would like it to have
> rounds. The spec covers all aspects of the final product and in this
> case there is more than just the mathematical correctness to consider
> and specify. Another equally important point to consider is the human
> aspect of actually applying this schedule. I would much prefer a
> schedule where it is absolutely trivial to modify its overall run time
> (by dropping a round) at the drop of a hat. This should not require any
> software to be re-run as it WILL happen when everyone is ablaze, there
> can be nothing simpler than just not running the final round.

I also like the notion of rounds (although I'm certain it doesn't belong in the spec...) but the FIRST Robotics schedule spec from which this stuff is derived has a slightly different understanding of rounds.

In particular, rounds can begin half way through a match - that is, the supply of teams left in a round being exhausted, the next round will begin after selecting, say, two of the four teams in a match.

This means that we do (potentially, if the number of present teams is not a multiple of 4) bias the match counts if we drop rounds in their entirety.

Alistair

[Rob Spanton]

Rich wrote:
> I would much prefer a schedule where it is absolutely trivial to modify
> its overall run time (by dropping a round) at the drop of a hat. This
> should not require any software to be re-run as it WILL happen when
> everyone is ablaze, there can be nothing simpler than just not running
> the final round.

Alistair wrote:
> In particular, rounds can begin half way through a match - that is,
> the supply of teams left in a round being exhausted, the next round
> will begin after selecting, say, two of the four teams in a match.
>
> This means that we do (potentially, if the number of present teams is
> not a multiple of 4) bias the match counts if we drop rounds in their
> entirety.

This is trivial to deal with! We've discussed surrogate appearances
many times (definition can be found again at bottom of this email). In
the event of dropping a round, we'd make sure we played the last match
of the round in its entirety. Any robot appearances in this match that
belonged to the dropped round would be treated as surrogate appearances.
This is a reasonable thing to do in an aflame situation.

Cheers,

Rob

The following surrogate appearances description is copied from
http://www.idleloop.com/matchmaker/

"Surrogate Appearances
In order to give all teams the required number of appearances on
the field (the number of rounds) it is sometimes necessary to
have some teams make an extra appearance. These are called
surrogate appearances. The outcome of these matches count for
the non-surrogate teams, but do not affect the standings of the
surrogates."