Fire Fighting: How the SR2013 Competition really worked

[Rob Spanton]

Hey Guys,

So this year's competition was good in that the user-facing side of
things worked out. Teams and their leaders have said how they were
happy with how the event went. However, this was all because of the 10
fire fighting volunteers [1] who worked relentlessly to make things work
out. This is up considerably from the ~3-5 fire fighters that we had at
SR2012.

What is "fire fighting"? When I say "fire fighting" here, I mean
sorting out of a thing that has been poorly managed and/or executed --
generally just minutes before or after it was supposed to be shipped.

From the "post competition discussion thread" and hearing things in
various pubs, it seems that most of our volunteers don't realise that
the SR2013 comp involved any fire fighting. When these fire fighters
hear these things said it is a face-palm moment. Wake up and see the
real picture! The competition was teetering on a knife-edge. The
network went up pretty much at the point competitors arrived. The match
schedule had to be manually tweaked by sleep deprived individuals. The
software for the displays had to be entirely rewritten in a few seconds,
as did the scoring code. The shepherding situation had to be engineered
to actually work over the first few hours. Both the refreshments and
music had to be scraped together at the last minute. A solution for
weighing down the pedestals had to be engineered on Thursday. I'm
certain there are lots of other things that I haven't mentioned above.

The reason that the competition went well was the fire fighters
perseverance, common sense, and understanding of what the competition
should be like.

"But the competition went well" I hear you say. "It all worked out, so
it doesn't matter." Sure, it worked, but at the expense of stressing
out those volunteers who became fire fighters. This is not a trade-off
worth making. All it does is piss off these people, and they are
significantly less likely to come back and help in the future. Saying
"it all worked out, so it doesn't matter" is pretty much synonymous with
saying "I don't care about those people's well-being." If you find
yourself saying those things, then it's quite likely the people who were
fire fighting will stop caring about the competition and SR.

I have also heard things from four of the fire-fighters that even when
they were working on sorting a lot of stuff out they received an awful
lot of obstruction and in one case insulting behaviour from other
volunteers. That type of behaviour is just not on.

It is clear that the organisation of the competition has a long way to
go before it no longer trades in people's well-being for success.

Cheers,

Rob

[1] Jeremy, Rich, Jeff, Howard, Elisabeth, Khilan, Chris K, Chris O,
Peter, Myself -- potentially some more people too.

[Jeremy Morse]

Hi,

I'm on board with all these things; additionally not one iota of thought
was given to how the pack-up/tear-down would be occurring after the
competition, leaving (as I understand it) a lot of equipment hanging
around, which could have seriously inconvenienced SUSU if they'd had
anything booked in on the Monday. Plus the building plans turned out to
be inadequate due to their rushed materialization (leading to several
bays being dissolved by the safety people -- not something that's
guaranteed to be raised in the future, leaving us liable).

On 17/04/13 14:08, Rob Spanton wrote:
> "But the competition went well" I hear you say. "It all worked out, so
> it doesn't matter." Sure, it worked, but at the expense of stressing
> out those volunteers who became fire fighters. This is not a trade-off
> worth making. All it does is piss off these people, and they are
> significantly less likely to come back and help in the future. Saying
> "it all worked out, so it doesn't matter" is pretty much synonymous with
> saying "I don't care about those people's well-being." If you find
> yourself saying those things, then it's quite likely the people who were
> fire fighting will stop caring about the competition and SR.

I wouldn't say I had a heart attack at the end, but some nausea and
anxiety occurred that wasn't particularly wanted (and kept me off work
for a day). Not an experience I wish to repeat.

--
Thanks,
Jeremy

[Chris Malton]

On 17/04/13 14:08, Rob Spanton wrote:

> The
> network went up pretty much at the point competitors arrived.

This was entirely my fault - caused by multiple people pissing me off
concurrently on the Friday, with me having had very little sleep for the
previous 6 days - it wasn't going to end well. The lack of sleep is
most definately my fault, the people complaining about my network loudly
behind my back (among other things), isn't entirely my problem, although
I'll acknowledge that parts of it not working are most definately my
problem!

While the network had been tested by me prior to its deployment, it is
quite evident that the tests I did, didn't find the issues we
encountered on the Friday.

I have since identified most of the issues we encountered on the Friday,
one (possibly more) of which was/were down to a switch used to extend up
to Level 4 which failed to do as expected, leaving me headscratching -
Orchard found this after running into similar issues to me on Friday
night/Saturday morning and dealt with it.

The second set of issues I was dealing with was down to an issue that
Orchard then discovered the root cause of: The Beagleboard XMs change
MAC address each boot. This caused havoc on the displays subnet and
resulted in displays not working as they'd run out of IP addresses.
This was an unexpected situation which I couldn't have tested, not
having a Beagleboard XM (or several) myself. I mistakenly assumed that
they always had the same MAC address.

I was also trying to fix Alistair's Macbook, which actually turned out
to be something he'd done earlier to it, which was leaving rogue DHCP
servers on VLANs.... and who knows what that was doing!

I'm perfectly prepared to stick my hands up and go "It broke, I know,
and it's my fault.". I'm already working to try and find a better
solution, and a simpler solution than this year, to which I think I
probably have an answer. I would probably have chosen to use Smallpiece
to prove that, but since I'm not on the Smallpiece team any more, that's
not going to happen.

I would like to once again, apologise for this complete debacle, which
is quite clearly my fault. Having had plenty of sleep Friday night, and
also Saturday night, I then turned up to help out where needed on Sat
PM/Sunday, and then stayed for the teardown, and did my bit to help out
where I could.

I hope you'll accept this as an apology.

Chris

[Tim Jones]

Wow - I'll admit that I was completely unaware of this. I'm inclined to say that this is testament to some frankly superb firefighting skills on the parts of some very valuable blueshirts, but this misses the point somewhat; we don't deliberately stab people because we know doctors exist. As for fixing this problem, I really can't see any reasonable solution other than extensive testing of everything - but this may well expose a great deal of my ignorance. I don't know what proportion of the competition, if any, can be rehearsed beforehand, but I'm fairly sure that, for example, we can set up a mock shepherding situation beforehand. We distribute ourselves around the competition venue with some designated shepherds, then send them to find people who will behave with varying levels of awkwardness when summoned, and see how long it takes for them to get down to the proposed arena.

I ask this with no sarcasm or rhetoric whatsoever, I am genuinely curious; how many of the competition-critical tasks were rehearsed prior to the competition, and how hard/expensive is it to set up these situations prior to the competition and see what happens?

>I have also heard things from four of the fire-fighters that even when 
>they were working on sorting a lot of stuff out they received an awful 
>lot of obstruction and in one case insulting behaviour from other 
>volunteers.  That type of behaviour is just not on. 

I appreciate that you're not naming names here, though I'm also aware that I can frequently act in a very insensitive way towards people, without realising that I'm doing it, or knowing that I've done it until it's pointed out. I am deeply sorry if this refers to me, and wish to apologise profusely if I found myself in the way of anything competition-critical, or insulted anyone who was trying to make the weekend work.

[Andy Busse]

Hi,

On 17/04/2013 22:06, Tim Jones wrote:
> and how hard/expensive is it to set up these situations prior to the
> competition and see what happens?

On a related note, in 3 and a bit months, we're running a summer school
- I have no problem if people want to use this as a testing ground for
new things. As there's fewer people, it's much more manageable to fall
back to whiteboards, guest eduroam logins, etc., plus all the teams will
be in one area.

I know it's not the perfect time of year to test things (obviously, a
rehearsal a month or two before the main event would be better), but
it's there.

Might it be possible to list the things which were forgotten as
individual {threads, trac tickets, whatever works best} so that we can
think about how and can engineer a situation whereby we can avoid such
stress in the future? (e.g. the shepherding situation - what was it like
to start with? what was engineered to fire-fight the situation? could
the job be shared with wardens, do we post shepherds to certain areas,
etc.). I know you've started that with this thread, but more information
on what went wrong would be good - learning from mistakes, etc.

> I appreciate that you're not naming names here, though I'm also aware
> that I can frequently act in a very insensitive way towards people,
> without realising that I'm doing it, or knowing that I've done it
> until it's pointed out. I am deeply sorry if this refers to me, and
> wish to apologise profusely if I found myself in the way of anything
> competition-critical, or insulted anyone who was trying to make the
> weekend work.

Same here. Apologies if I've been tactless in any way. Just trying to
help, here.

Thanks,
Andy