RuCTF 2014 Quals follow-up letter

[do...@hackerdom.ru]

Hi!

RuCTF 2014 Qualification event ended 10 days ago. Six days ago we published to this group our follow-up letter in Russian, and now we decided to translate it to English. This is the sixth year we hold a qualification event for RuCTF and it seems that each year we grow considerably. First of all, we would like to thank those who made this growth possible: everyone who made RuCTF in the past, the demiurges of RuCTF — Ilya Zelenchuk and Nikolay Zhuravlev, but most of all — those, who made this year qualification event.

This year developers were keen to make quality CTF, come up with interesting tasks, deliver as much joy as possible to our participants, and perform justified selection of teams to invite to the Final event. Every single person listed on http://ructf.org/2014/en/devteam committed a valuable input to the success of our event.

We are glad to have participants that come back to us every year. We also greet all the teams for whom this event was the first RuCTF (or even the first CTF). We sincerely hope that you enjoyed this event and that we will see you next year!

But the number of participants is not the only point of growth. We would like to note several directions we were working on this year.

First, we made the qualification event completely international. Last year, as an experiment, we allowed foreign teams to participate, and there were 218 teams registered (83 of those were Russian). This year we invited foreign teams and there were almost 400 registrations (110 of those were Russian). The dynamics of number of Russian teams (almost 30 new teams this year) makes us happy, as does almost doubled figure of all participants. Of course, not every registered team decided to solve tasks, but this figure is positive as well — this year there were 249 teams that solved at least one task, and there were only 96 such teams last year.

Second, we rewrote our checking system (thanks to Sergey Azovskov and Andrew Gein) to make it more convenient for both you and us. People that play RuCTF for a long time, might remember there were several issues with the previous checking system. The most severe one was the probability of loss of recently gained points. By the way, the new checking system is available at the GitHub: https://github.com/Hackerdom/qoala.

Third, our administrator Andrey Malets worked hard to improve RuCTF infrastructure and monitoring systems. It was not the easiest task to make the park of 20 virtual machines up and running on such a short notice.

Fourth, we put much effort on the quality and variety of tasks. This year we have two new categories, Hardware and Recon. We hope that everyone who opened tasks page found something interesting for himself. Having previous year feedback analysed, we tried to drastically reduce the number of tasks where you had to guess the answer. It seems, we succeeded to do so in most of tasks. When possible, we made flags start with 'RUCTF_' substring, and mentioned it in the task description. That way, when you found something that looked not like that string, it was clearly not the answer. Obviously, we have more things to improve, and we will make our tasks and descriptions better and better.

Just as in the last year, we hold RuCTF Quals Afterparty, that is we keep all the game infrastructure up for two more weeks. We develop all our projects on the GitHub (https://github.com/Hackerdom), and ructf2014-quals repository will be opened once Afterparty is over.

Sadly, we could not forecast and avoid all problems. We made several mistakes that brought inconveniences to teams. We apologise to anyone who suffered because of those.

First, we failed to start on time. Despite the fact that we started all our latter CTFs on time, we could not make so in this year because of various technical difficulties. Combined, they lead to whopping half an hour delay.

Second, we made mistakes in several tasks. We don't deny that. You all could see we did our best to fix these mistakes as soon as it was possible and to post the updates in the news section. As you can see at http://quals.ructf.org/board/, all the mistakes except for one were fixed before 9 AM MSK on March, 9th, that is about a quarter in the game. Moreover, most of mistakes were fixed in the first 3 hours after task was open, often even faster.

Third (this is one of the cases of the previous paragraph, but we want to emphasise it), we overlooked several easy (sometimes too easy) solutions. We spent considerable amount of time discussing what is the right thing to do in such a case, considered several options, and went with those, that, in our opinion, were right. What options did we have?

1. Let it be. Teams that figured the easy solution (not always obvious, to be honest), get their points with no effort. Reverse:200 and Web:200 were the worst in this terms, because the solution was obvious.

2. Decrease the score for erroneous task, and make another, fixed version. In this case we give you the opportunity to solve and enjoy the initial idea.

3. Fix the task and null the score for this task for any teams that were quick enough to post the flag (in case of Crypto:400 that was one team).

One of the main goals of our event is to bring fun and joy to our participants, that is why we decided to fix erroneous tasks so they would reflect our original ideas. In one case we left both simple and original version, in two other cases we just replaced simple versions.

In the feedback form we asked your advice on what we should have done in these three cases. It is important for us to know your opinion, so if you didn't fill the feedback form — please, do it.

Fourth (at last!), we assigned wrong scores for several tasks. Because of that, Reverse category was almost in the reverse order (could have been a feature :)). Same with few Admin tasks and Crypto:500 task. We have to mention that each year we spend quite a lot of time in discussing scoring rules. We can point out weak and strong sides of every scoring system, and of course 100-200-300-400-500 system is not ideal. It is possible that we will replace it with another one, more suitable. That could even happen in the next year.

Now, when the RuCTF 2014 Qualification event is over, it is time to ask ourselves if we could avoid those mistakes. We think, yes, we could avoid most of them, if we were even more accurate and thorough. We are sure that we can overcome these difficulties in the future.

One of minor issues was our English skills, that are far from perfect, to put it mildly. Despite the fact that all task descriptions were bilingual, most announcements were English-only, and that's where we all were lost in translation.

Once again, we want to heartily thank all of those who participated. We worked hard; we invented, programmed, tested, committed, pushed, deployed and automated all of those things for you. We hope that it was the pleasure for you to participate as it was for us to develop. Thank you and see you again!

--

Respectful and thankful for the great game,

RuCTF team