Question about contest

[marios mintzis]

Hello,

For many years the constest was about correct implementation and mostly performance. Even though tasks where focused sometimes on graph databases, query matching, disk optimization i/o, fast transaction/evaluation they always had something in common. The solutions where uploaded to the isolated testing inviroment, they run a "hidden" dataset(s) and if succeded shown on leaderboard. This made it more competitive, fun but also was almost showing the correctness of our solution since we were already running it on the testing machine. This year we will just submit the csv output and then only the top 5 or 10 (After 1 month) will be selected to run their solution on the machine ? Only then the hidden datasets will be run on a participants solution. Takes out the fun, the competitiveness, issues that could rise cannot be earlier found and it could introduce issues since maybe the solution of the teams needs a few tweaks to run correctly on the hidden datasets which could be prevented if at least they run it on the hidden small medium large before the final one ? I just want to know your thoughts.

Kind regards,
Marios

[Jakub Beránek]

I have to say that I was also a bit disappointed with this year's task and mainly the evaluation process, which is why I'm gonna skip this year.

But the contest hasn't started yet, so let's wait and see how it turns out. It's basically a Kaggle problem and that can also be pretty fun to solve :-) And it's too late to change the task anyway.

Jakub Beránek

[marios mintzis]

Hello Jakub,

Good job last year. It reminds me of kaggle yes and i am wondering since each year the organisers are supposed to be the previous year's winners how do they change the concept sometimes? Maybe its not up to them. If there is no evaluation machine running our submisions and its just a csv upload i might skip it this year as well even though it could still be interesting. Lets see what the organisers are thinking.

Marios 

[Xingyu Xie]

I'm confused about the change of the theme of contest. If SIGMOD contest focuses on deep learning rather than performance optimization, why people participate it? There're too many better deep learning competitions to attend, Kaggle, for example, with more money and higher quality.

Also decided to skip this year because of the task.

BTW, is there any other contest just like SIGMOD Programming Contest of past years, where participants try to optimize the performance of their solution in months?

Xingyu Xie

[marios mintzis]

I am glad we are all on the same page. I am not sure Xingyu Xie if there are other similar competitions to what SIGMOD Programming Contest used to be but if you find one please do let me know. Its so fascinating using the knowledge acquired over the years competing live with professionals all over the world with a live Leaderboard on complex solutions to achieve correctness, resource utilization with performance optimizations. Just the feeling of finding a new performance optimization uploading and waiting for the machine to run it and get the Green color and then immediately check leaderboard to see at what position you are now. 

Marios Mintzis 

[marios mintzis]

I am waiting for a response from the organisers. However ,Jakub and all from last year, remember how delayed last year's contest was and also the task changed a lot a few weeks after the page was up and the task was given. So maybe they have time to change the task or at least the evaluation process since the task seems ok even though the datasets seem very small. Last year we started middle of March i think so there is time but it all depends on when the conference is happening because top 5 will attend it as all previous years. Its not up to us but i am waiting for a response.

Marios

[alaska benchmark]

Hi all,

the submitted solutions will be evaluated with a hidden evaluation dataset. The results (in terms of Precision, Recall and F-measure) will be published on the leaderboard, which is updated every working day. Please find here the updated leaderboard!

After the final challenge deadline, we will reproduce the results of teams in the leaderboard in rank order and select as finalists the first 5 teams whose submission is reproduced. For more information please refer to the "Evaluation Process" in the task page.

Hope you will participate and have fun!

Andrea, Programming Contest Co-Chair

[marios mintzis]

So, Is there a Dashboard at least to show the results of the submissions ? I mean even though u decided to update the leaderboard every 24 hours (takes the fun out, and competitiveness) dont you think that there should be at least a dashboard so contestants can keep track of their submissions ? I mean why have the machine idle in the first place all this time ? In addition there are many points noted in this thread that are still not answered to make it completely clear. 

[marios mintzis]

Everything mentioned in the response from the organiser can be found on the page and believe me we read it 1000 times by now. This is how SIGMOD is. We read every single detail. Its a 2 months competition. We are very serious about it when we participate. We are waiting for a serious comprehensive answer to our questions/concerns.
1)Dashboard
2)Why the concept has changed so much ? for the first time ever (upload csv instead of running code on the machine, and machine learning)
3)Why leaderboard updates every 24 hours instead of updating with each submition? (SIGMOD was like this since 2013 that i started participating)
4)Can things change? At least run the codebase on the machine and then if better accuracy show in leaderboard?  or is it just another kaggle competition (sort of)
5)Are there many people on this ? It took 2 days to get an answer in the google group?
6)Arent the datasets a bit small ? Maybe on this context they are ok. Why are they ok ? 

Thanks