Re: Questions regarding final code/prediction submission re-opening for a week

[Nick Landia]

Hi,

Good question. To make it as fair as possible and also not overly complicated we are opening resubmission for all teams, so any team can submit fixes, as well as improvements if they wish.

I would ask everyone to spend some time double-checking the test set issue first though, and consider improvements after if you wish.

Best,

Nick

On Thu, Jun 30, 2022, 17:42 andrew y <andr...@kakaocorp.com> wrote:

Hello Nick,

I have a couple of questions regarding final code/prediction submission re-opening for a week.

I see that this could be interpreted in two ways:
1. Is the purpose of these resubmission only to correct the test set data being used? If so, will there be any verification process that the logic has not changed other than removing test set data?

2. If there are no verification process, then I assume it just means that everyone can modify their logic to assure the fairness of the competition. Is it possible for the teams that have not used test set data to resubmit the final code/prediction as well?

Sincerely,
Andrew

--
You received this message because you are subscribed to the Google Groups "RecSys-Challenge-2022" group.
To unsubscribe from this group and stop receiving emails from it, send an email to recsys-challenge...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/recsys-challenge-2022/614b13fe-70bc-4d4f-8f8f-7be9aae0f404n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ARS]

1. Based on an objective fact, it is a very arduous and time-consuming task to review whether the code of the competitor is illegal. The code and ideas of the toper are very complex, so depending reviewing the code will bring great uncertainty and  fairness to the result.
2. As far as I know, the final rules are modified 2-3 weeks before the end of the challenge. I accidently found the modifications of latest version rules on the official website at the end of May, and there was no email to notify me that the rules have changed. Considering that I have already invested  more than two months, and it is very unfair for me to change the rules before the end of the competition.
3. If the organizer really wants to ensure the fairness of the competition and does not introduce additional uncertainty caused by manual review, it is recommended to remove the rule number three  and reopen the competition for 1-2 weeks.

[iggy ll]

For me, the policy of the competition seems fine.

As far as I know, this rule is made before the competition begins.

• Do not use data from the test sessions in model training, use test session data only in the prediction step

The rule about using validation/final dataset is raised in this group quite early in the competition, 4 April https://groups.google.com/g/recsys-challenge-2022/c/rm4icXdhZVM 

and this rule is just clarified in detail in the rule on 20 May.

2022년 7월 1일 금요일 오후 6시 59분 16초 UTC+9에 ARS님이 작성:

[Benedikt S]

I participated in the competition, as well. I agree with kakaocorp.com. There was a version of the rules since the beginning. However, based on that version, another participants asked for more clarification in the groups and it was possible to know it since the beginning (April): https://groups.google.com/g/recsys-challenge-2022/c/rm4icXdhZVM

There were many discussions about the rules during May, as well.

We (and many participants) worked according to the original rules for 2 months without using the test data. I think it is not fair to change the rules and allow it now.

[Xingxiang Zhuo]

Couldn't agree more with ARS

[Nick Landia]

Hi,

as others have pointed out the rules were the same from the beginning. Unfortunately they were written in a way that allowed some room for interpretation, they could have been written clearer and I apologise for that. However, this question was raised on google groups very early on and clarified. The update of the wording on the website did not change the rules, it was to make it less prone to misunderstanding.

Best,

Nick