Guidelines for Subtask B

[Sapna Negi]

Dear participants

Here are the guidelines regarding using the trial test set for Subtask B.

- You cannot use SubtaskB_Trial_Test_Labeled.csv as a training set.

- You cannot use any other hand labeled dataset tagged with suggestions, from the hotel reviews domain, as a training set.

- You can look at the trial test dataset labels for validating your system, error analysis etc.

- In the case of systems which typically use a validation dataset for automatic hyperparameter tuning, you can use the trial test set as a validation set.  Please mention this in the description section and submit your source code (commenting where exactly is the trial test set used) in this case, using the Project URL section on CodaLab. See the screenshot attached.

However, if you would want to evaluate an approach using the trial set for training, please get in touch with us after the evaluation phase. 

We will not include it in the ranking, but we can consider preparing a separate leaderboard and mentioning it in the task description paper. You can also include the results from this approach in your system description papers.

Hope this helps with the clarification. Please reply to me on this email (not 'reply all') in case you want to confirm if your system adheres to these guidelines.

 

Regards

Organisers