New test split and codalab link

[Debayan Banerjee]

Dear participants,

We have a couple of big changes to announce:

1. We detected that the current test set has some wrong answers, because the dataset is LLM generated. We have curated a new test set, which is manually verified by 2 humans to be correct. We request all participants to abandon the existing competition link (https://codalab.lisn.upsaclay.fr/19140), and instead participate on the new link https://codalab.lisn.upsaclay.fr/competitions/19747. We had to create a new competition link because codalab does not allow adding new phases to an existing competition.

2.   In the new competition, we have Exact Match and F-Score as measures. We abandon the METEOR score as a measure.

3. To see both the metrics for your submitted solutions, you must make the submission public. This is a codalab limitation which we could not find a workaround for.

4. On the request of participants, we have extended the deadlines by a further 2 weeks. 

You will find the new dates, and all updated information on the challenge webpage https://kgqa.github.io/scholarly-QALD-challenge/2024/.  Please let us know if you have any questions. Apologies for the disruptions.

--

Regards,

Debayan Banerjee

[Şefika EFEOĞLU]

Hello, 

I received the following error below and tried to submit the results again after this error, but I could not upload my file.

Thanks.

Best regards.

Traceback (most recent call last):
  File "/tmp/codalab/tmpz6DnBT/run/program/qa_eval_em_f1.py", line 72, in 
    main()
  File "/tmp/codalab/tmpz6DnBT/run/program/qa_eval_em_f1.py", line 61, in main
    exact_match_accuracy, precision, recall, f1 = evaluate(gt_path, system_path)
  File "/tmp/codalab/tmpz6DnBT/run/program/qa_eval_em_f1.py", line 47, in evaluate
    gold_answers, predictions = load_data(gold_answers_path, system_answers_path)
  File "/tmp/codalab/tmpz6DnBT/run/program/qa_eval_em_f1.py", line 6, in load_data
    pred_data = json.loads(open(system_answers_path).read())
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/codalab/tmpz6DnBT/run/input/res/answers2.txt'

--
You received this message because you are subscribed to the Google Groups "scholarly-qald-2024" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scholarly-qald-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scholarly-qald-2024/CAMPgBOfkDw%2BoQeJLC-yd18xbYHN1PBHOVss_8yUi7Ts%3DXP8nJw%40mail.gmail.com.

[Debayan Banerjee]

Hi Sefika,

Apologies. Please write the answer now to a file named answers2.txt instead of answer.txt. We will update the instructions on the website shortly.

To view this discussion on the web visit https://groups.google.com/d/msgid/scholarly-qald-2024/CAL1r7Wn%3DP5jkgo%3DTGPqqs_dE63KWhdguJNc48-Ha%3D2bdEOJG8w%40mail.gmail.com.