Question related the submissions for LM-KBC

[Debanjali Biswas]

Hi, 

We also have some questions related to the submissions:

1. How can we submit our systems/models? Will the submission process take place through Kaggle or CodaLab? Can you provide us with a step-by-step guide on submission process?
   
2. During the testing phase, will we have access to a leaderboard to track the performance of our model?
3. What does "System description" refer to? Is it equivalent to a research paper? If so, what is the expected length of the paper? Are all participants' system descriptions published, or only a selected few?

Looking forward to your response. Thank you!

Regards,

Debanjali Biswas

(GESIS)

[LM-KBC Challenge]

Hi Debanjali,

Thanks for your questions!

1. We will release a Codalab link during the test dataset release (on 26th July). The submission process will be simple; you will upload your predictions.jsonl file, compressed in a .zip file, on Codalab. Then the system will evaluate your predictions and show your ranking on the leaderboard. During the test dataset release, we will also provide a sample prediction file, which can be used for formatting checks.

2. Yes, you can also make multiple submissions to Codalab, and the leaderboard will show your ranking based on your best f1-score. 

3. Please submit a GitHub repo link containing your entire codebase (models, metadata, etc.) and predictions.jsonl file. Also, a research paper style write-up (rough structure involving Abstract, Introduction, Related Work, Methodology, Experiments, Results and Conclusion) is required in LNCS format. There isn't a page length limit; however, mentioning your methodology in detail for reproducibility would be good. For reference, you can check last year's submissions on our website (https://lm-kbc.github.io/challenge2022/). All the participating team submissions will be reviewed and published.

Thanks,

The LM-KBC Team