Evaluation of multi-solution results

65 views

Skip to first unread message

irina.maslowski

unread,

Mar 7, 2019, 11:38:53 AM3/7/19

to BEA 2019 Shared Task: Grammatical Error Correction

Hi,

the page of the shared task (https://www.cl.cam.ac.uk/research/nl/bea2019st/#data) describing evaluation gives an example of three hypotheses for the error correction. It is not clear to me if it is only an example of ERRANT behaviour or if the files with results for evaluation can contain three hypotheses. If it is possible to submit multiple hypotheses for error correction, will they all be taken into account for system evaluation and how?

Thank you in advance!

BEA 2019 Shared Task Organisers

unread,

Mar 7, 2019, 11:56:13 AM3/7/19

to BEA 2019 Shared Task: Grammatical Error Correction

Hi Irina,

The table on the website is mainly an example of ERRANT behaviour. Since we only ask participants to produce plain text files as output, we do not take alternative hypotheses into account.

ERRANT does actually support multiple hypotheses in an M2 file, but that's not something we're using in the shared task.