Alternative correct versions of an incorrect sentence

[writeto...@gmail.com]

There could be multiple alternative correct versions of an incorrect sentence. How will such cases be resolved?

For CoNLL 2014 dataset, it was later proposed to handle such cases by having 8 additional annotators. How many annotators will be used for evaluation in this task?

 

[BEA 2019 Shared Task Organisers]

Yes, good question.

We had not announced it yet because annotating is still underway, but we expect to have 5 sets of annotations on the W&I+LOCNESS test set by the time we begin official evaluation.