Missing references in DART Test set

[Nivranshu Pasricha]

Hi,

It looks like some targets/references are missing from the DART test set. Examples which have an empty list as references have the 'dart_id' in the range 1391-3252 (inclusive). 

Regards,

Nivranshu Pasricha

[Nivranshu Pasricha]

There are missing references in the CommonGen test set and challenge set too.

I am using the gem_metrics evaluation script to get the scores for writing in the system description but cannot get the correct scores as references are missing. Would it be possible for the GEM benchmark organisers to provide results on our submissions before system description deadline? We submitted outputs on the test set and the challenge sets a few days ago on May 31st. 

Thanks,

Nivranshu

[Sebastian Gehrmann]

Hi Nivranshu, 

Sorry about the issues with DART - the file in the original repository was changed which means that the examples don't match up anymore. This is something we cannot easily fix, but I am currently looking into the issue. CommonGen and ToTTo have a private test set, so we unfortunately cannot distribute any infos here. 

For your system description, are you able to get scores on the validation set? If not, please open an issue on the metrics repository and we will try to help / run the evaluation for you. 

Best

Sebastian

--
You received this message because you are subscribed to the Google Groups "gem-benchmark" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gem-benchmar...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gem-benchmark/05a38e49-2339-4024-9683-65d37d55a7b1n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nivranshu Pasricha]

Thanks for your reply Sebastian. Yes, I can get the scores on the validation set and I will add these to our system description. I had posted an issue with the evaluation script a few weeks ago related to BERTScore but I have it working fine now. Thanks!