gem-benchmark

1–23 of 23

0 selected

Sebastian Gehrmann

Oct 7

Suggest Tasks for GEMv2

Dear all, As I mentioned in a previous announcement, we are now in the task collection phase for

unread,

Suggest Tasks for GEMv2

Dear all, As I mentioned in a previous announcement, we are now in the task collection phase for

Oct 7

Sebastian Gehrmann2

Oct 1

Re: The work on GEMv2 is starting!

Dear all, I just sent invitations to everyone who signed up for the respective groups. If you did not

unread,

Re: The work on GEMv2 is starting!

Dear all, I just sent invitations to everyone who signed up for the respective groups. If you did not

Oct 1

Sebastian Gehrmann2

Sep 2

Future planning for the GEM benchmark

Hi all, Over 50 of you already signed up for the next round of GEM and I wanted to kick off the next

unread,

Future planning for the GEM benchmark

Hi all, Over 50 of you already signed up for the next round of GEM and I wanted to kick off the next

Sep 2

Sebastian Gehrmann

Aug 6

GEM Workshop today

Dear presenters and co-organizers, We are very excited to start the Generation, Evaluation, and

unread,

GEM Workshop today

Dear presenters and co-organizers, We are very excited to start the Generation, Evaluation, and

Aug 6

Nivranshu Pasricha, Sebastian Gehrmann4

Jun 10

Missing references in DART Test set

Thanks for your reply Sebastian. Yes, I can get the scores on the validation set and I will add these

unread,

Missing references in DART Test set

Thanks for your reply Sebastian. Yes, I can get the scores on the validation set and I will add these

Jun 10

KV Aditya Srivatsa

Jun 9

Query regardind the system description format

Hi, would you be able to provide a rough format of what is expected in a system description? I was

unread,

Query regardind the system description format

Hi, would you be able to provide a rough format of what is expected in a system description? I was

Jun 9

A Shvets, Sebastian Gehrmann2

Jun 7

Dataset creation for GEM in local language

Hi Anna, Our plan is to have a yearly selection process for GEM (similar to our initial one) that

unread,

Dataset creation for GEM in local language

Hi Anna, Our plan is to have a yearly selection process for GEM (similar to our initial one) that

Jun 7

Dan Deutsch, … A Shvets3

Jun 4

Accessing baseline and system outputs

Hello Sebastian, Would you please send me those outputs as well? Thank you in advance! Anna On

unread,

Accessing baseline and system outputs

Hello Sebastian, Would you please send me those outputs as well? Thank you in advance! Anna On

Jun 4

Nick Doiron, Sebastian Gehrmann2

May 20

Including new datasets in the future?

Hi Nick, Thank you for reaching out about this. We will start the selection process for the next

unread,

Including new datasets in the future?

Hi Nick, Thank you for reaching out about this. We will start the selection process for the next

May 20

Monil Gokani, Sebastian Gehrmann3

May 16

Team submission

Hi, Thank you for the clarification!! On Mon, May 17, 2021, 12:38 AM Sebastian Gehrmann <gehrmann@

unread,

Team submission

Hi, Thank you for the clarification!! On Mon, May 17, 2021, 12:38 AM Sebastian Gehrmann <gehrmann@

May 16

KV Aditya Srivatsa, Sebastian Gehrmann3

May 13

Deadline confirmation for Shared Task Submission

Hi Sebastian, Thanks for the clarification. On Friday, 14 May 2021 at 01:17:48 UTC+5:30 gehrmann

unread,

Deadline confirmation for Shared Task Submission

Hi Sebastian, Thanks for the clarification. On Friday, 14 May 2021 at 01:17:48 UTC+5:30 gehrmann

May 13

Yimai Fang, Sebastian Gehrmann2

May 12

Checksum error in Huggingface datasets

Hi Yimai, Sorry for the delayed response. We updated the loader to match the new checksum. Everything

unread,

Checksum error in Huggingface datasets

Hi Yimai, Sorry for the delayed response. We updated the loader to match the new checksum. Everything

May 12

Michael White, Sebastian Gehrmann3

May 3

use of templates from Google paper and original SGD dataset?

Hi Sebastian Thanks for the clarification and updates! Attached is the script. It takes a path to the

unread,

use of templates from Google paper and original SGD dataset?

Hi Sebastian Thanks for the clarification and updates! Attached is the script. It takes a path to the

May 3

Jordi Clive, Sebastian Gehrmann3

May 3

Hi there, Regarding our baselines, most published ones use the hyperparameters (and code) posted in

unread,

Hi there, Regarding our baselines, most published ones use the hyperparameters (and code) posted in

May 3

Michael White, M, Simon5

Apr 29

intended use of challenge sets?

Hi Simon Thanks for confirming! I suggest adding a clarification to the Submitting Outputs section of

unread,

intended use of challenge sets?

Hi Simon Thanks for confirming! I suggest adding a clarification to the Submitting Outputs section of

Apr 29

bin, Simon Mille2

Apr 28

Non-English summarization

Hi Bin, as far as I know you can submit for whichever dataset you want; note however that, in your

unread,

Non-English summarization

Hi Bin, as far as I know you can submit for whichever dataset you want; note however that, in your

Apr 28

Sebastian Gehrmann

Apr 22

Final CfP for the Workshop on Generation, Evaluation, and Metrics (GEM) at ACL ’21

Final call for papers and shared task submissions for the Workshop on Generation, Evaluation, and

unread,

Final CfP for the Workshop on Generation, Evaluation, and Metrics (GEM) at ACL ’21

Final call for papers and shared task submissions for the Workshop on Generation, Evaluation, and

Apr 22

Michael White, gehrmann6

Apr 8

service and dialog order in SGD dataset?

Awesome 😎 will check it out! Many thanks Mike On Thu, Apr 8, 2021 at 6:19 AM 'gehrmann' via

unread,

service and dialog order in SGD dataset?

Awesome 😎 will check it out! Many thanks Mike On Thu, Apr 8, 2021 at 6:19 AM 'gehrmann' via

Apr 8

Sebastian Gehrmann

Mar 29

Second call for papers and shared task submissions for Workshop on Generation, Evaluation, and Metrics (GEM) at ACL ’21

Second call for papers and shared task submissions for Workshop on Generation, Evaluation, and

unread,

Second call for papers and shared task submissions for Workshop on Generation, Evaluation, and Metrics (GEM) at ACL ’21

Second call for papers and shared task submissions for Workshop on Generation, Evaluation, and

Mar 29

Nivranshu Pasricha

Mar 29

BLEU and ROUGE scores for DART

Hi, Can you please confirm if the results for DART in Table 2 in the GEM paper are correctly reported

unread,

BLEU and ROUGE scores for DART

Hi, Can you please confirm if the results for DART in Table 2 in the GEM paper are correctly reported

Mar 29

Michael White, odusek4

Mar 28

Dear Michael, I'm sorry for the delayed answer. I've looked more closely at the E2E scores

unread,

Dear Michael, I'm sorry for the delayed answer. I've looked more closely at the E2E scores

Mar 28

Leshem Choshen, … gehrmann3

Mar 23

Hi Leshem, As Emiel mentioned, all system outputs will be made available. We even have a specific

unread,

Hi Leshem, As Emiel mentioned, all system outputs will be made available. We even have a specific

Mar 23

Mar 9

Why the T5 baseline for ToTTo not match the result with <<Text-to-Text Pre-Training for Data-to-Text Tasks>>

Hello everyone, I am wondering the paper <<Text-to-Text Pre-Training for Data-to-Text Tasks>

unread,

Why the T5 baseline for ToTTo not match the result with <<Text-to-Text Pre-Training for Data-to-Text Tasks>>

Hello everyone, I am wondering the paper <<Text-to-Text Pre-Training for Data-to-Text Tasks>

Mar 9

Search

Clear search

Close search

Google apps

Main menu