Early-bird submission deadline approaching
23 views
Skip to first unread message
Johannes Kiesel
Dec 16, 2022, 5:24:07 PM12/16/22
to valu...@googlegroups.com
Hi all,
We are now close to the early-bird deadline! Submission is open as long
as it is December 16th anywhere on earth [1]. However, if you make a
submission on time and it fails for some reason, we will give you a bit
more time to fix the issue.
To participate in the early-bird submission (which is completely
optional), you have to have at least one run submitted to the
"arguments-test" dataset [2]. Both upload submissions and docker
submissions are welcome (and we already received both kinds). If you
made more than one submission to the test data, please send me a mail to
tell me which one should count.
As some teams struggled to prepare a valid file for upload submission,
we now created an online run file validator that you can use to quickly
check your file before you submit [3]. Feedback on this is welcome.
Since we plan for some time to fix errors (see above), please expect to
get your results on Monday.
If you have questions, please mail me, though note that it is soon
midnight for me and I will take some sleep. But do not worry if you run
into trouble. To encounter these troubles now instead of at the final
deadline is exactly what this early-bird deadline is for. And as I said
before: if you tried to submit but it failed, we will still try to fix
and include the run.
On a separate note: Unfortunately, it takes us longer than expected to
prepare the additional test dataset from religious texts. Since it is so
different to what we have already, the crowdworkers really struggled and
the quality of the annotations did not meet our standards. We are thus
now re-annotating the data. Though we hoped we could made it in time for
the early-bird deadline, it seems we will be missing it by a few days.
However, the dataset will definitely be available for the final
submission deadline as a completely optional second test dataset. We
hope you are also excited to find out whether your approach also works
on such arguments.
We hope you have a great time and have fingers crossed for all your
submissions!
Johannes, Milad, Nailia, Maximilian, Henning, and Benno
[1] https://time.is/Anywhere_on_Earth
[2] Select here:
https://www.tira.io/task/valueeval-at-semeval-2023-human-value-detection
[3] https://touche.webis.de/semeval23/touche23-web/run-validator.html
We are now close to the early-bird deadline! Submission is open as long
as it is December 16th anywhere on earth [1]. However, if you make a
submission on time and it fails for some reason, we will give you a bit
more time to fix the issue.
To participate in the early-bird submission (which is completely
optional), you have to have at least one run submitted to the
"arguments-test" dataset [2]. Both upload submissions and docker
submissions are welcome (and we already received both kinds). If you
made more than one submission to the test data, please send me a mail to
tell me which one should count.
As some teams struggled to prepare a valid file for upload submission,
we now created an online run file validator that you can use to quickly
check your file before you submit [3]. Feedback on this is welcome.
Since we plan for some time to fix errors (see above), please expect to
get your results on Monday.
If you have questions, please mail me, though note that it is soon
midnight for me and I will take some sleep. But do not worry if you run
into trouble. To encounter these troubles now instead of at the final
deadline is exactly what this early-bird deadline is for. And as I said
before: if you tried to submit but it failed, we will still try to fix
and include the run.
On a separate note: Unfortunately, it takes us longer than expected to
prepare the additional test dataset from religious texts. Since it is so
different to what we have already, the crowdworkers really struggled and
the quality of the annotations did not meet our standards. We are thus
now re-annotating the data. Though we hoped we could made it in time for
the early-bird deadline, it seems we will be missing it by a few days.
However, the dataset will definitely be available for the final
submission deadline as a completely optional second test dataset. We
hope you are also excited to find out whether your approach also works
on such arguments.
We hope you have a great time and have fingers crossed for all your
submissions!
Johannes, Milad, Nailia, Maximilian, Henning, and Benno
[1] https://time.is/Anywhere_on_Earth
[2] Select here:
https://www.tira.io/task/valueeval-at-semeval-2023-human-value-detection
[3] https://touche.webis.de/semeval23/touche23-web/run-validator.html
Johannes Kiesel
Dec 19, 2022, 8:39:36 AM12/19/22
to valu...@googlegroups.com
Hi all,
The early-bird deadline has passed and, after some checks, we now
published the results on the leaderboard [1].
"aristotle" shows our baselines, namely (from bottom to top): random
baseline, 1-baseline, BERT baseline from the ACL paper. All other
entries are participants who were brave enough to already submit some
approaches (the names should all start with "touch23-", but we made an
error for "augustine-of-hippo").
Even though some approaches are surely very early in their development,
it is good to see that all beat the random baseline (in terms of F1). We
are amazed that several teams have already beaten the BERT-based
approach from the ACL paper!
Moreover, TIRA has been very stable, so this early-bird deadline has
also been successful in terms of testing submissions. If you have not
done so yet, we would still advice you to do take a bit of time to make
a submission (e.g., for the validation dataset).
We are looking forward to your upcoming submissions! And then especially
to read your description papers [2] so that we get to know your approaches.
As a reminder, here are the upcoming important dates (see
valueeval.webis.de):
- January 24, 2023: Run submission closes.
- January 31, 2023: Run evaluation results out.
- In February 2023: Paper submission deadline.
You are allowed to submit up to four (further) runs for the test
dataset. If you submit more, simply tell me which four you want to have
considered.
We are excited to see how high your approaches will score. Surely you
can reach an F1 of 0.5!
Thank you all and have a great time!
> select dataset ARGUMENTS-TEST
[2] There is a section in the SemEval FAQ on papers, in case you are new
to SemEval: https://semeval.github.io/faq.html
--
Johannes Kiesel
Bauhaus-Universität Weimar
Bauhausstr. 9a, Room 106
99423 Weimar, Germany
Phone: +49 (0)3643 - 58 3720
The early-bird deadline has passed and, after some checks, we now
published the results on the leaderboard [1].
"aristotle" shows our baselines, namely (from bottom to top): random
baseline, 1-baseline, BERT baseline from the ACL paper. All other
entries are participants who were brave enough to already submit some
approaches (the names should all start with "touch23-", but we made an
error for "augustine-of-hippo").
Even though some approaches are surely very early in their development,
it is good to see that all beat the random baseline (in terms of F1). We
are amazed that several teams have already beaten the BERT-based
approach from the ACL paper!
Moreover, TIRA has been very stable, so this early-bird deadline has
also been successful in terms of testing submissions. If you have not
done so yet, we would still advice you to do take a bit of time to make
a submission (e.g., for the validation dataset).
We are looking forward to your upcoming submissions! And then especially
to read your description papers [2] so that we get to know your approaches.
As a reminder, here are the upcoming important dates (see
valueeval.webis.de):
- January 24, 2023: Run submission closes.
- January 31, 2023: Run evaluation results out.
- In February 2023: Paper submission deadline.
You are allowed to submit up to four (further) runs for the test
dataset. If you submit more, simply tell me which four you want to have
considered.
We are excited to see how high your approaches will score. Surely you
can reach an F1 of 0.5!
Thank you all and have a great time!
Johannes, Milad, Nailia, Maximilian, Henning, and Benno
[1]
https://www.tira.io/task/valueeval-at-semeval-2023-human-value-detection
[1]
> select dataset ARGUMENTS-TEST
[2] There is a section in the SemEval FAQ on papers, in case you are new
to SemEval: https://semeval.github.io/faq.html
Johannes Kiesel
Bauhaus-Universität Weimar
Bauhausstr. 9a, Room 106
99423 Weimar, Germany
Phone: +49 (0)3643 - 58 3720
Reply all
Reply to author
Forward
0 new messages