ValueEval results published

[Johannes Kiesel]

Hi all,

As some of you have already noticed: You should now be able to see the
scores of your approaches on the test sets. Please tell me if not or not
all.

Thank you all for the participation! This has been a very exciting
challenge for us and now we are extremely curious at the approaches
behind the numbers! To help you in interpreting your results, we also
prepared a dataset description paper together with everyone who
contributed data (thank you so much!) [1].


Everything below is for next steps.

https://touche.webis.de/semeval23/touche23-web/index.html#important-dates

Every team who submitted at least one run on one test set is allowed to
submit a description paper to SemEval. You do not need to attend the
workshop for it to be published. You will only be asked to review papers
of a few other team. But this peer-review is only to ensure your
description is understandable. We aim for 100% acceptance rate.

To assist you in the process of writing the paper, we prepared a paper
template based on the hints and guidelines of SemEval. It is linked in
the Important Dates section on our web page. I also submitted it to
Overleaf (in case you use that), and will add a link there once it got
accepted as a template to their gallery.

Moreover, we prepared a LaTeX table for each participating team [2] that
you can integrate into your paper ("\input{table-results}"), showing
your own results and a few others for comparison. Since there are a lot
of numbers in this task, we hope using this table can save you some
time. But do not hesitate to adjust it to your needs, especially to name
your submitted runs.

You can also submit more runs. If you do, please tell me and I will
unblind the result for you, though I'll also try to unblind everything
new daily. But the current submissions (the ones you selected in case
you submitted more than 4) will form the official leaderboard. I will
create a version of it for the task web page today or tomorrow. Until
then you can have a look in TIRA.

As said in a previous mail: we also highly recommend to create and
submit Docker images with your approach, so that other researches can
use it easily.



That is it from our side for now. Please do not hesitate to contact us
in case of further questions or if you need some numbers on the dataset
or the submitted approaches to include in your paper.

Thank you again for participation and now for preparing your description
paper!

Johannes, Milad, Nellie, Maximilian, Nicolas, Henning, Benno

[1] https://arxiv.org/abs/2301.13771
[2]
https://github.com/touche-webis-de/touche-code/tree/main/semeval23/human-value-detection/participant-tables

--
Johannes Kiesel

Bauhaus-Universität Weimar
Bauhausstr. 9a, Room 106
99423 Weimar, Germany

Phone: +49 (0)3643 - 58 3720

[Feynman Ma]

hi，

       Could you tell me if there will be an official ranking table？

Johannes Kiesel <johanne...@uni-weimar.de> 于2023年2月1日周三 22:23写道：

--
You received this message because you are subscribed to the Google Groups "ValueEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to valueeval+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/valueeval/e8073b73-a022-8249-a992-3f3a3ff3ca08%40uni-weimar.de.

[Johannes Kiesel]

Hi all,

I now added the promised tables:

https://touche.webis.de/semeval23/touche23-web/index.html#results

We will extend these tables over time extend with links to your source
code (if you tell us the location), Docker run instructions (if you put
an image into TIRA or elsewhere and tell us), and of course your
submitted paper.

Regards,
Johannes

On 02.02.23 12:24, Feynman Ma wrote:
> hi，
>        Could you tell me if there will be an official ranking table？
>
> Johannes Kiesel <johanne...@uni-weimar.de

> <mailto:johanne...@uni-weimar.de>> 于2023年2月1日周三 22:23写道：

>
> Hi all,
>
> As some of you have already noticed: You should now be able to see the
> scores of your approaches on the test sets. Please tell me if not or
> not
> all.
>
> Thank you all for the participation! This has been a very exciting
> challenge for us and now we are extremely curious at the approaches
> behind the numbers! To help you in interpreting your results, we also
> prepared a dataset description paper together with everyone who
> contributed data (thank you so much!) [1].
>
>
> Everything below is for next steps.
>

> https://touche.webis.de/semeval23/touche23-web/index.html#important-dates <https://touche.webis.de/semeval23/touche23-web/index.html#important-dates>

> [1] https://arxiv.org/abs/2301.13771 <https://arxiv.org/abs/2301.13771>
> [2]
> https://github.com/touche-webis-de/touche-code/tree/main/semeval23/human-value-detection/participant-tables <https://github.com/touche-webis-de/touche-code/tree/main/semeval23/human-value-detection/participant-tables>

>
> --
> Johannes Kiesel
>
> Bauhaus-Universität Weimar
> Bauhausstr. 9a, Room 106
> 99423 Weimar, Germany
>
> Phone: +49 (0)3643 - 58 3720
>
> --
> You received this message because you are subscribed to the Google
> Groups "ValueEval" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to valueeval+...@googlegroups.com

> <mailto:valueeval%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/valueeval/e8073b73-a022-8249-a992-3f3a3ff3ca08%40uni-weimar.de <https://groups.google.com/d/msgid/valueeval/e8073b73-a022-8249-a992-3f3a3ff3ca08%40uni-weimar.de>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "ValueEval" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to valueeval+...@googlegroups.com

> <mailto:valueeval+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/valueeval/CAGbF%3DTjh_vpVyK1-9LbtDPo_LvxdzQFysS3To%3D1hXSL3uLChug%40mail.gmail.com <https://groups.google.com/d/msgid/valueeval/CAGbF%3DTjh_vpVyK1-9LbtDPo_LvxdzQFysS3To%3D1hXSL3uLChug%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Johannes Kiesel]

Hi all,

I assume some of you who have submitted are already preparing their
system description papers, and the others plan to start soon. Thank you
so much! These papers are a really important aspect of the shared task,
much more so than the leaderboard.



I have a small update in this regard. Due to a bug in TIRA, two results
in the "Nahj al-Balagha" leaderboard are wrong. The TIRA team is
currently working on fixing the error.

But I already fixed the table on https://valueeval.webis.de and in the
LaTeX tables [1]. If you submitted for that dataset and already
downloaded your table: please update the rows "Best approach" and "Best
per category". In doubt, Github provides a visualization of the changes [2].

Thanks a lot to team Augustine of Hippo for pointing out the bug!

Sorry for the inconvenience and looking forward to your papers!
Johannes


[1]
https://github.com/touche-webis-de/touche-code/tree/main/semeval23/human-value-detection/participant-tables
[2]
https://github.com/touche-webis-de/touche-code/commit/8a0c017665fcd990a3def8c78a262fa1de862c5b

[Johannes Kiesel]

Hi all,

First of all: big thanks to all who are already working on your papers.
Some of you already contacted me, and I am happy to help. Also feel free
to ask if you have questions regarding making a Docker image. It is not
that hard [1], and you gain a lot in terms of making it easier for
others to use your software.


Since it caused some confusion: there are no arguments that resort to
Stimulation, Power: Resources, or Tradition in the New York Times
dataset. The leaderboard [2] now shows "-" instead of "0.00" for these.
I also updated the LaTeX tables in this regard [3]. There also was a
small error in the "Best per category" results for the NYT dataset in
this regard (thanks to team R. M. Hare for pointing it out!). Please
make sure to update that row in your table in case you submitted for New
York Times.


Thanks all!
Johannes


[1] https://docs.docker.com/get-started/
[2] https://touche.webis.de/semeval23/touche23-web/index.html#results
[3]
https://github.com/touche-webis-de/touche-code/tree/main/semeval23/human-value-detection/participant-tables