some issues for human_evaluation_results readme.md

4 views
Skip to first unread message

刀下留人

unread,
Jun 14, 2024, 3:53:16 PMJun 14
to textdetox-clef2024
Dear organizers,

I hope this letter finds you well. Upon reviewing the README.md file located at the provided GitHub repository link https://github.com/textdetox/textdetox_clef_2024/tree/main/human_evaluation_results . I have some issues and need your help🥹.

These issues have been highlighted in red for your convenience.

Texts Detoxification Outputs Human Evaluation Results
To consult the result of human evaluation of your system output navigate to the folder with the corresponding team name and check tsv file with the necessary language
 (*Could you tell me: how many workers are for each sample? 
toxic_sentence - original toxic sentence
neutral_sentence - detoxified sentence generated by the system 
toxic_fluency - fluency of the toxic sentence
neutral_fluency - fluency of the detoxified sentence
 (*Could you tell me: if the value labeled by a worker is 0 for NO, the text is difficult to understand; 0.5 for PARTIALLY, there are mistakes, but the text is intelligible; 1 for YES, there are no or only minor mistakes) 
fluency_score - overall fluency score (1 if neutral fluence is greater or equal to the neutral fluency and 0 otherwise)
(if is? : 1 if neutral fluence is greater or equal to the neutral fluency and 0 otherwise -> 1 if neutral_fluency is greater or equal to the toxic_fluency and 0 otherwise) 
content_score - score of semantic similarity between original and generated sentence
toxic_pairwise_score - score of toxicity (results from pairwise comparison of original and generated sentence)
 (*Could you tell me: if the value labeled by a worker is 0 for NO, the generated sentence is total toxic 0.5 for PARTIALLY, generated sentence have fixed some toxic word; 1 for YES, there are no toxic)  
To calculate the final metric for each language we
average the scores of fluency_score, content_score, toxic_pairwise_score
multiply averaged scores

Thank you very much for your support. I am looking forward to hearing back from you.

Best,

Jiangao Peng(cake, also is D1n910)

Foshan University
Reply all
Reply to author
Forward
0 new messages