Clarification on PsyDefDetect Evaluation and Inclusion of Class 0

[Eric Rudolph]

Dear organisers,

I have a question regarding the PsyDefDetect challenge evaluation.

The dataset contains 9 labels: seven hierarchical levels of defensive maturity and two auxiliary labels. On the leaderboard, the best reported result from the paper reaches an F1 score of 0.3148 and was achieved by fine-tuning Ministral-8B. However, the experimental setup section appears to indicate that the evaluation was performed only on the positive classes (1-8).

Could you please clarify whether class 0 should be excluded from the dataset for the challenge evaluation? If class 0 is included in the challenge setting, then the leaderboard entry from the paper may not be directly comparable and it might be worth clarifying or adjusting this on the leaderboard.

Best regards,

Eric Rudolph