Clarification — are zero-object (null) cases in the final ranking metric?

[Sachin Gupta]

Hi organizers,

Could you clarify how

zero-object (null-answer)

instances are scored in the

final competition ranking

?

The leaderboard results page states: "Zero-object cases are for reference only and will not be included in the final competition results." However, the provided

evaluate.py

scores a

(subject, relation)

pair with

an

empty gold

object set as precision=recall=F1=

1.0

when the prediction is also empty, and

includes

it in the macro average.

These imply different things, so we'd like to confirm for the

final ranking metric

:

1. Are pairs whose
   gold object set is empty
   excluded from the macro average entirely, or included and scored per
   evaluate.py
   (empty-empty → F1 = 1.0)?
2. For an empty-gold pair, if a system predicts a
   non-empty
   set, is that pair excluded from scoring, or scored as F1 = 0?

This determines whether correctly abstaining on null cases earns credit, and we want our local

evaluate.py

runs to match the official leaderboard.

Thanks!

[Jan-Christoph Kalo]

Dear Sachin,

You can find the evaluation script on the GitHub: https://github.com/lm-kbc/dataset2026

Best, 

Jan-Christoph 

From: akbc2026-s...@googlegroups.com <akbc2026-s...@googlegroups.com> on behalf of Sachin Gupta <sachi...@gmail.com>
Date: Friday, 7. August 2026 at 04:55
To: akbc2026-shared-task <akbc2026-s...@googlegroups.com>
Subject: Clarification — are zero-object (null) cases in the final ranking metric?

Hi organizers,

Could you clarify how zero-object (null-answer)instances are scored in the final competition ranking?

The leaderboard results page states: "Zero-object cases are for reference only and will not be included in the final competition results." However, the provided evaluate.pyscores a (subject, relation)pair withan empty goldobject set as precision=recall=F1=1.0when the prediction is also empty, and includesit in the macro average.

These imply different things, so we'd like to confirm for the final ranking metric:

1. Are pairs whose gold object set is emptyexcluded from the macro average entirely, or included and scored per evaluate.py(empty-empty → F1 = 1.0)?
2. For an empty-gold pair, if a system predicts a non-emptyset, is that pair excluded from scoring, or scored as F1 = 0?

This determines whether correctly abstaining on null cases earns credit, and we want our local evaluate.pyruns to match the official leaderboard. 

Thanks!

--
You received this message because you are subscribed to the Google Groups "akbc2026-shared-task" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akbc2026-shared-...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/akbc2026-shared-task/e0beabc1-f145-43ed-bb90-e681d59beae3n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simon Razniewski]

Clarification on the formulation on the "results" page: Yes, zero-object cases are counted within the "All relations" row.

What is meant by the footnote is that the final ranking goes by the "All relations" row, not the "zero-objects-only" row.

Hope this helps, thanks for asking!

Simon