Question About Changes in "best submitted" timestamp

[Wendi]

Hi,

I'm a member of DBGroup@SUSTech, our team found that our  "best submitted" timestamp changed abnormally recently. Few days before, our  "best submitted" timestamp was '03.26', but yesterday we found it became '03.14'. However, we have submitted many solutions that has a higher score than the one we submitted on 03.14 (seen from the changes in  "best submitted" timestamp). 

Is that because there are changes in the evaluation method of the secret dataset?

[alaska benchmark]

Hi Wendi,

Is that because there are changes in the evaluation method of the secret dataset?

no, the evaluation method has not changed nor the secret evaluation dataset. The "Best submitted" field considers your F-Measure rounded to two decimal points consistently with our ranking system.

Both your solutions (the one submitted on March 26 and the one submitted on March 13) yield exactly the same rounded F-Measure (0.99) and they have possibly been swapped when updating the leaderboard.

This means that submitting one or the other would have no impact in the leaderboard.

Kind regards,

Andrea - Programming Contest Co-Chair

[Wendi]

Hi,

Pretty thanks for your answering in patient. 

Since that the "Best submission" only considers the result rounded to two decimal points, we can hardly to tell whether the new solution is better than before. Is there a chance that the leaderboard will show the result in more details, like rounded to 3 or 4 decimal points?

Another question is that, how to rank two teams with same F1 measure in the leader board?  If two teams with same F1 measure, one in rank 5 and the other in rank 6  after ddl, will the rank 6 team have the opportunity to reproduce the solution?

[alaska benchmark]

Hi Wendi,

Since that the "Best submission" only considers the result rounded to two decimal points, we can hardly to tell whether the new solution is better than before. Is there a chance that the leaderboard will show the result in more details, like rounded to 3 or 4 decimal points?

we decided to show only 2 decimals of the F-measure in order to protect the secret dataset from disclosure attempts. Please note that this is consistent with our evaluation process, that considers a tie two solutions scoring for instance 0.991 and 0.994 respectively (both solutions round indeed to 0.99).

Another question is that, how to rank two teams with same F1 measure in the leader board?

Ties will be broken based on running time (plese see the dedicated "Evaluation Process" section in the task page).

If two teams with same F1 measure, one in rank 5 and the other in rank 6  after ddl, will the rank 6 team have the opportunity to reproduce the solution?

Sure, in that case we will reproduce the results of both teams: the solution with lower running time will be ranked 5th.