About the Leaderboard

[Francisco Ferraz]

Hello,

I've been working for a while with the TADPOLE datasets and I'm now trying to compare my results with the ones present on the Leaderboard: https://tadpole.grand-challenge.org/D4_Leaderboard/ . I have a couple questions:

- What is the difference between the table on the Results page (https://tadpole.grand-challenge.org/Results/) and the Leaderboard? 

- Why does the table assigned to the D2 prediction set have the same results for the repeated teams (i.e. Frog has mAUC of 0.931 and BCA of 0.849 in both tables) but then the table for D3 is completely different?

- If these results are the ones obtained forecasting D4, is it possible to obtain the results on the prediction sets D2 and D3?

- Were the winner models only trained using D1 or did they include D2 or D3 on their training set?

Hope you can help me, thanks in advance.

Fran.

[Razvan Marinescu]

Hi Francisco,

Here are the answers to your questions:

- What is the difference between the table on the Results page (https://tadpole.grand-challenge.org/Results/) and the Leaderboard? 

The Results page (https://tadpole.grand-challenge.org/Results/) only contains prize-eligible submissions that had to be sent to us by Nov 2017. The D4_Leaderboard (https://tadpole.grand-challenge.org/D4_Leaderboard/) contains some newer submissions (e.g. CBIL-MinMFa) that have been uploaded since.

- Why does the table assigned to the D2 prediction set have the same results for the repeated teams (i.e. Frog has mAUC of 0.931 and BCA of 0.849 in both tables) but then the table for D3 is completely different?

All entries sent to us by Nov 2017 (including Frog) appear in both tables. Results for D3 are all different, as they used a limited, cross-sectional training set which mimics a clinical trial. See the data page https://tadpole.grand-challenge.org/Data/#Datasets for details on D3.

- If these results are the ones obtained forecasting D4, is it possible to obtain the results on the prediction sets D2 and D3?

Not sure I fully understand the question. Since D2 and D3 are prediction sets (contain data of subjects that need to be forecasted), they are part of training, while D4 is the "ground-truth" test set. The results page shows results in both D2 prediction set + D4 test set (Overall scores — longitudinal dataset D2) as well as D3 prediction set + D4 test set (Overall scores — cross-sectional dataset D3).

- Were the winner models only trained using D1 or did they include D2 or D3 on their training set?

Almost all models were trained on both D1 + D2 or D1 + D3 (notable exception is the very simple BenchmarkLastVisit).  

Let us know if you have further questions.

Raz