TrojAI Round 1 Completion
TrojAI
Hello TrojAI Community,
We have exciting news!
Per the Round1 Success Criteria: 𝐶𝑟𝑜𝑠𝑠𝐸𝑛𝑡𝑟𝑜𝑝𝑦𝐿𝑜𝑠𝑠<0.3465
From https://pages.nist.gov/trojai/docs/overview.html#round-1
Round1 has been completed.
Congratulations to all participants so far!
The Round1 holdout dataset has been run against all containers which met the success criteria, producing the following results:
|
Team |
CrossEntropyLoss |
CrossEntropy95ConfidenceInterval |
ROC-AUC |
ExecutionTimeStamp |
|
IceTorch |
0.2248191088438034 |
0.09788817501068114 |
0.9708 |
20200724T042001 |
|
IceTorch |
0.23311187326908112 |
0.08348145508766175 |
0.9688 |
20200727T092001 |
|
Perspecta |
0.28113844990730286 |
0.11683407592773438 |
0.9199999999999999 |
20200725T153001 |
|
trojaicy |
0.39936232566833496 |
0.12342037153244018 |
0.8949999999999999 |
20200725T203002 |
|
Cassandra-XF |
0.4137641489505768 |
0.12689418935775756 |
0.8946000000000001 |
20200725T035001 |
Several of the teams beat the success criteria on the holdout dataset, so we collectively haven't over-fit the test dataset running on the ES leaderboard.
The NIST Test and Evaluation Team is in the process of standing up Round2 on the Leaderboard.
New data will be available shortly at http://data.nist.gov
Round1 test data (ran on the Leaderboard)
Round1 holdout data (determined detector generalization performance)
Round2 training data
Links to the data will be posted at https://pages.nist.gov/trojai/docs/data.html
The new data is in the process of being published. It is not likely to be available on http://data.nist.gov before 2020-08-10.
If you would like access before that, please email tro...@nist.gov with the Google Drive account email address you would like granted access and we will share the files with you.
Thank you,
The NIST Test and Evaluation Team