Sharing Simple Evaluation Scripts
13 views
Skip to first unread message
Nathaniel Weeks Rollings
Aug 29, 2022, 11:41:20 AM8/29/22
to TREC-IS
CrisisFACTS Organizers,
My team put together a handful of facts for the Lilac wildfire queries along with a simple scoring script. It is far from comprehensive, and some of the facts may not be entirely accurate, but we found that it gives a helpful sanity check on new models before diving into a more in-depth manual review. Would there be any issues if we share that here in case other folks would find it useful? Thanks.
- Nate
My team put together a handful of facts for the Lilac wildfire queries along with a simple scoring script. It is far from comprehensive, and some of the facts may not be entirely accurate, but we found that it gives a helpful sanity check on new models before diving into a more in-depth manual review. Would there be any issues if we share that here in case other folks would find it useful? Thanks.
- Nate
Cody Buntain
Aug 30, 2022, 1:58:30 AM8/30/22
to TREC-IS
Hi, Nate!
Indeed, no issue if you share them, and thank you for offering them to the community!
-Cody
Nathaniel Weeks Rollings
Sep 8, 2022, 9:38:05 AM9/8/22
to TREC-IS
Cody,
Great! Again, they are not terribly complex, but hopefully other teams will also find them useful for formative evaluations.
https://gitlab.com/nate.rollings/simple_crisisfacts_scoring
- Nate
Great! Again, they are not terribly complex, but hopefully other teams will also find them useful for formative evaluations.
https://gitlab.com/nate.rollings/simple_crisisfacts_scoring
- Nate
Reply all
Reply to author
Forward
0 new messages