TREC-IS 2021-A Reminder

[Cody Buntain]

Hello, all!

A few points:

1. Remember, runs are due on Monday, 3 May, at 11:59 PM AoE (if you need an extra day, please let me know, and we’ll work with you. We value participation more than timing  )
2. I’ll share the run-file check script later this afternoon (Thursday), so you can make sure your JSON submissions are going to be parsed correctly
3. Run submissions will be accepted via the NIST/TREC submission website. The server that runs that system is currently being upgraded and should be back online on Friday. I’ll post a note via Slack and via here with a link and when it’s available.

If you have any questions, comments, or ideas for annotation/future Incident Streams work, feel free to reach out.

-Cody

[Cody Buntain]

Evening, all.

As promised, I've created an updated version of the run-file format-checking script for the new JSON format. You can see it here: http://cody.bunta.in/bitbucket/trecis/check_incident.json.2021a.pl

This script checks whether the JSON has the appropriate fields per line, that each run-file has only a single unique run tag, correct formatting for information-type scores and labels, and a correct scale for the priority field. If you have any questions or run into a bug, please let me know.

We'll be sharing the link to the submission site on NIST's servers Friday afternoon.

-All the best,

Cody