New dataset, and more
Gregory Fiumara
Hello ELFT friends,
We have several updates for you today! This is a large update, so if you notice anything awry, please get in touch.
New Dataset
A big thank you goes out the US Department of Defense, which provided us with our largest collection of fingermarks to date. We've searched these images and performed initial analysis for all supported algorithms. Their report cards have been updated.
New Validation Checks
Several new checks were added to the validation script a few weeks ago. You are required to use the latest version of the script when submitting new software for testing. These new checks may help prevent you from experiencing common errors we've seen across multiple participants.
Two Submissions
Knowing that we planned to continuously add new datasets, we limited the analysis on our website to the latest algorithm per organization as a time-saving measure. Unfortunately, this does not necessarily allow for experimentation--when compared head-to-head with potential competitors, organizations will almost always choose to optimize for accuracy. To facilitate this, we will now allow for two submissions per organization.
You will notice that some participants still only have one submission, with others listed as Retired. Their first submission is dropped if:
- Their previous version no longer adheres to the latest timing requirements. Several algorithms were not able to meet timing requirements when we combined plain and rolled impressions last March.
- Software defects prevented complete execution of new data.
Submissions Per Year
The ELFT agreement states that participants may submit a new algorithm once every 120 days. Because our other fingerprint technology evaluations permit new submissions once every 90 days, we have mistakenly let a few organizations submit with the shorter duration. With the addition of this new dataset and two submissions per organization, we now must strictly enforce the 120-day limit to best utilize NIST resources.
Smaller Changes
There are a handful of other smaller changes, such as not reporting lower FPIR for smaller datasets and expansion of the Y-axis in some DET plots.
Coming Soon
We plan to have plenty more data and analysis updates coming soon. Please be sure your newer submissions support the several types of ELFT scenarios. We will add them as compute cycles and human resources allow.
Thanks,
-Greg