Threshold Average Precision (TAP-k): a measure of retrieval designed for bioinformatics

[biointelligence]

 biointelligence                  15th July 2010

 

 Threshold Average Precision (TAP-k): a measure of retrieval designed for bioinformatics

 

 

Since database retrieval is a fundamental operation, the measurement of retrieval efficacy is critical to progress in bioinformatics. This article points out some issues with current methods of measuring retrieval efficacy and suggests some improvements. In particular, many studies have used the pooled receiver operating characteristic for n irrelevant records (ROC_n) score, the area under the ROC curve (AUC) of a ‘pooled’ ROC curve, truncated at n irrelevant records. Unfortunately, the pooled ROC_n score does not faithfully reflect actual usage of retrieval algorithms. Additionally, a pooled ROC_n score can be very sensitive to retrieval results from as little as a single query.

Methods: To replace the pooled ROC_n score, we propose the Threshold Average Precision (TAP-k), a measure closely related to the well-known average precision in information retrieval, but reflecting the usage of E-values in bioinformatics. Furthermore, in addition to conditions previously given in the literature, we introduce three new criteria that an ideal measure of retrieval efficacy should satisfy.

Results: PSI-BLAST, GLOBAL, HMMER and RPS-BLAST provided examples of using the TAP-k and pooled ROC_n scores to evaluate sequence retrieval algorithms. In particular, compelling examples using real data highlight the drawbacks of the pooled ROC_n score, showing that it can produce evaluations skewing far from intuitive expectations. In contrast, the TAP-k satisfies most of the criteria desired in an ideal measure of retrieval efficacy.

Availability and Implementation: The TAP-k web server and downloadable Perl script are freely available at http://www.ncbi.nlm.nih.gov/CBBresearch/Spouge/html.ncbi/tap/

Regards,

Team Biointelligence

Biointelligence

Getting started with biosciences

Website: www.biointelligence.in

Email: in...@biointelligence.in
Tel: +91-731-420-2793

 

P please consider the environment before printing this email. Thank you.2