CompStats library for analyzing systems' performance on shared tasks now publicly available

37 views
Skip to first unread message

Hugo Jair Escalante

unread,
Mar 22, 2024, 2:14:34 PM3/22/24
to codalab-competitions, Mario Graff, Sergio Martín Nava Muñoz
Dear colleagues, 

We hope you are doing well. We are happy to share with the CodaLab community CompStats (https://compstats.readthedocs.io/en/latest/) a ready-to-use Python library for the statistical comparison of competitions' results.  CompStats allows you to easily evaluate the statistical-difference in performance among the submissions to your competitions, it also can be used to get insights into the difficulty of tasks. CompStats is based on bootstrapping analysis and it is very easy to use (you just need to load a cvs file with predictions and ground truth and import the library, sample code is available in the site above).

The details of this tool are reported in the following publication: 


We hope you find ComStats useful. If you use the tool, please cite our paper as follows:

@article{NAVAMUNOZ2024,
title = {Analysis of systems’ performance in natural language processing competitions},
journal = {Pattern Recognition Letters},
year = {2024},
issn = {0167-8655},
doi = {https://doi.org/10.1016/j.patrec.2024.03.010},
url = {https://www.sciencedirect.com/science/article/pii/S0167865524000783},
author = {Sergio Nava-Muñoz and Mario Graff and Hugo Jair Escalante},
keywords = {Performance, Bootstrap, Challenges},
}

Many thanks in advance for spreading the word. 


Best regards, 

Sergio, Mario and Hugo


Johann Petrak

unread,
Jul 29, 2024, 8:36:53 AM7/29/24
to codalab-competitions
Thank you, this looks like a nice package!
Unfortunately the documentation is very limited, there seems to be no git repo on github or
anywhere else, no issue tracker, no way to contribute, 
and even on PyPi there is not the information and the links one would expect for a package like this?
Is this maintained and are there plans to properly pubslish this package?

Cheers,
  Johann Petrak

Christine Nespor

unread,
Jul 29, 2024, 8:47:55 PM7/29/24
to Johann Petrak, codalab-competitions
Thank you for your email

--
You received this message because you are subscribed to the Google Groups "codalab-competitions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codalab-competit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codalab-competitions/6627cc55-306a-46a2-ada8-0f77d1f8d843n%40googlegroups.com.

Hugo Jair Escalante

unread,
Jul 29, 2024, 8:54:05 PM7/29/24
to Johann Petrak, codalab-competitions
Dear Johann

Thank you for your feedback. We are working indeed on making the library a python package. We will keep you posted.

Best
HJ

Reply all
Reply to author
Forward
0 new messages