CompStats library for analyzing systems' performance on shared tasks now publicly available

[Hugo Jair Escalante]

Dear colleagues, 

We hope you are doing well. We are happy to share with the CodaLab community CompStats (https://compstats.readthedocs.io/en/latest/) a ready-to-use Python library for the statistical comparison of competitions' results.  CompStats allows you to easily evaluate the statistical-difference in performance among the submissions to your competitions, it also can be used to get insights into the difficulty of tasks. CompStats is based on bootstrapping analysis and it is very easy to use (you just need to load a cvs file with predictions and ground truth and import the library, sample code is available in the site above).

The details of this tool are reported in the following publication: 

https://authors.elsevier.com/c/1ioJacAmyxPWF

We hope you find ComStats useful. If you use the tool, please cite our paper as follows:

@article{NAVAMUNOZ2024,
title = {Analysis of systems’ performance in natural language processing competitions},
journal = {Pattern Recognition Letters},
year = {2024},
issn = {0167-8655},
doi = {https://doi.org/10.1016/j.patrec.2024.03.010},
url = {https://www.sciencedirect.com/science/article/pii/S0167865524000783},
author = {Sergio Nava-Muñoz and Mario Graff and Hugo Jair Escalante},
keywords = {Performance, Bootstrap, Challenges},
}

Many thanks in advance for spreading the word. 

Best regards, 

Sergio, Mario and Hugo

[Johann Petrak]

Thank you, this looks like a nice package!

Unfortunately the documentation is very limited, there seems to be no git repo on github or

anywhere else, no issue tracker, no way to contribute, 

and even on PyPi there is not the information and the links one would expect for a package like this?

Is this maintained and are there plans to properly pubslish this package?

Cheers,

  Johann Petrak

[Christine Nespor]

Thank you for your email

--
You received this message because you are subscribed to the Google Groups "codalab-competitions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codalab-competit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codalab-competitions/6627cc55-306a-46a2-ada8-0f77d1f8d843n%40googlegroups.com.

[Hugo Jair Escalante]

Dear Johann

Thank you for your feedback. We are working indeed on making the library a python package. We will keep you posted.

Best

HJ