About ranking of participants

[Sandra Garcia Blasco]

Dear all,

We just received the results from the competition. We are now aware
that the ranking will not be revealed until the workshop. However, we
think that knowing the ranking has some advantages:

1) Shape the conclusions of the paper so they are in concordance to
the reality. It's necessary to know how good or bad was your system in
comparison with others facing the same problem with the same corpus.

2) In case I had a good ranking, I would like to include it in my
M.Sc. Thesis by July. I am guessing this might be a similar case for
other people.

3) Be able to sleep from here to September ;)

What do you think?
Best,

--
Sandra García
---------------------

[Faisal Chowdhury]

I completely agree with you.

Best,

Faisal

[Lan Man]

Totally agree. At least, the top one result, average and standard deviation can be released now, which would be helpful to tune the tone of submission.

Best,

LAN Man

[Stefania Rubrichi]

...totally agree with you!!!

DrIF team

[Tamara]

I agree as well!

On Jun 14, 2:08 pm, Stefania Rubrichi <stef...@gmail.com> wrote:
> ...totally agree with you!!!
>
> DrIF team
>

> 2011/6/14 Lan Man <lanman...@gmail.com>

>
> > Totally agree. At least, the top one result, average and standard deviation
> > can be released now, which would be helpful to tune the tone of submission.
>
> > Best,
> > LAN Man
>

> > 在 2011-6-14，19:26，Faisal Chowdhury <fmchowdh...@gmail.com> 写道：
>
> > I completely agree with you.
>
> > Best,
> > Faisal
>

[advaned]

Dear Participants,

We are still evaluating your files.

Best regards,
DDIExtraction2011 organizers.

[advaned]

Dear Participants,

Ranking results or score difference will be available the day of the
workshop. Always in this kind of tracks such as Cross Language
Evaluation Forum- CLEF (http://www.clef-campaign.org/) or ACE
(Automatic Content Extraction Evaluation, http://www.itl.nist.gov/iad/mig/tests/ace/ace07/)
only individual results are sent to participants. When the the paper
submission process ends, we will create a report wich documents the
evaluation identifying the participants and their scores achieved.
This report will be published with the rest of papers as workshop
proceedings.

Your paper describing your system shoud be focused on the techniques
used as well as the problems detected.


Regards,

DDIExtraction2011 organizers

[Santiago M. Mola]

Dear all,

On Jun 15, 4:11 pm, advaned <isegur...@gmail.com> wrote:
>
> Ranking results or score difference will be available the day of the
> workshop. Always in this kind of tracks  such as Cross Language

> Evaluation Forum- CLEF (http://www.clef-campaign.org/) [...]

> only  individual results are sent to participants.

Actually, this is not completely accurate. My personal experience at
CLEF was on PAN @ CLEF 2010, where all the results were published
before the deadline of the paper and even before the conference
registration. I'd say that the case was the same for ImageCLEF 2010
and WePS 3, although I haven't been able to check this accurately.

I'm not sure if peer-review was blind at PAN, and it is in this
competition, so I understand the natural concern about publishing the
ranking and then, reviewers being able to identify participants
because of their results. However, it is perfectly possible to publish
a blind ranking, just with the F-Scores and any other auxiliar
measures (e.g. Precision, Recall, Area Under Precision-Recall Curve)
and without names or any other data that could leak the identity of
the participants to the reviewers.

Best regards,
Santiago