Results Summarization Only

[a_ch...@esi.dz]

hi evry one
we are wondering when the results for sub-task Summarization Only will be sent to us?
We sent runs for both sub-task Pre-Filtered Summarization and Summarization Only. We received results for the former one but for the later we did not received results yet. Is it noraml ?
Thank you in adavance
Best regards

[Cristina Garbacea]

Hi,

I have the same question, but the for Pre-filtered Summarization sub-task. We have received the results for the 4 evaluated Summarization Only runs, but not for the run we submitted to Pre-filtered Summarization.

Also, we were wondering how the Summarization runs to be evaluated have been picked. Apparently we received results for runs we did not consider that important and annotated them accordingly when submitting.

Kind Regards,

Cristina Garbacea

[Fernando Diaz]

At the moment, we are only distributing evaluation results for runs that were in the pool.  The selection criteria were described previously,

The runs should use a positive numerical index (lower is more important) as part of the run id if you are planning on submitting lots of runs, such that we can limit if we find we have too many submissions. I

imagine that at least the first 4 runs will be evaluated, but likely more than that if you choose to do so (especially if they have high update overlap).  

If you feel like the missing runs should has qualified for judging according to this criteria, please let me know.

Fernando

---

From: tre...@googlegroups.com <tre...@googlegroups.com> on behalf of Cristina Garbacea <cr1st1na...@gmail.com>
Sent: Wednesday, October 21, 2015 10:27 AM
To: temporalsummarization
Cc: a_ch...@esi.dz
Subject: [TREC-TS] Re: Results Summarization Only

 

--
You received this message because you are subscribed to the Google Groups "temporalsummarization" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-ts+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Richard McCreadie]

To clarify further, due to the large increase in the number of particpants, we had to limit the number of pooled runs to 4 per group (note this is per group, not per task), selected as per the guidelines.

As per normal TREC procedure, we will release the actual assessments and evaluation script after the TREC conference such that participants can evalaute un-pooled runs.

RichardM

[a_ch...@esi.dz]

Hi
thanks for this clarification.
In fact, we sent two set of runs. The first one for Pre-Filtered Summarization in which the name of runs starts by "FS". The second set was for Summarization Only and the name of runs starts by "OS".

Due to the limitation of the number of runs per group and the way that we named our runs, it is apparent that our participation in sub-task Summarization Only is not considerded.

we would have preferred to have the results for the two sub-task to presented in the paper that we plane to submit to TREC conference
Best regards,
A chellal.

[Fernando Diaz]

Part of the reason we did not evaluate the unpooled runs is that we are not yet confident that the evaluation metrics are comparable. We hope to provide analysis at the workshop.

F

________________________________________
From: tre...@googlegroups.com <tre...@googlegroups.com> on behalf of a_ch...@esi.dz <a_ch...@esi.dz>
Sent: Wednesday, October 21, 2015 11:50 AM
To: temporalsummarization
Cc: cr1st1na...@gmail.com; a_ch...@esi.dz
Subject: Re: [TREC-TS] Re: Results Summarization Only

--
You received this message because you are subscribed to the Google Groups "temporalsummarization" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-ts+u...@googlegroups.com.

For more options, visit https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgroups.google.com%2fd%2foptout.&data=01%7c01%7cfdiaz%40microsoft.com%7cddbfb591a036499c4ac108d2da2f5d78%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=QJ1yu5sW%2fAg7R%2fjMhtqq%2fBRW5LFIApfOjLsxyORpjYk%3d