No. of runs

[Javier Parapar]

Hi everybody,

Any decision or thought about the number of runs allowed by team for
the author identification spi task?

Thanks in advance,
j

[Martin Potthast]

Dear Javier,

generally, you may submit as many runs as you like, but you should use
the training data in order to make a choice which parameter
settings/algorithm variants are most promising among the alternatives
available to you.

Best,
Martin

> --
> You received this message because you are subscribed to the Google Group "PAN".
> Visit this group at http://groups.google.com/group/pan-workshop-series
> To unsubscribe send email to pan-workshop-se...@googlegroups.com.

--
Martin Potthast
Bauhaus-Universität Weimar
www.webis.de  ---  www.netspeak.org

[Javier Parapar]

Thank you Martin, 

   We are already working on deciding that. 

Regards, 

j

Javier Parapar

Information Retrieval Lab
University of A Coruña
http://www.irlab.org

[Tim Snider]

hmmm ... is this realistic ??? it will be trivial for us (and probably others) to submit a hundred slightly different results

c u o, tim

[Martin Potthast]

> hmmm ... is this realistic ??? it will be trivial for us (and probably
> others) to submit a hundred slightly different results

True, though I would suggest you make a few reasonable selections
based on the training data. The training data is composed similarly to
the test data, so that this should be possible.

Martin

[J.M. Gomez]

Dear all

I guess that there is no official limit on the number of submissions for a given task, isn't it?

I assume that an order of magnitude of 10 is ok (e.g. at most 20). Please correct me is I am wrong.

Thank you and best regards

  Jose Maria

[Martin Potthast]

Dear Jose Maria,

> I guess that there is no official limit on the number of submissions for a
> given task, isn't it?
>
> I assume that an order of magnitude of 10 is ok (e.g. at most 20). Please
> correct me is I am wrong.

There is no limit on the number of runs you submit. However, we ask
you to limit yourself to a reasonable number of runs. Otherwise, the
people at our site might be forced to disregard some.

In previous years we have had the policy that the last run submitted
is the one that counts in the ranking.

Best,

[David Losada]

hi martin,

what do you mean by "last run"?

should we provide an order of preference for our runs?

or is it just that every group is scored by means of the poorest performing run?

d

> --
> You received this message because you are subscribed to the Google Group "PAN".
> Visit this group at http://groups.google.com/group/pan-workshop-series
> To unsubscribe send email to pan-workshop-se...@googlegroups.com.



--

David E. Losada, PhD. ACM Senior Member
Associate Professor (Prof. Titular de Universidad)
Dept. Electrónica y Computación.
Universidad de Santiago de Compostela, Campus Vida s/n.
15782 Santiago de Compostela, SPAIN
tel:+34 881813570, fax:+34 881 813602
david....@usc.es     http://www.gsi.dec.usc.es/ir
Colegiado num. 292 del C.P. Enx. Informáticos de Galicia
http://maps.google.com/maps/ms?ie=UTF8&hl=gl&t=h&om=1&doflg=ptk&msa=0&ll=42.876373,-8.554809&spn=0.004599,0.011373&z=17&msid=118004855173432402891.0004632efb9ebb10eee34

[Martin Potthast]

Hi David,

> what do you mean by "last run"?

This refers to the last run in chronological order.

> should we provide an order of preference for our runs?

If you submit more than one at a time, please provide an order of preference.

> or is it just that every group is scored by means of the poorest performing run?

No, it's not that way.

[David Losada]

ok.

let me then clarify things....

1) i understand that we are talking here about the sexual predator runs

2) i understand that if we want to send all of our runs together (e.g.
zipped) then we should
make explicit our preference
(e.g. numerating the files in order of preference, from your highest
promising run (1) to your lowest promising run (X, X>1) )

3) if you send several emails with different bulks of estimations
(i.e. different set of runs)
then the chronologically last email is the one that is
going to be used for ranking groups

thanks

d

[Martin Potthast]

> let me then clarify things....
>
> 1) i understand that we are talking here about the sexual predator runs
>
> 2) i understand that if we want to send all of our runs together (e.g.
> zipped) then we should
> make explicit our preference
>  (e.g. numerating the files in order of preference, from your highest
> promising run (1) to your lowest promising run (X, X>1) )
>
> 3) if you send several emails with different bulks of estimations
> (i.e. different set of runs)
> then the chronologically last email is the one that is
> going to be used for ranking groups
>
> thanks

Yes, this sums it up!

[Jose Maria Gomez Hidalgo]

Hi

Please take my comments as a personal opinion based on what I have understood from previous emails in this thread.

On my side, yes I am talking about sexual predator runs (but I do not see why it should not apply to other tasks...).

I understand 3), because it keeps you safe from not submitting nothing, by sending your first "baseline" approach (which incidentally may end up being your best result). But this approach is more typical from programming challenges like e.g. programming bots that play in a league.

I do not understand 2). If only the prioritized one will be taken into account, why should I send several runs? IMHO, this counts as sending just one. *My opinion* is that the best run from every team should count for group ranking, and every group should have a limited number of runs (e.g. 10); otherwise one should submit a baseline one to ensure something is submitted, and the best one according to internal optimization using the training data.

Thanks a lot

  Jose Maria

---

De: Martin Potthast <martin....@uni-weimar.de>
Para: pan-works...@googlegroups.com
Enviado: Lunes 4 de junio de 2012 11:42
Asunto: Re: [PAN'12] No. of runs

--
You received this message because you are subscribed to the Google Group "PAN".
Visit this group at http://groups.google.com/group/pan-workshop-series

To unsubscribe send email to pan-workshop-series+unsub...@googlegroups.com.

[Martin Potthast]

Hi Jose Maria,

> Please take my comments as a personal opinion based on what I have
> understood from previous emails in this thread.

No worries, discussion is always good.

> I understand 3), because it keeps you safe from not submitting nothing, by
> sending your first "baseline" approach (which incidentally may end up being
> your best result). But this approach is more typical from programming
> challenges like e.g. programming bots that play in a league.

Yes, that's the intention of this one.

> I do not understand 2). If only the prioritized one will be taken into
> account, why should I send several runs? IMHO, this counts as sending just
> one. *My opinion* is that the best run from every team should count for
> group ranking, and every group should have a limited number of runs (e.g.
> 10); otherwise one should submit a baseline one to ensure something is
> submitted, and the best one according to internal optimization using the
> training data.

We have thought about adopting the TREC limit of three runs. But then
again, why three? Why not four? If I had four excellent ideas and were
only allowed three runs that would be a shame, wouldn't it?

On the other hand, we do not wish people to drown us in data:
submitting dozens of runs, trying to effectively explore the parameter
space is not so nice (as you have observed, too). We want you to make
an educated guess as to which parameter setting or which
algorithm/model/process accomplishes the task best. Exploring the
parameter space is what you should do using the training data.

So, my suggestion to you is to submit as few runs as possible, picking
your best options.
If you have plenty of _paradigmatically different ideas_ of how to
approach the problem, however, then don't hesitate to submit a run for
each paradigm.

Best,
Martin

PS: I realize that leaving things undefined in this manner adds
confusion. If you really want us to define an explicit limit, we can
do so.

[Jose Maria Gomez Hidalgo]

First of all, thanks again for your fast answer.

My comments below **>

De: Martin Potthast <martin....@uni-weimar.de>
Para: pan-works...@googlegroups.com
Enviado: Lunes 4 de junio de 2012 14:16
Asunto: Re: [PAN'12] No. of runs

Hi Jose Maria,

> Please take my comments as a personal opinion based on what I have
> understood from previous emails in this thread.

No worries, discussion is always good.

**> Thanks :-)

> I understand 3), because it keeps you safe from not submitting nothing, by
> sending your first "baseline" approach (which incidentally may end up being
> your best result). But this approach is more typical from programming
> challenges like e.g. programming bots that play in a league.

Yes, that's the intention of this one.

**> Ok, thanks.

> I do not understand 2). If only the prioritized one will be taken into
> account, why should I send several runs? IMHO, this counts as sending just
> one. *My opinion* is that the best run from every team should count for
> group ranking, and every group should have a limited number of runs (e.g.
> 10); otherwise one should submit a baseline one to ensure something is
> submitted, and the best one according to internal optimization using the
> training data.

We have thought about adopting the TREC limit of three runs. But then
again, why three? Why not four? If I had four excellent ideas and were
only allowed three runs that would be a shame, wouldn't it?

**> I see your point.

On the other hand, we do not wish people to drown us in data:
submitting dozens of runs, trying to effectively explore the parameter
space is not so nice (as you have observed, too). We want you to make
an educated guess as to which parameter setting or which
algorithm/model/process accomplishes the task best. Exploring the
parameter space is what you should do using the training data.

**> For sure, 100% agree.

So, my suggestion to you is to submit as few runs as possible, picking
your best options.
If you have plenty of _paradigmatically different ideas_ of how to
approach the problem, however, then don't hesitate to submit a run for
each paradigm.

**> Ok, we will follow this approach, thanks.

**> Moreover, we are newcomers to this competition, we are just adding our 2 cents; the rules that are defined have been (successfully) used for years, so we do not want nor expect to change things to fit our taste :-)

Best,
Martin

PS: I realize that leaving things undefined in this manner adds
confusion. If you really want us to define an explicit limit, we can
do so.

**> Not needed, thanks.

--
Martin Potthast
Bauhaus-Universität Weimar
www.webis.de  ---  www.netspeak.org

[David Losada]

ok, then.

but what is the point of we ranking our runs?

i mean, i agree with jose maria, we should provide a number of runs
(as few as possible given our training experiments)
and, next, the organizers should rank the teams using the best run for
each team.

of course, we can give you our estimation for our variants given the
training data. but why should this ranking be used
for selecting our participating runs?



d

> pan-workshop-se...@googlegroups.com.

>
>
> --
> You received this message because you are subscribed to the Google Group
> "PAN".
> Visit this group at http://groups.google.com/group/pan-workshop-series
> To unsubscribe send email to

[Gunnar Eriksson]

> So, my suggestion to you is to submit as few runs as possible, picking
> your best options.
> If you have plenty of _paradigmatically different ideas_ of how to
> approach the problem, however, then don't hesitate to submit a run for
> each paradigm.

Is it a fair interpretation of the above to assume that is ok for us to
submit say two completely different runs - one of them being a student
group submission - without risking that the latter may be ignored of
performance reasons?


Gunnar Eriksson

[Thamar Solorio]

Hi Everyone,

Martin, your answer on this thread is a bit confusing. I understand why you have problems setting up a max number of submissions. But I also agree with the other people on this thread in that it's important to know how many runs per team you're planning on ranking. If in fact, a single team is trying to explore different paradigms, it's not clear how you guys will select "the one" for using in the ranking.
Maybe us, the participants, will be ok with having a limit on the number of runs we submit, but knowing that each one of them will enter the ranking and that the highest ranked within a team will be the one used in the overall ranking.

My two cents,

 

TS

 

--
You received this message because you are subscribed to the Google Group "PAN".
Visit this group at http://groups.google.com/group/pan-workshop-series

To unsubscribe send email to pan-workshop-se...@googlegroups.com.

[Martin Potthast]

Hi Thamar, Gunnar, David, and everyone else,

> [...] But I also agree with

> the other people on this thread in that it's important to know how many runs
> per team you're planning on ranking.

As to how many, this is simple: the number depends on our overall
workload. So, when submitting runs, please think about how much work
you wish to cause on our side. :-)

> If in fact, a single team is trying to
> explore different paradigms, it's not clear how you guys will select "the
> one" for using in the ranking.

Ha, that is an important question and in fact we do not wish to make
this decision for you. If you have different paradigms to approach the
problem at hand, isn't it up to you to place your bets on one of them?

Using the best of all submitted runs for an overall ranking is
certainly also a possibility, but it has it's downsides:
- it encourages submitting many runs with subtle parameter differences.
- it takes away an important and very real decision, everyone has to
make when developing a new technology: choosing the best approach.

So, you should make your last submission the one you believe to be the
best of all approaches you can come up with.

Your earlier submissions will be evaluated nonetheless as this has some upsides:
- it gives you the possibility to explore _wild ideas_ which you would
never dare to place your bets on in a competitive situation.
- it allows you to compete within your group: students vs. staff (as
suggested by Gunnar) and still pick the best alternative.
- it allows you to reflect upon your design decisions between runs.

The only downside I can think of is this: one of your earlier runs has
better performance than your latest run thus ranking you lower than if
you'd have made different call. But I must ask you this: how likely is
this gonna be if you evaluate the alternatives well during training?

> Maybe us, the participants, will be ok with having a limit on the number of
> runs we submit, but knowing that each one of them will enter the ranking and
> that the highest ranked within a team will be the one used in the overall
> ranking.

The question is why should we take away from you the risk (or fear?)
of loosing something by making a decision? No risk, no fun!

In fact, while discussing this I came to realize that our rule of
conduct is this:
You are allowed exactly one submission, no more
(plus some additional ones which run out of competition).

Is this agreeable?

Best,
Martin

[Tim Snider]

martin, i am in perfect agreement with your final decision ... even more so because of your rationale ... no risk no fun :-) i also appreciate your offer to score (a small number of) additional runs, but i hope you will be publishing the ground truth after the competition ... thanks again to all involved in putting this together

c u o, tim

[Martin Potthast]

Hi Tim,

> martin, i am in perfect agreement with your final decision ... even more so
> because of your rationale ... no risk no fun :-) i also appreciate your
> offer to score (a small number of) additional runs, but i hope you will be
> publishing the ground truth after the competition ... thanks again to all
> involved in putting this together

Sure, as always, the ground truth will be published after the competition.

[pro...@dsic.upv.es]

Hi everybody.

The easiest solution (especially for our German colleagues in charge
of evaluating aaalll the submissions) would be to allow just one.

I think that the possibility they give to allow more (although just
the one that each team bets on will be taken into account) encourages
exploring new ideas and understanding the impact of subtle parameter
differences. This should be of interest of every team, in primis those
with students who are doing their PhD on the topic because they will
have the feedback on all the approaches explored.

Danke Martin & co :-)

Auf wiedersehen

Paolo


Def. Quota Martin Potthast <martin....@uni-weimar.de>:

> --
> You received this message because you are subscribed to the Google
> Group "PAN".
> Visit this group at http://groups.google.com/group/pan-workshop-series
> To unsubscribe send email to
> pan-workshop-se...@googlegroups.com.
>

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

[David Losada]

yep, i like this 1-official run per team thing (as long as we can also
send other runs that
are evaluated and reported but which are not taken into for the
official ranking of teams)

therefore, i understand that sending a zip file with all the runs but
with indication of which
one is our "best shot" is good enough, right?

best

d

[Martin Potthast]

> therefore, i understand that sending a zip file with all the runs but
> with indication of which
> one is our "best shot" is good enough, right?

Yes, that will do.

[Thamar Solorio]

We'll go with whatever you guys decide. Perhaps one suggestion that can help alleviate the workload for you is that after the rankings have been made official, but before we submit our notebooks we can have the ground truth of the test files and then each participant team can run internal evaluations, as many as they need/want. But then the burden is no longer on your side.

 

Cheers,

TS