semeval 2012 and semeval 2013?
Ted Pedersen
I'm noticing that we now seem to have a Semeval-2012 and a
Semeval-2013? I'm a little puzzled by this, since I thought a decision
had be made to hold the third Semeval in 2013 (adhering to the usual 3
year cycle)?
Is Semeval-2012 now Semeval-3, with Semeval-2013 to be Semeval-4? I
just find it kind of confusing I guess, and am not sure how we reached
this point...can anyone post a short summary that gets us from the
earlier discussion about 2012 versus 2013 (either one or the other) to
2012 *and* 2013 (apparently both?)
Here's that discussion about 2012 and 2013 last I knew of it, where it
seemed there was fairly good support for 2013...
http://aclweb.org/aclwiki/index.php?title=SemEval_2012_versus_2013
Cordially,
Ted
--
Ted Pedersen
http://www.d.umn.edu/~tpederse
Deniz Yuret
The change was triggered by the introduction of the new *SEM
conference (http://ixa2.si.ehu.es/starsem). One of the main
motivations behind this conference is to reduce to fragmentation of
semantics workshops and related events and bring more people together.
As organizers of SemEval, we thought it would be appropriate to
associate our event with the *SEM conference as well. Whether to
start with the 2012 *SEM depended on whether there would be enough
task organizers willing to switch to a 2012 schedule. We sent the
following email to our current task organizers in September. We got
very positive responses about the association with the yearly *SEM
with 8 tasks willing to switch to 2012. Thus was born SemEval-2012
(http://www.cs.york.ac.uk/semeval-2012) and SemEval-2013
(http://www.cs.york.ac.uk/semeval-2013). The current plan is to
switch to a yearly schedule where not every task needs to run every
year.
best,
deniz
P.S. We recommend adopting a year convention for the SemEval name
(e.g. SemEval-2007, SemEval-2010, and SemEval-2012) rather than the
number convention (e.g. SemEval-3 etc.) to prevent any confusion.
Here is the email that was sent to the task organizers:
---
Dear Semeval task organizers,
Moved by the increasing fragmentation in the fields of lexical and
computational semantics, the SIGLEX and SIGSEM boards have decided to
organize some events in order to consolidate our fields.
- In 2012 a joint SIGLEX-SIGSEM conference called *SEM (pronounced
star sem) will be first held co-located with NAACL in Montreal.
- In 2013 SIGSEM will continue to organize its biennial conference,
IWCS, and SIGLEX will organize a conference collocated with a major
conference.
- In 2014 a second joint SIGLEX-SIGSEM conference will be held.
- In addition we want to organize a shared task every year, following
the CoNLL setup: there will be a competitive call for tasks, and the
task might be held for a few years in a row. The shared task will be
separate from SemEval tasks.
- The conference organizers plan to invite organizers of workshops in
our fields to join in this consolidation effort.
SensEval and SemEval have been building increasing momentum through
evaluation campaigns and workshops, and we would like the SemEval
evaluation exercise to be part of the new conferences. We think that
the new conferences will provide an ideal forum to gather the
community, increasing the potential impact of SemEval tasks.
One possibility for joining all these efforts would be to move SemEval
to a yearly cycle, where each task organizer can choose the cycle and
the years they want to hold their task, i.e. we would expect many of
the tasks not to happen every year.
SemEval is nothing without its task organizers and participants. We
would like to know your opinions on these issues:
- would you agree on the SemEval evaluation exercise being part of
these conferences?- would you agree on holding the SemEval evaluation
exercise every year?- would you be willing to move your task from 2013
to 2012?
Yours,
Eneko Agirre [SIGLEX President]Suresh Manandhar & Deniz Yuret
[SemEval-3 organizers]
Ted Pedersen
I think there are mixed messages here - one of them seems to be that
we'd like to have an event that features a single unifying shared task
(a la CoNLL). That seems like a good idea in fact, since one problem
with Semeval seems to be quite a few tasks, some with very modest or
even disappointing participation. That said, what we see here is
simply a mini-Semeval that takes place in 2012, with another Semeval
to take place in 2013. This seems like it might actually increase
rather than decrease fragmentation.
I'd like to suggest that we consider a model by which Semeval
continues on a 3 year cycle, and that there be a CoNLL style shared
task in the other two years. This might want to be named something
different from Semeval (SEMLEX shared task?), in order to make it
clear that it is just one task, selected via competitive proposal,
hopefully meant to unify the field somewhat, and that Semeval continue
to offer a forum for innovative new tasks and fairly broad
participation. Something like this seems to address several issues -
one is the fragmentation among shared tasks (so the SEMLEX shared task
could address that), but then we also want to allow for innovation (so
continuation of Semeval would also allow that).
What I'm gathering from Semeval-2012 and Semeval-2013 is that we are
just having more Semevals, but that we aren't really doing anything to
address the fragmentation among the shared tasks....Having said all
this, I recognize that 2012 and 2013 might be aberrations due to short
time lines. My suggestion would be to have a SEMLEX shared task (one
unifying task, competitive proposals, etc.) in 2014 and 2015, and then
another Semeval in 2016.
Cordially,
Ted
Deniz Yuret
I agree with you on the topic of a single unifying shared task (a la
CoNLL). In fact this has already been proposed by the *SEM organizers
and it is going to be named the *SEM shared task. The call for
proposals for the 2012 *SEM shared task went out a few days ago. This
will take place every year with *SEM and we may in fact decide to
focus on the same task several years in a row.
On the scheduling of the SemEval tasks there seem to be conflicting
opinions. The current plan is to move SemEval to a yearly cycle as
well, and let the organizers of each task type decide how often and in
what years they would like to run. Not every task type is expected to
run every year. Thus the average number and variety of tasks per year
(i.e. fragmentation) is expected to be comparable to the three year
cycle. The yearly cycle will give some more flexibility to the
organizers and participants in terms of timing. For example a two
year master's student will get a chance to participate in a SemEval
task regardless of her graduation year.
At the end, however, I think the final outcome will be determined by
the task organizers and participants. As long as there are a
sufficient number of interested organizers and enough participants
that will result in a productive SemEval every year I see no reason
why we should hold them back. If not, we will have to reconsider.
best,
deniz