Some more issues
Doug Oard
see a few new voices here as well. Have we given thought to whether
we should announce the availability of this forum broadly (e.g. on
SIGIR-List)?
Since at the outset here we're listing some of the big issues, let me
add to that discussion by suggesting that we be think though what we
mean by SIGIR when we say "SIGIR should ..." or "SIGIR should not
...". SIGIR is many things. For example:
SIGIR is more than just the research papers at one conference each
year. SIGIR the conference also includes demos, Salton lectures,
workshops, and posters, and of course it could contain other things if
we wanted it to (we used to have panels, for example). SIGIR the
organization sponsors (or coordinates in other ways with) several
conferences, including IR-focused
conferences such as ECIR, and "cross-over" conferences such
as CIKM, JCDL and WSDM. SIGIR the idea has even broader reach than that -
in some sense the journals whose editorial boards meet at SIGIR the
conference (IP&M, IRJ and TOIS) are an extension of SIGIR the idea,
and the evaluation activities our members have helped to create (TREC,
NTCIR, CLEF, INEX, MIREX, FIRE, and others) have a clear symbiotic
relationship with SIGIR the conference given our focus on evaluation.
Indeed, you can take this further since SIGIR the membership has a
broader vision of IR than we instantiate at SIGIR the conference -
that's one reason why you see SIGIR members who have prominent roles
in conferences like ASIST, WWW, and KDD.
SIGIR is a community. One sense of this is obvious; we meet somewhere
every year, and part of the reason that we do that is that we want to
spend time with each other. But there are other important senses as
well. Most notably, just as newspapers help a community to socially
construct some sense of what that community finds important, SIGIR is
one place where we work together to not just recognize good new work,
but also to recognize new research directions. Alan Smeaton did a
nice job at the banquet in Amsterdam describing how the focus of SIGIR
the conference has evolved over time. Of course, that evolution is
not fully controllable, but neither is it fully accidental. SIGIR the
conference is thus a part of the process by which we make those
choices, and by which we remain connected to a research community that
is shifting in emphasis over time.
SIGIR is an evaluation process. Like the other things that SIGIR is,
you may or may not like this one, but it is undeniable that decision
processes that have visible results help shape people's opinions about
other people. We see this perhaps most crassly in the direct use of
where people have published as part of nationwide research evaluation
exercises (for example, in the UK) and in individual hiring and
promotion decisions in academia generally. But of course evaluation
influences our thinking in much more subtle ways. For example, how
selective a conference is affects whether the best work will be
submitted there in the first place. How strong a conference's
reputation is has some affect on how likely wok published there is to
be found, read, and cited (and, rightly or wrongly, that act of
citation is treated in some aggregate sense as another form of
evaluation).
SIGIR is, of course, many other things as well. For example, SIGIR
the conference is in some sense a publisher (in an earlier era you
could draw fairly sharp lines between journals and the unindexed and
poorly distributed "grey literature" of conferences, but digital
distribution has erased some of those distinctions), SIGIR the
organization is a business (in the sense that it must generate at
least enough resources through its past and present activities to
support its activities in the future), and SIGIR the conference is a
venue for professional development (through at least the mentoring
program, the doctoral consortium, and the networking that happens over
coffee).
I don't mean to say that all of these aspects of SIGIR should equally
important in shaping our thinking - surely SIGIR as a business exists
only because of the value of the other things that SIGIR does, for
example. But each of these things is important in some way to at
least some people, and many of them are interdependent. This
particular discussion started as a discussion about how the decision
process for full papers at SIGIR the conference should be managed.
But some of the points that have been made have broader scope than
that. I am not suggesting that we start every sentence with a full
specification of what we have in mind, but I would like to suggest
that our discussion will not be complete until we take the ideas that
result and map them back to the many things that SIGIR is at some
appropriate level of specificity.
I'll just take a moment here at the end to close the loop on a
discussion with Gord that started on the SPC list in case there is
more on that that people want to say here. The discussion had been
developing around such issues as the repeatability of experiments and
the centrality of the scientific method to what we did, and my
contribution to the discussion had been to suggest that research was
broader than just science (with observational user studies as one such
example). Briefly paraphrasing Gord's response, it was that in his
view science involves the generation and testing of claims, and if the
resulting claim is not testable then he questioned its value. Bruce
then asked that we take the discussion elsewhere (since it was growing
beyond SPC business), so there the issue lay. Let me take up the
issue again by just saying that Gord and I agree completely. For
those who think of hypothesis generation as within the scope of the
scientific method, I would be happy to agree that focusing full papers
at SIGIR the conference on "scientific research" is entirely
appropriate (and we can then proceed to debate under what
circumstances, if any, papers that lack strong experiment results
should be accepted). But too often I have heard the scientific method
used in this broader debate (which started long before the SPC took
the issue up) in a way that sounds to me like all the speaker has in
mind is hypothesis testing. Activities like systematic observations
of user behavior, theorizing, or system design that you might think of
as outside of hypothesis testing, but not necessarily outside of
hypothesis generation, may or may not be appropriate as the sole focus
of a SIGIR full paper. That's something that we can debate. But I
would not want to see those issues defined as being outside of our
scope because our scope encompasses only the scientific method. If we
do want to make that statement, let's make sure we all have the scope
of science that Gord has suggested in mind.
Doug