Review criteria - no major changes, please?

[Fuhr]

Dear all,

as the discussion seems to dry out, the last posts to this forum have
led me to the conlcusion that there are in fact two different views on
where this discussion should be going (or have gone):

1. Given the partly unsatisfying results of this year's reviewing
process, there is a desire to improve things - both wrt. to the
criteria as well as to the process. The general attitude is to
optimize the outcome. On the other hand, no need for drastic changes
is seen.

2. Alternatively, one can see this as the (long overdue) point in time
for major changes: Given the limited scientific standards present in
many of the accepted papers, more rigid rules should be formulated to
change the character of the conference and raise its quality.
This will have severe consequences, of course. I have seen similar
discussions in other fields, where people said 'we have had enough of
this - no more significant progress is visible'. Would we have the
courage to say e.g. 'no more papers on adhoc retrieval - at least not
of the *how* style'?
Of course, these would be just the consequences - but the real issue
is about the criteria to be applied, which would finally lead to such
consequences.

I think it is clear that I favor the second approach. We habe enough
scientific potential in our field, which allows us to do better than
at the moment. At the moment, IR has a good reputation, due to its
many popular applications - but this will not last forever. As soon as
people take a closer look at the methods we are applying, we will be
in trouble.

However, my opinion seems to be a minority position, so I'd better
keep my mouth shut.

Norbert

[gvc]

I think a broad rethink is necessary. My only quibble with what
you're proposing is, I think it is too narrow. See my comments in
another thread concerning the broad issues, as I see them. Review
process and standards for reporting results may well be deck chairs on
the Titanic.