Re: [alignathon:66] The Reviews have returned,

[Javier Herrero]

Dear Dent et al,

Indeed, reviewer 2 is the more critical of the work.

I also find very difficult to address his/her first point about the biases
in the whole project. Yes, EVOLVER has been used to train/develop CACTUS
and to generate the sequences used in the competition. I don’t find any
way around it apart from re-writing CACTUS from scratch or re-start the
competition with different sequences. You could read a stronger message
in-between lines. This might or might not be what the editor is referring
to when she says, "they also had considerable concerns about the work as
it stands, which were more strongly stated in their comments to the
Editors”. Again, I don’t think there is much to do about it, but you may
want to clear that point with the editor ASAP.

The second point is easier to address, it is simply about running a few
more on the analyses. I think there is little excuse for not doing so.

Re point #4 I must say that we were strongly encouraged to submit
alignments even if we were not going to tackle all the data sets. The
manuscript expresses some frustration related to this (“Unfortunately, not
all algorithms / pipelines were run for all test sets") and I fear the
reviewer is simply picking up on that. We probably missed the opportunity
of being more positive on that point.

I don’t think s/he is really requesting that the new version includes only
algorithms with results on all datasets. The comment refers to “any future
competition”:
I would encourage the contest organizers to require submissions on all
datasets in any future competitions.
That said, probably everyone is aware of some of my frustrations with this
project, so if you want to leave EPO out that is absolutely fine by me
(but you should clear that with Stephen and Kathryn as well who did the
work).

I hope you can get the details on how MUGSY and PSAR-align were run.

I think we could have some chances to soften reviewer 2 by modifying the
title. Maybe by having something in the lines of “Alignathon: Lessons
learned during a competitive assessment of whole genome alignment methods”
would explicitly avoid having to single out a winner, would justify the
value of the work despite the biases of the analysis and the missing data
sets for the evaluation.

I hope this helps

PD: Could you please update my email address in the google group from
jher...@ebi.ac.uk to javier....@ucl.ac.uk?

Javier
—
Javier Herrero, PhD
Head of the Bill Lyons Informatics Centre
UCL Cancer Institute
http://www.ucl.ac.uk/cancer/