New Evaluation Measures
0 views
Skip to first unread message
Martin Potthast
May 11, 2009, 4:49:00 AM5/11/09
to pan09-co...@googlegroups.com
Dear all,
on the competition Web page you will find new evaluation measures that we will use to asses your performance in the competition.
If you don't see it, try deleting your browser cache.
The new measures reflect the criticism the old ones got, and they are much simpler to understand. Both, precision and recall, are being measured at the case-level rather than the document level. This has the advantage that there is no problem anymore with detections in documents which do not contain plagiarism. If you detect something in a document which does not contain plagiarism it will harm your precision.
Another thing is the treatment of external plagiarism cases. As you know you are required to accurately detect the plagiarized passage and the source passage. The characters of both sides are counted, and your overlap with them is measured to compute the precision and recall for each case. A detection which does not overlap with by at least one char with both sides of the corresponding plagiarized passage will not increase your recall and harm your precision.
As before, if you happen to detect the same plagiarized passage more than once this will harm your performance with respect to the average granularity.
The formula to compute the your overall performance is now given explicitly and not only in the rules.
Best regards,
Martin Potthast
PS: My thanks to all of you who pointed out the problems with the old measures, and my apologies to all of you for the inconvenience.
--
Martin Potthast
Bauhaus-Universität Weimar
www.webis.de - netspeak.webis.de
If you do things right, people won't be sure you've done anything at all.
on the competition Web page you will find new evaluation measures that we will use to asses your performance in the competition.
If you don't see it, try deleting your browser cache.
The new measures reflect the criticism the old ones got, and they are much simpler to understand. Both, precision and recall, are being measured at the case-level rather than the document level. This has the advantage that there is no problem anymore with detections in documents which do not contain plagiarism. If you detect something in a document which does not contain plagiarism it will harm your precision.
Another thing is the treatment of external plagiarism cases. As you know you are required to accurately detect the plagiarized passage and the source passage. The characters of both sides are counted, and your overlap with them is measured to compute the precision and recall for each case. A detection which does not overlap with by at least one char with both sides of the corresponding plagiarized passage will not increase your recall and harm your precision.
As before, if you happen to detect the same plagiarized passage more than once this will harm your performance with respect to the average granularity.
The formula to compute the your overall performance is now given explicitly and not only in the rules.
Best regards,
Martin Potthast
PS: My thanks to all of you who pointed out the problems with the old measures, and my apologies to all of you for the inconvenience.
--
Martin Potthast
Bauhaus-Universität Weimar
www.webis.de - netspeak.webis.de
If you do things right, people won't be sure you've done anything at all.
J.A. Palkovskii Plagiarism-Detector Project Leading Programmer
May 11, 2009, 3:26:53 PM5/11/09
to PAN'09 Competition on Plagiarism Detection
Hello Matrin!
I've teamed up with my team and found out that the new formula is a
err... less understandable than the previous one.
I can hardly call it "simpler to understand" :-(.
Could you please make some additional explanations?
Two ideal things I would like you to do are:
1. Present the new formula's with complete "variable-meaning" legend.
e.g.: |S| - is a ... . To get this value you need to ...
2. Present the new formula in a Coding-variable style:
Corretcly_Detected_Plagiarism_Chars_count /
Totally_Detected_Plagiarism_Chars_Count
etc.
Looking forward to your reply!
On 11 май, 11:49, Martin Potthast <martin.potth...@uni-weimar.de>
wrote:
> Bauhaus-Universität Weimarwww.webis.de- netspeak.webis.de
I've teamed up with my team and found out that the new formula is a
err... less understandable than the previous one.
I can hardly call it "simpler to understand" :-(.
Could you please make some additional explanations?
Two ideal things I would like you to do are:
1. Present the new formula's with complete "variable-meaning" legend.
e.g.: |S| - is a ... . To get this value you need to ...
2. Present the new formula in a Coding-variable style:
Corretcly_Detected_Plagiarism_Chars_count /
Totally_Detected_Plagiarism_Chars_Count
etc.
Looking forward to your reply!
On 11 май, 11:49, Martin Potthast <martin.potth...@uni-weimar.de>
wrote:
Martin Potthast
May 11, 2009, 6:20:53 PM5/11/09
to pan09-co...@googlegroups.com
Hi Yurii,
I am sorry to hear that...
I'll do my very best.
Maybe I am mistaken, but the test right above the formulas does describe all variables:
Best,
Martin
--
Martin Potthast
Bauhaus-Universität Weimar
www.webis.de - netspeak.webis.de
I've teamed up with my team and found out that the new formula is a
err... less understandable than the previous one.
I can hardly call it "simpler to understand" :-(.
I am sorry to hear that...
Could you please make some additional explanations?
I'll do my very best.
Two ideal things I would like you to do are:
1. Present the new formula's with complete "variable-meaning" legend.
e.g.: |S| - is a ... . To get this value you need to ...
2. Present the new formula in a Coding-variable style:
Corretcly_Detected_Plagiarism_Chars_count /
Totally_Detected_Plagiarism_Chars_Count
etc.
Looking forward to your reply!
Maybe I am mistaken, but the test right above the formulas does describe all variables:
"Let s denote a plagiarized passage from the set S of all plagiarized passages. Let r denote a detection from the set R of all detections and let S_R be the subset of S for which detections exist in R. Let |s|, |r| denote the char lengths of s, r and let |S|, |R|, |S_R| be the sizes of the respective sets."
Best,
Martin
--
Martin Potthast
Bauhaus-Universität Weimar
Mike Sanders
May 12, 2009, 2:13:06 AM5/12/09
to pan09-co...@googlegroups.com
Hello, Martin!
We are not native English speakers and we do require some explanation due to the fact that we (me in particular) hardly
understand English hardcore math language - I'm a linguist having zero mathematical background - so I need to consult a wiki
before getting to understand ascii variant of ranged sigma sign or harmonic mean :-).
So I will highly appreciate the description of the following:
1. What is meant under: "detection"?
2. What is meant under: "r"?
3. What is meant under: "R"?
4. What is meant under: "S_R"?
These are the point I do have problems understanding.
Thank you in advance!
We are not native English speakers and we do require some explanation due to the fact that we (me in particular) hardly
understand English hardcore math language - I'm a linguist having zero mathematical background - so I need to consult a wiki
before getting to understand ascii variant of ranged sigma sign or harmonic mean :-).
So I will highly appreciate the description of the following:
Let r denote a detection from the set R of all detections and let S_R be the subset of S for which detections exist in R. Let |s|, |r| denote the char lengths of s, r and let |S|, |R|, |S_R| be the sizes of the respective sets.
2. What is meant under: "r"?
3. What is meant under: "R"?
4. What is meant under: "S_R"?
These are the point I do have problems understanding.
Thank you in advance!
2009/5/12 Martin Potthast <martin....@uni-weimar.de>
Martin Potthast
May 12, 2009, 3:09:48 AM5/12/09
to pan09-co...@googlegroups.com
Hi Yurii,
Never mind, I'm happy to help. You may also want to look up the general ideas of precision, recall, and F-measure, e.g., here.
A "detection" is what your software reports as plagiarism, i.e., from (document X, offset 1000, length 1000) to (document Y, offset 5000, length 700). As you can see there is a direct correspondence between the term detection and the feature-tags named "detected-plagiarism" you have to produce.
"r" is a variable representing one detection.
"R" is a set in which all of your detections on the whole corpus are collected.
The set S contains all plagiarized passages, s, which we inserted in the corpus.
Now, S_R is a subset of S, that is, a selection of elements of S for which a particular constraint holds.
The constraint is that only those plagiarized passages are selected for S_R which were detected by you. As explained above, the set R collects all of your detections so that we can compute the subset of S simply by looking at all of your detections which overlap by at least one char with a plagiarized passage and the source passage.
I hope I could help, but still, don't hesitate to ask more questions!
Best,
Martin
PS: Looking at your mail address: Is it Yurii or is it Mike, or both?
We are not native English speakers and we do require some explanation due to the fact that we (me in particular) hardly
understand English hardcore math language - I'm a linguist having zero mathematical background - so I need to consult a wiki
before getting to understand ascii variant of ranged sigma sign or harmonic mean :-).
So I will highly appreciate the description of the following:
Never mind, I'm happy to help. You may also want to look up the general ideas of precision, recall, and F-measure, e.g., here.
1. What is meant under: "detection"?Let r denote a detection from the set R of all detections and let S_R be the subset of S for which detections exist in R. Let |s|, |r| denote the char lengths of s, r and let |S|, |R|, |S_R| be the sizes of the respective sets.
A "detection" is what your software reports as plagiarism, i.e., from (document X, offset 1000, length 1000) to (document Y, offset 5000, length 700). As you can see there is a direct correspondence between the term detection and the feature-tags named "detected-plagiarism" you have to produce.
2. What is meant under: "r"?
"r" is a variable representing one detection.
3. What is meant under: "R"?
"R" is a set in which all of your detections on the whole corpus are collected.
4. What is meant under: "S_R"?
The set S contains all plagiarized passages, s, which we inserted in the corpus.
Now, S_R is a subset of S, that is, a selection of elements of S for which a particular constraint holds.
The constraint is that only those plagiarized passages are selected for S_R which were detected by you. As explained above, the set R collects all of your detections so that we can compute the subset of S simply by looking at all of your detections which overlap by at least one char with a plagiarized passage and the source passage.
These are the point I do have problems understanding.
Thank you in advance!
I hope I could help, but still, don't hesitate to ask more questions!
Best,
Martin
PS: Looking at your mail address: Is it Yurii or is it Mike, or both?
J.A. Palkovskii Plagiarism-Detector Project Leading Programmer
May 15, 2009, 8:37:55 AM5/15/09
to PAN'09 Competition on Plagiarism Detection
Hello Martin!
Thank you for most detailed answer!
No more questeions as for the formula.
--
Best Regards Yurii
On 12 май, 01:20, Martin Potthast <martin.potth...@uni-weimar.de>
wrote:
>
> > passages. Let *r* denote a detection from the set *R* of all detections
> > and let *S_R* be the subset of *S* for which detections exist in *R*. Let
> > |*s*|, |*r*| denote the char lengths of *s*, *r* and let |*S*|, |*R*|, |*
> > S_R*| be the sizes of the respective sets."
Thank you for most detailed answer!
No more questeions as for the formula.
--
Best Regards Yurii
On 12 май, 01:20, Martin Potthast <martin.potth...@uni-weimar.de>
wrote:
> Hi Yurii,
>
> I've teamed up with my team and found out that the new formula is a
>
> > err... less understandable than the previous one.
>
> > I can hardly call it "simpler to understand" :-(.
>
> I am sorry to hear that...
>
> > Could you please make some additional explanations?
>
> I'll do my very best.
>
> > Two ideal things I would like you to do are:
>
> > 1. Present the new formula's with complete "variable-meaning" legend.
> > e.g.: |S| - is a ... . To get this value you need to ...
> > 2. Present the new formula in a Coding-variable style:
>
> > Corretcly_Detected_Plagiarism_Chars_count /
> > Totally_Detected_Plagiarism_Chars_Count
> > etc.
>
> > Looking forward to your reply!
>
> Maybe I am mistaken, but the test right above the formulas does describe all
> variables:
>
> "Let *s* denote a plagiarized passage from the set *S* of all plagiarized
>
> I've teamed up with my team and found out that the new formula is a
>
> > err... less understandable than the previous one.
>
> > I can hardly call it "simpler to understand" :-(.
>
> I am sorry to hear that...
>
> > Could you please make some additional explanations?
>
> I'll do my very best.
>
> > Two ideal things I would like you to do are:
>
> > 1. Present the new formula's with complete "variable-meaning" legend.
> > e.g.: |S| - is a ... . To get this value you need to ...
> > 2. Present the new formula in a Coding-variable style:
>
> > Corretcly_Detected_Plagiarism_Chars_count /
> > Totally_Detected_Plagiarism_Chars_Count
> > etc.
>
> > Looking forward to your reply!
>
> Maybe I am mistaken, but the test right above the formulas does describe all
> variables:
>
>
> > passages. Let *r* denote a detection from the set *R* of all detections
> > and let *S_R* be the subset of *S* for which detections exist in *R*. Let
> > |*s*|, |*r*| denote the char lengths of *s*, *r* and let |*S*|, |*R*|, |*
> > S_R*| be the sizes of the respective sets."
>
> Best,
> Martin
>
> --
> Martin Potthast
> Bauhaus-Universität Weimarwww.webis.de- netspeak.webis.de
> Best,
> Martin
>
> --
> Martin Potthast
Martin Potthast
May 15, 2009, 9:53:58 AM5/15/09
to pan09-co...@googlegroups.com
Hi Yurii,
that's nice to hear!
Best,
Martin
that's nice to hear!
Best,
Martin
--
Martin Potthast
Bauhaus-Universität Weimar
www.webis.de - netspeak.webis.deMartin Potthast
Bauhaus-Universität Weimar
Reply all
Reply to author
Forward
0 new messages