Task 3 evaluation+training error
avram.and...@gmail.com
Sasha Spala
Hi Avram,
Check that you’re submitting only the zipped files and are not compressing the folder that contains the files.
Best,
Sasha
From: <semeval-202...@googlegroups.com> on behalf of "avram.and...@gmail.com" <avram.and...@gmail.com>
Date: Tuesday, March 3, 2020 at 12:44 PM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Task 3 evaluation+training error
Hello,
When I try to submit for task 3, I get the following error:
Can you take a look at it?
Thank you,
Avram Andrei
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/semeval-2020-task-6-all/aae7d012-90a5-4fbf-9f80-8201007dbc1b%40googlegroups.com.
avram.and...@gmail.com
Hi Avram,
Check that you’re submitting only the zipped files and are not compressing the folder that contains the files.
Best,
Sasha
From: <semeval-202...@googlegroups.com> on behalf of "avram.an...@gmail.com" <avram.an...@gmail.com>
Date: Tuesday, March 3, 2020 at 12:44 PM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Task 3 evaluation+training error
Hello,
When I try to submit for task 3, I get the following error:
Can you take a look at it?
Thank you,
Avram Andrei
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.
avram.and...@gmail.com
Hi Avram,
Check that you’re submitting only the zipped files and are not compressing the folder that contains the files.
Best,
Sasha
From: <semeval-202...@googlegroups.com> on behalf of "avram.an...@gmail.com" <avram.an...@gmail.com>
Date: Tuesday, March 3, 2020 at 12:44 PM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Task 3 evaluation+training error
Hello,
When I try to submit for task 3, I get the following error:
Can you take a look at it?
Thank you,
Avram Andrei
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.
Sasha Spala
Thanks for bringing this up. I’ll have a moment in about an hour to check in on this – I’ll get back to you then.
From: <semeval-202...@googlegroups.com> on behalf of "avram.and...@gmail.com" <avram.and...@gmail.com>
Date: Thursday, March 5, 2020 at 3:48 PM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Re: Task 3 evaluation+training error
I managed to solve the filename match problem, but now i get the following error:
I checked and the "Refers-To" token is in the evaluated tokens.
On Tuesday, 3 March 2020 20:55:21 UTC+2, Sasha Spala wrote:
Hi Avram,
Check that you’re submitting only the zipped files and are not compressing the folder that contains the files.
Best,
Sasha
From: <semeval-202...@googlegroups.com> on behalf of "avram.an...@gmail.com" <avram.an...@gmail.com>
Date: Tuesday, March 3, 2020 at 12:44 PM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Task 3 evaluation+training error
Hello,
When I try to submit for task 3, I get the following error:
Can you take a look at it?
Thank you,
Avram Andrei
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/aae7d012-90a5-4fbf-9f80-8201007dbc1b%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/semeval-2020-task-6-all/307adcd5-b8f8-4e7d-a7b0-9cff9f88f562%40googlegroups.com.
Sasha Spala
Should be fixed now.
Best,
Sasha
To view this discussion on the web visit
https://groups.google.com/d/msgid/semeval-2020-task-6-all/2306B2E5-8FEE-47DD-98CA-0280AE2B2AF0%40adobe.com.
avram.and...@gmail.com
Sasha Spala
On Mar 7, 2020, at 4:11 AM, "avram.and...@gmail.com" <avram.and...@gmail.com> wrote:
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/09c2c76a-ff2f-4e5a-a50d-5efaa39052ba%40googlegroups.com.
Sasha Spala
Avram, are you still facing this problem? I couldn’t reproduce it on evaluated relations and I haven’t heard anything so far from others so I’m assuming it is just the result of predicting a relation for an unevaluated label.
Sasha
From: Sasha Spala <ssp...@adobe.com>
Date: Saturday, March 7, 2020 at 10:13 AM
To: "avram.and...@gmail.com" <avram.and...@gmail.com>
Cc: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Re: Task 3 evaluation+training error
Sure, I’ll check it out. I’ll be back at my laptop this evening.
For supplements, make sure you’re not trying to predict relations for a non-evaluated token label.
We are only evaluating on labels that are both in the evaluated token labels and the evaluated relation labels. In any case, I’ll take a look and make sure the evaluation script is doing what we expect.
Sasha
Sent from my iPhone
On Mar 7, 2020, at 4:11 AM, "avram.and...@gmail.com" <avram.and...@gmail.com> wrote:
Thanks! I managed to submit today and the submission finished running. However, I get a -1 overall score, altough the macro metrics are different from 0 and got no error. Could you please check this out?
Could you please also update the script for the training subtask 3? I get the same error, but with "Supplements" this time.
Thank you very much for your effort!
On Tuesday, 3 March 2020 19:44:48 UTC+2, avram.an...@gmail.com wrote:
Hello,
When I try to submit for task 3, I get the following error:
Can you take a look at it?
Thank you,
Avram Andrei
avram.and...@gmail.com
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.
Sasha Spala
Yep! You should not be predicting relation labels for labels that are not included in the task 2 evaluated labels. This was brought up on Github and we opted to evaluate only Supplements for the evaluated labels (i.e. Qualifiers, which is mixed in with Supplements relations for Secondary Definitions). This should be the only case where there is overlap with the evaluated relations in subtask 3 and unevaluated labels from subtask 2.
Sasha
Error! Filename not specified.
Can you take a look at it?
Thank you,
Avram Andrei
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/09c2c76a-ff2f-4e5a-a50d-5efaa39052ba%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/semeval-2020-task-6-all/f9730217-8b0c-48d7-ab89-58289c1583e3%40googlegroups.com.
avram.and...@gmail.com
ValueError: Encountered unknown relation in the following rows [(104, 'Refers-To'), (105, 'Refers-To'), (106, 'Refers-To'), (119, 'Supplements'),
(120, 'Supplements'), (121, 'Supplements'), (122, 'Supplements'), (123, 'Supplements'), (124, 'Supplements'), (125, 'Supplements'), (126, 'Supplements'),
(127, 'Supplements'), (128, 'Supplements'), (129, 'Supplements'), (130, 'Supplements'), (131, 'Supplements'), (132, 'Supplements'), (133, 'Supplements'),
(134, 'Supplements'), (135, 'Supplements'), (136, 'Supplements'), (137, 'Supplements'), (138, 'Supplements'), (139, 'Supplements'), (140, 'Supplements'),
(141, 'Supplements'), (142, 'Supplements'), (606, 'Supplements'), (607, 'Supplements'), (608, 'Supplements'), (609, 'Supplements'), (610, 'Supplements'),
(611, 'Supplements'), (612, 'Supplements'), (613, 'Supplements'), (614, 'Supplements'), (615, 'Supplements'), (616, 'Supplements')]
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/09c2c76a-ff2f-4e5a-a50d-5efaa39052ba%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.
Sasha Spala
Is this in Codalab or are you running locally?
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/09c2c76a-ff2f-4e5a-a50d-5efaa39052ba%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/f9730217-8b0c-48d7-ab89-58289c1583e3%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/semeval-2020-task-6-all/95b07a7a-1fc3-4d1f-af98-94d390833acb%40googlegroups.com.
avram.and...@gmail.com
On Tuesday, 10 March 2020 22:46:07 UTC+2, Sasha Spala wrote:
Is this in Codalab or are you running locally?
From: <semeval-202...@googlegroups.com> on behalf of "avram.an...@gmail.com" <avram.an...@gmail.com>
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/09c2c76a-ff2f-4e5a-a50d-5efaa39052ba%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/f9730217-8b0c-48d7-ab89-58289c1583e3%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.
avram.and...@gmail.com
ValueError: Row length mismatch: Pred 641 Gold 661
I checked the number of lines of my predictions with the latest version of the dev on github and it looks like they match.
Sasha Spala
I’ve just re-uploaded the dev reference data just in case it’s a version behind what’s on Github. Let me know if that fixes things.
Sasha
From: <semeval-202...@googlegroups.com> on behalf of "avram.and...@gmail.com" <avram.and...@gmail.com>
Date: Tuesday, March 10, 2020 at 5:16 PM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Re: Task 3 evaluation+training error
Yeah, I was trying to see what I was doing wrong with my submissions on the training set because I have already submitted 5 times for the evaluation, but with a score of -1...
Now I get the following error:
ValueError: Row length mismatch: Pred 641 Gold 661
I checked the number of lines of my predictions with the latest version of the dev on github and it looks like they match.
On Tuesday, 3 March 2020 19:44:48 UTC+2, avram.an...@gmail.com wrote:
Hello,
When I try to submit for task 3, I get the following error:
Can you take a look at it?
Thank you,
Avram Andrei
--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/semeval-2020-task-6-all/88f4a429-9597-4fa8-9a39-fe10e1f2c7de%40googlegroups.com.
Sasha Spala
Actually, your comment on the Github issue tipped me off that the git repo might be a version behind. Try it with the data I just pushed – these should also contain the fixes we talked about before for the overlapping/missing relationships you brought up.
Best,
Sasha
From: <semeval-202...@googlegroups.com> on behalf of "avram.and...@gmail.com" <avram.and...@gmail.com>
Date: Tuesday, March 10, 2020 at 5:16 PM
To: DeftEval 2020 <semeval-202...@googlegroups.com>
Subject: Re: Task 3 evaluation+training error
Yeah, I was trying to see what I was doing wrong with my submissions on the training set because I have already submitted 5 times for the evaluation, but with a score of -1...
Now I get the following error:
ValueError: Row length mismatch: Pred 641 Gold 661
I checked the number of lines of my predictions with the latest version of the dev on github and it looks like they match.
On Tuesday, 3 March 2020 19:44:48 UTC+2, avram.an...@gmail.com wrote:
Hello,
When I try to submit for task 3, I get the following error:
Can you take a look at it?
Thank you,
Avram Andrei
--
avram.and...@gmail.com
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.
Sasha Spala
On Mar 11, 2020, at 6:26 PM, "avram.and...@gmail.com" <avram.and...@gmail.com> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/961a49c8-837b-4303-a8f1-939cf7bb446f%40googlegroups.com.
avram.and...@gmail.com
On Thursday, 12 March 2020 00:39:37 UTC+2, Sasha Spala wrote:
-1 in the leaderboard you mean? As long as you’re getting the expected results in your results.txt output, that’s what matters.
Sent from my iPhone
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/961a49c8-837b-4303-a8f1-939cf7bb446f%40googlegroups.com.