Task2 Evaluation error

[Mukesh kumar Mehta]

 Yesterday my submission was successful but today I got this error. There is  something wrong with the gold data. The length of Gold data shouldn't be 441 please resolve it ASAP.

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "/tmp/codalab/tmpRpLteW/run/program/semeval2020_06_evaluation_main.py", line 83, in 
    main(cfg)
  File "/tmp/codalab/tmpRpLteW/run/program/semeval2020_06_evaluation_main.py", line 68, in main
    task_2_report = task_2_eval_main(ref_path, res_path, output_dir, cfg['task_2']['eval_labels'])
  File "/tmp/codalab/tmpRpLteW/run/program/evaluation_sub2.py", line 192, in task_2_eval_main
    temp_y_gold, temp_y_pred = get_gold_and_pred_labels(ref_path.joinpath(child.name), child)
  File "/tmp/codalab/tmpRpLteW/run/program/evaluation_sub2.py", line 135, in get_gold_and_pred_labels
    validate_data(gold_rows, pred_rows)
  File "/tmp/codalab/tmpRpLteW/run/program/evaluation_sub2.py", line 111, in validate_data
    validate_length(gold_rows, pred_rows)
  File "/tmp/codalab/tmpRpLteW/run/program/evaluation_sub2.py", line 50, in validate_length
    raise ValueError("Row length mismatch: Pred {} Gold {}".format(len(pred_rows), len(gold_rows)))
ValueError: Row length mismatch: Pred 27 Gold 441

[Sasha Spala]

Hi Mukesh,

Nothing changed on the Github or in Codalab today or yesterday. Perhaps you tried to submit through the wrong tab? Codalab does default to a different tab every time you refresh the page so I would check to make sure you’re submitting to the correct phase.

Sasha

Sent from my iPhone

On Feb 23, 2020, at 6:06 AM, Mukesh kumar Mehta <mukesh...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "DeftEval 2020" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/c3525c0d-8c86-407b-9382-933bf3ab92fe%40googlegroups.com.

[Mukesh kumar Mehta]

Hi Sasha,

I have faced another error in task2.

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
/tmp/codalab/tmpTTuEaT/run/program/evaluation_sub2.py:79: UserWarning: Labels exist in the reference files that will not be scored given the current evaluation labels.
  warnings.warn("Labels exist in the reference files that will not be scored given the current evaluation labels.")
Traceback (most recent call last):
  File "/tmp/codalab/tmpTTuEaT/run/program/semeval2020_06_evaluation_main.py", line 83, in 
    main(cfg)
  File "/tmp/codalab/tmpTTuEaT/run/program/semeval2020_06_evaluation_main.py", line 68, in main
    task_2_report = task_2_eval_main(ref_path, res_path, output_dir, cfg['task_2']['eval_labels'])
  File "/tmp/codalab/tmpTTuEaT/run/program/evaluation_sub2.py", line 212, in task_2_eval_main
    report = evaluate(y_gold, y_pred, eval_labels)
  File "/tmp/codalab/tmpTTuEaT/run/program/evaluation_sub2.py", line 151, in evaluate
    validate_res_labels(eval_labels, y_pred)
  File "/tmp/codalab/tmpTTuEaT/run/program/evaluation_sub2.py", line 86, in validate_res_labels
    raise ValueError(f"Encountered unknown or unevaluated label: {label}")
ValueError: Encountered unknown or unevaluated label: B-Secondary-Definition

Sasha

To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task-6-all+unsub...@googlegroups.com.

[Sasha Spala]

Looks like you’re trying to predict a secondary definition, but Codalab (or your config file if you’re running locally) is not evaluating on secondary definitions.

Sasha

Sent from my iPhone

On Feb 23, 2020, at 11:04 AM, Mukesh kumar Mehta <mukesh...@gmail.com> wrote:



To unsubscribe from this group and stop receiving emails from it, send an email to semeval-2020-task...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semeval-2020-task-6-all/851ba5a9-e937-4cf1-a14c-14f49d8fe041%40googlegroups.com.