About failed submissions

[jing ling]

Dear Organizers:

Due to various reasons, our team has failed to submit many times. May I ask if the failed submission consumes the number of times the team can submit?

Best wishes,

Alice

[Jennifer D'Souza]

Dear Alice,

Do you see a count on the number of remaining submissions against your team account?

Also, there are 10 attempts left for each team for tomorrow (Sunday) as well. The current evaluation phase ends on Jan 17th at midnight UTC timezone. So if you were able to make even one successful submission, would it help you use that as a reference for error checking for any subsequent submissions you would like to make? Our motivation in placing the 10 attempts was indeed in some way to create a workaround for submission errors to Codalab.

In case, you haven't been able to make any successful submission to Codalab and so do not have a reference submission, please let us know your team name/username as additional info. And we will try to make it possible for you to have your team submission made. Additionally, we share a reference here https://github.com/ncg-task/evaluation-phase1indicating the expected directory structures etc. for a valid submission. Note your top submission directory should be named "submission." This is not required for our evaluation program but seems necessary for the way Codalab itself is setup.

Best wishes,

Jennifer

[Anna Martin]

Hello organizers and Alice,

I have a similar issue; I have tried submitting to this phase a few times and always get the error "Could not find submission file /tmp/codalab/tmpaj4Q_P/run/input/res/constituency_parsing". Do you think there is an issue with Codalab? Alice I was wondering if this is the same error your team is getting.

Thank you,

Anna

[Official Task]

Dear Anna,

I see that your entry has appeared on the Leaderboard now. So then we assume your issue is resolved. I suppose in case, indeed, Alice was facing a similar problem, it would be very helpful I suppose if you would be willing to share what changed between your earlier failed submissions and the successful one.

Thank you for your submission.

With best wishes,

Jennifer

--
You received this message because you are subscribed to the Google Groups "NLPContributionGraph Shared Task at SemEval 2021" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ncg-task-semeval...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ncg-task-semeval-2021/3556e4d4-219b-4364-9f13-7d0d53642d8dn%40googlegroups.com.

[mjs...@illinois.edu]

Dear organizers,

We are getting the same issue:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. Could not find submission file /tmp/codalab/tmpB3uzyu/run/input/res/constituency_parsing

We named our top directory as 'submission' as instructed.

submission/

└──constituency_parsing/

    ├── 0/

    │   ├── sentences.txt

    │   └── entities.txt

    │   └── triples/

    │   │   └── research-problem.txt

    │   │   └── model.txt  

    │   │   └── ...

    │   └── ... 

    └── ...

Thank you,

Janina

[Official Task]

Dear Janina,

We have received your offline submission that you sent us before the deadline. We will consider it as an exception since we want to acknowledge participants' efforts in building their systems.

You should be able to make successful submissions for the remaining two evaluation phases. The only change needed in your present format is the following: The top-level zipped folder should be called submission. So if your directory structure looks as shown below, the competition site will not accept the submission.

submission.zip

    submission/

    └──constituency_parsing/

        ├── 0/

        │   ├── sentences.txt

        │   └── entities.txt

        │   └── triples/

        │   │   └── research-problem.txt

        │   │   └── model.txt  

        │   │   └── ...

        │   └── ... 

        └── ...

Instead, it should be as shown below.

submission.zip

└──constituency_parsing/

    ├── 0/

    │   ├── sentences.txt

    │   └── entities.txt

    │   └── triples/

    │   │   └── research-problem.txt

    │   │   └── model.txt  

    │   │   └── ...

    │   └── ... 

    └── ...

Wish you good luck in submitting to the leaderboard in the next phases. Let us know if any further concerns.

With best wishes,

Jennifer

To view this discussion on the web visit https://groups.google.com/d/msgid/ncg-task-semeval-2021/fd0b2644-bd2d-4a99-af2e-2fad1d30d37bn%40googlegroups.com.

[Ammaar Ahmad]

Hello, I am trying to submit my output in a post competition link. I have got 2 failed submissions. I have only completed Task A of identifying contribution sentences. So I submit in the following form.

submission.zip

└──constituency_parsing/

    ├── 0/

    │   ├── sentences.txt

    │   └── ... 

    └── ...

Is this a valid submission or I need to create empty files for Subtask B and C