How many submissions can be done by a team and how frequently?

68 views
Skip to first unread message

Raphaël Troncy

unread,
Jun 22, 2022, 4:35:22 AM6/22/22
to Sem-Tab Challenge
Dear  Jiaoyan, all SemTab organizers,

We are the DAGOBAH team and we intend to improve our current solution with this year challenge. From https://sem-tab-challenge.github.io/2022/, you indicate that "Participants can test and develop their systems on the given ground truth (validation set). They can weekly upload their annotations corresponding to the targets (test set)." but the link behind the "upload" is referring to the same page. How can we actually submit results for evaluation?

Furthermore, can you further specify what you mean by "weekly submissions"? Does this mean that each teams are entitled to a SINGLE submission every week? Are they calendar weeks (from Monday to Sunday) or are they sliding window of 7 days from a given submission? 

In the case of DAGOBAH, we are working on several systems and modules that can be combined, starting from our competitive DAGOBAH-SL solution published last year to embeddings-based modules we are now developing. Knowing how many submissions we can do would help us to prioritize our strategy to test our numerous ideas beyond the tests we can do on the very small "validation" set you have provided.
Best regards.

Raphael Troncy on behalf of the DAGOBAH team

Vasilis Efthymiou

unread,
Jun 28, 2022, 8:15:53 AM6/28/22
to Sem-Tab Challenge
Dear Raphael,

sorry for the late response (not sure if someone else has responded already). We are glad to have the DAGOBAH team participate again.

We are aware of the broken google form link and we are currently working to resolve it. I dare to say it's a matter of hours to be fixed (fingers crossed).

We were thinking a 7-day sliding window. You can submit as many times as you want, but we will only consider the last submission for each weekly evaluation. We will send in more details, when the google form is finalized, along with more instructions on the submission process.

Thanks for your patience.

Jiaoyan Chen

unread,
Jul 1, 2022, 12:59:33 PM7/1/22
to Raphaël Troncy, Sem-Tab Challenge, Vasilis Efthymiou
Dear Raphael and all the participants,

Just to be a bit more clear. Our evaluation will be run on Sunday (UK time) and take the latest submission of that week. We will update the result on Monday. 

If you make a submission late on Sunday, it may be evaluated in the following week. We will record the submission file from which the results are generated and the evaluation time  in the result document. 

Regards,
Jiaoyan

-- 
You received this message because you are subscribed to the Google Groups "Sem-Tab Challenge" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sem-tab-challe...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/sem-tab-challenge/3d012151-6112-435b-bba9-a86aacad276bn%40googlegroups.com.

Reply all
Reply to author
Forward
0 new messages