Semeval task 1 - Codalab URGENT action requested

[Eneko Agirre]

Dear participants,

thanks for your patience!

We now fixed all issues with ill-formed runs, and will soon be able to
publish the official results. Unfortunately, CODALAB is very rigid, so
we need as soon as possible your collaboration:

- please login to Codalab

- make sure that your team name is the one you used for registration.
This is the team name that will be used in your paper title.

- in case you had several Codalab users with the same team name, please
leave only one and delete the rest.

- select all your runs as being "public"

- choose the run with the highest primary score for the leaderboard

This way, two result tables will be shown in the Codalab website: the
"results" leaderboard with the best primary run per participant, and the
"public submissions" one, with all runs.

Let us know if you have any issues (unless you already sent an e-mail,
no need to resend again).

Those participants with post-competition fixes, we ask for your
patience, we will incorporate your runs as soon as the official results
are settled. They will be shown on a different leaderboard, called
"Submission correction". We will contact you in due time.

Best

the organizers

[Basma Hassan]

Hi Eneko,

I signed in and made my 3 submissions as public and is able to see them in "Public Submissions" results tab. But i wonder how can i choose the run with the highest primary score for the leaderboard ? 

Codalab is choosing the third run by default and i didn't find any option that allow me to change this selection to the highest score one.

Thanks alot for your great efforts.

Regards,

----
Basma Hassan Kamal,
Assistant Lecturer,
Computer Science Department,
Faculty of Computers and Information,
Fayoum University, Egypt

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com
To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com
For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
--- You received this message because you are subscribed to the Google Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[AL-NATSHEH Hussein]

Dear organizers,

I can see in the public submission results table that some runs was submitted in a February time-stamp (7 examples could be found from 3 users) How could that be possible? Are the pending correction submissions would be treated the same?

After many tries to re-submit our corrected run using the CodaLab submission form, I directly contacted the organizers describing the submission mistake we did and asking how to submit the correction using CodaLab. Since I understood that there was a technical issue with CodaLab, I finally attached the correction to another follow-up email. So I am wondering how did the 3 users manage to have their submissions on a February time-stamp while we did not.

P.S. I am not able to "- choose the run with the highest primary score for the leaderboard" using the CodaLab available options. It seems that it is automatically the 3rd run and I am not able to change that. Please advice. Thanks a lot.

Best regards,

Hussein AL-NATSHEH
CodaLab username: natsheh
Team name: UdL

________________________________________
From: sts-s...@googlegroups.com [sts-s...@googlegroups.com] on behalf of Eneko Agirre [e.ag...@ehu.eus]
Sent: 07 February 2017 08:19
To: ilopez077
Subject: [STS] Semeval task 1 - Codalab URGENT action requested

Dear participants,

thanks for your patience!

Best

the organizers

--

--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

[Eneko Agirre]

Dear Hussein,

in order to correctly evaluate ill-formed runs, we had to run again the
evaluation script, which changed the timestamp.

That is the cause of the February timestamps. Note that we have not
accepted late runs/corrections yet, and no one was allowed to submit new
runs for the official table.

As we have just mentioned, the corrected/late runs will be dealt with soon.

best

eneko

PD. An email with instructions to choose runs for the leaderboard will
follow.


02/07/2017 11:55 AM(e)an, AL-NATSHEH Hussein igorleak idatzi zuen:

--

Eneko Agirre
Euskal Herriko Unibertsitatea
Universidad del Pais Vasco
University of the Basque Country
http://ixa2.si.ehu.eus/eneko

[Ergun Bicici]

That's not that surprising since the competition's current status is correction:

Submission correction

Feb. 6, 2017, 6 p.m. UTC

Best Regards,
Ergun

Ergun Biçici

http://bicici.github.com/

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

[Iñigo Lopez Gazpio]

Dear participants,

the process to choose which submission to show in the leaderboard is the following

1.- Go to competition homepage and click "participate"
2.- Select the phase you wish to make changes, "Official evaluation"



As only one submission can be shown in the leaderboard, just one submission will be checked, as shown in the picture.

To change it do as follows:

3.- Click the "+" symbol
4.- Click either "Remove from leaderboard" or "submit to leaderboard"





P.S. You can also check results of the submission under "View scoring output log"

[Basma Hassan]

Hi Inigo,
Thanks for sending the instruction to submit a run on leaderboard.

Unfortunately the option "Submit to Leaderboard" and "Remove from LeaderBoard" are not shown at my side. So what is wrong please?  

----
Basma Hassan Kamal,
Assistant Lecturer,
Computer Science Department,
Faculty of Computers and Information,
Fayoum University, Egypt

[l.gazpio]

Hi,

could you please try some other browser, and if the problem persists
send me an email

thanks

On 02/07/2017 02:48 PM, Basma Hassan wrote:
> Hi Inigo,
> Thanks for sending the instruction to submit a run on leaderboard.
>
> Unfortunately the option "Submit to Leaderboard" and "Remove from
> LeaderBoard" are not shown at my side. So what is wrong please?
>

> Inline images 1
>
> ----
> Basma Hassan Kamal,
> Assistant Lecturer,
> Computer Science Department,
> Faculty of Computers and Information,
> Fayoum University, Egypt
>
> On 7 February 2017 at 15:41, Iñigo Lopez Gazpio <l.ga...@gmail.com

> <mailto:l.ga...@gmail.com>> wrote:
>
> Dear participants,
>
> the process to choose which submission to show in the leaderboard is
> the following
>
> 1.- Go to competition homepage and click "participate"
> 2.- Select the phase you wish to make changes, "Official evaluation"
>
>
>

> As only one submission can be shown in the leaderboard, just one
> submission will be checked, as shown in the picture.
>
> To change it do as follows:
>
> 3.- Click the "+" symbol
> 4.- Click either "Remove from leaderboard" or "submit to leaderboard"
>
>
>
>
>

> <mailto:sts-s...@googlegroups.com>

> To unsubscribe, send email to

> sts-semeval...@googlegroups.com
> <mailto:sts-semeval%2Bunsu...@googlegroups.com>

> For more options,
> http://groups.google.com/group/sts-semeval?hl=en?hl=en
> <http://groups.google.com/group/sts-semeval?hl=en?hl=en>
> ---
> You received this message because you are subscribed to the Google
> Groups "STS SemEval" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to sts-semeval...@googlegroups.com
> <mailto:sts-semeval...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
> --
> Website of task, http://alt.qcri.org/semeval2017/task1/
> To post to this group, send email to sts-s...@googlegroups.com

> To unsubscribe, send email to sts-semeval...@googlegroups.com

> For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
> ---
> You received this message because you are subscribed to the Google
> Groups "STS SemEval" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to sts-semeval...@googlegroups.com
> <mailto:sts-semeval...@googlegroups.com>.

[Basma Hassan]

Hi,

I tried Google Chrome, Mozilla Firefox, and Microsoft edge browsers and all are the same.

Just a small hint that may help:
The screen shot you sent is related to "Submission Correction" Phase, which is the current active phase on the competition.However, the "Official Evaluation" phase has ended (as shown in screenshot bellow that it is closed). So, may be this is the difference and the reason why we are not apply to make changes in "official evaluation" phase submissions? This is just a guessing from me.


​

----
Basma Hassan Kamal,
Assistant Lecturer,
Computer Science Department,
Faculty of Computers and Information,
Fayoum University, Egypt

    <mailto:sts-semeval@googlegroups.com>

    To unsubscribe, send email to

    sts-semeval+unsubscribe@googlegroups.com
    <mailto:sts-semeval%2Bunsubscri...@googlegroups.com>

    For more options,
    http://groups.google.com/group/sts-semeval?hl=en?hl=en
    <http://groups.google.com/group/sts-semeval?hl=en?hl=en>
    ---
    You received this message because you are subscribed to the Google
    Groups "STS SemEval" group.
    To unsubscribe from this group and stop receiving emails from it,

    send an email to sts-semeval+unsubscribe@googlegroups.com
    <mailto:sts-semeval+unsubscribe...@googlegroups.com>.

    For more options, visit https://groups.google.com/d/optout
    <https://groups.google.com/d/optout>.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
---
You received this message because you are subscribed to the Google
Groups "STS SemEval" group.
To unsubscribe from this group and stop receiving emails from it, send

an email to sts-semeval+unsubscribe@googlegroups.com
<mailto:sts-semeval+unsubscribe...@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/
To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval+unsubscribe@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en
--- You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval+unsubscribe@googlegroups.com.

[l.gazpio]

Thanks for reporting this issue, we'll work on it and let you know.

Additionally, we would like to know if someone else is also suffering
the very same issue. If positive, please reply to this thread.

Thanks again for your time

Iñigo

On 02/07/2017 03:09 PM, Basma Hassan wrote:
> Hi,
>
> I tried Google Chrome, Mozilla Firefox, and Microsoft edge browsers and
> all are the same.
>
> Just a small hint that may help:
> The screen shot you sent is related to "Submission Correction" Phase,
> which is the current active phase on the competition.However, the
> "Official Evaluation" phase has ended (as shown in screenshot bellow
> that it is closed). So, may be this is the difference and the reason why
> we are not apply to make changes in "official evaluation" phase
> submissions? This is just a guessing from me.
>
>
> ​
>

> ----
> Basma Hassan Kamal,
> Assistant Lecturer,
> Computer Science Department,
> Faculty of Computers and Information,
> Fayoum University, Egypt
>
> On 7 February 2017 at 15:58, l.gazpio <l.ga...@gmail.com

> <mailto:l.ga...@gmail.com>> wrote:
>
> Hi,
>
> could you please try some other browser, and if the problem persists
> send me an email
>
> thanks
>
> On 02/07/2017 02:48 PM, Basma Hassan wrote:
>
> Hi Inigo,
> Thanks for sending the instruction to submit a run on leaderboard.
>
> Unfortunately the option "Submit to Leaderboard" and "Remove from
> LeaderBoard" are not shown at my side. So what is wrong please?
>
> Inline images 1
>
> ----
> Basma Hassan Kamal,
> Assistant Lecturer,
> Computer Science Department,
> Faculty of Computers and Information,
> Fayoum University, Egypt
>
> On 7 February 2017 at 15:41, Iñigo Lopez Gazpio
> <l.ga...@gmail.com <mailto:l.ga...@gmail.com>

> sts-s...@googlegroups.com <mailto:sts-s...@googlegroups.com>
> <mailto:sts-s...@googlegroups.com
> <mailto:sts-s...@googlegroups.com>>

> To unsubscribe, send email to

> sts-semeval...@googlegroups.com
> <mailto:sts-semeval%2Bunsu...@googlegroups.com>
> <mailto:sts-semeval%2Bunsu...@googlegroups.com
> <mailto:sts-semeval%252Buns...@googlegroups.com>>

> For more options,
> http://groups.google.com/group/sts-semeval?hl=en?hl=en
> <http://groups.google.com/group/sts-semeval?hl=en?hl=en>
> <http://groups.google.com/group/sts-semeval?hl=en?hl=en
> <http://groups.google.com/group/sts-semeval?hl=en?hl=en>>
> ---
> You received this message because you are subscribed to the
> Google
> Groups "STS SemEval" group.
> To unsubscribe from this group and stop receiving emails
> from it,

> send an email to sts-semeval...@googlegroups.com
> <mailto:sts-semeval%2Bunsu...@googlegroups.com>
> <mailto:sts-semeval...@googlegroups.com
> <mailto:sts-semeval%2Bunsu...@googlegroups.com>>.

> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
> --
> --
> Website of task, http://alt.qcri.org/semeval2017/task1/

> <http://alt.qcri.org/semeval2017/task1/>
> To post to this group, send email to

> sts-s...@googlegroups.com <mailto:sts-s...@googlegroups.com>

> To unsubscribe, send email to

> sts-semeval...@googlegroups.com
> <mailto:sts-semeval%2Bunsu...@googlegroups.com>

> For more options,
> http://groups.google.com/group/sts-semeval?hl=en?hl=en
> <http://groups.google.com/group/sts-semeval?hl=en?hl=en>
> ---
> You received this message because you are subscribed to the Google
> Groups "STS SemEval" group.
> To unsubscribe from this group and stop receiving emails from
> it, send

> an email to sts-semeval...@googlegroups.com
> <mailto:sts-semeval%2Bunsu...@googlegroups.com>
> <mailto:sts-semeval...@googlegroups.com
> <mailto:sts-semeval%2Bunsu...@googlegroups.com>>.

> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> --
> Website of task, http://alt.qcri.org/semeval2017/task1/

> <http://alt.qcri.org/semeval2017/task1/>
> To post to this group, send email to sts-s...@googlegroups.com

> <mailto:sts-s...@googlegroups.com>

> To unsubscribe, send email to

> sts-semeval...@googlegroups.com
> <mailto:sts-semeval%2Bunsu...@googlegroups.com>

> For more options,
> http://groups.google.com/group/sts-semeval?hl=en?hl=en
> <http://groups.google.com/group/sts-semeval?hl=en?hl=en>
> --- You received this message because you are subscribed to the
> Google Groups "STS SemEval" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to sts-semeval...@googlegroups.com
> <mailto:sts-semeval%2Bunsu...@googlegroups.com>.

[Cyril Goutte]

Same issue here. In addition "Only show results currently on leaderboard” tickbox is not working (or not doing what is expected!), and all “Submission Correction" panes are empty.

Cyril.

> To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Amr Rekaby]

Same issue here, exactly like what Basma Hassan sent.

[I-Ta Lee]

same issue here.

>         <mailto:sts-semeval%252Bunsubscribe@googlegroups.com>>

>              For more options,
>         http://groups.google.com/group/sts-semeval?hl=en?hl=en
>         <http://groups.google.com/group/sts-semeval?hl=en?hl=en>
>              <http://groups.google.com/group/sts-semeval?hl=en?hl=en
>         <http://groups.google.com/group/sts-semeval?hl=en?hl=en>>
>              ---
>              You received this message because you are subscribed to the
>         Google
>              Groups "STS SemEval" group.
>              To unsubscribe from this group and stop receiving emails
>         from it,
>              send an email to sts-semeval...@googlegroups.com
>         <mailto:sts-semeval%2Bunsu...@googlegroups.com>

>              <mailto:sts-semeval+unsub...@googlegroups.com

>         <mailto:sts-semeval%2Bunsu...@googlegroups.com>>.
>              For more options, visit https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>
>              <https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>>.
>
>
>         --
>         --
>         Website of task, http://alt.qcri.org/semeval2017/task1/
>         <http://alt.qcri.org/semeval2017/task1/>
>         To post to this group, send email to
>         sts-s...@googlegroups.com <mailto:sts-s...@googlegroups.com>
>         To unsubscribe, send email to
>         sts-semeval...@googlegroups.com
>         <mailto:sts-semeval%2Bunsu...@googlegroups.com>
>         For more options,
>         http://groups.google.com/group/sts-semeval?hl=en?hl=en
>         <http://groups.google.com/group/sts-semeval?hl=en?hl=en>
>         ---
>         You received this message because you are subscribed to the Google
>         Groups "STS SemEval" group.
>         To unsubscribe from this group and stop receiving emails from
>         it, send
>         an email to sts-semeval...@googlegroups.com
>         <mailto:sts-semeval%2Bunsu...@googlegroups.com>

>         <mailto:sts-semeval+unsub...@googlegroups.com

[eneko]

Dear participants,

In view of Codalab's limitations, we will need to drop the leaderboard, sorry for the hassle.

We will be using the "public submissions" tab, which unfortunately cannot be sorted. We will thus manually copy the "public submissions" data into a google drive spreadsheet periodically. We will do the copying until all runs are made public (BTW, Codalab only shows user names, so we will manually copy the team names there in due time).

Please check the current version: https://docs.google.com/spreadsheets/d/1a5ZNg5IqKnBLaNKHMyVvmAn2_xc79FqwbXLL5z4ABAk/edit#gid=0

For those of you who did not yet made public your runs, and/or selected the appropriate team name, we repeat the instructions here (removing the leaderboard option):

- please login to Codalab

- make sure that your team name is the one you used for registration.
This is the team name that will be used in your paper title.

- in case you had several Codalab users with the same team name, please
leave only one and delete the rest.

- select all your runs as being "public"

Let us know if you have any issues (unless you already sent an e-mail,
no need to resend again).

Those participants with post-competition fixes, we ask for your
patience, we will incorporate your runs as soon as the official results
are settled. They will be shown on a different leaderboard, called
"Submission correction". We will contact you in due time.

Best

the organizers

ar., 2017.eko otsren 07a 16:37(e)an, Amr Rekaby igorleak idatzi zuen:

--
--
Website of task, http://alt.qcri.org/semeval2017/task1/

To post to this group, send email to sts-s...@googlegroups.com

To unsubscribe, send email to sts-semeval...@googlegroups.com

For more options, http://groups.google.com/group/sts-semeval?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "STS SemEval" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 

Eneko Agirre
Euskal Herriko Unibertsitatea
University of the Basque Country
http://ixa2.si.ehu.eus/eneko 

[peter....@gmail.com]

Yes, same here.

>         <mailto:sts-semeval%252Bunsubscribe@googlegroups.com>>

>              For more options,
>         http://groups.google.com/group/sts-semeval?hl=en?hl=en
>         <http://groups.google.com/group/sts-semeval?hl=en?hl=en>
>              <http://groups.google.com/group/sts-semeval?hl=en?hl=en
>         <http://groups.google.com/group/sts-semeval?hl=en?hl=en>>
>              ---
>              You received this message because you are subscribed to the
>         Google
>              Groups "STS SemEval" group.
>              To unsubscribe from this group and stop receiving emails
>         from it,
>              send an email to sts-semeval...@googlegroups.com
>         <mailto:sts-semeval%2Bunsu...@googlegroups.com>

>              <mailto:sts-semeval+unsub...@googlegroups.com

>         <mailto:sts-semeval%2Bunsu...@googlegroups.com>>.
>              For more options, visit https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>
>              <https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>>.
>
>
>         --
>         --
>         Website of task, http://alt.qcri.org/semeval2017/task1/
>         <http://alt.qcri.org/semeval2017/task1/>
>         To post to this group, send email to
>         sts-s...@googlegroups.com <mailto:sts-s...@googlegroups.com>
>         To unsubscribe, send email to
>         sts-semeval...@googlegroups.com
>         <mailto:sts-semeval%2Bunsu...@googlegroups.com>
>         For more options,
>         http://groups.google.com/group/sts-semeval?hl=en?hl=en
>         <http://groups.google.com/group/sts-semeval?hl=en?hl=en>
>         ---
>         You received this message because you are subscribed to the Google
>         Groups "STS SemEval" group.
>         To unsubscribe from this group and stop receiving emails from
>         it, send
>         an email to sts-semeval...@googlegroups.com
>         <mailto:sts-semeval%2Bunsu...@googlegroups.com>

>         <mailto:sts-semeval+unsub...@googlegroups.com

[Eneko]

hi peter, martyna,

as mentioned in a previous mail we found no way for codalab to allow for selecting runs. please see the email which I copy here for your convenience:

----

Dear participants,

In view of Codalab's limitations, we will need to drop the leaderboard, sorry for the hassle.

We will be using the "public submissions" tab, which unfortunately cannot be sorted. We will thus manually copy the "public submissions" data into a google drive spreadsheet periodically. We will do the copying until all runs are made public (BTW, Codalab only shows user names, so we will manually copy the team names there in due time).

Please check the current version: https://docs.google.com/spreadsheets/d/1a5ZNg5IqKnBLaNKHMyVvmAn2_xc79FqwbXLL5z4ABAk/edit#gid=0

For those of you who did not yet made public your runs, and/or selected the appropriate team name, we repeat the instructions here (removing the leaderboard option):

- please login to Codalab

- make sure that your team name is the one you used for registration.
This is the team name that will be used in your paper title.

- in case you had several Codalab users with the same team name, please
leave only one and delete the rest.

- select all your runs as being "public"

[AL-NATSHEH Hussein]

Hello,

Same here, I am still not able to choose which run to be in the leader board. However, I think the organizers said that they will use a Google Drive Spreadsheet instead in order to handle that. No?

Regards,

Hussein AL-NATSHEH

CodaLab usernale: natsheh
Team: UdL

---

From: sts-s...@googlegroups.com [sts-s...@googlegroups.com] on behalf of peter....@gmail.com [peter....@gmail.com]
Sent: 09 February 2017 16:26
To: STS SemEval
Cc: basm...@gmail.com
Subject: Re: [STS] Re: Semeval task 1 - Codalab URGENT action requested

To unsubscribe from this group and stop receiving emails from it, send an email to sts-semeval...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eneko]

Hi Hussein, please see my previous post

best

eneko