EvoAPPS Session & chair allocation, and order of papers in sessions

[Anna Isabel Esparcia-Alcazar]

Dear all,

I have created some slots and sessions in MyReview and have (randomly) assigned you as chairs.

I'm assuming 36 long talks plus 24 short, as follows:

track	long talks	short talks
EvoBAFIN	1	0
EvoBIO	3	3
EvoCOMNET	3	1
EvoCOMPLEX	2	2
EvoENERGY	3	2
EvoGAMES	4	3
EvoIASP	4	5
EvoINDUSTRY	2	3
EvoKNOW	0	1
EvoNUM	2	0
EvoPAR	1	0
EvoROBOT	4	2
EvoSET	2	1
EvoSTOC	4	1
General	1	0
TOTAL	36	24

Is this correct?

Now, given the following restrictions:

1.- a long talk is 20 min+5min questions, 

2.- a short talk is 10 min, no questions,

3.- all short talks must go before the poster sessions, i.e. on Wednesday 

4.- when there are parallel sessions, the presentations must be synchronised so that people can move between the rooms

5.- The first session on Wed should end early, so that we have plenty of time for our AGM.

6.- The last session on Thurs should end early, if possible, so that we have plenty of time to take the bus to the conference dinner

(restrictions 3&4 are due to numerous complaints we had last year)

I have come up with a slot/session distribution as follows:

Slots	time	session	Room 2	Room 3
Wed slot 1=  2 x130 min = (4long +1 short)x2	1110-1320	1&2	EvoNUM, EvoSET	EvoSTOC
Wed slot 2= 2x110' = 10+10 (20) short talks	1430-1620	3&4	EvoBIO,  EvoCOMNET, EvoENERGY, EvoINDUSTRY, EvoKNOW	EvoGAMES, EvoIASP, EvoROBOT
Wed slot 3= 1x95' = 3 long + 2 short	1640-1815	5	EvoPAR, EvoCOMPLEX
Thurs slot 1 = 2x 100' =  4+4 (8) long talks	0930-1110	6&7	EvoGAMES	EvoIASP
Thurs slot 2 = 1 x110' = 4 long talks	1130-1320	8	EvoBIO, EvoBAFIN
Thurs slot 3 = 1x100' = 4 long talks	1420-1600	9	EvoROBOT
Thurs slot 4 = 1x100' = 3 long talks	1620-1800	10	EvoINDUSTRY, General,
Fri slot 1 = 2x90' = 3+3 (6) long talks	1000-1130	11&12	EvoENERGY	EvoCOMNET

If you're not happy with this arrangement please let me know.

Otherwise, what I am asking you now is (1) choose the order of your papers in the sessions and (2) let us know your candidates for best paper nominations. 

Regarding the order of papers, you need to do as follows:

Go to the Editors menu, then to the "experimental fetch camera-ready version", and then you will see the page you used previously to get the camera ready versions, which happens to also be the one where papers get assigned to sessions. There, if the track has been set to either "Any" or your own track, you will see your papers.

Now, if someone else has set the track menu to anything else, you won't see anything. In that case you need to go to the Editors menu again and choose another option, such as "List of papers", set the track to the right one there, click "apply" and go back to the "Fetch camera ready versions" menu, which will then show your track. Then you can set the order of a paper in a session by assigning  it a number in the white box next to the drop down menu (or modifying it, in case it already has one).

Please DO NOT change the sessions because that will mess up the whole programme. If you have a problem with your session, please let us know.

Regarding Best Paper Awards, previously we have had complaints about the transparence and fairness of the process (because it's next to impossible to schedule all nominations in the same track and therefore the people who vote for one candidate have probably not seen the others). This year we are proposing to organise an open vote among ourselves with the following rules: (a) Each chair gets one vote, (b)   chairs cannot vote for the papers in their own track (although they can voice their opinion), (c) a chair cannot vote for papers that they have authored, (d) everybody commits to reading all nominations and (d) in the event of a tie, the Coordinator (i.e. Kevin) casts the deciding vote. The voting could be done in the AGM (lunch time on Wednesday).

Alternatively, we could make it compulsory for best paper nominees to present a poster and then the audience can vote at the poster session. But this can raise the usual complaint of being a "beauty contest" for posters and, furthermore, we already have a Best Poster prize.

Best,

A*

[Giovanni Iacca]

Hi Anna,

How do you save the order of papers? The "validate conf. session assignment" button is for admins only, not for editors.

Cheers,

Giovanni

 

Regarding the order of papers, you need to do as follows:

Go to the Editors menu, then to the "experimental fetch camera-ready version", and then you will see the page you used previously to get the camera ready versions, which happens to also be the one where papers get assigned to sessions. There, if the track has been set to either "Any" or your own track, you will see your papers.

Now, if someone else has set the track menu to anything else, you won't see anything. In that case you need to go to the Editors menu again and choose another option, such as "List of papers", set the track to the right one there, click "apply" and go back to the "Fetch camera ready versions" menu, which will then show your track. Then you can set the order of a paper in a session by assigning  it a number in the white box next to the drop down menu (or modifying it, in case it already has one).

Please DO NOT change the sessions because that will mess up the whole programme. If you have a problem with your session, please let us know.

--

Giovanni Iacca, Ph.D.

[Jaume Bacardit]

Anna,

About best papers, I guess it all still starts with the nomination of papers by each track. How many papers can we nominate?

Best regards,
Jaume

On 11/02/2018 19:01, Anna Isabel Esparcia-Alcazar wrote:

Regarding Best Paper Awards, previously we have had complaints about the transparence and fairness of the process (because it's next to impossible to schedule all nominations in the same track and therefore the people who vote for one candidate have probably not seen the others). This year we are proposing to organise an open vote among ourselves with the following rules: (a) Each chair gets one vote, (b)   chairs cannot vote for the papers in their own track (although they can voice their opinion), (c) a chair cannot vote for papers that they have authored, (d) everybody commits to reading all nominations and (d) in the event of a tie, the Coordinator (i.e. Kevin) casts the deciding vote. The voting could be done in the AGM (lunch time on Wednesday).

-- 
-------------------------------------------------------------------
Jaume Bacardit, PhD
Reader in Machine Learning
Newcastle University

The Interdisciplinary Computing and Complex BioSystems research group.
Web: http://www.ico2s.org/
Twitter: @ico2s
School of Computing, Newcastle University.
1 Science Square, Science Central, Newcastle upon Tyne, NE4 5TG, UK

Email: jaume _dot_ bacardit _at_ newcastle.ac.uk _dot_ ac _dot_ uk
Web: http://homepages.cs.ncl.ac.uk/jaume.bacardit
Twitter: @jaumebp
-------------------------------------------------------------------- 

[Anna Isabel Esparcia-Alcazar]

We could do this per track: there are 14 APPS, so one nomination each. 

Still, 14 nominations looks like a bit much for 86 submissions, especially considering that some tracks only got 1 or 2 submissions. I would say 6-7 is the maximum in order to make ourselves look "respectable".

In the system I found 11 papers for which at least one reviewer ticked the "best" box.

These are as follows: 

BAFIN (1), BIO (1) COMNET (2) COMPLEX (1) GAMES (2) IASP (2) NUM (1) STOC(1)

If we ask the COMNET, GAMES and IASP chairs to choose between their two candidates, it would give us a list of 8, which is still doable. 

And if we wanted to restrict even further, we could set a limit of, say, 6 submissions in order to get a nomination. In this way, BAFIN and COMPLEX would drop out of the list and we would get 6 nominations.

Any other ideas?

---

From: evo...@googlegroups.com <evo...@googlegroups.com> on behalf of Jaume Bacardit <jaume.b...@newcastle.ac.uk>
Sent: Monday, February 12, 2018 10:01 AM
To: evo...@googlegroups.com
Subject: Re: EvoAPPS Session & chair allocation, and order of papers in sessions

 

--
You received this message because you are subscribed to the Google Groups "evoAPPS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to evoapps+u...@googlegroups.com.
To post to this group, send email to evo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/evoapps/WM!5fd089dddfc1e6bb965a04f879eea0d12b5978e7642c9a084494361f68fef03413c5851d05d1d40899061e42c2ca5bd8!%40mailhub-mx1.ncl.ac.uk.
For more options, visit https://groups.google.com/d/optout.

[Anna Isabel Esparcia-Alcazar]

Hmm, just send me your modifications then :-/

---

From: evo...@googlegroups.com <evo...@googlegroups.com> on behalf of Giovanni Iacca <giovann...@gmail.com>
Sent: Monday, February 12, 2018 7:31 AM
To: EvoApps

Subject: Re: EvoAPPS Session & chair allocation, and order of papers in sessions

--
You received this message because you are subscribed to the Google Groups "evoAPPS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to evoapps+u...@googlegroups.com.
To post to this group, send email to evo...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/evoapps/CABtRnVpGmMjHp-mU-D%2BdMfKxEY1JkBOg0udEBkN-1TBQYG836Q%40mail.gmail.com.

[Anna Isabel Esparcia-Alcazar]

I  forgot to add: another option could be to filter per points.

E.g. we pick those candidates with overall mark >= 5 (7 candidates) or >5 (4 candidates)

---

From: evo...@googlegroups.com <evo...@googlegroups.com> on behalf of Anna Isabel Esparcia-Alcazar <aesp...@hotmail.com>
Sent: Monday, February 12, 2018 10:33 AM
To: Jaume Bacardit; evo...@googlegroups.com

To view this discussion on the web, visit https://groups.google.com/d/msgid/evoapps/DB4PR03MB07338363BC5CE3BAC27E2EFFB0F70%40DB4PR03MB0733.eurprd03.prod.outlook.com.

[Anna Isabel Esparcia-Alcazar]

(I'm including the whole list in the reply because Alberto's question is relevant to you all)

> I have another question: I did not understand the procedure to select the best paper. I mean, the procedure is clear, but how

> do we get the initial list of best paper candidates? Do we just look at the ones who were recommended by the reviewers, 

> does every  track pick one, or...?

Looking at the reviewers' recommendations is my first approach to start the ball rolling. (Another approach: filter by overall score >=5, and I get 9 papers. As it happens, out of the 4 papers with the highest overall scores, only one got a best paper nomination by a reviewer. And another one belongs to the General track, which I had not considered so far :-S )

So you can go farther. Perhaps you have a paper in your track that got high marks in general but was not nominated by the reviewers. Then you could consider nominating it yourselves. 

For instance, ROBOT has 10 submissions but no nominations, and their highest-scoring paper has a 5.5 score; they might want to consider nominating it (but then they would require to re-review it themselves, becasue just now it only has two reviews).

In GAMES you don't have a similar case, as the two highest-scoring papers have already been nominated by reviewers. However, I just realised one of these nominations must be a mistake, because the reviewer who ticked it also ranked the paper a "3 - Probably accept as poster". So that just goes to show you, you cannot even trust the nominations...

Can everybody check their possible candidates, whether or not selected by the reviewers?

A*

---

From: Alberto Tonda <albert...@gmail.com>
Sent: Monday, February 12, 2018 12:17 PM
To: Anna Isabel Esparcia-Alcazar

Subject: Re: EvoAPPS Session & chair allocation, and order of papers in sessions

 

Checked, it's all right.

I have another question: I did not understand the procedure to select the best paper. I mean, the procedure is clear, but how do we get the initial list of best paper candidates? Do we just look at the ones who were recommended by the reviewers, does every track pick one, or...?

Thank you for your time (and patience :-D)

[cag...@ce.unipr.it]

I would make it a three-stage process.

I suggest that we first generate a shortlist based on track chairs'
nominations, with possible minimal requirements (e.g., a nomination as
best paper by reviewers OR a mark above a given threshold) with 1 paper
nominated per track (if above threshold) or possibly 2 only in case of
great uncertainty.

After all, the final award is ONE, so sensibly, if track chairs have a
preference for one, there is no real reason for adding any other paper
to the list.

Then, we should make the shortlist public to all chairs, making a table
(Excel sheet) with at least the marks visible to all.
If also the reviewers' comments were there, the better.

At that point, chairs can vote (possibly in the same way we did for the
invited speakers) and finally the first 3-4 papers will be nominated
officially as best paper candidates, with the last word left to the
attendees (and to the presenters, since the final judgement will be
based also on how well the paper contents have been exposed).

Stefano

PS I would possibly allow track chairs of track xxx to unofficially
appoint their shortlisted paper(s) as "Evoxxx nominees for best paper
award" or something alike, possibly with a note aside the abstract in
the track page, where usually abstracts placed, if I am not wrong.

[Anna Isabel Esparcia-Alcazar]

Dear Stefano,

I see two main problems with your proposal:

- it will take a long time (which we don't have, as we need all the nominations ready for the booklet)

- it does not solve the problem of the bad image we have been giving.

The second is by far the more serious. Having people vote for best papers is doable when they can be scheduled in the same session. It also makes sense if all nominations are more or less in the same area and the audience can be expected to be knowledgeable in that area. But this is not the case here, and it makes us look really bad that candidates are presented in different sessions with different sizes of audiences.

We would also have the classical "beauty contest" problem: it is not the worth of the paper that gets measured (because the audience have not read it) but the quality of the speaker, or worse still, how many friends s/he has. 

I'm not saying that we should not have a Best Presentation award (even better, a "Best session presentation" award), but it should be a different kettle of fish altogether.

Last GECCO, I happened to ask a tricky question to one of the BP nominees in a track. He did not win, even though his paper was probably the best, and I cannot help but think that the "negative vibes" raised by my question influenced the outcome.

> PS I would possibly allow track chairs of track xxx to unofficially

> appoint their shortlisted paper(s) as "Evoxxx nominees for best paper

> award" or something alike, possibly with a note aside the abstract in
> the track page, where usually abstracts placed, if I am not wrong.

As well as the bad image problem (if evoBAFIN has one accepted paper, and that paper happens to be the BP candidate for evoBAFIN, what value does that information have? what does it tell about us?) I can see this as a source of confusion (people will be asking "so, is this a BP candidate or not?")

Sorry to be so negative, but I think a qualified vote by us is by far the best option.

A*

---

From: evo...@googlegroups.com <evo...@googlegroups.com> on behalf of cag...@ce.unipr.it <cag...@ce.unipr.it>
Sent: Monday, February 12, 2018 1:44 PM
To: evo...@googlegroups.com

Subject: Re: EvoAPPS Session & chair allocation, and order of papers in sessions

[cag...@ce.unipr.it]

Dear Anna, I understand your points, with which I mostly agree. I was
just thinking we were giving for granted we would go on as we have
traditionally gone.

However, if a "qualified vote" is necessary, I think that a
shortlist/nomination mechanism, unless we go for a purely 'quantitative
criterion' based on the reviewers' votes, would probably be the fastest
and, I would add, fairest.

Stefano

[Alberto Tonda]

Dear all,

me and Antonio (EvoGAMES) also like the idea of a "quantitative criterion" mentioned by Stefano: we could go just by reviewers' votes. Not only because it's fairer, but because it's easier :-D

--

Alberto Tonda

[Anna Isabel Esparcia-Alcazar]

Here's a spreadsheet to make things easier:

https://docs.google.com/spreadsheets/d/1oLssXJyvbwkF72ULBlw4uE2xnsoPLot_KGwkwYOlIVg/edit#gid=1530206292

it includes papers with marks above 5, with indication of the 3 of them that have a BP nomination by a reviewer, and also of conflicts of interest (4 have been co-authored by chairs)

---

From: evo...@googlegroups.com <evo...@googlegroups.com> on behalf of Alberto Tonda <albert...@gmail.com>
Sent: Monday, February 12, 2018 5:52 PM

To: evo...@googlegroups.com
Subject: Re: EvoAPPS Session & chair allocation, and order of papers in sessions

--
You received this message because you are subscribed to the Google Groups "evoAPPS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to evoapps+u...@googlegroups.com.
To post to this group, send email to evo...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/evoapps/CAAMUxvzeyscdN9n5Jz0zzGvi85czgLqXXXj9yjAFRMYHqLe80A%40mail.gmail.com.

[Nguyen Trung Thanh]

Hi Anna,

Thank you very much for compiling the very useful list. I had already asked Michalis to handle the best paper nomination in EvoSTOC. I cannot do that due to conflict of interest.

Best regards,

Thanh

On 12 February 2018 at 19:34, Anna Isabel Esparcia-Alcazar <aesp...@hotmail.com> wrote:

Here's a spreadsheet to make things easier:

https://docs.google.com/spreadsheets/d/1oLssXJyvbwkF72ULBlw4uE2xnsoPLot_KGwkwYOlIVg/edit#gid=1530206292

it includes papers with marks above 5, with indication of the 3 of them that have a BP nomination by a reviewer, and also of conflicts of interest (4 have been co-authored by chairs)

---

From: evo...@googlegroups.com <evo...@googlegroups.com> on behalf of Alberto Tonda <albert...@gmail.com>
Sent: Monday, February 12, 2018 5:52 PM
To: evo...@googlegroups.com
Subject: Re: EvoAPPS Session & chair allocation, and order of papers in sessions

 

Dear all,

me and Antonio (EvoGAMES) also like the idea of a "quantitative criterion" mentioned by Stefano: we could go just by reviewers' votes. Not only because it's fairer, but because it's easier :-D

On Mon, Feb 12, 2018 at 3:27 PM, <cag...@ce.unipr.it> wrote:

Dear Anna, I understand your points, with which I mostly agree. I was
just thinking we were giving for granted we would go on as we have
traditionally gone.

However, if a "qualified vote" is necessary, I think that a
shortlist/nomination mechanism, unless we go for a purely 'quantitative
criterion' based on the reviewers' votes, would probably be the fastest
and, I would add, fairest.

Stefano

--

Alberto Tonda

--
You received this message because you are subscribed to the Google Groups "evoAPPS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to evoapps+unsubscribe@googlegroups.com.

To post to this group, send email to evo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/evoapps/CAAMUxvzeyscdN9n5Jz0zzGvi85czgLqXXXj9yjAFRMYHqLe80A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "evoAPPS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to evoapps+unsubscribe@googlegroups.com.

To post to this group, send email to evo...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/evoapps/DB4PR03MB0733DF7FAB58E944D3755D0FB0F70%40DB4PR03MB0733.eurprd03.prod.outlook.com.