Fw: Best papers zip

[Anna Isabel Esparcia-Alcazar]

Dear all,

Paul has kindly prepared this zip file containing the four nominees for BPA.

-Late Acceptance Hill Climbing for Constrained Covering Arrays, Mosab Bazargani, John H. Drake, Edmund K. Burke
-Evolving a Repertoire of Controllers for a Multi-Function Swarm, Sondre A. Engebraten, Jonas Moen, Oleg Yakimenko, Kyrre Glette
-A Multi-Objective Time-Linkage Approach for Dynamic Optimization Problems with Previous-Solution Displacement Restriction, Danial Yazdani, Trung Thanh Nguyen, Juergen Branke, Jin Wang
-A fast metaheuristic for the design of DVB-T2 networks, Fabio D'Andreagiovanni, Antonella Nardin

Please read them all so that we (with the exception of Fabio, Kyrre and Thanh) can vote on the EvoAPPS AGM in Parma.

You have 3 weeks!

Best,

A*

[Anna Isabel Esparcia-Alcazar]

[Anna Isabel Esparcia-Alcazar]

In just over a week we will be discussing this so please remember to READ THE PAPERS!

...and enjoy your holidays, should you have any :-)

a*

---

From: Anna Isabel Esparcia-Alcazar <aesp...@hotmail.com>
Sent: Tuesday, March 13, 2018 2:31 PM
To: evo...@googlegroups.com
Subject: Fw: Best papers zip

 

[Anna Isabel Esparcia-Alcazar]

Dear all,

Below please find the reviews for the 4 BP nominees, in case they help you with your decision.

See you for dinner! 

A*

Reviews for paper #75 "Late Acceptance Hill Climbing for Constrained Covering Arrays"

Reviewer: 1

   Originality : Good

   Quality : Good

   Relevance : Very Good

   Presentation : Good

Summary: The authors use late acceptance hill climbing in place of simulated annealing in the CASA system, to create a variant they call CALA. These systems solve the combinatorial interaction testing problem. They evaluate on a small number of example problems and report the results.

Details: * The paper is very well written, and I enjoyed reading it.

* The use of LAHC reduces the number of hyper-parameters for consideration, which is of great benefit to an end user. The LAHC algorithm is an elegant and interesting concept, so I am excited to see its successful application here.

* CASA is a suitable choice for comparison, being a well-known tool that reasonably represents state-of-the-art in CIT.

* The use of the acronym "ISIL" is non-standard in the literature and should be removed, especially given its unfortunate connotations. It is not necessary to introduce new nomenclature for such a simple concept as "number of iterations".

* the experimental setup is principled and well-justified.

* Bolding the best results in Table 1 would make it more readable.

* The results in Figure 2 don't seem to statistically significant. It may be fair to say "there is little difference" but not to make any stronger claims.

* The discussion of the results does not give a fair comparison, because the amount of computational effort used by different algorithms was unequal. Plotting a pareto front of CE vs results would be illuminating (Figure 3 shows the number of iterations, not the number of function evaluations)

Lesser Comments

* "software development is shifting from producing an entire application from scratch to produing components of related products" - this is an extremely general claim, and there are numerous conter-examples of monolithic products. It needs to be qualified or backed up with rigorous evidence.

* "from the core set of a software functionality"

* "for constrained CIT" ... "the constrained CIT"

* "it uses SA tries to improve"

* "instancesApache"

Candidate for the best paper award? : No

Does the paper include up to five keywords? : Yes

=====================================

Reviewer: 2

   Originality : Good

   Quality : Very Good

   Relevance : Very Good

   Presentation : Very Good

Summary: The Late Acceptance Hill-Climbing (LAHC) algorithm is a one-point search metaheuristic with a single parameter. It is a simpler algorithm than Simulated Annealing, and does not require complex parameter setting. The authors have studied the application of LAHC to the generation of covering arrays for Combinatorial Interaction Testing (CIT). The authors replace the use of Simulated Annealing with LAHC in the CASA framework. Their experimental evaluation indicates that LAHC yields similar or better results to SA in 34 of the 35 studied benchmarks.

Details: Simulated Annealing is a common approach to solving CIT problems, and evaluations using other heuristic search methods are relatively rare - despite the potential for improvements in the results. Given the increased prevalence of software product lines, improvements to CIT solving techniques are potentially quite important. As this is the first approach to apply the LAHC algorithm to the CIT problem, it is both relatively novel and potentially important in kickstarting further research developments. (actually, this paper is my first introduction to the LAHC technique, and I found it quite interesting - I could see plenty of applications for a lean, memory-light technique like this)

This paper was well-written and easy to follow. The background serves as a good introduction to the topic, and appropriate related work is cited. The work seems sound, and the evaluation choices are well-justified. I can easily recommend this for acceptance.

I do have a few questions and comments I would like to see addressed in the final paper:

- The history length appears to have a significant impact. How was an initial value of “32” chosen? The authors state that small values are not desirable, but why 32?

- Suboptimal moves can be chosen, which will help prevent the search from becoming stuck, but does the history mechanism eliminate the issue? Can LAHC still get stuck?

- I would recommend that the authors make their alterations to CASA available through an open-source repository, and that they add a link to the paper. This would allow other researchers to use and extend their work.

Minor comments:

==============

  - Page 6-7 transition: “chance revisiting those ranges” -> “chance of revisiting those ranges”

Candidate for the best paper award? : No

Does the paper include up to five keywords? : Yes

=====================================

Reviewer: 3

   Originality : Good

   Quality : Good

   Relevance : Good

   Presentation : Good

Summary: This paper describes an approach and experiment of replacing simulated annealing with the late acceptance hill climbing search strategy in the CASES tool, which generates combinatorial interaction test suites. In 14 out of 35 cases tested in the experiments, LAHC was better than SA, while there was only one case where SA was better.

Details: I did not know the Late Acceptance Hill Climbing algorithm, and I learned something new in the paper. The application context of combinatorial interaction testing is suitable, replacing SA with this heuristic seems like a useful idea for evaluation, and the experiments are largely done well. Overall, this is a nice paper.

The paper repeatedly stresses that CASA was optimised for integrating SA. It is not clear to me how. The search algorithm should be like a black box, replacing SA with any other algorithm should be straight forward, why would SA have some kind of advantage? If the argument is that the experimental results are somehow influenced by this (and that LAHC had some kind of disadvantage) then the paper should be clearer as to what exactly is the advantage of SA in CASA. Or alternatively, some of the many repetitions of this claim should be removed.

Related to this, I wonder if replacing one local search algorithm with another justifies calling this a "new framework". I have no problem with replacing one heuristic with another and running some experiments, I just wonder at what point there really is a "new framework", and at which point the credit should remain with the original authors of CASA (in particular since SA is only a small part of CASA, and just replacing a small part doesn't make a new framework!) Maybe there is something non-trivial about switching the heuristics - probably related to my previous point. Again, this could be clarified.

The result presentation needs some improvement. I don't understand the reasoning for picking the best out of 8 different configurations. I rather think that for each configuration the algorithms should be compared in a fair way. There is no problem if CALA isn't always best, since there is a good explanation why CALA keeps improving given more time (Figure 3). Also, each comparison should be done using a statistical test. Table 1 could also use some highlighting, e.g. by making the best result in each row bold.

Candidate for the best paper award? : No

Does the paper include up to five keywords? : Yes

---

Reviews for paper #80 "Evolving a Repertoire of Controllers for a Multi-Function Swarm"

Reviewer: 1

   Originality : Very Good

   Quality : Good

   Relevance : Very Good

   Presentation : Very Good

Summary: In the paper "Evolving a repertoire of controllers for a multi-function swarm” the authors present a study of MAP-Elites for the domain of swarm robotics. The MAP-Elites bin is filled following two dimensions: an exploration task and a "dispersion-but-stay-together" task.

Details: Overall the paper is well written, easy to follow, and the paper addresses a highly relevant approach of applying MAP-Elites to swarms. I think, the paper should be published because it is a valuable contribution to evolutionary swarm robotics. I have only one main concern: the paper does not contain a comparison to an alternative approach, so it is unclear whether the application of MAP-Elites constitutes an improvement to the state of the art. Alternatively, the authors could have tested an approach similar to the popular MAP-Elites application of using the MAP-Elites bin at runtime. For example, the required task could be switched or even continuously changed over time by a user and the robots would need to switch controllers online.

A few detailed comments:

The abstract mentions “perimeter surveillance and network creation” which are not well-known / not well-defined tasks in swarm robotics. Better you refer (as later in the paper) to exploration and dispersion with keeping sensor connections (or similar).

Your approach of “simple weighting of input components or forces” is basically the motor schema approach by Ronald Arkin, you should refer to his framework (also cf. behavior-based robotics).

p. 3 “a numer of application. most…”: change to “applications”

p. 4 P-controller: Please be more precise and name the process variable and the set point to make the discussion more compatible to the terminology of feedback control.

p. 4, “4. Direction to the least visited square surrounding the agent”: Please be more specific: is it a Moore or von Neumann neighborhood?

p. 8, “fixed range of 200m”: This is the first time in the paper (or did I miss something?) that an actual distance in meters is given. Please describe in a way that the reader has a chance to understand.

p. 8, I don’t understand why you define your fitness function relative to the controller type - or at least that’s how you communicate it. There should be an effective vector that is independent of the controller type. What vector affects the robot?

p. 9: I’m not sure whether you specify your 2 populations: individuals in your parallel MAP-Elites and the population of simulated robots.  Please be more precise here.

p. 10, “random seed”: What is randomized in your simulation besides the robot’s initial position/heading?

Ref 1: check capital letters ({UAV}s...)

Candidate for the best paper award? : No

Does the paper include up to five keywords? : No

Comments for PC

very good paper, self-contained and nice for EvoROBOT, comparison to state of the art would have been nice but maybe not possible in a conference paper

=====================================

Reviewer: 2

   Originality : Good

   Quality : Good

   Relevance : Good

   Presentation : Good

Summary: An interesting paper handling the problem of how to develop a swarm's microscopic rules towards a given macroscopic goal.

Details: An interesting paper handling the problem of how to develop a swarm's microscopic rules towards a given macroscopic goal. As this is not the first approach using potential fields for swarm algorithms a bit more literature work should be done. For example in the I-SWARM project many such algorithms were published on I-SWARM and JASMINE robots in real hardware and in simulation.

The paper should also clarify in more detail the issue of how much self-localization the robots need. Do they have to know their own global coordinates? Do they have to know their global headings? do they have to be able to assess their relative headings and distances to each other? If this is achieved by IR sensors, what "cones" do these sensors and light emitters cover, as this gives the maximum angular resolution of these relative data.

Other issues to improve:

- Throughout the text it should be "Equation" and not "Formula". In the U.S. a "formula" is a drug you take, for example for sport performance improvement.

- Many figures require color code legends of the shades of blue that are used in the figures.

- The tasks "Exploration", "Repulsion" and "Clustering" should be described in detail in the text or the figure legend before Fig.1 refers to them.

- Throughout the article: Units/dimensions should be given for all variables and constant parameters to enable unit check.

- Equation 4 seems to be corrupted, a closing parenthesis is missing at least, but maybe more is missing?

- Fig.2: increase font size of numbers at axis and give all axis a label

- Equation 6: This equation is a bit unclear, is "Fitness" a function of time? On the right-hand-side of the equation there seem to be all constants (as "b" is not indicated as being a function of time), so how can fitness differ between runs and over time?

- Fig. 5,7,9: Make axes texts better readable (bigger).

- Conclusions seem to be drawn without any statistical tests (e.g., t-test, Mann-Whitney-test, Chi-square test, correlation tests, ...) to support those conclusions based on the reported data.

- It should be shown how fitness evolves over time during the evolutionary runs.

Candidate for the best paper award? : No

Does the paper include up to five keywords? : Yes

---

Reviews for paper #83 "A Multi-Objective Time-Linkage Approach for Dynamic Optimization Problems with Previous-Solution Displacement Restriction"
Reviewer: 1

   Originality : Good

   Quality : Very Good

   Relevance : Very Good

   Presentation : Good

Summary:

Details: This paper aims to tackle dynamic optimization problems with previous-solution displacement (PSDR). A PSO-based method is proposed dealing with two objectives of PSDR and the experiment results are competitive. For me this paper makes significant contribution. It is well-written and the experiment is well-designed. It is recommended to accept the paper as talk. The minor point is that the main contribution and the performance of the algorithm should be highlighted in the introduction.

Candidate for the best paper award? : No

Does the paper include up to five keywords? : No

=====================================

Reviewer: 2

   Originality : Good

   Quality : Good

   Relevance : Very Good

   Presentation : Good

Summary:

Details: In general, the paper proposes a hybrid method based on PSO for DOPs with previous-solution displacement restriction (PSDR). The work is interesting, and the paper is well written.  

Some suggestions are given below:

For the experimental design, it would be better to see some test problems from the real world with the PSDR requirement. At least, the paper should give some discussions on potential real-world problems.

Keywords: Not required for a conference paper.

Page 2: "we investigate on DOPs" should be "we investigate DOPs";

Page 4, "that already identified" should be "that have already been identified".

Candidate for the best paper award? : No

Does the paper include up to five keywords? : Yes

---

Reviews for paper #88 "A fast metaheuristic for the design of DVB-T2 networks"

Reviewer: 1

   Originality : Good

   Quality : Very Good

   Relevance : Very Good

   Presentation : Very Good

Summary: The paper modeled the design of DVB-T2 networks as a Mixed Integer Linear Programming problem, termed DVB-MILP, and proposed a bio-inspired metaheuristic for its solution. Test results showed that the proposed metaheuristic outperformed the commercial CPLEX optimization solver on realistic problem instances.

Details: The paper is technically sound and well written. The design problem of DVB-T2 network is well motivated and nicely formulated into a MILP problem. To solve realistic instances of such problem with moderate sizes, the paper proposed a metaheuristic by combining a probabilistic variable fixing procedure with an exact large neighborhood search.

Corrections to be made:

The paper actually has 6 keywords.

There are missing section numbers (3, 4, and 5) in the last paragraph of Section

1.

On the top of p. 6, Intuitively, for "a" given TP, "a" server DS, and "a" subset

of ...

The 2nd sentence of the 1st paragraph of Section 3 mightbe revised as ... may be

too challenging to be optimally solved ...

The "non-existent" symbol in Definition 1 needs to be fixed.

Comments for PC

The paper is very well articulated and the test results demonstrated the solution approach to be very effective and efficient. The paper is a good candidate for the best paper award.

Candidate for the best paper award? : Yes

Does the paper include up to five keywords? : No

=====================================

Reviewer: 2

   Originality : Good

   Quality : Very Good

   Relevance : Very Good

   Presentation : Very Good

Summary: The paper presents a MILP-based metaheuristic for the design of DVB-T2 networks.

The paper is well written, the method applied is clearly explained and the results are good.

Details: Typos:

P2L4: "o European"

P2L-7: 3 missing section references

P8: z_{sl)} should be fixed

Candidate for the best paper award? : No

Does the paper include up to five keywords? : Yes

=====================================

Reviewer: 3

   Originality : Good

   Quality : Good

   Relevance : Very Good

   Presentation : Very Good

Summary: In this paper, firstly a mixed integer linear programming problem for DVB-T2 design is presented, then a meta-heuristic is proposed to solve this optimization problem in a more efficient way. The results are very good compared to CPLEX.

Details: The paper presents the problem of designing a DVB-T2 network as the Mixed Integer Linear Programming Problem (DVB-MILP). Then the formulation of DVB-MILP is strengthened by using the method proposed in the literature [6, 10]. After that, a metaheuristic is proposed to solve this problem in an efficient way. The proposed algorithm is based on the works [8,9]. Finally the experiments are carried out and compared with CPLEX. In a given time limit, the proposed method performs much better than CPLEX. To sum up, by using/adapting some methods in the literature, DVB-T2 problem is solved efficiently.

In general, the paper is written very well. Few suggestions are :

* Move the contribution 3 to the end of the contribution 2.

* Section numbers are missing in the last paragraph of Introduction.

Candidate for the best paper award? : No

Does the paper include up to five keywords? : Yes

---

From: evo...@googlegroups.com <evo...@googlegroups.com> on behalf of Anna Isabel Esparcia-Alcazar <aesp...@hotmail.com>
Sent: Tuesday, March 27, 2018 4:27 PM
To: EvoAPPS Chairs
Subject: Reminder: Best papers zip

 

--
You received this message because you are subscribed to the Google Groups "evoAPPS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to evoapps+u...@googlegroups.com.
To post to this group, send email to evo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/evoapps/DB4PR03MB07330D32A371AACA3F230437B0AC0%40DB4PR03MB0733.eurprd03.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.