Active Members - please introduce yourself here

[Carola Doerr]

Dear all,

You can use this thread to briefly introduce yourself and your interest in benchmarking iterative optimization heuristics.

Best wishes,
Carola Doerr, Pascal Kerschke, Boris Naujoks, Mike Preuss, Vanessa Volz

[Vanessa Volz]

Hello,

I am Vanessa Volz. I am now a researcher at a start-up in Copenhagen (Denmark) called modl.ai.

My main research focus is the application of evolutionary algorithms to problems in the games industry, e.g. for automatic tuning of game parameters or procedural content generation. Unfortunately, there was no benchmark for this area of application when I started my PhD, so I set one up together with colleagues (Tea and Boris) called GBEA [1]. I am convinced that this benchmark is not only interesting in the context of games, but could also be a way to benchmark algorithms targeting real-world applications. This is because the GBEA simulates things like expensive fitness function evaluations, non-symmetric noise and non-regular landscapes.

I am also interested specifically in best practices for benchmarking and how these could be "baked into" benchmarking software. Further, I think we as a community could gain a lot by integrating the various little benchmarks everyone sets up for themselves through a common framework.

Cheers,

Vanessa

[1] http://www.gm.fh-koeln.de/~naujoks/gbea/

[Markus Wagner]

Hi!

My name is Markus, https://cs.adelaide.edu.au/~markus/, and some of my work is on applications. What I find interesting is that there are some application papers that stop at "we've done a few repetitions of only our algorithm on a small number of instances, and here are the averages" -- okay, I'm simplifying this a bit (and some people have valid excuses), but you get the idea. Long story short: there is much love to be shared -- across communities. I'm always happy to learn a few new tricks, too.

Cheers,

Markus

[Aleš Zamuda]

Dear all,

a hello from me as well. My name is Aleš Zamuda and my works have been in
attending merely all CEC competitions on benchmarking continuous optimizers
and discussing these benchmarks. In meanwhile, other types of features evolved
at these competitions and I also took place at several of these, but also
mostly continuous. However, all these benchmarking approaches can benefit from
well defined benchmarking statistics as well as test instances.
I was in program committees for various competitions at the conferences like
CEC and GECCO, also reviewed many articles on these subjects. I published
papers on benchmarks in few domains besides the bare CEC functions, like:
- underwater trajectory planning for robotic vehicles,
- 3D animation of trees and geometry reconstruction,
- powerplant energy scheduling, and
- healthcare in-patient feature stability selection for machine learning,
- (also, other real-world problems in the CEC 2011 functions - interatomic
potential optimization, materil design, space trajectory planning, etc.)

Btw., I like the Sörensen's paper on "Metaheuristics - the metaphor exposed",
saying it does not matter to publish a nice story on an algorithm (=metaphor),
I think this is social sciences (and, very helpful perhaps in teaching, but
not in publishing technical improvement). I consider that it matters to show
that the algorithm actually is technically better (well benchmarked/proven).
With Tome and Stjepan, at CEC we are organizing a special session BEADO:
http://beado.feri.um.si/

Also, we wrote an overview survey on benchmarking with Christine Zarges at
GECCO, many ideas on improving benchmarking are already listed there. I am
vice-chair at WG3: Benchmarks at ImAppNIO COST action. My webpage:
http://aleszamuda.si

All best to all members of Benchmarking Network!

Best Regards.

--
doc. dr. Aleš Zamuda, PhD | Assistant Professor
University of Maribor | Faculty of Electrical Engineering and Computer Science
Institute of Computer Science | Computer Architecture and Languages Laboratory
Koroška cesta 46, 2000 Maribor, Slovenia, EU | Phone: +386 2220 7404
E-mail: ales....@um.si, aza...@acm.org, ales....@ieee.org
IEEE Senior Member | IEEE Slovenia Section Vice Chairman & YP Chairman
Web: http://labraj.feri.um.si/~ales, http://aleszamuda.si

On day petek, 13. december 2019 at 09:14:14 CET you Carola Doerr have written:

[Katherine Malan]

Hello all,

I am Katherine Malan. Down here in South Africa we have already started on our long summer break and I am currently on holiday with my family at the sea. I am an associate professor at the University of South Africa in the Department of Decision Sciences. I am really interested in understanding complex optimisation problems so that we can more intelligently and automatically apply algorithms to solving new problems. So, benchmarking is an essential part of that. I am looking forward to being part of the conversation.

Cheers,

Katherine

[Katherine Malan]

Like Ales, I am a fan of Kenneth Sorensen's metaphor paper. In the many papers that I review on "novel" metaheuristics I always point them to that paper.

[Luís Paquete]

Dear all,

I am Luís Paquete from Coimbra. I have been looking into multiobjective combinatorial problems and most of my work is about 

understanding the difficulty of solving these problems, both for exact and heuristic approaches. Benchmarking on these problems 

seems to be difficult as well, so I am glad to contribute :-)

Best wishes,

Luís 

[Arnaud Liefooghe]

Hello,

I'm Arnaud Liefooghe, from the University of Lille, France.

My main focus is on multi-objective (combinatorial) optimization.

I'm interested in benchmarking in order to improve our understanding
of problem difficulties and of the behavior / performance of search
heuristics, and ultimately to (automatically) select and configure
algorithms based on high-level landscape features.

I'm looking forward to collaborating with you!

Best,
Arnaud

> --
> You received this message because you are subscribed to the Google Groups "benchmarking-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-net...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/benchmarking-network/293ab664-e248-48fc-ae21-c548281f8e3d%40googlegroups.com.

[Steffen Finck]

Dear all,

I am Steffen Finck and I am a researcher at the FH Vorarlberg University of Applied Sciences in Austria. I was involved in the early BBOB competitions and I use different benchmarks regularly for testing algorithms or check results in some review papers. I am interested in different benchmarks, currently I am working with a student with the CEC'13 multi-modal benchmark. I am interested in what we can learn from the results of benchmarking and how we can make sure the benchmark covers interesting problems.

Best,

Steffen

[Petr Pošík]

Hello,

I am Petr Pošík, researcher from Czech Technical University in Prague.

I am primarily (but not only) interested in real-valued optimization, EDAs and applying ML methods to improve EC.I participated in the preparation of the first few BBOB workshops, but then I had to inhibit some research activities due to an increased administrative and teaching load. But my interest in benchmarking still survives, since - apart from building theoretical foundations - it is IMHO the closest we can get to an objective assessment of search heuristics.

And since it is almost Xmas - I wish you all calm and relaxing Xmas with your beloved ones and happy and successful new year!

Cheers, Petr

[Martin Zaefferer]

Dear all,

I'm a researcher at TH Köln, with a focus on surrogate model-based optimization / expensive real-world optimization problems.

Hence, I'm interested in benchmarking algorithms in that very same area.

Particularly, I'd like to investigate further how benchmarks with real-world relevance can be generated, possibly using data-driven models.

Thanks for setting this up and contributing!

Best wishes

Martin

[Marius Lindauer]

Hi everyone,

I am Marius Lindauer from the University of Hannover, Germany.

My research focus is on meta-algorithmic approaches, incl. AutoML, algorithm configuration, algorithm selection and similar topics.

Over the years, we proposed the algorithm configuration benchmarking library AClib [1] and the algorithm selection benchmarking library ASlib [2]. 

Furthermore, I'm also a bit involved in the hyperparameter optimization benchmarking library HPOlib [3].

In general, I'm very interested in how benchmarking should be done in a scientific, repeatable and reliable way. 

To this end, I have a paper on best practices for algorithm configuration [4] and one on neural architecture search [5] --- but I think these barely scratch the surface of the topic.

I'm looking forward to the meeting at the PPSN.

Best regards,

Marius

[1] https://bitbucket.org/mlindauer/aclib2/

[2] http://www.coseal.net/aslib/

[3] https://github.com/automl/HPOlib2

[4] https://arxiv.org/abs/1705.06058

[5] https://arxiv.org/abs/1909.02453

[Tea Tušar]

Hello,

A Happy New Year to all!

I'm Tea Tušar, a research fellow at the Jožef Stefan Institute in Ljubljana, Slovenia.

Benchmarking has been one of my core research interests in the past few years. I have been contributing to the development of the open-source COCO platform for Comparing COntinuous Optimizers (https://github.com/numbbo/coco). Most recently (at GECCO 2019) we have introduced the following new suites of benchmark problems:

- Mixed-integer problems (a single- and a bi-objective suite of problems), see https://dl.acm.org/citation.cfm?id=3321868

- Real-world problems from the domain of games (Mario GAN problems and TopTrumps problems, both in single- and bi-objective variants), see https://dl.acm.org/citation.cfm?id=3321805

I'm also part of a group of researchers trying to better understand the properties of real-world problems in order to be able to create better benchmarks for optimization algorithms. We are collecting information on any kind of real-world optimization problems in this survey: https://tinyurl.com/vhakphu

You are cordially invited to take part in it. If you indicate so in the survey, we will acknowledge your contribution in any publications/reports based on this survey.

Best regards,
Tea