What’s Happening in AI Evaluation (Before the End of the Year)

[Wout Schellaert]

Hi

This is a general overview of events on the topic of AI Evaluation that are happening before the end of 2022.

There are (at least) these 5 workshops

• GENEA: Generation and Evaluation of Non-verbal Behaviour for Embodied Agents ICMI  - 07/11/2022 - https://genea-workshop.github.io/2022/workshop/
• Eval4NLP: Evaluation & Comparison of NLP Systems
  AACL - 20/11/2022 - https://eval4nlp.github.io/2022/index.html
• HEGM: Human Evaluation of Generative Models
  NeurIPS - 03/12/2022 - https://humaneval-workshop.github.io/
  
• GEM: Generation, Evaluation & Metrics
  EMNLP  - 07/12/2022 - https://gem-benchmark.com/workshop
• Benchmarking in Robotic Manipulation
  CoRL - 15/12/2022 - https://sites.google.com/view/corl22benchmarkingworkshop/home

And similar to last year, NeurIPS 2022 also has a Dataset & Benchmark track.

There is also some general news:

• At the end of June, Huggingface announced their Evaluation on the Hub initiative, and they have recently published an accompanying paper.
• PyTorch releases alpha version of a built-in evaluation & metrics library TorchEval
• MLSafety announces SafeBench, a competition with up to $500,000 in prizes for ML safety benchmark ideas.

It is likely I have missed events or news, so feel free add them & post them in this mailing list.

Regards

Wout Schellaert

[Jose Hernandez Orallo]

Dear all,

We're happy to announce and invite all of you to the following initiative and associated event. Feel free to spread the word as well.

"Predictable AI" (predictable-ai.org) is an initiative to study the predictability of present and future AI systems, and their interplay throughout human-AI ecosystems. On the one hand, the predictability of AI systems requires the re-understanding of evaluation as an inference problem that anticipates how AI systems will behave for each (co-)operative condition (problem and stakeholders), by extracting cognitive capabilities or by learning ML assessors from test data. On the other hand, the predictability of future AI as an ecosystem calls for discovering robust patterns and scaling laws at different aggregation levels that ultimately connect predictability with control, liability and risks.

The initiative will start on 8th March 2023 with a singular event consisting of invited talks, panels, short highlights and time for networking.

Confirmed speakers (in-person):

- Irina Rish (Mila, Quebec AI Institute, Montreal)
- Lucy Cheke (U. Cambridge)
- Emilia Gómez (JRC, European Commission)
- Ida Momennejad (Microsoft Research, NYC)
- Joel Leibo (Deepmind, London)
- Seán O'hEigeartaigh (U. Cambridge)

The event is co-organised and will count with a numerous representation of colleagues from the CFI (http://lcfi.ac.uk/) and CSER (https://www.cser.ac.uk/) in Cambridge, and ValGRAI (https://valgrai.eu/) and VRAIN (https://vrain.upv.es/) in Valencia.

More information about the initiative, the event and the agenda of the day at https://predictable-ai.org. Registration is open (and free of charge) at https://forms.office.com/e/r6yVZRC30Z (deadline 31st January 2023, but we recommend to register as soon as possible due to limited capacity).

Best wishes,

Jose Hernandez-Orallo