[Job] 1 Year PostDoc in Reliable Evaluation of Generative AI

[Heiko Paulheim]

Postdoctoral Researcher (m/f/d) in Reliable Evaluation of Generative AI

University of Mannheim, Data and Web Science Group (DWS)
Fixed-term position (1 year, full-time)
Starting date: As soon as possible

About the Project

The Data and Web Science Group at the University of Mannheim is seeking a highly motivated Postdoctoral Researcher (m/f/d) to join the research project"Reliable Evaluation of Generative AI (ZEGeKI)": https://www.uni-mannheim.de/dws/research/projects/reliable-evaluation-of-generative-ai-zegeki/

Generative AI systems such as ChatGPT are rapidly becoming integral parts of both everyday applications and complex software systems. At the same time, evaluating these systems reliably has become increasingly challenging. Traditional benchmark-based evaluations suffer from problems such as benchmark overfitting, benchmark leakage, and limited applicability to modern AI systems that continuously learn or interact with external information sources.

The ZEGeKI project addresses these challenges through a novel paradigm for AI evaluation based on dynamic, runtime-generated benchmarks. Building on the GET methodology (Generate, Evaluate, Trash), the project investigates how generative AI itself can be used to continuously create fresh evaluation tasks that cannot be memorized in advance by the systems being tested.

Your Responsibilities

• Conduct cutting-edge research on the evaluation of large language models and generative AI systems.
• Design and implement novel methodologies for dynamic benchmark generation.
• Investigate issues such as benchmark leakage, data contamination, and evaluation robustness.
• Develop and experimentally validate new AI evaluation frameworks.
• Publish results in leading international conferences and journals.
• Contribute to research proposals and the strategic development of the group's research agenda.
• Supervise student researchers and contribute to teaching activities where appropriate.

Your Profile

We are looking for candidates who have:

• A completed PhD in Computer Science, Data Science, Artificial Intelligence, Machine Learning, or a related discipline.
• A strong publication record in relevant venues.
• Experience with large language models, generative AI, machine learning, or AI evaluation.
• Excellent programming skills, preferably in Python and modern AI frameworks.
• Strong analytical and scientific writing skills.
• The ability to work independently while contributing to a collaborative research environment.
• Excellent English communication skills (German is a plus, but not required).

The following will be considered an advantage:

• Experience with benchmark design and evaluation methodologies.
• Knowledge of responsible AI, AI safety, or trustworthy AI.
• Experience leading research projects or supervising students.

What We Offer

• The opportunity to help shape a novel and rapidly emerging field of research.
• A highly visible project at the intersection of Generative AI, AI evaluation, and trustworthy AI.
• A stimulating international research environment at one of Germany's leading information systems and data science groups.
• Excellent opportunities for scientific publishing and networking.
• Competitive salary according to the German public service pay scale (TV-L, depending on qualifications).

Application

Applications should include:

• Cover letter describing research interests and motivation.
• Curriculum vitae.
• List of publications.
• Contact information for two academic references.
• Copies of relevant certificates.

All those should be sent to the contact below in one PDF file.

Applications will be reviewed on a rolling basis until the position is filled.

For informal inquiries, please contact:

Prof. Dr. Heiko Paulheim
Data and Web Science Group
University of Mannheim
heiko.p...@uni-mannheim.de
https://www.uni-mannheim.de/dws/people/professors/prof-dr-heiko-paulheim/