Online Social Choice and Welfare Seminar: Jobst Heitzig, Tuesday 23 September
8 views
Skip to first unread message
Marcus Pivato
Sep 16, 2025, 10:57:39 AMSep 16
to social-choice-a...@googlegroups.com, com...@duke.edu
[with apologies for cross-posting]
Dear all,
The next presentation in the Online Social Choice and Welfare Seminar will be next Tuesday (23 September). Here are the details.
Time: 5PM GMT (10AM San Francisco, 1PM Toronto/Montréal, 2PM Rio de Janeiro, 6PM London, 7PM Berlin, 8PM Istanbul, 10:30PM New Delhi) [Note unusual time]
Speaker: Jobst Heitzig (Potsdam Institute for Climate Impact Research)
Title: "Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power"
Abstract: Power is a key concept in AI safety: power-seeking as an instrumental goal, sudden or gradual disempowerment of humans, power balance in human-AI interaction and international AI governance. At the same time, power as the ability to pursue diverse goals is essential for wellbeing.
This paper explores the idea of promoting both safety and wellbeing by forcing AI agents explicitly to empower humans and to manage the power balance between humans and AI agents in a desirable way. Using a principled, partially axiomatic approach, we design a parameterizable and decomposable objective function that represents an inequality- and risk-averse long-term aggregate of human power. It takes into account humans' bounded rationality and social norms, and, crucially, considers a wide variety of possible human goals.
We derive algorithms for computing that metric by backward induction or approximating it via a form of multi-agent reinforcement learning from a given world model. We exemplify the consequences of (softly) maximizing this metric in a variety of paradigmatic situations and describe what instrumental sub-goals it will likely imply. Our cautious assessment is that softly maximizing suitable aggregate metrics of human power might constitute a beneficial objective for agentic AI systems that is safer than direct utility-based objectives.
(Joint work with Ram Potham; working paper available here)
Dear all,
The next presentation in the Online Social Choice and Welfare Seminar will be next Tuesday (23 September). Here are the details.
Time: 5PM GMT (10AM San Francisco, 1PM Toronto/Montréal, 2PM Rio de Janeiro, 6PM London, 7PM Berlin, 8PM Istanbul, 10:30PM New Delhi) [Note unusual time]
Speaker: Jobst Heitzig (Potsdam Institute for Climate Impact Research)
Title: "Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power"
Abstract: Power is a key concept in AI safety: power-seeking as an instrumental goal, sudden or gradual disempowerment of humans, power balance in human-AI interaction and international AI governance. At the same time, power as the ability to pursue diverse goals is essential for wellbeing.
This paper explores the idea of promoting both safety and wellbeing by forcing AI agents explicitly to empower humans and to manage the power balance between humans and AI agents in a desirable way. Using a principled, partially axiomatic approach, we design a parameterizable and decomposable objective function that represents an inequality- and risk-averse long-term aggregate of human power. It takes into account humans' bounded rationality and social norms, and, crucially, considers a wide variety of possible human goals.
We derive algorithms for computing that metric by backward induction or approximating it via a form of multi-agent reinforcement learning from a given world model. We exemplify the consequences of (softly) maximizing this metric in a variety of paradigmatic situations and describe what instrumental sub-goals it will likely imply. Our cautious assessment is that softly maximizing suitable aggregate metrics of human power might constitute a beneficial objective for agentic AI systems that is safer than direct utility-based objectives.
(Joint work with Ram Potham; working paper available here)
To obtain the Zoom link, please subscribe to the Seminar Mailing List, or contact one of the organisers.
Reminder: On the seminar website you can find the video recordings, slides and supplementary materials for all past presentations, as well as information about future presentations.
--
Marcus Pivato
Centre d'Économie de la Sorbonne
Centre d'Économie de la Sorbonne
Université Paris 1 Panthéon-Sorbonne
Reply all
Reply to author
Forward
0 new messages