GSAI Seminar July 2025 – Engineering Rational Cooperative AI via Inverse Planning and Probabilistic Programming (Tan Zhi Xuan)
29 views
Skip to first unread message
Orpheus Lummis
Jun 2, 2025, 8:14:44 AMJun 2
to guarantee...@googlegroups.com
You are invited to the July 2025 edition of the Guaranteed Safe AI Seminars. It will be on July 10, 13:00 EDT. To join: https://lu.ma/yldjxmej.
Engineering Rational Cooperative AI via Inverse Planning and Probabilistic Programming
Tan Zhi Xuan - Incoming Assistant Professor in the National University of Singapore’s Department of Computer Science with a joint appointment at the A*STAR Institute of High Performance Computing (IHPC). PhD in the the MIT Probabilistic Computing Project and Computational Cognitive Science lab.
How can we build cooperative machines that model and understand human minds — machines that assist us with our goals, coordinate on shared plans, infer the intentions behind our words, and even learn our norms and values? In this talk, I will introduce a scalable Bayesian approach to building such systems via inverse planning and probabilistic programming. By combining online model-based planners and sequential Monte Carlo inference into a single architecture, Sequential Inverse Plan Search (SIPS), we can infer human goals from actions in faster-than-real-time, while scaling to environments with hundreds of possible goals and long planning horizons that have proved intractable for earlier methods. SIPS can additionally make use of large language models (LLMs) as likelihood functions within probabilistic programs, allowing us to build AI assistants and copilots that reliably infer human goals from ambiguous instructions, then provide assistance under uncertainty with much higher success rates than LLMs can on their own. By applying this Bayesian approach in many-agent environments, we are also able to design agents that rapidly learn cooperative social norms from others' behavior, achieving mutually beneficial outcomes with orders of magnitude less data than model-free deep RL. I will conclude by charting out how this research program could deliver a new generation of cooperative AI systems grounded in rational AI engineering, while illuminating the computational foundations of human cooperation and addressing fundamental challenges in building human-aligned AI.
Tan Zhi Xuan - Incoming Assistant Professor in the National University of Singapore’s Department of Computer Science with a joint appointment at the A*STAR Institute of High Performance Computing (IHPC). PhD in the the MIT Probabilistic Computing Project and Computational Cognitive Science lab.
How can we build cooperative machines that model and understand human minds — machines that assist us with our goals, coordinate on shared plans, infer the intentions behind our words, and even learn our norms and values? In this talk, I will introduce a scalable Bayesian approach to building such systems via inverse planning and probabilistic programming. By combining online model-based planners and sequential Monte Carlo inference into a single architecture, Sequential Inverse Plan Search (SIPS), we can infer human goals from actions in faster-than-real-time, while scaling to environments with hundreds of possible goals and long planning horizons that have proved intractable for earlier methods. SIPS can additionally make use of large language models (LLMs) as likelihood functions within probabilistic programs, allowing us to build AI assistants and copilots that reliably infer human goals from ambiguous instructions, then provide assistance under uncertainty with much higher success rates than LLMs can on their own. By applying this Bayesian approach in many-agent environments, we are also able to design agents that rapidly learn cooperative social norms from others' behavior, achieving mutually beneficial outcomes with orders of magnitude less data than model-free deep RL. I will conclude by charting out how this research program could deliver a new generation of cooperative AI systems grounded in rational AI engineering, while illuminating the computational foundations of human cooperation and addressing fundamental challenges in building human-aligned AI.
Orpheus Lummis
Jul 3, 2025, 12:30:44 PM (11 days ago) Jul 3
to guaranteed-safe-ai
Xuan shared the following papers that the talk is based on, as reading materials:
- Online Bayesian Goal Inference for Boundedly Rational Planning Agents https://proceedings.neurips.cc/paper/2020/hash/df3aebc649f9e3b674eeb790a4da224e-Abstract.html
- Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning https://arxiv.org/abs/2402.17930
- Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games https://arxiv.org/abs/2402.13399
- Online Bayesian Goal Inference for Boundedly Rational Planning Agents https://proceedings.neurips.cc/paper/2020/hash/df3aebc649f9e3b674eeb790a4da224e-Abstract.html
- Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning https://arxiv.org/abs/2402.17930
- Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games https://arxiv.org/abs/2402.13399
Looking forward to the presentation & discussion!
Orpheus Lummis
Jul 11, 2025, 5:33:43 AM (4 days ago) Jul 11
to Orpheus Lummis, guaranteed-safe-ai
The recording is now available: https://www.youtube.com/watch?v=uiJ1dmmNL0k
--
You received this message because you are subscribed to the Google Groups "guaranteed-safe-ai" group.
To unsubscribe from this group and stop receiving emails from it, send an email to guaranteed-safe...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/guaranteed-safe-ai/6336e731-c641-4832-a675-78d4b3263f52n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages