Research update: The Future of Life Institute

[Ram Rachum]

Hi everyone!

I'm writing this update while flying over the Arabian Sea on my way to New Zealand. We're an hour and a half in on a 16 hour flight... it's going to be tough.

Also: The video of my dominance hierarchies talk was uploaded.

Retrospective on goals for last month

In last month's update I outlined a list of goals. Here's my update on these goals:

1. Prepare a short talk for the ALA and COINE workshops: ✅ Done

   The ALA and COINE workshops ended up requiring different time slots for their talks: ALA requires 5 minutes and COINE requires 15 minutes. I prepared two separate sets of talks and slides.

   I've also given a full-length version of the dominance hierarchies talk at a local community called NLP IL, and a 10 minute version of a talk at a new local meetup called MARL-IL.

   Here's a video of the long-form talk in the UMD MARL reading group.

2. Learn how to run Opponent Shaping algorithms: 🔄 Ongoing

   I finally made meaningful progress on this.

   I initially wanted to use the M-FOS algorithm, because it seems to be the latest and greatest in OS algorithms. However, it has an intricate meta-game mechanism which I believe is a complex solution to a problem I don't have. I'm trying a different OS algorithm called POLA, written by Stephen Zhao.

   I looked at POLA's reference implementation and it's pretty complex. Part of the difficulty is that it's Jax-based, and I haven't used Jax before. I'm learning it now and it's pretty cool. The Jax stack is an alternative to TensorFlow and PyTorch. In reinforcement learning, it's able to achieve a very high efficiency, because it can reduce RL environments to matrix operations. Intuitively, Jax feels well designed. The Flax package provides neural networks implemented in Jax.

   I've taken the POLA code and put it in a new repo. I'm very carefully refactoring it and cleaning up lots of redundant code. Stephen and Georg are helping me when I get stuck. I'm gradually getting a better understanding of how POLA works.

More stuff

CAIF grant proposal rejected

This one hurts... I've put a lot of effort into the CAIF grant proposal, but unfortunately I got a notification that it was rejected. I did get specific feedback:

We have reviewed your proposal Opponent Shaping for AI Interpretability and Corrigibility, and unfortunately we will not be able to offer you funding for this proposal.

The reviewers appreciated the comprehensive application. However, they did not think that your proposal centered around cooperative AI problems in the sense that is in focus for the mission of CAIF, but saw this proposal rather as using multi-agent methods to solve an (important) single-alignment problem.

This is fair. The grant proposal for Nonlinear is still pending decision, so I hope that one will work out.

Accepted to FLI community, received $3K travel grant

I was accepted to The Future of Life Institute's community of AI Safety researchers! My bio page there. I was also approved for a $3K grant for my trip to AAMAS! This is great news, because I've been running on my savings since the last grant ended, and this gives me a breath of air to continue.

About FLI from Wikipedia:

The Future of Life Institute (FLI) is a nonprofit organization which aims to steer transformative technology towards benefiting life and away from large-scale risks, with a focus on existential risk from advanced artificial intelligence (AI). FLI's work includes grantmaking, educational outreach, and advocacy within the United Nations, United States government, and European Union institutions.

Their AI Safety community has about 100 researchers, which include Max Tegmark and Stuart Russell, so it's a good list to be on. They've got a fortnightly meeting which I'll attend when I get back from my trip. I hope that this will open new grant opportunities for me. Interestingly, I saw that FLI funded The Foresight Institute in 2023. If you remember, I tried and failed to get funding from Foresight 6 months ago. I guess I'm moving upstream.

My goals for this month

1. Present my research at AAMAS.

   • May 6th: Attend the ALA workshop and give a talk and a poster session about my research.

   • May 7th: Attend the COINE workshop and give a talk and a poster session about my research.

   • May 8th: Attend AAMAS talks, show my poster at the poster session.

   • May 9th - May 10th: Attend AAMAS talks

2. Have fun in New Zealand and Japan.

   I've never been to the southern hemisphere before. Besides my trip to Jordan in my twenties, I haven't been anywhere east or south of Israel. I'm excited about this trip. After AAMAS I'll have a week in New Zealand to do some hiking. Then I'll fly to Tokyo. I'll spend 5 days there and then 2-3 weeks in Osaka. If you're in any of these places, feel free to reach out!

That's it for now. See you next month!

Ram.