Research update: Gridworld environments, CAIF grant

12 views

Skip to first unread message

Ram Rachum

unread,

Feb 27, 2025, 3:31:21 PMFeb 27

to ram-rachum-res...@googlegroups.com

Hi everyone!

Two topics to update about this month: I've been working on getting team formation in gridworld environments, and I've progressed to the next phase in my application for the CAIF grant.

Team formation in gridworld environments

In last month's update, I've shown good results of team formation in an environment that was specifically tailored to induce team formation. That's a good start, and the next step is to get similar results in more neutral environments, i.e. environments which have less specific rules.

I figured I should try gridworld environments. If you're not familiar, gridworld is a general term for games in which agents move around a 2-dimensional grid. Think of them as the MDP version of games like Pacman or Sokoban. Many RL researchers use gridworld environments for their research, and I figured they'd be a good choice here.

One complication with gridworlds is that they require convolutional neural networks, since they are spatial. This is challenging because up to now I haven't combined CNNs with my Viola algorithm.

I designed a gridworld environment in which agents move around and eat berries. I won't reveal the rules, but here's a screenshot:

Not quite as fun as Pacman, I admit. This game is 1-on-1, but I've trained populations of agents to play it in random pairs.

I hoped I'd see team formation right away, but I didn't. In fact, I didn't even see reciprocity, which is a required condition for the kind of team formation I'm after. I'm currently debugging why reciprocity doesn't emerge.

CAIF grant

On the last day of 2024, I sent an update saying that I applied to The Cooperative AI Foundation (CAIF)'s new round of grants. They have a new structure for their grants, where applicants first submit a 2-page pre-proposal, and only after being selected do they need to write a proposal. My pre-proposal was selected :)

Now I'm facing the difficult task of expanding my pre-proposal to a 5-page proposal, along with some more details I need to add on their application forms. Just in time, CAIF released a long and detailed report which details what kind of problems they're interested in. I'm reading through that and attempting to apply it to my proposal.

CHAI Internship

In the background, I'm doing some logistics for my internship at CHAI, UC Berkeley. I bought flights, and I'm going through the long and arduous process of obtaining a J-1 visa to the US.