Research update: My paper is almost ready...
Ram Rachum
Hi everyone!
Retrospective on goals for last month
In last month's update I outlined a list of goals. Here's my update on these goals:
Finish 80% of the dominance hierarchies paper: ✅ Done
Woohoo! I've made substantial progress on the paper this month, and 80% sounds like an accurate estimate. More details below.
Thank you ALTER!
Before I start talking about the progress with the paper, I want to note that my fellowship at ALTER is ending in five days from now. ALTER have been funding me for the last six months, and without their generous support I couldn't have written my paper. Thank you David Manheim for all the help and advice, and Naham Shapiro for the meticulous logistics work.
Now I should start seeking a new source of funding. On David's recommendation, I applied to the Foresight Institute's AI Safety fund. I'm planning on applying for more grants from more organizations after I finish the paper.
If you know of any relevant funding opportunity, please let me know.
My paper
I've been teasing my paper a lot, so I might as well tease you with the title and abstract:
Emergent Dominance Hierarchies in Reinforcement Learning Agents
Modern Reinforcement Learning (RL) algorithms are able to outperform humans in a wide variety of tasks. Multi-agent reinforcement learning (MARL) settings present additional challenges, and successful cooperation in mixed-motive groups of agents depends on a delicate balancing act between individual and group objectives. Social conventions and norms, often inspired by human institutions, are used as tools for striking this balance.
In this paper, we examine a fundamental, well-studied social convention that underlies cooperation in both animal and human societies: Dominance hierarchies.
We adapt the ethological theory of dominance hierarchies to artificial agents, borrowing the established terminology and definitions with as few amendments as possible. We demonstrate that populations of RL agents, operating without explicit programming or intrinsic rewards, can invent, learn, enforce, and transmit a dominance hierarchy to new populations. The dominance hierarchies that emerge have a similar structure to those studied in chickens, mice, chimpanzees, and other species.
I'm rereading this abstract for the thousandth time now and experiencing mixed feelings. I'm proud of it and I think it will succeed. I also see all the points where I changed something because of one of my mentors' suggestions, and then changed it again because of another mentor, and then again until it reached a stable condition. It reminds me of a stone riverbank being shaped by centuries of water flowing through it. It hasn't quite been centuries in this case, but this paper, and especially its abstract, probably hold the title of having the highest attention-per-word ratio out of everything I ever wrote.
Let's talk a bit about my progress. At this point my paper draft is 5.5 pages long, with another page for references and a half-page for appendices. The final paper will be 8 pages long, not including references and appendices, because that's the AAMAS limit. The Intro, Background and Related Work sections are 90% finished. The Definitions section is 70% finished. The Results and Discussion And Future Work sections are 40% finished. The repo is 90% finished.
One thing I've been working on in the last couple of days is the plots. At first I used hand-drawn plots, which is a great idea for the writing period because they serve as good placeholders while being very easy to make. I've now started replacing them with plots produced using Plotly. Here's one of these plots with no context:
There are more major things to do for the paper. I have to run a couple of experiments that I haven't run, I have to find more references for claims I'm making, I have to tone down some of the claims possibly, I have to cut a few paragraphs I love out of the intro and background because of the page limit... Lots of work.
Meme.
My goals for this month
Finish the dominance hierarchies paper and submit it to AAMAS.
I've got nine days to wrap this paper up and submit it to AAMAS. My excitement for this paper is eclipsed only by the anticipation for October 10th, which is the day that I can officially say I'm not working on the paper anymore. Until review season comes, of course 🫠
Apply for funding with several foundations.
Once I'm done submitting the paper I should apply for at least two different sources of funding.
If you know of any relevant funding opportunity, please let me know.
That's it for now. See you next month!
Ram.