Research update: Our paper was accepted as an extended abstract!
Ram Rachum
Hi everyone!
Let's start with the bottom line: Our dominance hierarchies paper was accepted to AAMAS as an extended abstract! 🎉
At this point, those of you who aren't in research are asking themselves "what the hell is an extended abstract, and what does it mean for a paper to be accepted as one?"
An extended abstract is a short paper: Two pages of content and one page of references. Barely enough to present an idea and maybe one experiment. When a paper gets accepted to a conference as an extended abstract, it's like the conference is saying "We like what you got... But it's not enough for a full paper. Squish it to 2+1 pages and you're in." So it isn't a full acceptance, but at this point in my career I'll snag it without thinking twice.
More details below.
Retrospective on goals for last month
In last month's update I outlined a list of goals. Here's my update on these goals:
Learn Pax and/or JaxMARL and attempt to run experiments: 😕 Postponed
I didn't have time to work on Pax this month, which bums me out because right now it's the most interesting research direction I've got. It's going to have to wait until February.
Process AAMAS reviews for my dominance hierarchies paper:
On November 30th I got the paper reviews from AAMAS. There were three reviews. I scrolled down that email with great anxiety.
The first reviewer said that the paper was so-so, but admitted that they were not very knowledgeable in this area. Not a good start.
I scrolled down to the second reviewer. (Cue the R2 jokes.) They were not happy with the paper. Their main objection was that it's not clear what the practical benefits of the paper are. That is unfortunately a good point. An additional misfortune is that this reviewer marked themselves as knowledgeable in this area, increasing the impact of their negative review.
At this point I was 90% sure that I was dead in the water. I scrolled down to the third review. The third reviewer was enthusiastic about the paper! They said it's original and insightful, a significant contribution, well-designed, convincing... And other words that bring tears to my eyes 🥹 We had a fighting chance.
We wrote a rebuttal. It's my first one, so it was a learning process as well. We answered the reviewers' questions. The most important concern was the usefulness of our research. I brought up a project I recently heard about: Microsoft's AutoGen. It uses multiple LLMs in a sort of organizational hierarchy, and LLMs are all the rage right now. I argued that because we're putting LLMs in organizational hierarchies, which are somewhat similar to dominance hierarchies, it could benefit us to have better terminology and definitions regarding hierarchies of RL agents.
I sent that rebuttal and prayed.
On December 21st I got the notification from AAMAS:
We are delighted to inform you...
Cheers of joy!
...that your AAMAS 2024 submission “Emergent Dominance Hierarchies in Reinforcement Learning Agents” (#21) has been accepted for publication as an extended abstract, and poster presentation at the conference. Congratulations!
It took me a few confused minutes to remember what extended abstracts are. So it wasn't a full acceptance, but that's still pretty good.
This is a big milestone in my journey to being a researcher. It also means that I have a lot of work towards the conference. More details about that below.
The paper, and my coauthors
I can now share the dominance hierarchies paper with you: Emergent Dominance Hierarchies in Reinforcement Learning Agents. This isn't the final version, we're going to add a few fixes, but it's 98% there.
I can also share the code for the environment I developed: github.com/cool-RR/chicken-coop
I think I haven't introduced all of my coauthors yet:
Yonatan Nakar: I've mentioned Yonatan before. Yonatan is a long-time friend of my younger brother, who became a friend of mine. He's doing a PhD in computer science in Tel Aviv University, though in a different field than mine.
Bill Tomlinson: When I was doing a literature review of the intersection of dominance hierarchies and computer science, I found Bill's paper How Is an Agent Like a Wolf? Dominance and Submission in Multi-Agent Systems. Bill is a Professor of Informatics and Education at the University of California, Irvine, and a researcher in the California Institute for Telecommunications and Information Technology. I emailed him to talk about my work, and he was excited to help me write the paper.
Nitay Alon: I found Nitay Alon when I was looking for multi-agent reinforcement learning reading groups. I found a page for a discontinued reading group on the Hebrew University of Jerusalem's website, and Nitay was listed as the contact. I emailed him to introduce myself. We had a video meeting and we both realized that we know each other from high school! Nitay was one grade below me. While I had gone on to be an engineer, he started on an academic career. Nitay is currently doing a PhD supervised by both Jeffrey Rosenschein at HUJI and Peter Dayan at the Max Planck Institute. He's interested in the intersection of multi-agent reinforcement learning, theory of mind and deception
Reuth Mirsky: I think I presented Reuth before; she's my kind-of advisor at Bar-Ilan University. Reuth heads the GOLD lab and does research on the intersection of multi-agent systems and robotics. She leads the RaD-AI workshop that I've participated in at AAMAS 2023.
There are also many other people who helped, and I've mentioned most of them in the acknowledgements.
Besides the excitement of the success of the research, the paper's acceptance also means that I'm going to go to Auckland, New Zealand! I've never been anywhere close to that part of the world yet. The easternmost point I've been to is Jordan, which is a stone's throw away from my home in Tel Aviv.
I already bought the flight to New Zealand. I'm excited to go. One caveat is that the conference is in May, which is off-season for that part of the world. Around half of the days will be rainy. But I'll have to make the most out of it. I imagine that other conference goers will also be excited about New Zealand and will want to plan hikes together after the conference.
Since I'm flying to such a far-away part of the world, and since I'll have to sit my butt on a plane for 15 hours straight (after a 2 hour warmup flight and then spending the night in a sleeping pod at Dubai airport), I'll want to make the best use of my time while I'm there. I think I'll spend a total of two weeks in New Zealand, and then two weeks in Thailand. But I decided not to get tickets in advance. I'll fly there and do the conference, and only then decide where I want to go.
My first citation!
If the paper's acceptance wasn't enough, here's some more good news: On December 10th I got a notification email from Semantic Scholar letting me know that a citation was made to my stubbornness paper! I'm very excited. This is the first time a paper of mine is cited. It was cited by "How should an AI trust its human teammates? Exploring possible cues of artificial trust" by Carolina Centeio Jorge.
Here is the relevant passage:
Human’s trust and trustworthiness is also affected by the artificial teammate’s behavior, see e.g., [14]. It would also be interesting to investigate the human’s trustworthiness in situations where the artificial teammate behaves differently. For example, situations where the artificial teammate does not obey to its immediate human teammate have been recently studied, see e.g., [53, 59]. Such situations can have mixed motives and knowledge, e.g., the human may want to go straight whereas the artificial teammate knows that it is dangerous (e.g., the human does not have the skills required for what’s ahead), so it opposes. [...]
My paper is [53]. It's cool to think that someone out there was slaving away on their paper, and then they were happy to find my paper as a source to lean on. I hope that many more citations will come.
I feel more like a real researcher this month 😊
My goals for this month
Prepare 4 versions of the dominance hierarchies paper.
Until a week ago, I thought my paper was just one paper. Then it was accepted as an extended abstract, and it became two papers, as I needed to prepare a short version of it. Then Reuth suggested that I submit my paper to the ALA and COINE workshops at AAMAS, each of which requires a different paper format.
Now my paper has become four papers 😪 I have to prepare the four versions and make sure they're consistent. The main challenge would be the extended abstract, because I have to find a way to cut the paper down to 30%-40% of its length while still retaining the interesting bits and keeping it readable.
Prepare 1-2 versions of the talk for the dominance hierarchies paper.
I've been working for a while on a 50 minute talk for this paper. I think I have 40% of the talk ready. I have to finish preparing that talk, and then make a 15 minute version of it, in case I do get accepted to either the ALA or COINE workshops.
Prepare a poster.
I'm going to present my paper as a poster in the AAMAS poster sessions. I need to design a nice poster, and then see whether I can get it printed remotely in Auckland so I won't have to carry it on the plane.
That's it for now. See you next month! And happy new year 🎉
Ram.