Trends in NeuroAI - Leaving Stability AI (blog post)
Paul Scotti
https://www.paulscotti.com/blog/leavingSAI
I resigned from Stability AI after over a year working as Head of NeuroAI. I’m now looking into new ways to foster fully transparent open-source science and conduct research on neuroAI and medical AI more broadly.
A personal retrospective below touches on my journey with open science communities and how Stability AI’s role in the “science-in-the-open” movement was such a pivotal phenomenon to have come and gone.
My roots with open science & OnNeuroBack in 2017 I had an idea—what if I could coordinate an international network of likeminded neuroscientists and psychologists together through an online community? We could share our ideas and progress as they happened, and stay in touch through regular online journal clubs.
I had just moved to Columbus, Ohio for my PhD studies and was feeling quite isolated. Relatedly, I was thinking about how academics typically work in isolation within their lab. Usually the earliest academics share their ideas and work outside of their own lab is at infrequent academic conferences. And if they don’t share at a conference then they might not share their findings until years after the work was completed, as a finished peer-reviewed research publication. It just seems like an obviously inefficient system for science progress as a whole. I wanted to build an online community of neuroscientists where we could all learn and grow together as emerging academics.
Nowadays this idea seems ordinary; online journal clubs are commonplace and there’s plenty of internet communities dedicated to academic subfields. But remember, this is back in 2017! Before the COVID pandemic and before the boom of Discord research communities! The notion that anyone could go on their laptop and attend a live presentation and do Q&A with an esteemed professor across the globe was not at all a regular occurrence.
I tapped into my limited networks, advertised at academic conferences, and built a modest group of researchers who regularly attended OnNeuro meetings. I also expanded from online journal clubs to live-streaming all the neuro lectures at my university that I could, with recordings uploaded to YouTube. I tried to convince people at other institutions to do the same, but I struggled because it was a heavy undertaking for them to communicate with IT, get approval from speakers for their lecture to be recorded and shared online, and get approval from university departments.
When COVID hit in 2020, Tim Vogels and Panos Bozelos started an initiative called WorldWideNeuro. They essentially had the same idea as me but they started at the perfect time to tap into the pandemic ceasing travel, and they actually had the networking power to convince academics across tons of universities to start sharing their in-person lectures and seminars online. 2020 also happened to be the year I was finishing up my Ph.D. and so I was limited in free time. This demotivated me from continuing to put effort into OnNeuro, and I ultimately gave it away to the Ohio State University CCBBI Student Group to continue recording meetings and putting the content on YouTube (which I’m happy to see they are still doing!).
Transitioning to neuroAII feel I owe a lot of my grad school success to open science because I believe it was talking about OnNeuro in my personal statement that helped award me an NSF GRFP fellowship. This fellowship funded my PhD studies where my dissertation involved computational modeling to reconstruct visual perception (simple features like color and orientation) from fMRI brain activity. This work helped land me a postdoc, where I left Ohio State University and joined Princeton University in 2022.
My postdoc project tackled decoding of visual memory. I wanted to look into using AI image generation models to help reconstruct images from fMRI brain patterns using a new dataset called the Natural Scenes Dataset1. This started my transition from computational neuroimaging into the newly coined neuroAI field, as I took to the internet to improve my deep learning expertise.
From OnNeuro to LAIONI discovered the LAION Discord server and figured that seemed like a good place to connect with other passionate AI researchers and improve my skillset so I could best tackle my postdoc project. But who would have thought that there would already be a channel inside of the LAION Discord specifically tackling reconstruction of visual perception from brain activity?
I introduced myself to the group and quickly thereafter became the project lead for what later became MindEye2. Tanishq (aka ilovescience) welcomed me into the group and our small group quickly started making progress with weekly meetings and frequent asynchronous updates in the Discord thread. Every so often a new person would show up and join the team, where we all were simply contributing to a shared public GitHub repository and posting about our updates as we tried different ideas out. One such team member was Atmadeep (aka aka Atom_101) who contributed significantly and became co-first author with me on the MindEye paper. I learned a ton during this time—clearly much more than I would have if I were independently researching as a postdoc—our scrappy mishmash team of online volunteers shared ideas and trained large-scale multi-gpu neural networks and our own denoising diffusion models from scratch.
After a few months, Tanishq was hired as research director at Stability AI and he created his own Discord server named MedARC to host various medical AI research projects. Tanishq offered me the opportunity to join Stability AI as a full-time employee where my job would be to lead and organize neuroAI open-source projects. It seemed like a no-brainer—I would continue to work on the research I was most passionate about, contribute to open science, and I would get paid more. So the group moved over to MedARC and we resumed work.
In 2023, MindEye was accepted (and featured as spotlight) to NeurIPS. We achieved state-of-the-art performance reconstructing visual perception from fMRI brain patterns. It received a great deal of public attention and I was excited that this novel science-in-the-open approach seemed to be working so well. NeurIPS was also the first time I had a chance to meet some of the MedARC community in person!
Tanishq, Stepan, and I presenting our poster at NeurIPS.We subsequently developed a follow-up, MindEye23, which was accepted to ICML and showcased further improvements to state-of-the-art and also specifically improved performance in low-sample settings via shared subject models (as opposed to previous work which trained models independently for every participant in the dataset).
Reconstructed images from fMRI brain activity collected during perception. These results were obtained using only 1 hour of training data, which reflects just 2.5% of the full training dataset used in other works (like MindEye1).
In hindsight, our situation was indeed a bit too good to be true. Emad Mostaque founded Stability AI and was the one responsible for hiring Tanishq and sharing Stability compute to support not just MedARC’s endeavors but a whole host of other open science Discord channels (e.g., EleutherAI, OpenBioML, CarperAI). Stability AI provided free compute to volunteers working on open-source research in these communities and it enabled us to fully leverage crowd-sourced transparent science without the barriers typical to these sorts of projects. We used the Stability HPC as a shared file system and contributors could easily run different ablations and model trainings using the same compute setup.
I guess Emad’s generosity may have been seen as counter productive to the incentives of a for-profit startup. The external Stability HPC supported dozens of projects, consumed a ton of expensive compute, and it wasn’t directly leading to additional revenue for Stability. Emad left Stability AI and shortly thereafter the external HPC was understandably shut down and non-employees were kicked off the cluster. This drastically decreased productivity in MedARC and all the other Discord science communities that were relying on this compute. We had to shut down some projects and our rate of progress was never quite the same.
ConclusionStability AI was my first step outside of the traditional academic pipeline and I am incredibly grateful for the opportunity to spend over a year working as the principal investigator of the MedARC Neuroimaging and AI Lab, collaborating across academia, industry, and an international group of passionate and talented research volunteers. I am convinced that the fully transparent science-in-the-open approach, if supported in the right way, still hasn’t reached its full potential for scientific breakthroughs. And now, having left Stability AI, I am also in a new phase of leaving stability in my life overall. Stability AI and academia have been stable pillars I relied on and now my career is inherently less stable—I figure I need to be more comfortable with instability if I wish to wholeheartedly continue embracing my ideals surrounding open science.
Thanks to Patrick Mineault and Tanishq Abraham for inspiring me to start writing thoughts in blog form.
Allen, E. J., et al. (2021). A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 24(8), 1164–1176. https://www.nature.com/articles/s41593-021-00962-x ↩
Scotti, P. S., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Cohen, E., Dempster, A. J., Verlinde, N., Yundler, E., Weisberg, D., Norman, K. A., & Abraham, T. M. (2023). Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2305.18274 ↩
Scotti, P. S., Tripathy, M., Torrico Villanueva, C. K., Kneeland, R., Chen, T., Narang, A., Santhirasegaran, C., Xu, J., Naselaris, T., Norman, K. A., & Abraham, T. M. (2024). MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. International Conference on Machine Learning. https://arxiv.org/abs/2403.11207 ↩