Trends in NeuroAI - Brain-optimized inference improves fMRI-to-image
1 view
Skip to first unread message
Paul Scotti
Dec 31, 2023, 1:17:39 PM12/31/23
to MedARC Neuroimaging & AI
Happy New Years everyone! I hope you've all had a wonderful holiday season. Looking forward to entering 2024 together by delving deeper into the ever-evolving topic of NeuroAI 😊
Our first journal club of the new year will be Friday, Jan 5 11:00am ET!
Our first journal club of the new year will be Friday, Jan 5 11:00am ET!
We have first author Reese Kneeland presenting on his preprint "Brain-optimized inference improves reconstructions of fMRI brain activity". This work shows state-of-the-art reconstructions of seen images from fMRI activation by hooking up MindEye to a brain-optimized encoding model. Reese is a PhD student at University of Minnesota working in the Naselaris lab (https://www.reesekneeland.com).
Title:
Brain-optimized inference improves reconstructions of fMRI brain activity
Abstract:
The release of large datasets and developments in AI have led to dramatic improvements in decoding methods that reconstruct seen images from human brain activity. We evaluate the prospect of further improving recent decoding methods by optimizing for consistency between reconstructions and brain activity during inference. We sample seed reconstructions from a base decoding method, then iteratively refine these reconstructions using a brain-optimized encoding model that maps images to brain activity. At each iteration, we sample a small library of images from an image distribution (a diffusion model) conditioned on a seed reconstruction from the previous iteration. We select those that best approximate the measured brain activity when passed through our encoding model, and use these images for structural guidance during the generation of the small library in the next iteration. We reduce the stochasticity of the image distribution at each iteration, and stop when a criterion on the "width" of the image distribution is met. We show that when this process is applied to recent decoding methods, it outperforms the base decoding method as measured by human raters, a variety of image feature metrics, and alignment to brain activity. These results demonstrate that reconstruction quality can be significantly improved by explicitly aligning decoding distributions to brain activity distributions, even when the seed reconstruction is output from a state-of-the-art decoding algorithm. Interestingly, the rate of refinement varies systematically across visual cortex, with earlier visual areas generally converging more slowly and preferring narrower image distributions, relative to higher-level brain areas. Brain-optimized inference thus offers a succinct and novel method for improving reconstructions and exploring the diversity of representations across visual brain areas.
Speaker:
Reese Kneeland (University of Minnesota)
Paper link:
https://arxiv.org/abs/2312.07705
To access the Zoom link you can add the event to your Calendar (Google Calendar | iCal | View in browser)
Or here's the direct Zoom link: https://princeton.zoom.us/j/97664762824
Best,
Paul
Brain-optimized inference improves reconstructions of fMRI brain activity
Abstract:
The release of large datasets and developments in AI have led to dramatic improvements in decoding methods that reconstruct seen images from human brain activity. We evaluate the prospect of further improving recent decoding methods by optimizing for consistency between reconstructions and brain activity during inference. We sample seed reconstructions from a base decoding method, then iteratively refine these reconstructions using a brain-optimized encoding model that maps images to brain activity. At each iteration, we sample a small library of images from an image distribution (a diffusion model) conditioned on a seed reconstruction from the previous iteration. We select those that best approximate the measured brain activity when passed through our encoding model, and use these images for structural guidance during the generation of the small library in the next iteration. We reduce the stochasticity of the image distribution at each iteration, and stop when a criterion on the "width" of the image distribution is met. We show that when this process is applied to recent decoding methods, it outperforms the base decoding method as measured by human raters, a variety of image feature metrics, and alignment to brain activity. These results demonstrate that reconstruction quality can be significantly improved by explicitly aligning decoding distributions to brain activity distributions, even when the seed reconstruction is output from a state-of-the-art decoding algorithm. Interestingly, the rate of refinement varies systematically across visual cortex, with earlier visual areas generally converging more slowly and preferring narrower image distributions, relative to higher-level brain areas. Brain-optimized inference thus offers a succinct and novel method for improving reconstructions and exploring the diversity of representations across visual brain areas.
Speaker:
Reese Kneeland (University of Minnesota)
Paper link:
https://arxiv.org/abs/2312.07705
To access the Zoom link you can add the event to your Calendar (Google Calendar | iCal | View in browser)
Or here's the direct Zoom link: https://princeton.zoom.us/j/97664762824
Best,
Paul
Reply all
Reply to author
Forward
0 new messages