Trends in NeuroAI - Meta's MEG-to-image reconstruction
112 views
Skip to first unread message
Paul Scotti
Nov 30, 2023, 2:58:13 PM11/30/23
to MedARC Neuroimaging & AI
Next reading group will be Thursday, Dec 7 11:00am ET!
Title:
Brain decoding: toward real-time reconstruction of visual perception
Abstract:
In the past five years, the use of generative and foundational AI systems has greatly improved the decoding of brain activity. Visual perception, in particular, can now be decoded from functional Magnetic Resonance Imaging (fMRI) with remarkable fidelity. This neuroimaging technique, however, suffers from a limited temporal resolution (≈0.5 Hz) and thus fundamentally constrains its real-time usage. Here, we propose an alternative approach based on magnetoencephalography (MEG), a neuroimaging device capable of measuring brain activity with high temporal resolution (≈5,000 Hz). For this, we develop an MEG decoding model trained with both contrastive and regression objectives and consisting of three modules: i) pretrained embeddings obtained from the image, ii) an MEG module trained end-to-end and iii) a pretrained image generator. Our results are threefold: Firstly, our MEG decoder shows a 7X improvement of image-retrieval over classic linear decoders. Second, late brain responses to images are best decoded with DINOv2, a recent foundational image model. Third, image retrievals and generations both suggest that MEG signals primarily contain high-level visual features, whereas the same approach applied to 7T fMRI also recovers low-level features. Overall, these results provide an important step towards the decoding - in real time - of the visual processes continuously unfolding within the human brain.
Speaker:
Dr. Paul Scotti (Stability AI, MedARC)
Paper link:
https://arxiv.org/abs/2310.19812
Meta's blog post about the paper:
This will be an informal journal club where we discuss Meta's MEG-to-image paper "Brain decoding: toward real-time reconstruction of visual perception". I will lead discussion walking us through the paper.
Reconstructing seen images from MEG, rather than fMRI, is a pretty big deal because MEG has worse spatial (but improved temporal) resolution compared to fMRI. It's kind of a middle-ground between EEG and fMRI. That means high-quality image reconstruction is probably harder with MEG than fMRI, but if you can get it to work, it means you could ultimately do stuff like video reconstruction. Or maybe even have a real-time demo where you reconstruct images near-instantaneously like SDXL Turbo. Now despite the title of this paper, they aren't doing any real-time reconstruction here. But the potential is there.
Reconstructing seen images from MEG, rather than fMRI, is a pretty big deal because MEG has worse spatial (but improved temporal) resolution compared to fMRI. It's kind of a middle-ground between EEG and fMRI. That means high-quality image reconstruction is probably harder with MEG than fMRI, but if you can get it to work, it means you could ultimately do stuff like video reconstruction. Or maybe even have a real-time demo where you reconstruct images near-instantaneously like SDXL Turbo. Now despite the title of this paper, they aren't doing any real-time reconstruction here. But the potential is there.
Title:
Brain decoding: toward real-time reconstruction of visual perception
Abstract:
In the past five years, the use of generative and foundational AI systems has greatly improved the decoding of brain activity. Visual perception, in particular, can now be decoded from functional Magnetic Resonance Imaging (fMRI) with remarkable fidelity. This neuroimaging technique, however, suffers from a limited temporal resolution (≈0.5 Hz) and thus fundamentally constrains its real-time usage. Here, we propose an alternative approach based on magnetoencephalography (MEG), a neuroimaging device capable of measuring brain activity with high temporal resolution (≈5,000 Hz). For this, we develop an MEG decoding model trained with both contrastive and regression objectives and consisting of three modules: i) pretrained embeddings obtained from the image, ii) an MEG module trained end-to-end and iii) a pretrained image generator. Our results are threefold: Firstly, our MEG decoder shows a 7X improvement of image-retrieval over classic linear decoders. Second, late brain responses to images are best decoded with DINOv2, a recent foundational image model. Third, image retrievals and generations both suggest that MEG signals primarily contain high-level visual features, whereas the same approach applied to 7T fMRI also recovers low-level features. Overall, these results provide an important step towards the decoding - in real time - of the visual processes continuously unfolding within the human brain.
Speaker:
Dr. Paul Scotti (Stability AI, MedARC)
Paper link:
https://arxiv.org/abs/2310.19812
Meta's blog post about the paper:
https://ai.meta.com/blog/brain-ai-image-decoding-meg-magnetoencephalography/
To access the Zoom link you can add the event to your Calendar (Google Calendar | iCal | View in browser)
Or here's the direct Zoom link: https://princeton.zoom.us/j/97664762824
Best,
Paul
To access the Zoom link you can add the event to your Calendar (Google Calendar | iCal | View in browser)
Or here's the direct Zoom link: https://princeton.zoom.us/j/97664762824
Best,
Paul
Reply all
Reply to author
Forward
0 new messages