Reconstructing Images from Brain Activity

Four extensions to MindEye2, probing what actually helps when rebuilding seen images from fMRI.

MindEye2 reconstructs the image a person is looking at from their fMRI brain activity. For the Advanced Deep Learning seminar at Leiden University, our group of four took the published system of Scotti and colleagues as a starting point and ran various experiments to determine what actually benefits performance, and what just trades one kind of quality for another. We built four extensions and measured each against a matched baseline on a single subject.

Our four extensions on top of MindEye2.

Links · 💻 Code · 🤗 Checkpoints · 📄 Paper

The four extensions

  1. A semantic auxiliary loss that pushes the model toward the right meaning during training.
  2. Two brain optimised inference variants that score candidate reconstructions in the model’s own latent space rather than in voxel space.
  3. A systematic sweep of both diffusion start timepoints, which trades low level pixel accuracy against high level meaning.
  4. A caption ablation that swaps the refinement caption for a vision language model under several prompting schemes.

What we found

The honest headline is that most of these do not simply make reconstruction better. The semantic loss helps retrieval but hurts several pixel level metrics. The inference variants raise brain correlation at the cost of perceptual quality. The timepoint sweep and the caption study expose clear trade offs rather than clean wins. It reads less like a leaderboard climb and more like a careful look at where the gains in a system like this really come from, and what they cost. Everything is on one subject and one hour of data, which the paper is upfront about.

The diffusion timepoint sweep. The same six Subject 2 test images reconstructed at each SDXL refinement start point, from a low one that stays close to the brain derived structure up to a high one that lets the diffusion prior take over.