Mind-Video is an AI tool that aims to reconstruct high-quality videos from brain activity captured through continuous functional magnetic resonance imaging (fMRI) data. It is an extension of the previous fMRI-Image reconstruction work called Mind-Vis. The tool addresses the challenge of recovering continuous visual experiences in the form of videos from non-invasive brain recordings.Mind-Video employs a two-module pipeline that bridges the gap between image and video brain decoding. The first module focuses on learning general visual fMRI features through large-scale unsupervised learning with masked brain modeling and spatiotemporal attention. It then distills semantic-related features using multimodal contrastive learning with an annotated dataset.In the second module, the learned features are fine-tuned through co-training with an augmented stable diffusion model, specifically designed for video generation guided by fMRI data.The tool’s contribution lies in its flexible and adaptable pipeline, which consists of an fMRI encoder and an augmented stable diffusion model trained separately and finetuned together. It employs a progressive learning scheme that enables the encoder to learn brain features through multiple stages. The resulting videos demonstrate high semantic accuracy, including motions and scene dynamics, outperforming previous state-of-the-art approaches.Attention analysis of the transformers decoding fMRI data reveals the dominance of the visual cortex in processing visual spatiotemporal information and the hierarchical nature of the encoder’s layers in extracting structural and abstract visual features. The fMRI encoder also shows progressive improvement in assimilating more nuanced semantic information throughout its training stages.Mind-Video utilizes data from the Human Connectome Project and acknowledges the contributions of collaborators and supporters in the development of the tool.
