PAPER

Mind‑to‑Image is an end‑to‑end pipeline developed at Obvious which translates the brain’s fMRI signals into pictures a person is merely imagining. While earlier projects could only rebuild what someone was looking at, here the participants were asked to picture scenes in their minds, and we collected six hours of brain‑scan data, and worked on the basis of a top‑tier brain‑to‑image system to build one which could handle the richer signals of pure imagination. The result is a first demonstration that both remembered scenes and purely invented scenes can be turned into recognizable portraits or landscapes straight from neural data.

Mind-to-Image

We collected six hours of brain‑scans, and worked on the basis of a top‑tier brain‑to‑image system to build one which could handle the richer signals of pure imagination. The result is a demonstration that remembered and purely invented scenes can be turned into visuals straight from neural data.

Mind‑to‑Image is an end‑to‑end pipeline developed at Obvious which translates the brain’s fMRI signals into pictures a person is merely imagining. While earlier projects could only rebuild what someone was looking at, here the participants were asked to picture scenes in their minds.

An introduction to Mind-to-Image

Once this mapping was established, it could be applied to the imagination phase, allowing the reconstruction of scenes that existed only in thought.

The model was trained using several hours of fMRI data collected while an artist first viewed, then remembered, and finally imagined images inside the scanner. From the first two phases, it learned to associate distinct patterns of brain activation with visual features such as shape and composition.

A 3-step process for reconstructing imagination

The picture is meant to correspond to the imagined content: whether a portrait, a landscape, or something more abstract. In this way, the system turns purely mental images into visible ones, offering a direct glimpse of imagination itself.

When a participant lies in the MRI and imagines a scene, Mind-to-Image interprets the resulting pattern of brain activity and converts it into a “visual fingerprint.” That fingerprint guides an image generator to produce a picture.

Image generation based on brain scan analysis

The mind-to-image protocol inscribes itself as a scientific follow up of techniques such as hypnosis, automatic writing and exquisite corpse, allowing creation to emerge from the mind and be given a physical form.

This technique inscribes itself as a following of the surrealist movement. Surrealism is a cultural movement which explores the expression of the subconscious, and the techniques which allow for the suppression of the different filters between the realm of the mind, and physical artistic expression.

The extension of the surrealist movement

Projecting Visual Mental Imagination of the Brain from fMRI

OnlyFlow

Traditional approaches struggle to combine an arbitrary motion with an unrelated prompt, while OnlyFlow tackles this challenge by using optical flow as a conditioning signal.

OnlyFlow is a method developed by Obvious to improve control over video diffusion models. Its goal is to generate a video that follows both a text prompt and a desired movement, such as a camera pan or object motion.

Motion Conditioning for Video Diffusion Models

Schema OnlyFlow1

The pipeline works by first extracting optical flow from an auxiliary video, which describes how pixels move between frames. This motion is encoded through a dedicated motion encoder, and the resulting features are injected into the temporal attention layers of the video diffusion model.

Optical-Flow-Guided Motion

The continuity in the movement used to generate the videos create a global coherence, and the artwork acts both as a proof of work, and a visual demonstration of what can be done using OnlyFlow.

“Be Water”, an artwork created using OnlyFlow, was showcased in an immersive space at Artechouse NYC and at Kanvas Dubaï, highlighting the creative potential of this tool. The video artwork is composed of four independently generated videos, each one of them displayed on a wall.

Research

PAPER