Natural vision

Decoding Scenes from V1 During Natural Vision

Reconstructing static natural scenes from marmoset V1 activity during free viewing.

Authors Tejasvi Kothapalli¹, Bruno Olshausen¹, Jacob Yates¹

Affiliation ¹ UC Berkeley Redwood Center for Theoretical Neuroscience

Published Aug 15, 2026

Tags Decoding, Denoising, Natural Scenes, Transformers, 1/f², Free-viewing, V1

Abstract

To our knowledge, decoding a full static natural scene from V1 activity during natural vision has never been done. Here, we present two complementary methods for reconstructing a static scene from V1 activity while a marmoset freely views it. We record neural activity from marmoset foveal V1. Due to retinotopy and the recording location, all recorded cells respond to a ~0.5° × 0.5° region of visual space near the center of gaze. Our first, principled two-stage method uses ridge regression to predict this gaze-contingent patch at every fixation. Eye position places each decoded patch in scene coordinates; patches accumulated across fixations and sessions are then combined with a 1/f prior to reconstruct the full scene. Our second method uses state-of-the-art deep learning to build a foundation decoder: a transformer consumes streamlined 33 ms neural tokens and directly reconstructs the scene, without explicit fixation detection, saccade excision, or hand-tuned event rules. The interactive visualization compares both methods for each animal. Every reconstruction is fully held out—the decoder was never trained on the scene it reconstructs.

Animal

Each animal has its own sessions, decoder and reconstructions.

Scenes

Loading scenes… Select a scene, then scroll down to watch its reconstruction unfold.

Selected scene

Loading…

Total —

Current — Presentation — — cells

Click a point in the original scene to inspect the separate local patch decoder.
←→ step through fixations · hold to accelerate

Static natural scene Hull 0.35° Decoded patch Eye position

Click to inspect a patch

34.1° 19.2°

Decoded patches

1/f reconstruction Contrast restored

End-to-end transformer reconstruction 33 ms neural tokens + gaze position → full scene directly No patch decoder · No fixation segmentation · No 1/f reconstruction 33 ms tiled tokens

Fixation0 / 0 Viewing time0:00 / 0:00 Speed

Loading scene data…

Acknowledgments

Thanks Eero Simoncelli for inspiring me to do this project at CSHL and for incredibly helpful conversations about how to implement this.

Thanks Bruno Olshausen for suggesting a 1/f prior, which was crucial for cleaning up the reconstruction.

Thanks Jacob Yates for the infinite wisdom and support, as always.

Thanks to my best friend Ashwin Nathan for giving me access to Codex.