Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2026 Mar 5;9(1):314.
doi: 10.1038/s41746-026-02496-7.

Shifting the retinal foundation models paradigm from slices to volumes for optical coherence tomography

Raphael Judkiewicz  1 , Eran Berkowitz  2   3   4 , Meishar Meisel  2 , Tomer Michaeli  5 , Joachim A Behar  6
Affiliations

Shifting the retinal foundation models paradigm from slices to volumes for optical coherence tomography

Raphael Judkiewicz et al. NPJ Digit Med. .

Abstract

Optical Coherence Tomography (OCT) is essential in ophthalmology for cross-sectional imaging of the retina. Pretrained foundation models facilitate task-specific model development by enabling fine-tuning with limited labeled data. However, current foundation models rely on a single B-scan (usually the central slice), overlooking volumetric context. This research investigates video foundation models to capture full 3D retinal structure and improve diagnostic performance. V-JEPA, a state-of-the-art video foundation model, was benchmarked against retinal foundation models (RETFound, VisionFM) and a natural image foundation model (DINOv2). All were fine-tuned to detect Age-related Macular Degeneration or Glaucomatous Optic Neuropathy using five OCT datasets. V-JEPA consistently equaled or outperformed image-based models, achieving an average AUROC of 0.94 (0.80-0.99), versus 0.90 (0.76-0.98) for the best image model, a statistically significant improvement (p < 0.001). To our knowledge, this is the first application of transformer-based video models to volumetric OCT, highlighting their promise in 3D medical imaging.

PubMed Disclaimer

Conflict of interest statement

Competing interests: The authors declare no competing interests.

Figures

Fig. 1
Fig. 1. Overview of the research.
a Geographic distribution of the datasets used in the experiments. b Comparison between video and image foundation models for OCT analysis. c Key results: Left: Performance benchmark of video, general image, and retinal image foundation models. Middle: Model performance as a function of training set size. Right: Visualization of attention maps from the video foundation model across the full OCT volume. BioRender.com was used to produce this figure.
Fig. 2
Fig. 2. Results.
a Benchmark of state-of-the-art transformer based image (DINOv2, RETFound, VisionFM) and video (V-JEPA) models on the five datasets. Each bar represents the mean value and error bars indicate the minimum and maximum AUROC computed for each test fold. P-values from pairwise DeLong tests comparing each model to V-JEPA, corrected using the Holm-Bonferroni method, are shown above the bars. b Performance as a function of training set size. This experiment is performed on the largest dataset, namely CirrusOCT.
Fig. 3
Fig. 3. Explainability.
a Attention maps from V-JEPA across multiple B-scans of an AMD OCT, illustrating how the model dynamically tracks relevant anatomical features over the full volume of the retina. b Comparison of attention of V-JEPA and DINOv2 on the same middle slice (Slice 8) shows that V-JEPA focuses more precisely on clinically relevant structures compared to a single-frame image-based model, highlighting the benefit of spatial context in volumetric interpretation.
Fig. 4
Fig. 4. Distance in slices between the middle slice and the closest one showing signs of pathology (Drusen or CNV).
Showed for 2 groups, DINOv2 true positive, and the intersection between DINOv2 false negative and V-JEPA true positive. True positives and false negatives were found using threshold maximizing F1 score.
Fig. 5
Fig. 5. Analysis of performance and computational cost for different volumetric methods.
a Benchmarking V-JEPA against a 3D CNN trained from scratch and DINOv2 adapted to a 3D setting. b Influence of input slice count on V-JEPA performance. c Computational footprint of the five models, tokens, GFLOPs, latency, and peak GPU memory for a single forward pass.
See this image and copyright information in PMC

References

    1. World Health Organization. World report on vision. https://www.who.int/publications/i/item/world-report-on-vision (2019).
    1. Poplin, R. et al. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng.2, 158–164 (2018). - DOI - PubMed
    1. Ting, D. S. W. et al. Artificial intelligence and deep learning in ophthalmology. Br. J. Ophthalmol.103, 167–175 (2019). - DOI - PMC - PubMed
    1. Cheung, C. Y. et al. A deep learning model for detection of alzheimer’s disease based on retinal photographs: a retrospective, multicentre case-control study. Lancet Digit. Health4, E806–E815 (2022). - DOI - PubMed
    1. Srivastava, O., Tennant, M., Grewal, P., Rubin, U. & Seamone, M. Artificial intelligence and machine learning in ophthalmology: a review. Indian J. Ophthalmol.71, 11–17 (2023). - DOI - PMC - PubMed

LinkOut - more resources