Modern Recurrent Neural Networks have become a competitive architecture for 3D reconstruction due to their linear-time complexity. However, their performance degrades significantly when applied beyond the training context length, revealing limited length generalization. In this work, we revisit the 3D reconstruction foundation models from a Test-Time Training perspective, framing their designs as an online learning problem. Building on this perspective, we leverage the alignment confidence between the memory state and incoming observations to derive a closed-form learning rate for memory updates, to balance between retaining historical information and adapting to new observations. This training-free intervention, termed TTT3R, substantially improves length generalization, achieving a 2\( \times \) improvement in global pose estimation over baselines, all while operating at 20 FPS with just 6 GB of GPU memory to process thousands of images.
TTT3R improves length generalization and mitigates forgetting. For sequences beyond 1000 frames, we incorporate TTT3R with a state reset process, please see Sec. A.1 of Sup.Mat. for details.
Real-world applications often require handling an arbitrary number of images.
Recent feed-forward methods (e.g., VGGT, Point3R) suffer from high memory consumption. Notably, only CUT3R achieves constant memory usage with RNN design. However, as illustrated above, CUT3R fails to generalize to long sequences due to training on most 64-frame sequences.
To address forgetting, we retain the cross-attention formulation but introduce a per-token learning rate $\beta_t$, derived from alignment confidence between state and observations. This acts as a soft gate, improving long-context extrapolation.
Instead of updating all states uniformly, we incorporate image attention (i.e., $\mathbf{Q}_{\mathbf{S}_{t-1}} {\mathbf{K}^{\top}_{\mathbf{X}_t}}\in\mathbb{R}^{n\times(h\times w)}$) as per-token learning rates $\beta_t\in \mathbb{R}^{n \times 1}$. High-confidence matches get larger updates, while low-quality updates are suppressed for better performance.
TTT3R: 3D Reconstruction as Test-Time Training
@article{chen2025ttt3r,
title={TTT3R: 3D Reconstruction as Test-Time Training},
author={Chen, Xingyu and Chen, Yue and Xiu, Yuliang and Geiger, Andreas and Chen, Anpei},
journal={arXiv preprint arXiv:2509.26645},
year={2025}
}