TrackingWorld: World-centric Monocular 3D Tracking

👥 Authors and Affiliations

Jiahao Lu¹, Weitao Xiong^1,5, Jiacheng Deng², Peng Li¹, Tianyu Huang³, Zhiyang Dou⁴, Cheng Lin⁶, Sai-Kit Yeung¹, Yuan Liu^1†

¹ The Hong Kong University of Science and Technology ² University of Science and Technology of China

³ The Chinese University of Hong Kong ⁴ The University of Hong Kong ⁵ Xiamen University

⁶ Macau University of Science and Technology

📝 Abstract

TrackingWorld is a novel approach for dense, world-centric 3D tracking from monocular videos. Our method estimates accurate camera poses and disentangles 3D trajectories of both static and dynamic components — not limited to a single foreground object. It supports dense tracking of nearly all pixels, enabling robust 3D scene understanding from monocular inputs.

🌍 World-Centric Pose

Estimates accurate camera poses for consistent 3D world coordinate system anchoring.

🔄 Disentangled Trajectories

Separates 3D motion for static background and dynamic foreground components.

👀 Dense Pixel Coverage

Supports tracking of nearly all pixels, moving beyond sparse keypoints.

🧩 Pipeline and Methodology

Overview

Dense 2D Trajectory Generation
From a monocular video sequence, TrackingWorld first extracts dense 2D point trajectories that track both static background structures and newly emerging dynamic objects.
Optimization-based 4D Reconstruction
The 2D trajectories are lifted into a world-centric 3D space through a unified optimization framework consisting of:
- Initial Camera Pose Estimation
  We estimate coarse camera poses at the clip level, providing an essential initialization for subsequent refinement and 3D trajectory lifting.
- Dynamic Background Refinement
  Potentially dynamic regions are identified and filtered out, enabling accurate pose optimization using stable background correspondences.
- World-centric 3D Trajectory Reconstruction
  With refined poses and dense correspondences, we reconstruct dense 3D trajectories that capture the motion of both static scene elements and dynamic objects.

Figure: Overview of TrackingWorld Framework.

🎥 Teaser and Main Results

TrackingWorld provides dense, world-centric 3D trajectories for almost every visible pixel, enabling complete 4D scene reconstruction with high accuracy and robustness.

Figure: TrackingWorld 3D reconstruction and 2D/3D tracking demonstration.

💻 Interactive 3D Demos (Click to select scene)

📄 Citation

If you find TrackingWorld useful for your research or applications, please consider citing our paper:

@inproceedings{
    lu2025trackingworld,
    title={TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels},
    author={Jiahao Lu and Weitao Xiong and Jiacheng Deng and Peng Li and Tianyu Huang and Zhiyang Dou and Cheng Lin and Sai-Kit Yeung and Yuan Liu},
    booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
    year={2025},
    url={https://openreview.net/forum?id=vDV912fa3t}
}