ST-4DGS: Spatial-Temporally Consistent 4D Gaussian Splatting for Efficient Dynamic Scene Rendering

Dynamic scene rendering at any novel views continues to be a difficult but important task, especially for high-fidelity rendering quality at real-time rendering speed. The recent 3D Gaussian Splatting, i.e., 3DGS, shows great success for static scene rendering with impressive quality at very efficient speed, however, the extension of 3DGS from static scene to dynamic as 4DGS is still challenging, especially to maintain the spatial-temporally persistent dynamic rendering quality. This paper proposes a novel spatial-temporally consistent 4D Gaussian Splatting, i.e., ST-4DGS, for real-time dynamic scene rendering which especially aims at the spatial-temporally persistent dynamic rendering quality and maintain the real-time efficiency. The key ideas of ST-4DGS are two new novel mechanisms: (1) a spatial-temporal encoder for 4D Gaussians representations but with a motion-aware shape regularization, and (2) a spatial-temporally joint density control, which are very effective to prevent the compactness degeneration for the 4D Gaussians representation during the dynamic scene learning, thus leading to spatial-temporally consistent dynamic rendering quality while in efficient manner. With extensive evaluation on public datasets, our ST-4DGS can achieve much better dynamic rendering quality than previous dynamic rendering approaches, such as 4DGS, HexPlane, K-Plane etc, and in a more efficient way for real-time dynamic rendering. To our best knowledge, we are the first 4D Gaussian Splatting for high-fidelity dynamic rendering at real-time speed, especially ensuring the spatial-temporally consistent rendering quality.

Download paper here

Recommended citation: Deqi Li, Shi-Sheng Huang, Zhiyuan Lu, Xinran Duan, and Hua Huang. 2024. ST-4DGS: Spatial-Temporally Consistent 4D Gaussian Splatting for Efficient Dynamic Scene Rendering. In ACM SIGGRAPH 2024 Conference Papers (SIGGRAPH ‘24). Association for Computing Machinery, New York, NY, USA, Article 83, 1–11. https://doi.org/10.1145/3641519.3657520