Sathira Silva*, Savindu Wannigama*, Prof. Roshan Ragel
$\dagger$ , Gihan Jayatilaka$\ddagger$
* Equal contribution
- [25/01/2024]: Paper (preprint) released in arxiv
- [04/01/2024]: Paper (preprint) released in ResearchGate
Temporal reasoning holds equal importance to spatial reasoning in a cognitive perception system. In human perception, temporal information is crucial for identifying occluded objects and determining the motion state of entities. A system proficient in spatiotemporal reasoning excels in making inferences with high temporal coherence. While previous works emphasize the significance of temporal fusion in 3D object detection, earlier attempts at 3D Semantic Occupancy Prediction (3D SOP) often overlooked the value of incorporating temporal information. The current state-of-the-art in 3D SOP literature seldom exploits temporal cues. This is evident in TPVFormer’s SOP visualizations, where adjacent prediction frames lack temporal coherence as they rely solely on the current time step for semantic predictions.
This work introduces S2TPVFormer, a variant of TPVFormer, which utilizes a spatiotemporal transformer architecture inspired by BEVFormer, for dense and temporally coherent 3D semantic occupancy prediction. Leveraging TPV (Tri-Perspective View) representation, the model’s spatiotemporal encoder generates temporally rich embeddings, fostering coherent predictions. The study proposes a novel Temporal Cross-View Hybrid Attention mechanism, enabling the exchange of spatiotemporal information across different views. To illustrate the efficacy of temporal information incorporation and the potential of the new attention mechanism, the research explores three distinct temporal fusion paradigms.
To summarize, this work contributes in the following ways,
- We pioneer the use of TPV representation for embedding spatiotemporal information in 3D scenes within the domain of vision-centric SOP and the broader 3D perception literature.
- We introduce a novel temporal fusion workflow for TPV representation, analyzing how CVHA facilitates the sharing of spatiotemporal information across the three planes.
- The lower parameter model of our method achieves a significant 3.1% improvement in mIoU for 3D SOP when evaluated on the nuScenes validation dataset with TPVFormer’s sparse pseudo-voxel ground truth, compared to TPVFormer.
Our code is based on TPVFormer and BEVFormer. Many thanks to them!
- E/17/331 - SILVA H.S.C. - [email protected]
- E/17/369 - WANNIGAMA S.B. - [email protected]
If this work is helpful for your research, please consider citing:
@article{silva2024s2tpvformer,
title={S2TPVFormer: Spatio-Temporal Tri-Perspective View for temporally coherent 3D Semantic Occupancy Prediction},
author={Sathira Silva and Savindu Bhashitha Wannigama and Roshan Ragel and Gihan Jayatilaka},
journal={arXiv:2401.13785},
year={2024},
}