Pseudo-Simulation for Autonomous Driving

Paper | Leaderboard v2 | Warmup Leaderboard v2 | Challenge Results (CVPR Phase)

Pseudo-Simulation for Autonomous Driving

Wei Cao^3,5, Marcel Hallgarten^1,3,6, Tianyu Li⁴, Daniel Dauner¹, Xunjiang Gu⁶, Caojun Wang⁴, Yakov Miron³,
Marco Aiello⁵, Hongyang Li⁴, Igor Gilitschenski^6,7, Boris Ivanovic², Marco Pavone^2,8, Andreas Geiger¹, and Kashyap Chitta^1,2

¹University of Tübingen, Tübingen AI Center, ²NVIDIA Research, ³Robert Bosch GmbH
⁴OpenDriveLab at Shanghai Innovation Institute, ⁵University of Stuttgart, ⁶University of Toronto, ⁷Vector Institute, ⁸Stanford University

The main branch contains the code for NAVSIM v2, used in the 2025 NAVSIM challenge. For NAVSIM v1, as well as its navtest leaderboard, which are also part of this repository, please check the v1.1 branch.

NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

Daniel Dauner^1,2, Marcel Hallgarten^1,5, Tianyu Li³, Xinshuo Weng⁴, Zhiyu Huang^4,6, Zetong Yang³,
Hongyang Li³, Igor Gilitschenski^7,8, Boris Ivanovic⁴, Marco Pavone^4,9, Andreas Geiger^1,2, and Kashyap Chitta^1,2

¹University of Tübingen, ²Tübingen AI Center, ³OpenDriveLab at Shanghai AI Lab, ⁴NVIDIA Research
⁵Robert Bosch GmbH, ⁶Nanyang Technological University, ⁷University of Toronto, ⁸Vector Institute, ⁹Stanford University

Advances in Neural Information Processing Systems (NeurIPS), 2024
Track on Datasets and Benchmarks

Highlights

🚀 TL;DR: We introduce Pseudo-Simulation, a novel AV evaluation methodology that combines the efficiency of open-loop evaluation with the robustness of closed-loop evaluation. By augmenting real data with synthetic observations near the planned trajectory, pseudo-simulation achieves strong correlation with closed-loop simulation while being much faster and easier to scale.

🤔 Motivation: Current AV evaluation methods face critical trade-offs: closed-loop simulation is resource-intensive and requires model access rather than just model predictions, while open-loop evaluation overlooks important factors such as error recovery and behavior deviation from the expert path. An evaluation paradigm bridging the gap is required for large-scale, rapid validation.

🏆 Highlights: Pseudo-simulation achieves a strong correlation with traditional, computationally expensive closed-loop simulations while requiring 6x less compute. Unlike traditional closed-loop simulation, pseudo-simulation is neither sequential nor interactive, enabling the open-loop computation of all evaluation metrics in our leaderboard. It will serve as the primary evaluation framework for the AGC2025 NAVSIM End-to-End Driving Challenge.

Table of Contents

1. Highlights
2. Getting started
3. Changelog
4. License and citation
5. Other resources

Getting started

• Download and installation
• Understanding and creating agents
• Understanding the data format and classes
• Dataset splits vs. filtered training / test splits
• Understanding the Extended PDM Score
• Understanding the traffic simulation
• Submitting to the Leaderboard

(back to top)

Changelog

• [2025/04/28] NAVSIM v2.2 release (official devkit version for AGC 2025)
  • Release of private_test_hard dataset (see splits) for the HuggingFace NAVSIM v2 End-to-End Driving Challenge 2025 Leaderboard.
    • The submission deadline is 2025-05-11 00:00:00 UTC
    • You are limited to one upload per day on the challenge leaderboard, which should take approximately 2 hours to evaluate after a succesful submission.
  • Fixed bug in openscene_meta_datas for navhard and warmup
    • If you used navhard_two_stage/openscene_meta_datas or warmup_two_stage/openscene_meta_datas to evaluate your model, please re-download and use the new data.
  • ⚠️ IMPORTANT: Using the test/navtest/navhard_two_stage/warmup_two_stage/private_test_two_stage splits for training your challenge submissions is not allowed.
    • Using any other publicly available datasets or pretrained weights is allowed.
    • Furthermore, to be eligible for awards, the use of data must be described explicitly in the technical report for your submission.
• [2025/04/24] NAVSIM v2.1.2 release
  • Release of navhard_two_stage dataset (see splits)
  • Updated Extended Predictive Driver Model Score (EPDMS) for the Hugging Face Warmup leaderboard. See see metrics for details regarding the implementation.
• [2025/04/13] NAVSIM v2.1.1 release
  • Updated dataset for the warmup leaderboard with minor fixes
• [2025/04/08] NAVSIM v2.1 release
  • Added new dataset for the Hugging Face Warmup leaderboard (see submission)
  • Introduced support for two-stage reactive traffic agents (see traffic simulation)
• [2025/02/28] NAVSIM v2.0 release
  • Extends the PDM Score with more metrics and penalties (see metrics)
  • Adds a new two-stage pseudo closed-loop simulation (see metrics)
  • Adds support for reactive traffic agent policies (see traffic simulation)
• [2024/09/03] NAVSIM v1.1 release
  • Leaderboard for navtest on Hugging Face
  • Release of baseline checkpoints on Hugging Face
  • Updated docs for submission and paper
• [2024/04/21] NAVSIM v1.0 release (official devkit version for AGC 2024)
  • Parallelization of metric caching / evaluation
  • Adds Transfuser baseline (see agents)
  • Adds standardized training and test filtered splits (see splits)
  • Visualization tools (see tutorial_visualization.ipynb)
• [2024/04/03] NAVSIM v0.4 release
  • Support for test phase frames of competition
  • Download script for trainval
  • Egostatus MLP Agent and training pipeline
• [2024/03/25] NAVSIM v0.3 release
  • Adds code for Leaderboard submission
• [2024/03/11] NAVSIM v0.2 release
  • Easier installation and download
  • mini and test data split integration
  • Privileged Human agent
• [2024/02/20] NAVSIM v0.1 release (initial demo)
  • OpenScene-mini sensor blobs and annotation logs
  • Naive ConstantVelocity agent

(back to top)

License and citation

All assets and code in this repository are under the Apache 2.0 license unless specified otherwise. The datasets (including nuPlan and OpenScene) inherit their own distribution licenses. Please consider citing our papers if they help your research.

@article{Cao2025ARXIV,
	title={Pseudo-Simulation for Autonomous Driving}, 
        author={Wei Cao and Marcel Hallgarten and Tianyu Li and Daniel Dauner and Xunjiang Gu and Caojun Wang and Yakov Miron and Marco Aiello and Hongyang Li and Igor Gilitschenski and Boris Ivanovic and Marco Pavone and Andreas Geiger and Kashyap Chitta},
	journal = {arXiv},
    	volume = {2506.04218},
	year = {2025},
}

@inproceedings{Dauner2024NEURIPS,
	title = {NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking},
	author = {Daniel Dauner and Marcel Hallgarten and Tianyu Li and Xinshuo Weng and Zhiyu Huang and Zetong Yang and Hongyang Li and Igor Gilitschenski and Boris Ivanovic and Marco Pavone and Andreas Geiger and Kashyap Chitta},
	booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
	year = {2024},
}

(back to top)

Other resources

• SLEDGE | tuPlan garage | CARLA garage | Survey on E2EAD
• PlanT | KING | TransFuser | NEAT

(back to top)