SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries

Wei-Jer Chang1, Francesco Pittaluga2, Masayoshi Tomizuka1, Wei Zhan1, Manmohan Chandraker2,3
1University of California Berkeley, 2NEC Labs America, 3University of California San Diego
Paper arXiv Code
ECCV 2024

We will realease the inference code soon.

Abstract

Evaluating the performance of autonomous vehicle planning algorithms necessitates simulating long-tail safety-critical traffic scenarios. However, traditional methods for generating such scenarios often fall short in terms of controllability and realism; they also neglect the dynamics of agent interactions. To address these limitations, we introduce SAFE-SIM, a novel diffusion-based controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We develop a novel approach to simulate safety-critical scenarios through an adversarial term in the denoising process of diffusion models, which allows an adversarial agent to challenge a planner with plausible maneuvers while all agents in the scene exhibit reactive and realistic behaviors. Furthermore, we propose novel guidance objectives and a partial diffusion process that enables users to control key aspects of the scenarios, such as the collision type and aggressiveness of the adversarial agent, while maintaining the realism of the behavior. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability. These findings affirm that diffusion models provide a robust and versatile foundation for safety-critical, interactive traffic simulation, extending their utility across the broader autonomous driving landscape.

Video

Overall Framework

Figure

SAFE-SIM evaluates a planner within scenarios featuring multiple controllable reactive agents. These agents have two distinct roles:adversarial agents, which actively challenge the planner by exhibiting controllable adversarial behaviors such as specific collision types and levels of aggressiveness, and non-adversarial agents, which follow normal driving behavior to maintain the realism of the entire scene.

How did we achieve this?

Figure

With recent advances in trajectory diffusion models, we introduce a novel approach that incorporates an adversarial term in the denoising process. This allows an adv agent to challenge the planner with controllable adversarial behaviors. This process optimizes the adversarial agent’s trajectory using the adversarial cost function \(J_{\text{adv}}\) to influence the ego vehicle, while \(J_{\text{reg}}\) is used for maintaining realism.

Controllability on Time-to-Collision (TTC)

By varying the level of \(J_{\text{control}}\), we are able to vary the adversarial behavior within the same scenario to challenge the planner.

Partial Diffusion: User Control Flexibility

Figure

Partial Diffusion process enables users to control key aspects of the scenarios. During the denoising process, instead of starting from pure noise, we begin with trajectories specified by humans or heuristics to incorporate human knowledge. Noise is then added and subsequently denoised. Users can adjust noise levels to balance between user control and the model’s data distribution. We generate proposals based on domain knowledge, such as collision types.

Figure

Partial Diffusion: Control Collision Types

Legend

Base on different trajectory proposals, we can control the collision types in the scene.

We can also control the distance to conflict point to generate different interactive scenarios for the planner evaluation.

Partial Diffusion: Control Distance to Conflict Point

Key Findings

  • Controllable Adversarial Behavior: SAFE-SIM enables the multiple controllable factors, allowing for diverse scenario variations within the same scenario.
  • Partial Diffusion for Explicit Control: Constructing tailored guidance functions for specific targets can be challenging (e.g. collision types). Partial Diffusion provides a straightforward method for user explicit control.

BibTeX

@misc{chang2024safesimsafetycriticalclosedlooptraffic,
      title={SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries}, 
      author={Wei-Jer Chang and Francesco Pittaluga and Masayoshi Tomizuka and Wei Zhan and Manmohan Chandraker},
      year={2024},
      eprint={2401.00391},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2401.00391}, 
}