LSRE Latent World Model for Semantic Risk Estimation
Figure 1 : Overall pipeline of LSRE. A pretrained vision–language model (VLM) provides sparse semantic-risk supervision for key frames. A recurrent state-space world model encodes multi-view observations into latent states with temporal dynamics, and generates short-horizon rollouts. A lightweight latent classifier, trained under VLM supervision, evaluates both instantaneous and predicted future latent states to produce a real-time semantic risk signal for the driving stack.
Paper context
Paper title: LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Driving Abstract: Real-world autonomous driving must adhere to complex human social rules that extend beyond legally codified traffic regulations. Many of these semantic constraints, such as yielding to emergency vehicles, complying with traffic officers' gestures, or stopping for school buses, are intuitive for humans yet difficult to encode explicitly. Although large vision-language models (VLMs) can interpret such semantics, their inference cost makes them impractical for real-time deployment. This work proposes LSRE, a Latent Semantic Rule Encoding framework that converts sparsely sampled VLM judgments into decision boundaries within the latent space of a recurrent world model. By encoding language-defined safety semantics into a lightweight latent classifier, LSRE enables real-time semantic risk assessment at 10 Hz without per-frame VLM queries. Experiments on six semantic-failure scenarios in CARLA demonstrate that LSRE attains semantic risk detection accuracy comparable to a large VLM baseline, while providing substantially earlier hazard anticipation and maintaining low computational latency. LSRE further generalizes to rarely seen semantic-similar test cases, indicating that language-guided latent classification offers an effective and deployable mechanism for semantic safety monitoring in autonomous driving. Passages referencing this figure: Figure 1 : Overall pipeline of LSRE.
