Grounding Driving VLA
via Inverse Kinematics

KAIST
NeurIPS 2026

Abstract

Counterfactual obstacle insertion exposes blind planning in representative Driving VLAs: their trajectories remain largely unchanged despite a vehicle appearing directly ahead. An empirically calibrated bias–variance analysis identifies how ego-predictable training examples admit a shortcut through ego status and commands. We address this failure with an inverse-kinematics-inspired architecture that anchors trajectory decoding on current and predicted future visual states. First, a next visual state prediction objective that requires the LLM to predict the future visual scene provides dense visual supervision and suppresses shortcut paths. Second, a separate Inverse Kinematics Network (a cross-attention-based conditional diffusion model) that takes only the current and future visual states as input is designed to suppress reliance on ego status and textual shortcuts during trajectory decoding. Our 0.5B model achieves state-of-the-art trajectory planning performance on NAVSIM-v1, NAVSIM-v2, and nuScenes using cameras alone, without LiDAR or a BEV encoder. Against the matched OpenDriveVLA* baseline, it improves NAVSIM PDMS and EPDMS by 19.0 and 20.4 points, respectively. Controlled ablations and intervention analyses support improved use of visual features, with additional benefits from future-state prediction in dynamic situations such as turning.

Key Contributions

  • Diagnosing blind planning. Controlled visual interventions reveal that representative Driving VLAs can retain nearly identical trajectories after an obstacle is inserted. A bias–variance analysis characterizes the ego-status shortcut under an empirically calibrated majority-regime assumption.
  • Grounding actions in visual states. Next visual state prediction provides dense visual supervision. An Inverse Kinematics Network decodes trajectories from only the current and predicted future visual states, removing direct ego-status and command inputs from the trajectory decoder.
  • Strong performance with a 0.5B backbone. Using three front cameras, our model achieves 92.2 PDMS on NAVSIM-v1, 90.6 EPDMS on NAVSIM-v2, and 0.06 m average L2 error on nuScenes under the ST-P3 evaluation protocol.

Overall Methods

The visual encoder extracts current visual states from three front cameras. The LLM predicts future visual states. The IK Network conditions on these two visual states to generate a trajectory.
Overview of our model. The vision encoder extracts current visual tokens from three front cameras. The LLM predicts the future visual state and answers driving-scene questions. The Inverse Kinematics Network generates a trajectory conditioned only on the current and predicted future visual states.

Ego status and route commands guide future-state prediction through the LLM, while the trajectory decoder has no direct access to them. The IK Network is a cross-attention conditional diffusion model, with trajectory-loss gradients stopped at its visual conditioning inputs.

“Inverse kinematics” describes the idea of inferring a trajectory between a start and a target visual state; it is an architectural analogy rather than a strict vehicle-dynamics formulation.

Video Overview

A 2:42 silent, captioned overview of the motivation, architecture, and results. Method scenes use the original paper figures with animated highlights and explanatory text. Download video.

Quantitative Results

NAVSIM

Our 0.5B model improves over the matched OpenDriveVLA* baseline by 19.0 PDMS and 20.4 EPDMS points. Selected camera-based VLA comparisons from the paper are shown below.

OpenDriveVLA* shares the backbone scale, training data, hyperparameters, and SFT → GRPO schedule with our full model; the next visual state objective and IK Network differ. Our model uses perspective-camera features without LiDAR or a BEV encoder. A dash indicates a result not included in this comparison.

nuScenes

nuScenes validation, ST-P3 protocol. L2 error in meters; lower is better.
Method 1 s ↓ 2 s ↓ 3 s ↓ Average ↓
DriveVLM 0.180.340.680.40
EMMA 0.140.290.540.32
OpenDriveVLA-0.5B 0.150.320.570.35
OpenDriveVLA-7B 0.150.310.550.33
Impromptu-VLA-3B 0.130.270.520.30
Ours-0.5B 0.040.06 0.100.06

The nuScenes OpenDriveVLA results are reported original baselines, not the NAVSIM matched variant. Under the separate UniAD evaluation protocol, our average L2 error is 0.13 m. See the paper for full metrics and comparisons.

Qualitative Results

Counterfactual Obstacle Insertion

Three original and obstacle-inserted camera views. Before and after trajectories largely overlap for the representative VLA baselines; our model changes its predicted trajectory.
Does a visual change alter the plan? A vehicle is inserted into the front-camera image while non-visual inputs are held fixed. Green and red trajectories show predictions before and after insertion. The selected examples illustrate the responses of representative Driving VLAs and our model.
Obstacle stitching on nuScenes validation, as reported in the paper.
Model Deceleration ratio Avg. displacement change
Impromptu-VLA 3B38.8%−0.00 m
Alpamayo-R1 10B54.5%−0.12 m
OpenDriveVLA 0.5B44.2%−0.06 m
Ours 0.5B63.8%−1.04 m

Negative displacement change indicates a shorter predicted 3 s endpoint range, not a measured braking rate. These aggregate results are distinct from the individual examples shown in the figure and videos.

Driving Examples

Left-turn, right-turn, urban, and nighttime examples comparing saved baseline and model predictions. Playback is interpolated between available camera frames and saved predictions. Clip-level errors are not dataset-level benchmark averages. Download video.

Obstacle Position and Scale

Near, far, and sky insertions with exact saved predictions averaged over K = 50 samples. Two illustrative scenes are distinguished from the dataset means over 6,019 frames. Far and the size-matched sky insertion use the same sprite scale. Download video.

BibTeX

Download .bib
@misc{park2026grounding,
  title  = {Grounding Driving VLA via Inverse Kinematics},
  author = {Park, Junsung and Shim, Hyunjung},
  year   = {2026},
  note   = {Preprint}
}