Executive Summary

Key figures

Source: Kenny, Dharmavaram, Lee, Phan-Minh, Rajesh, Hu, Major, Tomov, Shah, Explainable deep learning improves human mental models of self-driving cars, Nature (2 September 2026)

Hundreds of companies worldwide build autonomous driving technology, and the market splits roughly in two. On one side sit fully driverless ride-hailing fleets; on the other, driver assistance in consumer vehicles with a person in the seat. The second is this paper's stage. Machine learning has raised performance considerably, but a learned planner still cannot always determine the correct action in unusual or challenging situations, and a human has to step in. The rarity of those failures is itself the problem: because they are infrequent, the driver is unlikely to be braced for them.

Communication between car and driver broke down, and multiple high-profile incidents followed, some of them fatal. The paper cites those cases in its opening as grounds for calling the interpretability of ML planners urgent. Prior work existed. Some studies used surveys and simulated scenarios, some had a human driver emulate the AV, and some had language models produce natural-language rationales for the driving policy. The authors level four diagnoses at that whole lineage: those studies were theoretical, did not provide causally faithful explanations, were evaluated only in simulation, or did not convincingly show the practical utility of the explanations to end users. It is the last of the four that this paper takes aim at.

That leaves one open question: how do you provide explanations that are understandable, useful and faithful to the decision-making process of the AV in a realistic setting? The authors also state the working definition of explainable AI they borrowed. It is AI systems that can explain their rationale to a human user, characterize their strengths and weaknesses, and convey an understanding of how they will behave in the future. Take that last clause as the scorecard and the grading scheme in section 4 follows naturally. The paper was received on 24 November 2024, accepted on 24 July 2026 and published on 2 September. Peer review alone took a year and eight months.

The part they touched is a single module in the autonomous driving stack: the planner. It takes the scene information produced by the perception module and outputs the trajectory that the downstream control module will follow. Inside, it has four pieces. A scene encoder compresses the scene into a vector, a trajectory generator produces a large batch of candidates, a scene-trajectory encoder binds scene and trajectory together, and a reward layer scores each candidate. There are 146 candidates at every step. A heuristic generator produces 143 jerk-optimal trajectories toward anchor waypoints along the route, where jerk is the rate of change of acceleration, and the pretrained prediction model used as the scene encoder contributes 3 more. The highest-scoring one is selected. What the score means is human-likeness: the planner was trained by inverse reinforcement learning on 80 hours of human expert driving, so that trajectories a person would have driven score high.

CW-Net lifts out exactly one of those four pieces, the final reward layer. In its place goes a concept classifier, and behind that a new reward layer. The classifier takes the bundled scene-trajectory embedding and assigns probabilities to labels a person can read: whether the car is approaching a stopped vehicle, whether it is within three metres of another vehicle, whether it is close to a cyclist. The new reward layer computes trajectory scores from that probability vector alone. The first three pieces are frozen during training, and the only things being learned are the classifier and the new reward layer.

The authors trace the idea back to case-based reasoning, a classical artificial intelligence method rooted in cognitive models of human reasoning and memory. The notion is that you match the present scene against remembered cases and retrieve the behaviour attached to that case. The second scene in section 3 shows that structure operating in the open.

The post hoc techniques in wide use today attach to an already trained model from the outside and estimate the reason for a decision. By construction, nothing guarantees that the estimate matches the model's actual decision process. In CW-Net, the concept probabilities are the only input that reaches the final decision module. Concepts determine the car's behaviour, and so the authors call this explanation causally faithful.

What happened to performance? Run side by side with the original black-box planner in closed-loop simulation on the nuPlan benchmark, CW-Net came in under 1% difference on every measure, and under 0.01 in average L2 distance. The original planner's own record is worth noting. When following the lane or decelerating from high speed, it made more than 93% of human driving distance, stayed more than 90% collision-free, and ended up less than a metre from the human expert trajectory at 5 seconds. Starting from a stop, those fell to 74% and 81%, and the deviation widened to 1.2 metres. Rather than hide the weakness, the authors wrote that it gave them a chance to explain undesirable behaviour.

The authors did not pit CW-Net against other explanation techniques, and they give two reasons. One is that at the time of writing they were unaware of other work capable of modelling interpretable-by-design inverse reinforcement learning systems. The other is that to generalize across the myriad concept-based techniques scattered through the literature, they judged it better to compare concept-based explanation against no explanation at all. So what this paper shows is not that CW-Net beats competing techniques, but that having an explanation beats having none. The same design decision is why the control group in the online study that follows saw speed and steering data rather than a rival explanation.

The vehicle testing took place on a private track. Three people rode in the car: a safety driver who engaged and disengaged self-driving mode and took over when things got unsafe, a support engineer who deployed CW-Net and set scenario destinations, and a researcher who directed the testing. The dashboard showed a map with the perception module's object detections and the chosen output trajectory overlaid, and beside it the probability of each concept rendered as a percentage.

The scenes were not staged in advance. The researchers dictated only high-level plans for each day and waited for surprising things to happen on their own. They call this a semi-naturalistic study. In each of the three scenes that emerged, the safety driver first said their own guess out loud, said it again after seeing the explanation, and a counterfactual test came in between.

In the second scene there was no stopped vehicle in front of the car. The planner nonetheless matched the present situation against training scenarios labelled ASV and selected the stopping behaviour associated with them. The explanation does not describe the world; it describes the model. The authors are explicit about this. Regardless of the fact that no vehicle was ahead, the explanation is causally faithful in that it correctly identifies why the planner stopped.

The follow-up analysis of the third scene moved the fault from one module to another. The perception system was detecting the cyclist perfectly well. The planner was the problem: it was not configured to consume inputs for cyclists, and so it chose trajectories that would have collided with the cyclist. The car stopped every time because a separate built-in safety backup unrelated to CW-Net commanded a brake when collision was imminent. Without the explanation, the driver would have gone on believing the car handles cyclists well. A single concept probability broke that misconception.

Three scenes amount to a handful of observations by one safety driver on a private track, which is thin ground for generalization. So the authors added an online study that replayed the same scenes as forward-camera video with concept probabilities overlaid. For each video, participants answer two things: a nearest-neighbour task, in which they pick the reason for the car's behaviour closest to their own belief, and a prediction task, in which they say what the car would do under changed conditions. Both are multiple choice followed by a confidence score, and a free-form text rationale was attached to the prediction task only. The ordering is what matters. Participants answered once before seeing the explanation and again after. It mirrors the sequence the driver in the car went through.

Participants were split into two groups and run separately. Nine safety drivers, test engineers and test specialists from Motional formed the expert group, and thirty randomly sampled US citizens from Prolific formed the non-expert group. All nine experts work for the company that builds this car and volunteered without pay. The thirty non-experts were paid USD 15 per hour. On the nearest-neighbour task, 8 of 9 experts and 27 of 30 non-experts improved. Free-form responses were graded with an LLM-as-a-judge, GPT-5, and the authors disclose that the prompt was tuned three times on expert responses and then applied once to the non-expert responses.

The free-form result comes from exact binomial tests on the non-expert responses. Before seeing the explanation, responses were close to what the safety driver in the car had believed beforehand; afterwards they shifted toward the driver's final belief and toward the ground-truth reason for the car's behaviour. The three P values are below 10⁻⁹, below 0.0002 and below 10⁻⁵ respectively. People who only watched a video retraced, in the same order, the shift in understanding that happened inside the car.

This is where the paper's scorecard gets set. A significant relationship emerged between how much a participant's judgement improved and how much their prediction accuracy rose. For experts the coefficient was 2.02 with a standard error of 0.87 and P = 0.02; for non-experts, 9.86 with a standard error of 2.07 and P below 0.001. These are linear mixed-effects models with individual participant variation controlled as a random effect. The explanation corrected the model in people's heads, and the corrected model produced genuinely better predictions. The decision to measure by predictive ability rather than accuracy earns its justification here.

Run the same analysis on the free-form measure, though, and the picture weakens by a notch. Non-experts came in at 5.03 with a standard error of 1.23 and were significant, but experts landed at 1.70 with a standard error of 0.91 and P = 0.06, short of the threshold. Summarizing their hypotheses, the authors themselves note that only the free-form text response category failed to reach significance with experts. It is also worth remembering that 8/9, the number from this paper most likely to be quoted, comes from nine experts.

4.1Rerunning Situational Awareness on Public-Road Footage

A private track is a controlled space, so the authors also gathered data on public roads in Las Vegas. The conditions, though, were quite different. Owing to safety reasons and the experimental nature of the planner, the safety driver operated the vehicle in manual mode for several hours while CW-Net ran in the background recording concept probabilities only. That produced 1 hour 3 minutes of ASV footage, 51 minutes of CLOSE and more than 3 hours involving cyclists. Neither cones nor cyclists occur naturally on public roads, so pedestrians took the place of the cone and the PEDESTRIAN concept took the place of BIKE.

This footage was run through the standard procedure for measuring situational awareness. SAGAT freezes a scenario and asks the participant three things: what the inputs to the car are (perception), why the car is doing that (comprehension), and what the car would do if conditions changed (projection). Previous benchmark applications of SAGAT in driving research were conducted almost exclusively within controlled driving simulators. Running it on recordings of real public-road operation is the difference the authors claim, and it answers a long-standing concern about the ecological validity of simulator-only assessments. The target was 100 participants and 99 remained after attention checks. The experimental group of 51 saw concept probabilities; the control group of 48 saw speed and steering data, a placeholder explanation that shows nothing substantive while balancing cognitive load. The stimuli comprised 6 surprising events, 6 unsurprising ones and 1 attention check.

Each time the video froze, a blackout screen came up with six yes-or-no questions. Four related to perception, one to comprehension and one to projection. That is why the three numbers below cannot be weighted equally: perception is an average over four items, while projection rests on a single one.

The thresholds for counting a concept as active were also set by people. They were 0.5 for ASV and PEDESTRIAN and 0.94 for CLOSE, derived from the earlier private-track data. Meanwhile the distribution of concept activations did not change meaningfully between the private-track tests and the public-road tests. The two rounds were more than a year apart, on different vehicles, with different software stacks and under different conditions, and the authors take this as evidence that the algorithm is robust.

In surprising events all three dimensions rose. Perception came in at Cohen's d of 1.290 with a 95% confidence interval of 0.857 to 1.723, comprehension at 0.996 with 0.578 to 1.413, and projection at 0.606 with 0.203 to 1.009. The first two are large effects and projection is a medium one. In unsurprising events, no significant difference survived Bonferroni correction. The two results have to be read together for either to mean anything. Had the explanations helped all the time, that would have been suspicious in its own right, because information that fills the screen when it is not needed adds cognitive load and erodes situational awareness.

Given how much the explanations helped, you would expect the concept classifier to be fairly accurate, and the numbers say otherwise. On a 5% holdout from the training datasets, mean accuracy was 0.54, precision 0.23, recall 0.77 and F1 0.31. The spread across concepts is wide. SLOW is caught well, at F1 0.82, while BIKE is close to zero. That number is why BIKE stayed under 1% throughout section 3. The authors wrote that the cyclist concept is perhaps not well encoded or understood by the car.

The concepts the authors name when they discuss concept separation are CLOSE and PEDESTRIAN. In the experimental prototype they tested, those two had poor precision and high recall, relatively speaking, which means they fired often in situations that were not the situation. CLOSE is precisely the concept that carried the first scene in section 3. The case where the explanation proved useful and the case where the classifier was shaky sit on top of the same label.

The training material was not scarce. There were two datasets: 500,000 scenarios with 8 concept labels, and 3,000,000 scenarios with 10. Since each scenario carries 146 trajectories, the concept classifier saw somewhere between 73 million and 438 million data points, and a single data point can carry more than one concept at once. It is hard to argue that volume was the problem.

The problem is not volume but definition. Run down the list in Methods and CLOSE means within three metres of another vehicle, SLOW means driving between 1 and 2 metres per second, FAST means faster than 2 metres per second, and PUDO means being inside a pedestrian pickup–drop-off zone. They wear labels a person can read, but underneath they are distance and speed thresholds plus scenario-type distinctions. Which eight or ten concepts to use, and where to draw the line at three metres, are not things the model learned. They are things the research team decided. Accuracy metrics do not measure the quality of that decision. They measure only how well the pre-set labels are hit.

The authors do not treat this low accuracy as a defect. In Methods, under 'Concept separation', they head off the obvious reading of a 0.54. Their argument runs in two parts, and the citations they attach to the first part point to work on unsupervised concept separation in large models.

As far as classification accuracy goes, then, the authors are betting on the model side. Yet the future work the same paper names in its conclusion is a different problem. It is extending CW-Net to a larger set of concepts, and doing so probably in an unsupervised manner in order to overcome the challenges of labelling. However good the model gets, a set of eight labels does not become ten by itself. Covering the vast array of concepts relevant to driving means someone has to erect new axes.

The domains the paper points to for extension inherit the same condition. The conclusion names human–robot interaction systems that require real-time explanations, including AI wingmen, drone navigation systems and robotic surgeons, and on the architecture side it names end-to-end learning systems and vision–language–action models. Regulation comes attached. Many regulatory bodies have already made explainable AI a core component of their legislation, and the paper expects AVs to follow suit as they are widely deployed. Drones or surgical robots, someone has to write the list of human-readable concepts for that domain from scratch.

The three items below are not in the paper. They are questions we would carry over to the data side of any system being asked to explain itself.

• How many labels does our system show to a person, and is there a record of who set that list and on what grounds? If thresholds are used, where did those numbers come from?
• Are we looking at per-label performance separately? A single average accuracy will not reveal that one particular label is effectively non-functional. In this paper, the fact that BIKE has an F1 near zero was buried inside a mean of 0.54.
• What are we measuring the value of an explanation with? Judged by model metrics, this paper's concept classifier is a failure. Judged by how well a person predicts the next behaviour, the result comes out differently.

Core paper

• 1.Kenny, E. M., Dharmavaram, A., Lee, S. U., Phan-Minh, T., Rajesh, S., Hu, Y., Major, L., Tomov, M. S., Shah, J. A. (2026). "Explainable deep learning improves human mental models of self-driving cars." Nature. doi:10.1038/s41586-026-10950-5. Every figure and quotation in this article comes from here. Received 24 November 2024, accepted 24 July 2026, published 2 September 2026. Data and code for reproducing the figures are in the GitHub repository, and the videos shown to participants are on the project site.

Work the paper builds on

• 2.Phan-Minh, T. et al. (2023). "DriveIRL: Drive in Real Life with Inverse Reinforcement Learning." Proc. IEEE ICRA 2023, 1544–1550. The original architecture of the black-box planner CW-Net wraps, discussed in section 2.
• 3.Tomov, M. S. et al. (2026). "TreeIRL: Safe Urban Driving with Tree Search and Inverse Reinforcement Learning." Proc. IEEE ICRA 2026 (in the press). A successor planner in the same line.
• 4.Yuksekgonul, M., Wang, M., Zou, J. (2023). "Post-hoc Concept Bottleneck Models." Proc. ICLR 2023. The concept bottleneck lineage referenced in section 2.
• 5.Karnchanachari, N. et al. (2024). "Towards Learning-Based Planning: The nuPlan Benchmark for Real-World Autonomous Driving." Proc. IEEE ICRA 2024, 629–636. The benchmark used for the driving performance comparison in section 2.
• 6.Heim, M., Suárez-Ruiz, F., Bhuiyan, I., Brito, B., Tomov, M. S. (2025). "Lab2Car: A Versatile Wrapper for Deploying Experimental Planners in Complex Real-World Environments." Proc. IEEE ICRA 2025. The wrapper used for the real-vehicle deployment in section 3.