Reasoning AI under scrutiny

How to make explainability verifiable

24.03.2026

Artificial intelligence, Validation / Regulatory Affairs

Imagine an autonomous vehicle braking and explaining when asked: “I detected the ball on the side of the road and assumed with high likelihood that it was a child.” This exact scenario was the focus of a panel discussion at NVIDIA’s GTC conference in March 2026. AI systems should not only act but also provide comprehensible justifications for their actions. Marco Pavone, NVIDIA’s director of autonomous vehicle research, describes reasoning as the process of “breaking down a complex problem into smaller, more manageable parts and planning an action step by step.”

For regulated industries, this is a promising approach. At the same time, however, it creates a new challenge for quality assurance. The reasoning itself becomes the test subject.

Explanations are not automatically reliable

Article 13 of the EU AI Act requires that high-risk AI systems be “sufficiently transparent.” Reasoning traces seem to elegantly meet this requirement. But how reliable are a model’s justifications?

In a study published at NeurIPS 2023, Turpin et al. demonstrated that LLMs systematically conceal their actual decision-making reasons. When bias factors were introduced, the models altered their answers by up to 36 percentage points without mentioning the bias in their chains of thought.
In 2025, Anthropic tested reasoning models and found that these models disclosed hidden clues in their chains of thought only 25 to 39 percent of the time.
Barez et al., from the University of Oxford, conclude in their analysis that equating chain-of-thought with explainability represents “misplaced trust.”

The problem is architectural in nature: parallel computation in Transformer networks cannot be translated into a sequential narrative without loss. NVIDIA addresses this problem through a post-training step that firmly links reasoning and action. This is an important advancement, but it does not fully solve the problem.

Establishing reasoning as a quality metric

For companies in highly regulated industries, such as medical technology, automotive, and finance, this presents an opportunity. Those who document reasoning traces and systematically validate them create a new dimension of quality. This principle is familiar from the deterministic world. Every statement requires independent verification. Applied to AI reasoning, this means specifically:

Test reasoning consistency across semantically equivalent inputs, comparable to morphological testing.
Compare CoT traces against quantitative XAI methods, such as SHAP, to verify content plausibility.
Monitor the quality of reasoning throughout the product lifecycle to detect degradation early.

This three-pronged approach complements existing QA methods. At GTC 2026, NVIDIA’s Yejin Choi made the crucial point: Models that merely imitate fail in rare or new situations. Reasoning capability mitigates this risk. Validating this capability ensures its benefits.

Those who build expertise now will succeed

No single procedure meets all the requirements of the EU AI Act, MDR, ISO 26262, or DORA. A robust approach combines reasoning traces for understandable communication with quantitative methods for verifiable results. The ability to independently verify AI results and explanations is becoming a key competency in quality assurance within regulated industries.

Companies that develop this competency early on will be better prepared when auditors first ask about the validation of reasoning traces. sepp.med can help you integrate these new requirements into your existing QA strategy.

Reasoning AI under scrutiny

Explanations are not automatically reliable

Explanations are not automatically reliable

Establishing reasoning as a quality metric

Those who build expertise now will succeed

FAQ

Sources

Get in touch with us!