Is the explainability in the product, or in the slide deck?
"We have explainability" is not a yes-or-no answer. What matters is where the explanation lives: inside the mechanism that makes the decision, or in a layer added afterwards — because the two fail in different ways.
“We have explainability” is not a yes-or-no answer, and treating it as one is how a due diligence misses a rewrite. The question that matters is where it lives. An explanation can be a property of how the system decides, or it can be produced by a layer or procedure added after the fact. Both satisfy a slide. They behave very differently when a customer, an auditor or a regulator asks why a particular decision was made, and they cost very different amounts to put in place.
Two places an explanation can come from
Most explainability work attaches to a model after it is trained. You have a black box that performs well, and you build something alongside it that produces an account of its behaviour — a saliency map, a surrogate, a simpler policy distilled from the original. This is convenient, because it leaves the performing system untouched, and it is the default for a reason.
None of this makes one choice right and the other wrong. Post-hoc explanation is adequate in plenty of systems, and a model that is legible by construction can still fail to give a particular user the account they need. What changes with the location is the kind of risk you carry, what you are able to verify, and what it costs to change your mind later.
The alternative to explaining afterwards is to build the model so that its decision procedure is legible in the first place. In a system my group published this year, we did that for continuous control: instead of a policy that maps a situation to an action through an opaque computation, the agent learns a set of prototypes — representative situations — and acts by comparing where it is now to those prototypes, blending their associated behaviours by similarity. The account of the decision is not reconstructed afterwards. It is the decision: this situation resembles that one, so act accordingly.
The result that matters commercially is the one people assume is impossible. It matched the performance of the standard black-box algorithm across eight continuous control tasks. Not beat it — matched it. The transparency did not cost anything measurable on those benchmarks.
I want to be careful about how far that carries. Eight benchmark tasks are not a production system, and there are certainly domains where legibility by construction costs real performance. But that is exactly the point: it is a quantity, and it can be measured. In assessments I hear “explainability would cost us too much performance” stated as though it were a law. It is a claim about a specific system, and most of the teams saying it have not run the experiment.
Why the location changes the risk
A post-hoc explanation is produced outside the model’s decision procedure, which means the explanation and the computation that produced the decision can come apart. The cases where they come apart are precisely the ones I want to look for — unusual or shifted cases, because those are also the cases in which explanations are most likely to be scrutinised. The explanation is often best validated on routine cases and least established exactly where scrutiny becomes highest. That is the same problem as explanations validated where the system looks best, one level up.
There is also a plain engineering consequence. If the explanation is part of the architecture, it constrains the design and it is present from the first training run. If it sits on top, then the day a customer or a regulator demands faithful explanations you may be adding a new explanation stack — or, in the harder cases, replacing the model and part of what sits downstream of it. Teams rarely price that possibility because it sits in a future they consider unlikely, right up until a procurement questionnaire arrives.
What I ask
Where does the explanation come from — the decision mechanism, or a post-hoc layer or procedure? The answer is usually immediate, and it determines what follows.
How faithful is it, and how do you know? Faithfulness is something you can test: does the explanation track the behaviour of the decision mechanism under controlled changes, or does it merely produce a plausible story? A team that has never tested it has an explanation of unknown accuracy, which is a strange thing to put in front of an auditor.
Where has the explanation been checked? Typical cases or hard ones. An explanation validated only on the easy majority tells you least about the situation where someone will demand it.
What happens if a customer requires faithful explanations next year? A configuration change, an added component, or a retrain. This is the same question as the cost of the next customer’s requirement, and here the answer is often larger than the team expects.
Who is the explanation for? An explanation legible only to the ML engineers who built the system is not usable by the person who has to sign off on the decision. In a regulated setting that gap can become part of the compliance burden, and better visualisations do not close it.
The investment version of this
For most companies, explainability is not the primary reason a customer buys the product. What it can do is determine which markets you can enter, and how expensive entering them turns out to be — health, finance, public sector, insurance, safety-critical control, and other settings where an institution has to justify the decision downstream.
So the question I am really asking is not whether the team cares about interpretability. It is whether the constraint was considered at design time or will be discovered at sales time. The first is an architecture decision that has already been paid for. The second is a retrofit — sometimes a rewrite — arriving in the middle of a growth plan, and it will not appear on any slide until the quarter it does.
This is the kind of question I work through in technical due diligence — where a capability actually lives, what it cost, and what it would cost to add later.
Evidence
- ICML 2026 senior author