Does this team know why their system works, or only that it does?
An explanation that has only been illustrated looks exactly like one that has been verified. The question is not whether a team's account is true, but how much of it they have tried to falsify.
Every team can tell you why their system works. The explanation is usually coherent, usually confident, and often correct. What is harder to establish, and what actually matters when you are deciding whether to fund the thing, is whether the explanation has ever been tested — because an explanation that has only been illustrated looks exactly like one that has been verified.
This is not a question about honesty. The teams I have seen get this wrong were not overselling. They believed their account of their own system, and the account was true over the range they had looked at.
Explanations get checked where the system looks best
A few years ago my group worked on a method for explaining what individual units inside a neural network respond to — the kind of statement that reads “this unit detects water, or the sky at dusk”. The existing approach produced explanations like that by searching for a logical combination of concepts that lined up with the unit’s behaviour, and it worked: the explanations were readable, they matched the data, and people used them.
The problem was where they had been checked. The alignment was measured only against the unit’s highest activations — the cases where it fires hardest. That is a narrow slice of what the unit does, and it is the slice on which any explanation looks its most convincing. Our contribution was a way to compute these explanations across the whole range of a unit’s behaviour instead, and a set of criteria for comparing what different methods return.
What we found when we looked at the rest of the range was not a refinement. One unit we inspected responds to windmills, to closets, and to amusement parks, depending on how hard it is firing — three concepts with nothing to do with each other, in the same unit. Another turns out to be recognising streets, but in progressively more specific contexts as the activation rises, so the narrow explanation captures the most specific case and silently drops the rest.
The scale of it is the part worth carrying. Earlier work by Mu and Andreas, looking only at the highest activations, found that around 31% of units fire for unrelated concepts. Looking across the full range, that figure is roughly 85%. Only about one unit in seven is doing a single coherent thing everywhere. The estimate involves manual inspection and should be read as an indication rather than a measurement — but the direction is not subtle. The same question, asked over a wider window, returns an almost inverted answer.
And the ignored range is not idle. When we masked out each band of activations and measured how often the network’s prediction changed, the middle bands — the ones no explanation method was looking at — changed it about as often as the top band did. Only the very lowest band mattered noticeably less. The explanations were covering the part of the behaviour that was easiest to describe, not the whole of the part that was doing work.
The finding I care about here is not the method. It is the structure of the mistake, which is the same one from the reward story: the explanation was validated in the regime where the thing looked best, and nothing about the explanation announced that it was partial. It was not wrong. It was true over a range nobody had stated.
The question is not what the team understands. It’s what they have falsified
I want to be careful here, because it is easy to overreach. Nothing above shows that a team with good explanations has causal understanding of its product. What it shows is narrower and more useful: an explanation carries an implicit domain, and the only way to know the domain is to have gone outside it.
So the question I am really asking is how much of a team’s account of their own system has been tested against something that could have contradicted it.
That matters commercially because of what it predicts about the next twelve months. A team that has tested its own account can usually make better guesses about where the system will fail, and can state the limits of its claim before deployment rather than after. A team that has only ever confirmed its account is more likely to meet the boundary in production, and may have little prepared beyond collecting more data, retraining, or narrowing the claim — which is a roadmap carrying an unknown number of cycles against problems nobody can yet name.
What I ask
How was the explanation checked, and over what range? Any explanation of a system is a claim, and every claim has an implicit domain. If nobody can state the domain, the explanation has been illustrated rather than tested. This is a fair question and a good team enjoys it.
Where does the explanation stop being true? A better question than whether it is true. A team that has looked will usually have a boundary in mind, even a rough one. If the answer stays entirely qualitative, I want to know why.
What did you expect to see, and what did you see instead? A team that has genuinely investigated its own system can usually point to something that changed its view of how the system worked. That change is the evidence that they looked before they knew what they would find.
What does your method return when there is nothing to find? This is the one I would ask that most teams have never been asked. In the work above we ran the explanation algorithm on untrained networks — random weights, no learned structure — to see what it produces when there is genuinely nothing to explain. It produces confident, readable labels anyway, and knowing what those look like is what let us recognise the same labels in the trained networks and discount them. An interpretation method can return a structured-looking answer when the underlying signal is weak or absent. Unless you establish the null case, you do not know what a false positive looks like — and therefore cannot say which of your findings are findings.
Who checked it — the people who built it, or someone else? Not a governance question. An explanation produced by the same people who designed the system is worth checking independently, because it is likely to be framed using the same assumptions the design was built around — the failure mode above, one level up.
The uncomfortable version of this
I should be honest about the limit of the argument. It is entirely possible to ship a valuable product without understanding why it works, and plenty of companies have. Understanding is not a precondition for revenue.
What testing your own account buys you is a bounded claim — some sense of where the thing is likely to fail, and therefore of what you can safely promise a customer. For a company whose value rests on a technical advantage rather than distribution, that boundary is close to the whole investment case. If the team cannot tell you where they expect the advantage to stop, the case is carrying more uncertainty than the headline result suggests. That is not a reason to say no. It is a reason to price it.
This is one of the things I look at in technical due diligence — not whether a team’s account of their system is plausible, but whether it has been tested anywhere except where the system is at its best.
Evidence
- NeurIPS 2023 senior author