What I look for when a startup shows me its best result
A result that is excellent on the headline metric and strangely poor somewhere else is more informative than a mediocre headline. The odd secondary number is usually where the specification failure is hiding.
The headline number is rarely where the problem is. When I assess a learning system, the thing I look for first is a result that is excellent on the metric the team is optimising and quietly strange somewhere else. That combination tells me more than a mediocre headline ever could, because a specification failure does not look like a failure. It looks like success, plus something odd that nobody has explained yet.
I know the shape of this because I have been on the wrong side of it.
The metric was right. It was answering the wrong question
On a system I led, we trained agents for Horizon Forbidden West to perform a fixed task while taking a runtime instruction about how to perform it — fight this way rather than that way. The opponents were machines with removable components, so one family of instructions asked the agent to strip those components off rather than simply deal damage. We scored those runs by counting the components removed.
It worked. Under those instructions the agent removed roughly twice as many components as under any other. On the metric we had written down, the result was unambiguous, and for a while that was the metric we watched.
Two things were wrong with it, and they were different kinds of wrong.
The first was in what the metric counted. We credited a component as removed whether it had been detached or destroyed, and the agent found the cheapest way to satisfy that definition: an explosive weapon that takes several components off at once. Crude, effective, entirely within the rules we had written. The definition permitted a solution we had not imagined, which is what definitions do.
The second was worse, because it cost the agent the actual task. The reward for removing components was large relative to the reward for winning the fight. In some training runs that produced a policy that stalked the weaker opponents around the arena, waiting to line up a clean shot at a component, until the clock ran out. It was maximising what we measured and losing fights it should have won.
The tell was in a number nobody was optimising
Neither failure showed up in the metric we were watching. The count of removed components was fine. What was not fine was the rate — how long it took to detach them — and the episode length, which was running longer than it had any reason to. Then the win rate, broken down by opponent, showed the thing that made no sense at all: the agent was doing worse against the weaker machines. Real degradation does not work that way round.
The numbers looked strange before we could say what was strange about them. What settled it was watching the footage. Playing back the episodes, you could see the agent circling a machine it could plainly have finished, waiting for a line on a component, running the clock down. The aggregates we had could tell us a number was wrong. They could not tell us what the agent was doing with its time, and that was the gap the video closed.
We did not get there quickly. The first hypotheses were the ordinary ones: an exploration problem, or a policy that hadn’t trained long enough. Both are reasonable and both are things you can act on by turning a knob. It took the footage to move the suspicion from the learning process to the thing being learned.
That order repeats, and it is worth knowing about. The specification is the last place anyone looks, because it is the part that was decided deliberately and written down. Everything else in the system is something that happened; the objective is something you chose, which makes it feel like the fixed point rather than a hypothesis. Meanwhile the metric it defines is the one the system is being pushed towards, which is exactly why it is poorly placed to tell you whether the push is working.
What this means when you’re the one deciding
If a company reports that its agent improves task completion by thirty per cent, what I want is not the thirty per cent but the breakdown behind it. I would ask for it split by environment, by task difficulty, by customer, by random initialisation, by model version, and by failure class — and I would read the difficulty split first, looking for the same inversion: does anything get worse where it should be getting easier?
Three questions follow from this, and they work on any learning system regardless of domain:
What does the objective actually credit? Not what it’s for — what it counts. Then: what else satisfies that definition? Every team has a version of this answer. The good ones can name theirs without being prompted, because they have already gone looking.
Which numbers are you not optimising, and what do they do? A team that only tracks its target metric has no instrument capable of detecting this class of failure. A team that maintains a set of unoptimised diagnostic metrics has told you something about its engineering maturity before answering a single question.
What did you find when you went looking for it? This is the one that separates people. A team that reports where its system degrades has been searching. A clean set of results with no anomalies anywhere is not reassurance, it’s a reason to ask what was tested — and how much of it there was. In the work above the evaluation covered roughly 57,000 episodes, and that scale mattered: the anomaly appeared in the breakdown by opponent type and would have vanished into noise in a smaller run.
The part that doesn’t generalise, and the part that does
I am not claiming this specific failure is common. Reward specification in the setting above is unusually explicit — most systems bury their objective in a loss function, a ranking, or a business KPI, where it is harder to inspect and therefore easier to get wrong without anyone writing it down.
What generalises is the diagnostic posture. A learning system will exploit whatever structure in the objective makes success easier, including paths that satisfy the letter of what you wrote and defeat the purpose. You can reduce the problem by writing a better objective, but you cannot assume you have eliminated it. You still need signals that tell you when the system has found a path you did not anticipate — which means watching the numbers nobody is pushing on, because nothing is holding those up.
When a team shows me its best result, that result is the least informative thing in the room. I want the second-best one, and the one that got worse.
This is the kind of question I work through in technical due diligence — claim by claim, on the evidence a company can actually produce.
Evidence
- arXiv 2026 first author, project lead