When is a benchmark result evidence of a company advantage?
A benchmark tells you a ceiling is reachable. It does not tell you that the thing which ships is the thing that was measured — and the distance between them is usually where the company actually lives.
A benchmark result establishes that something is possible. That is genuinely valuable information, and it is not nothing — most claims fail before they get that far. What it does not establish is that the system which reaches the benchmark is the system a customer will use, or that the team can get from one to the other.
The gap between those two is not a detail of execution. In my experience it is usually where most of the remaining work is, and it is almost never what gets presented.
The result and the product were different systems
I was one of many co-authors on an agent that beat the world’s best drivers at a racing simulator, published in Nature in 2022. It is the best-known thing I have been part of, and it is a good example precisely because the headline is so clean: a superhuman result, against ranked human champions, in a real commercial environment rather than a toy one.
Then it had to become a feature in a shipped game, and the problem stopped being the one the benchmark had measured.
The agent that beat the champions was superhuman on a small number of car and track combinations. The version that shipped as a permanent feature covered 340+ cars across nine tracks, later well over 500 — a different problem, because generalising across vehicles is not the same task as mastering a few. It had to run inside a console in real time, which constrains what the network can be. And it could not simply be superhuman: a permanently unbeatable opponent is not necessarily a useful game feature, and the shipped version was built to show a wider range of behaviours so that players of every level got a good race — which is not the objective the benchmark measured. Verifying that required automated tests running across more than a thousand consoles in parallel, infrastructure that did not exist when the paper was written.
The first playable release came about a year after the Nature paper, inside a company with unusual resources and a partner studio that owned the environment. That is the reference class I hold in mind when a team tells me a benchmark result is nearly a product.
None of this diminishes the original result. The point is narrower: the benchmark proved the ceiling was reachable. It did not prove the ceiling was shippable, and the work between the two was substantial, different in kind, and largely invisible from outside.
What a benchmark can and cannot support
A benchmark result supports three claims. That the problem is solvable at all. That this team could solve it once. And that their approach is at least competitive on the axis the benchmark measures.
Those are worth having. But investors are usually being asked to believe something else: that the result implies a durable advantage. For that, the benchmark has to survive four questions it cannot answer by itself.
Is the benchmark objective the product objective? In the case above it was not — the product needed something the benchmark had not been designed to measure. This is common, and often nobody has noticed the mismatch because the benchmark was inherited rather than chosen.
Does the result hold across the space the product spans? Excellent on the evaluation set, adequate on the customer’s data, is the ordinary outcome. The question is how much of the product’s range has been measured.
What are the deployment constraints, and were they in scope? Latency, memory, cost per inference, the target hardware. Constraints that appear after training tend to be expensive, because they change what the model can be rather than how it is tuned.
What of this can a competitor reproduce, and how quickly? If the benchmark is public and the method is published, the result alone is unlikely to be the moat. The moat, if there is one, may be in what surrounded it — the environment, the data, the evaluation infrastructure, the relationship with whoever owns the deployment surface.
That last one is worth dwelling on. In the case above, what was hard to replicate was not the algorithm, which is published. It was the access to the environment, the scale of the training and testing apparatus, and the partnership that let the result become a feature. A competitor reading the paper gets the method. They do not get the rest.
What I ask
Who chose this benchmark, and when? A benchmark chosen after the results were known is a different object from one committed to in advance. Both happen; only one is evidence.
What is the gap between the benchmark objective and the customer’s objective? If a team has never articulated the difference, they have not yet met the customer’s problem.
What would it take to run this in the customer’s environment? Ask for the list of things that would have to be built. The length and character of that list tells you more about the state of the company than the result does.
What is the strongest version of this result a competitor could reach with the published method and no access to your data or environment? A team that has thought seriously about defensibility should be able to give you a bounded answer.
The version I would actually believe
The benchmark result I find most persuasive is not the highest one. It is the one accompanied by a clear-eyed account of what it does not cover — which distributions were not tested, which constraints have not been faced, what the gap to the deployed setting consists of, and how long that gap took to close last time.
That account is difficult to produce and easy to check, which is what makes it informative. A headline number is easy to produce and hard to check, which is what makes it a starting point rather than a conclusion.
This is the kind of question I work through in technical due diligence — what a result establishes, what it doesn’t, and which experiment would settle the difference.
Evidence
- Nature 2022 co-author