Notes and teardowns
Notes on evaluating reinforcement learning, agentic, and optimisation systems, and claim-by-claim teardowns of systems whose technical claims are public and checkable.
-
The contract says it will never do that. Where is that written in the system?
A penalty expresses a preference; an enforced constraint can provide a guarantee. Most promises a company makes about what its system will never do are implemented as preferences, and the difference shows up only in the cases the promise exists for.
-
When does a research team become an engineering risk?
Almost every team says their approach generalises. Very few have paid to find out, because the experiment costs money and produces no headline — it only tells you whether you have a method or a solution to one problem.
-
Does this company have proprietary technology, or a proprietary evaluation?
The cheapest thing to improve is not the system but the measurement. A metric a company controls can move without the product moving — and the same fact makes a good private evaluation one of the more defensible things a team can own.
-
When is a benchmark result evidence of a company advantage?
A benchmark tells you a ceiling is reachable. It does not tell you that the thing which ships is the thing that was measured — and the distance between them is usually where the company actually lives.
-
Is the explainability in the product, or in the slide deck?
"We have explainability" is not a yes-or-no answer. What matters is where the explanation lives: inside the mechanism that makes the decision, or in a layer added afterwards — because the two fail in different ways.
-
The demo is impressive. What would make it investable?
A demo is one point. What matters is how far you can move from it, what moving costs, and whether the point sits inside the region the system was built for.
-
Does this team know why their system works, or only that it does?
An explanation that has only been illustrated looks exactly like one that has been verified. The question is not whether a team's account is true, but how much of it they have tried to falsify.
-
What I look for when a startup shows me its best result
A result that is excellent on the headline metric and strangely poor somewhere else is more informative than a mediocre headline. The odd secondary number is usually where the specification failure is hiding.