Evaluation is the product
As model weights commoditize, the durable asset in AI biotech is the loop that tells you whether a prediction was any good.
Foundation models are becoming a input. Not free, not trivial, but increasingly interchangeable at the level of architecture and pretraining corpus. What is not interchangeable is the answer to a simpler question: did this prediction help us make a better decision?
In most AI biotech companies, evaluation is treated as a QA function. A team trains a model, someone runs a benchmark, the number goes in the deck, and everyone moves on. That framing gets the economics backwards. In a domain where the cost of being wrong is measured in months and millions, the evaluation loop is not a check on the product. It is the product.
Why benchmarks decay
A public benchmark is a snapshot of what the field could measure at a particular moment. The moment it is published, it starts to become a target. Models are tuned against it, leaderboard positions consolidate, and the gap between benchmark performance and real-world utility widens.
This is not a failure of any particular benchmark. It is what happens when a measurement becomes a goal. The benchmarks that stay useful are the ones tied to a decision someone actually has to make, refreshed with new data, and paired with a clear account of what they do not cover.
The three layers of an evaluation asset
A serious evaluation function has three layers, and most companies have at most one.
Public benchmarks establish a baseline and a shared vocabulary. They are necessary for hiring, communication, and sanity checks. They are not evidence of product value.
Internal held-out data is where things get interesting. Proprietary assays, curated negative results, and well-documented failures are hard to collect and harder to keep clean. This is the layer that tells you whether your model generalizes to the questions your company actually cares about.
Prospective experiments are the only layer that can validate a claim about the future. A prediction registered before an experiment, scored after it, with the endpoint fixed in advance. This is slow. It is also the only thing that turns a model into a scientific instrument.
The companies that compound are the ones that invest in all three and keep them connected. A public benchmark result should be traceable to an internal evaluation, which should be traceable to a prospective decision. When those links break, the numbers become theatre.
Evaluation as strategy
There is a strategic point hiding here. If model weights commoditize, then the defensible asset is not the model. It is the data, the experimental capacity, and the institutional knowledge required to know whether a prediction was right.
That has a few implications:
- Evaluation design is a product decision. The choice of endpoint determines what the model optimizes. It should be made by people who understand the biology and the commercial question, not delegated to whoever happens to own the test set.
- Negative results are inventory. A well-documented failure is more valuable than a cleaned-up success, because it constrains the next hypothesis. Companies that discard negative results are throwing away their most expensive data.
- The loop is the moat. A competitor can download the same pretrained model. It cannot download three years of prospective assays, calibrated against the decisions your team actually made.
A practical test
If you run an AI biotech company, try this exercise. Pick the model your team is most proud of. Ask: what decision did this model change in the last six months, and what did we learn from the outcome?
If the answer is a benchmark number, you have a research project. If the answer is a decision and a result — positive or negative — you have a product.
The model is the easy part to demo. The loop is the hard part to build. It is also the part that lasts.