Idea 02/AI & Computation/Validation protocol/3 min

The Failed-Trial Gauntlet

Before a cell world model nominates a target, make it predict the trials that already failed.

Virtual-cell and world-model claims are validated on held-out perturbations drawn from the same distribution, which cannot distinguish a model of biology from a model of the dataset. A gauntlet built from historical clinical failures would.

01What changed

2026 brought the first credible claim of a full world model of the human cell, built to predict how cells respond to perturbation rather than merely interpolate between measured profiles.1 GenBio AI cell world model, 2026
Perturbation atlases have grown to the point where held-out prediction is a meaningful test, and foundation models trained on them now report respectable correlations on unseen perturbations.2 Arc Institute tooling, 2026
Meanwhile the clinical failure record is large, public and strikingly uninformative: most phase III efficacy failures were predicted to work by every preclinical model used at the time.3 Cancer Discovery, 2026

02The gap

There is no out-of-distribution benchmark for cell models that is both real and adversarial. Every headline number comes from the same kind of data the model trained on. The field is optimising interpolation and calling it prediction.

03The idea

Build the benchmark out of defeats. Assemble 40 to 60 compounds that entered phase II or III and failed for lack of efficacy, with their mechanism, their transcriptomic signature in relevant cell systems, and the trial outcome. Freeze the answer key. Then ask a cell world model to rank these compounds against matched controls that succeeded.

The key design choice is the control set. Compare each failure not to a random drug but to a drug in the same pathway that worked, so the model cannot win by learning that a target class is bad. What remains is exactly the question that matters: can the model tell a viable perturbation from a dead one within the same biology?

Score with a proper scoring rule, not accuracy, and report a calibration curve. A model that says everything is 55 percent likely to work is uninformative even at high accuracy.

Publish the gauntlet as a held-out set with a submission server. Models that clear it earn the right to nominate novel targets; models that do not are still useful, but for hypothesis generation only. The gauntlet also produces something the field lacks: a per-target error map showing where virtual cells are structurally blind.

04Why now

Both halves now exist. Cell world models are mature enough to be tested, and the failure record is digitised and linkable through trial registries and published mechanism tables. The benchmark could be assembled by a small team in a quarter.

05The decisive test

Do this

Assemble 40 efficacy failures with pathway-matched successes, freeze the key, and score three public cell models plus a pathway-level base rate.

Success looks like

At least one model beats the pathway base rate with a Brier skill score above 0.15, and its errors cluster by mechanism rather than at random.

Kills it

If no model beats the pathway base rate, the current generation of virtual cells is a retrieval system, not a predictor, and should be described that way in public.

06What would break it

Selection bias in which failures were published with usable transcriptomic data.
Models may have seen the trials in pretraining, which contaminates the test unless access dates are audited.
A gauntlet built from old failures may be too easy or too hard for genuinely new mechanisms.

07If it works

Drug discovery gets a gate that correlates with clinical reality, and virtual-cell claims become falsifiable before they are marketed.