Idea 03/AI & Computation/Theory + controller/3 min

Experiment-Optimal Stopping

Reasoning and measuring are both purchases. Nobody has priced them against each other.

Give an AI scientist a budget in dollars rather than tokens, and the optimal policy stops being 'think harder' and becomes an explicit market between inference, cheap assays and decisive experiments.

01What changed

Test-time compute now has a real theory. A 2026 result shows a single long chain of thought can be worth exponentially many short attempts, which reframes inference as a search with returns that depend on how the budget is spent.1 Long CoT vs many short attempts, 2026
Autonomous systems have started producing publishable biological findings, and agentic labs have been demonstrated on solid-state quantum spin experiments, so the loop from reasoning to physical action is closed in at least two domains.2 WASP autonomous AI scientist, 2026
Closed-loop materials platforms now use language models to redefine their own search space between rounds, which is effectively a heuristic version of the allocation problem.3 Communications Materials, 2026

02The gap

Every autonomous-lab paper reports results at a fixed compute and a fixed experiment budget. Almost none report the marginal value of the next experiment versus the next thousand tokens. Without that exchange rate, autonomy is a demo, not an optimisation.

03The idea

Formalise it as a two-armed bandit with a third arm that costs money. At each step the controller chooses: think (cost per token, reduces uncertainty about which experiment to run), assay (cheap, noisy, narrow), or experiment (expensive, precise, slow). The state is a posterior over hypotheses from a Bayesian model of the domain.

The exchange rate is measurable. Run the same task at several token budgets and several experiment budgets; fit a surface of outcome quality against both. The gradient of that surface gives the local price of a token in experiments. That single number is what an autonomous lab needs and never computes.

Then the interesting prediction: for mature domains with good simulators, the optimal policy spends most of its budget thinking, because a simulated experiment is nearly free. For domains with bad simulators, the optimum inverts and the controller should stop reasoning early and start measuring. This predicts a domain-dependent crossover that can be tested directly.

The controller also gives a clean way to compare AI scientists: hold total cost constant and compare the achieved outcome. Papers that report only success rate at unspecified cost are not comparable and should stop being published that way.

04Why now

Agentic labs exist, inference is metered, and cloud labs sell experiments by the well. For the first time all three arms have a price tag attached, which is the precondition for doing economics on them.

05The decisive test

Do this

Instrument one protein-expression optimisation task and one materials task with a controller that chooses among reasoning, cheap assay and definitive experiment under a fixed dollar budget.

Success looks like

The adaptive controller reaches the same target 30 percent or more cheaper than the best fixed policy, and the fitted exchange rate predicts the crossover between the two tasks.

Kills it

If the outcome surface is flat in token budget across domains, reasoning is not a controllable input and the market framing collapses.

06What would break it

Cost accounting for reasoning is opaque when providers price by token class.
A domain-specific controller may not transfer, leaving a collection of anecdotes.
Optimising cost can select for cheap, low-information experiments and degrade the science.

07If it works

Autonomous labs get a budgeting theory, and 'AI scientist' claims become comparable on a single axis: discovery per dollar.