Incoprea research · leads batch P · 3 November 2026

Statistical inference for scientific measurement: 200 leads, three computations and one proof

0 · The structure being generalised

A measurement produces a number. A scientific claim is a statement about the process that produced it, and the bridge between the two is inference: a model, a design, an estimator, an interval, and an aggregation rule. Every failure catalogued on this page sits at one of four joints of that bridge.

AG  identifiability      is the parameter determined by the data at all?
AH  validation           was the number checked against data it had not seen?
AI  uncertainty          does the interval describe the procedure, or only the arithmetic?
AJ  aggregation          were the units distinct, and was each counted once?

The first joint is exact and can be settled before any data are collected. For a regular model with parameter θ and log-likelihood ℓ, the Fisher information is

I(θ) = E[ ∇ℓ(θ) ∇ℓ(θ)ᵀ ]

and the proof below shows that I(θ) singular is the same statement as θ not being determined by the data: a null vector v gives a direction in which the distribution does not move at all. Everything else on this page is either a consequence of that fact or a failure to check it.

The remaining three joints are statistical rather than algebraic, and each has a characteristic failure mode. Validation fails when the split leaks information through a grouping the analyst did not model. Uncertainty fails when a variance term is dropped or a nuisance is treated as known. Aggregation fails when the same unit is counted twice, or when the literature that supplies the units was selected on the answer. The four sections below take them in that order.

AG · Identifiability: when a model's parameters are determined by the data

Identifiability is a property of the model and the design, not of the sample. Three kinds of failure recur. Exact non-identifiability: two parameter values give the same likelihood, so no estimator can separate them and the information matrix is singular. Practical non-identifiability: the likelihood is formally curved, but the curvature is small enough that the estimate is unusable at any achievable sample size — a mixture with two similar rates, an Arrhenius fit over a narrow temperature range. Identification by assumption: a parameter pinned not by the data but by an exclusion restriction, a distributional form, or a normalisation convention. The three are often reported identically, which is the point of separating them here.

AH · Validation: splits, leakage, and what held-out really means

A held-out set is a promise that its rows played no part in producing the prediction. The promise is broken by every step that touches the whole dataset: feature selection, scaling, imputation, hyperparameter search, early stopping, seed choice, and the analyst's memory of the last run. It is also broken whenever the rows are not exchangeable — the same subject measured twice, the same specimen re-run, the same batch, the same site, the same week. A random split is a valid estimator of generalisation only under exchangeability, and the leads below are about how far real data are from that, how the resulting optimism scales, and what a validated number actually claims.

AI · Uncertainty: intervals that are too narrow, and the ones that cannot be computed

An interval is a claim about a procedure: repeat the whole experiment, and the procedure covers the truth at the stated rate. Two failures are worth separating. The interval can be too narrow, because a variance term was dropped, a dependence between observations was ignored, or a nuisance parameter was treated as known. Or it can be uncomputable, because the parameter is not identified, the denominator is not distinguishable from zero, or the variance sits on the boundary of its space. The distinction is practical: a too-narrow interval is an arithmetic or modelling error, while an uncomputable one is a fact about the design, and no widening repairs it.

AJ · Evidence aggregation: meta-analysis failure modes and derived-variable double counting

Aggregation assumes that the units being added are distinct and that they measure the same thing. Both assumptions fail quietly, and both failures have the same signature: an interval that is too narrow because the effective number of independent units is smaller than the count. A cohort can enter a meta-analysis twice through duplicate publication, overlapping samples, or a shared control arm. A derived variable can enter a model alongside its own denominator, so one quantity is counted as two. And the selection that produced the literature — significance, review, the file drawer — is not observed, so the pooled estimate is a property of the selection as much as of the effect.

Proof · joint unidentifiability is a singular Fisher information matrix

[PROVED] Let p(y; θ) be a parametric model with θ ∈ Θ ⊆ Rᵈ and Fisher information I(θ) = E_θ[∇ℓ ∇ℓᵀ]. Suppose I is finite and continuous near θ₀ and Θ is open. Then

(i)   I(θ₀) nonsingular        =>  θ is locally identifiable at θ₀
(ii)  I(θ₀) v = 0, v ≠ 0        =>  the information for the scalar vᵀθ is zero,
                                    its Cramér–Rao bound is infinite, and no regular
                                    √n-consistent estimator of a functional moving
                                    along v exists
(iii) I(θ₀ + t v) v = 0 near 0  =>  p(·; θ₀ + t v) = p(·; θ₀) exactly: the two
                                    parameters are jointly unidentifiable outright

Assumptions. (A1) The support of p(·; θ) does not depend on θ, so differentiation under the integral is legitimate. (A2) ℓ is twice differentiable in θ and dominated, so that E_θ[∇²ℓ] = −I(θ) and E_θ[∇ℓ] = 0. (A3) I is finite and continuous in a neighbourhood of θ₀. (A4) Θ is open, so θ₀ + t v ∈ Θ for small t.

Proof of (i), and of (iii). (i) Suppose p(·; θ₁) = p(·; θ₀) with θ₁ ≠ θ₀ near θ₀. The Kullback–Leibler divergence is then zero. By a second-order Taylor expansion with the information identity, D(θ₀ ‖ θ₁) = ½ (θ₁ − θ₀)ᵀ I(θ*) (θ₁ − θ₀) for some θ* on the segment, which is strictly positive because I is positive definite in a neighbourhood by (A3). A contradiction, so θ₁ = θ₀ locally. (iii) If vᵀ ∇ℓ(θ) = 0 for every θ in a neighbourhood, then for each y the function t ↦ log p(y; θ₀ + t v) has zero derivative on that neighbourhood, so it is constant; hence p(·; θ₀ + t v) = p(·; θ₀) as distributions. This is the case of a parameter that enters only through a combination — only α + β, only a product, only a ratio — and it is the case that matters for section AG.

Proof of (ii). Suppose I(θ₀) v = 0 with v ≠ 0. Then vᵀ I(θ₀) v = Var_θ₀(vᵀ ∇ℓ) = 0, and a random variable with zero variance equals its mean; the mean of the score is zero by (A2), so vᵀ ∇ℓ(θ₀) = 0 almost surely. The score in the direction v therefore has zero mean and zero variance at θ₀, the Fisher information for the scalar vᵀθ is zero, and the Cramér–Rao bound for it is infinite. The same statement in asymptotic form: for the local sequence θ_t = θ₀ + t v/√n, the divergence between n observations is n D(θ₀ ‖ θ_t) = ½ vᵀ I(θ₀) v + o(1) = o(1), so no test has power against that sequence and no regular estimator separates the two parameter values at rate √n.

Note what (ii) does and does not say. It does not say the distribution is unchanged at a finite distance: a model can have a singular information matrix at an isolated point and still be identifiable there, which is why (iii) is stated separately. What singularity at θ₀ buys is that the data carry no first-order information along v, so the parameter is not determined to the order the data can resolve — and in the structural case, not determined at all.

Corollary (directional Cramér–Rao). For an unbiased estimator T of a differentiable functional ψ, Var(T) ≥ ∇ψᵀ I⁻¹ ∇ψ. When I is singular the bound is infinite in every direction outside the row space of I, and a finite bound survives only for functionals with ∇ψ in that row space. That is exactly why α + β is estimable in the first example below while α and β are not.

Example 1 (exactly singular). y = α + β + ε with ε ~ N(0,1). The score is (1, 1)ᵀ ε, so I = [[1,1],[1,1]], det = 0, and the null vector is (1, −1)/√2. The profile log-likelihood in α is constant: the computed spread over α ∈ [−20, 20] is 1.1e-28, which is floating-point zero.

Example 2 (singular at zero spread). y_i = a + b x_i + ε_i with ε ~ N(0, σ²) gives I = σ⁻² [[n, Σx],[Σx, Σx²]] and

det I = σ⁻⁴ ( n Σx² − (Σx)² ) = (n²/σ⁴) Var(x)

so b is identified exactly when x is not constant. The table takes x_i = s·t_i with t fixed on [−1, 1], n = 50, σ = 1, and computes I numerically.

[COMPUTED]

A hundredfold reduction in the design spread reduces det I by 10,000× — the square, as the formula requires — and widens the 95% slope interval by 100×. At s = 0 the matrix is exactly singular, the rank drops to one, and the interval is unbounded. Nothing about the estimator has changed; the design has simply stopped carrying information about b.

[CONJECTURED] For a fixed k, the leakage gap of test 1 is non-decreasing in the within-group correlation, and it is bounded above by the share of outcome variance that is constant within group. The computed gap of 0.347 at ρ = 0.99 against a between-group share of 0.760 is consistent with the bound (the gap is 0.46 of it) but does not establish it, and the bound says nothing about how fast the gap approaches it.

Tests run

Three computations, all in incoprea_nov3/code/statmeas_tests.py, writing incoprea_nov3/results/statistical_measurement.json. All three are seeded simulations with stipulated parameters; no measured data are used anywhere on this page, and every number in the tables and in this prose is read from that JSON.

1 · The leakage gap between a random and a grouped split

simulation · stipulated parameters · no measured data

One hundred groups of eight rows. Each group has a latent vector z in R⁵. A row is x = √ρ · z + √(1−ρ) · noise, so ρ is the within-group correlation of the features, and the target is y = z₀ + 0.7 z₁² + N(0, 0.8²). The same data are split twice: a random 5-fold, and a group 5-fold that never puts a group in both training and test. The learner is 5-nearest-neighbour regression, and the control is ridge regression on the same splits. Twenty replications per cell; the ± column is the standard error over replications.

What the table says. The gap rises from 0.020 at ρ = 0 to 0.347 at ρ = 0.99, a factor of 17.8. At ρ = 0 the features carry no information about the target, so both splits score below the mean of the fold (−0.194 and −0.213) and the gap collapses to 0.020. It does not reach zero: 0.020 is 2.8 standard errors, because a random split still shares group composition between folds and a grouped split does not. At ρ = 0.99 the features identify the group almost exactly, a random split places each test row's own group in the training fold, and the random-split score reaches 0.706 against an honest 0.359, a difference of 0.347 in R² on data where no group label was ever supplied to the learner.

The control. Ridge regression cannot memorise a group: it has five coefficients and no representation of group identity. Its gap is 0.018 at ρ = 0 and 0.070 at ρ = 0.99 — 5.0× smaller than the k-NN gap, but not zero.

The expectation this contradicted. I expected the linear control to show a gap of exactly zero, isolating memorisation as the whole mechanism. It does not: 0.070 of the 0.347 survives with a five-parameter linear learner. Sharing a group between train and test is a leak in itself, whatever the capacity of the learner, because the group's contribution is in the training fit either way; capacity multiplies the leak rather than creating it. The honest statement is that the gap is a split effect plus a learner effect, and at ρ = 0.99 they stand in the ratio 1:4.0.

2 · How a derived ratio inflates an apparent effect size

simulation · stipulated parameters · no measured data

Take X = exp(σ_x Z₁) and Y = exp(σ_y Z₂) with Z₁ and Z₂ independent standard normal. X and Y are independent by construction, so their true association is exactly zero. Form the derived ratio r = Y/X. Then ln r = ln Y − ln X is normal with variance σ_x² + σ_y² and cov(ln r, ln X) = −σ_x², so

corr(ln r, ln X) = −σ_x / √(σ_x² + σ_y²)      exactly, for all σ_x, σ_y

The table compares that closed form with a 2,000,000-draw simulation, then reports the apparent effect size between the top and bottom tertiles of X, and the rejection rate of H₀: corr(r, X) = 0 at n = 30 using the Fisher z test (4,000 replications per cell).

What the table says. The closed form and the simulation agree to 0.0011 in correlation units across all 24 cells, which is a check on the arithmetic rather than a finding. The finding is the fifth and sixth columns. At σ_x = σ_y = 0.3 the correlation is exactly −0.7071, and the top tertile of X has a mean ratio 1.70 standard deviations below the bottom tertile, from two variables that were generated independently. At the corner σ_x = 0.30, σ_y = 0.10 the apparent effect reaches 3.05 standard deviations, and the n = 30 test rejects the null of no association in 100.0% of replications. The smallest apparent effect anywhere on the grid is 0.20 standard deviations, at σ_x = 0.05, σ_y = 0.50.

Why it happens, stated without the simulation. r is a function of X. Any covariate that predicts X therefore predicts r, and the sign is negative because X sits in the denominator. Testing r against X is testing one quantity against itself, and the test is not wrong about the arithmetic — it is wrong about the estimand. The same structure applies to any derived variable with a measured denominator: normalised signal, fold change, per-capita rate, yield per unit input, intensity normalised to a reference channel.

The expectation this contradicted. I expected a raw-scale analysis to be much less misleading than a log-scale one, because the raw correlation is strongly attenuated at high σ_x: at σ_x = 0.8, σ_y = 0.1 the exact log correlation is −0.992 and the raw correlation is −0.521. Attenuation of a correlation is not attenuation of an effect size. At that corner the tertile difference is still 1.97 standard deviations, and the largest apparent effect on the grid — 3.05 SD — sits at σ_x = 0.30, σ_y = 0.10, where the raw correlation is only −0.865. A weaker correlation between two variables is not a weaker claim about their relationship when the variables are ratios.

3 · Coverage when a nuisance parameter is fixed rather than fitted

simulation · stipulated parameters · no measured data

y = θx + νz + e with θ = ν = 1, x and z standard normal with correlation ρ, and e standard normal. Four analyses of the same data: oracle (ν known and subtracted), fixed 0 (ν fixed at 0, which is the covariate omitted), fixed 1.1 (ν fixed at 1.1, a 10% error in a nuisance believed known), and fitted (ν estimated jointly). Nominal 95% Wald interval for θ, 2,000 replications per cell; the last column is the ratio of the mean interval width under fixed 0 to the width under the fitted analysis.

What the table says. With ν fitted, coverage runs from 0.939 to 0.958 across all twelve cells, which is what a nominal 95% interval should do at 2,000 draws (Monte Carlo standard error about 0.005). With ν fixed at 0 and ρ = 0 the coverage is also correct — 0.959 at n = 400 — because the omitted variable is independent of x: the estimator stays unbiased and only its variance changes. The width there is 1.41× the fitted width at n = 400, which is √2 to three digits: the nuisance's variance moves into the residual and nothing else changes. (At n = 40 the same ratio is 1.39, because the two fits spend n − 2 and n − 3 degrees of freedom.) Omitting a nuisance that is independent of the target regressor costs precision and nothing else.

At high correlation the same omission destroys coverage. With ρ = 0.95 and ν fixed at 0, coverage is 0.001 at n = 40 and 0.000 at n = 400. The bias of θ̂ is ν·ρ ≈ 0.95, which is 5.6 standard errors at n = 40 and grows as √n: the interval is not too narrow, it is in the wrong place, and more data makes it miss more reliably.

The 10% error is the interesting column. At ρ = 0.95 coverage falls from 0.909 at n = 40 to 0.528 at n = 400. The induced bias is fixed at 0.094 — 10% of ν times ρ — while the standard error at n = 400 is 0.050, so the interval sits about 1.9 standard errors away from the truth. A 10% error in a nuisance "known" from a datasheet is not a small error once the sample is large.

The expectation this contradicted. I expected a fixed nuisance to produce intervals that are too narrow, in the way that dropping a variance component does. It does the opposite at low correlation: the interval becomes 1.41× wider and keeps its coverage, and the failure at high correlation is a displacement, not a narrowing. So the two directions of error are not symmetric, and the practical rule is not "always fit everything": fitting a nuisance you could have fixed costs precision, and fixing a nuisance you should have fitted costs validity — and only the second gets worse as the data accumulate.

Counts

Tags are editorial: test marks a lead that can be attacked with a computation, a dataset, or a re-analysis already to hand; note marks one that needs a literature sample, a new measurement, or an institutional change. A "test" tag is not evidence that the lead is true.

Limits

• Every number here is simulated or closed-form. No measured data are used. The three tests are seeded Monte Carlo runs with stipulated parameters, and the tables describe those simulations rather than any material, instrument or literature.
• Test 1 is one generative model. One hundred groups of eight rows, a five-dimensional latent, k = 5, twenty replications per cell. The negative R² at ρ = 0 is real and expected: the features carry no information about the target there, so every learner does worse than the mean. The gap is monotone in ρ in this model, not in general — a different functional form or a different k could break the monotonicity, and the 5.0× k-NN-to-ridge ratio is a property of this design.
• Test 2's tertile d is a descriptive statistic computed on the simulated sample, not a population effect size; it is reported because it is the quantity a reader would compute, not because it is estimable in general. The exact correlation formula holds for independent log-normal factors; a different noise structure gives a different formula with the same sign and the same mechanism.
• Test 3 is linear, Gaussian and homoskedastic, and the nuisance is fixed at two particular wrong values. The qualitative result — that a fixed nuisance trades precision for validity — is standard; the specific coverages are properties of these cells, and 2,000 replications put a Monte Carlo standard error of about 0.005 on each.
• The proof is local. Nonsingular information gives local identifiability; global identifiability needs injectivity of θ ↦ p(·;θ) and is a separate question. Step (iii) uses (A1) and (A3) to integrate the vanishing score along a segment; without a fixed support the argument fails at the boundary of the support. The computed checks are one design and one exactly singular model.
• The conjecture is labelled. The bound in the conjecture is not proved; test 1 is consistent with it but covers one value of k, one group size and one latent dimension.
• No literature search. Several leads are standard results under other names — Manski bounds, Fieller's theorem, the design effect, Brown's method for correlated p-values, the winner's curse — and are stated here as falsifiable claims rather than as discoveries.
• All 200 leads are untested. The tags describe intent, not status. Nothing on this page is a measurement, and all three tests contradicted an expectation I brought to them.

If one sentence survives: identifiability is not a matter of precision but of whether the distribution moves at all when the parameter does, so a singular information matrix, a leaked split, a fixed nuisance and a doubled unit are the same failure in four costumes — each one leaves the arithmetic intact while the interval describes a quantity that is not the one being measured.

Back to the research hub · incoprea_nov3/code/statmeas_tests.py · incoprea_nov3/results/statistical_measurement.json