Test kit

Test kit

Seven checks that audit a numeric catalogue for the defects that survive review.

What this is for. These are the failure modes that generated the
sixteen entries in the corrections register, written as code so they can be
run against your data rather than mine. Pure Python and NumPy, no dependencies. Each check
returns a small dictionary; each is under twenty lines, so it can be read and argued with.

The checks

Check Question it answers Fails when
threshold_audit Does a threshold comparison silently drop boundary values? boundary values exist
quantisation_audit Do reported values carry a rounding convention that biases a mean? χ² is large
derived_column_audit Is one column a deterministic transform of another? residual σ ≤ 0.10 dex
conditioning_audit Can this many unknowns be separated by this many measurements? rank deficient or condition > 10³
count_bound_audit Is the model identified at all, before any question of precision? points < parameters
estimator_audit How much do standard estimators of one quantity disagree? spread > 1.05
per_scale_audit Is a pooled statistic hiding a mixture of populations? per-group spread > 1.5

Worked example, on real data

Run against 24,897 USGS earthquake records (2024, M ≥ 2.5). Every number below came out of the
toolkit, not out of a document.

1. Threshold

The completeness threshold was 4.4. A comparison written as m >= 4.35 + 0.05 evaluates
its right side to 4.400000000000001, and drops the events sitting exactly on the boundary.

Boundary events present Kept by naive comparison Kept by corrected comparison Dropped
2,060 6,351 8,411 2,060

Effect on the b-value: 1.252 → 0.980, a 28 % error, from a comparison operator.

2. Quantisation

Last-digit counts over the same 24,897 magnitudes give χ² = 3,150 against a uniform expectation,
with a 37 % excess at digit 5. Reported magnitudes carry a rounding convention that a mean will inherit.

3. Estimators

Aki–Utsu maximum likelihood 1.252 against cumulative least squares 1.142 — a spread of 1.10 on the
same 8,411 events. Any single reported b hides this choice.

4. Per-scale mixing

Magnitude scale Events b
Md 2,179 0.728
Mww 1,163 0.995
ML 8,168 1.047
Mwr 705 1.562
mb 12,478 1.742
Spread — 2.39×

The pooled value for the same file is 1.252 — inside the range but belonging to no single scale.

5. Conditioning

Fitting two similar exponential sensitivities on five temperatures gives a condition number of
26.9. Adding a third, nearly collinear sensitivity and letting it carry its own
parameter raises that to 2,001.8 — a 74-fold penalty. The same third measurement added
as a redundant observation instead of a new parameter lowers it. The penalty belongs to the
parameter count, not the measurement count. This is the correction recorded as #15.

Known limitations, including the one found while building this

  • The derived-column check produces false positives on monotone inverse pairs. Run on
    a sorted column against its own reverse, it flags “derived” when the two are merely anti-correlated. The
    0.10 dex rule was calibrated on forward transforms. Use it to triage, not to conclude. This defect was
    discovered during the build of this page and is entry 16 in the corrections register.
  • Completeness uses maximum curvature, which is crude and can underestimate the threshold.
  • Seven checks do not cover everything. Kriging weights, interpolation leverage, closed-sum
    and selection effects are absent.
  • Each check reports a diagnostic, not a verdict. A large χ² means “look here”, not “the data are wrong”.

Use

import auditkit as ak

ak.threshold_audit(values, mc=4.4)          # -> dropped boundary events
ak.quantisation_audit(values)               # -> last-digit chi-square
ak.derived_column_audit(x, y)               # -> residual spread in decades
ak.conditioning_audit(kernel_matrix)        # -> rank, condition number
ak.count_bound_audit(params=3, points=3)    # -> identifiable at all?
ak.estimator_audit(magnitudes, mc=4.4)      # -> estimator spread
ak.per_scale_audit(mags, types)             # -> pooled-mixture detection

The point of a test kit is not that it finds faults. It is that a fault found once can be
checked for everywhere else in minutes, which is the only way an inspection becomes routine.

auditkit.py is 70 lines and ships with a demo script and its JSON
output. 4 November 2026.