Theory 36/Society, Computation & Engineering /Autonomous systems
Autonomy Is a Tail Problem
Progress is reported as miles per disengagement. Safety is decided by the thousandth case.
Autonomous vehicle safety is set by the frequency of rare scenarios rather than by average disengagement, so a system with better averages can be less safe if its failure tail is heavier.
01The evidence
Disengagement rates have improved faster than incident rates, which suggests the two measure different things.
02The tension
Metrics use average disengagements, which are available and comparable and do not represent the tail that determines safety.
03The mechanism
Safety is a function of the probability of the rare scenario multiplied by the consequence. Improvements in common scenarios barely move that product, while a single unresolved rare class dominates it.
That makes the tail the determinant, and predicts incident rates should be predicted from rare-scenario coverage rather than from averages.
04What this is not
Those metrics summarise the middle. This targets the tail, which is where safety is decided.
05Predictions
- P1Incidents should track rare-scenario coverage.
- P2Average improvements should not predict incidents.
- P3Tail coverage should separate systems.
06Refutation
Incidents tracking average disengagement would favour the average metric.
07What it changes
It would change how autonomy is certified.