Theory 20/Exploration vs Exploitation/Reinforcement learning
Exploration Is Paid in Regret
The claim
Exploration in learning trades against cumulative regret, so the optimal exploration rate is set by the horizon rather than by the task alone.
01Mechanism
Each exploratory action forgoes known reward; the total cost of learning is bounded by the horizon over which the knowledge is used.
02Prediction
Exploration should fall as the horizon shortens.
03Refutation
Exploration independent of horizon would remove the trade-off.