Reasoning and inference Read this week

The Race Is Decided Early! A Kernel Law Predicts Lock-In in Majority-Vote Test-Time RL

Hwang et al.

In test-time reinforcement learning rewarded by majority vote, each question locks into one answer early. The authors derive the probability that the runner-up wins as a normal tail probability of the initial logit gap, corrected through the kernel of the shared parameters.

Filed under Reasoning and inference, Probability

Test-time reinforcement learning trains a language model on unlabelled questions by rewarding the samples that agree with its most frequent answer. During training each question locks into one answer, on hard questions often a wrong one, and until now which answer won was known only afterwards.

The authors show the outcome is decided early, in a short race driven by sampling noise between answers that start close together. With a separate logit for each answer, the runner-up wins with a normal tail probability of the initial gap, scaled by an explicit noise constant. In a network the logits share parameters, so updates to trailing answers also move the gap between the two leaders; a closed form computes the shifted probability from a kernel probe and the starting logits. In LoRA experiments on three models from two families, predictions made before training, with the kernel frozen at its starting value, forecast the locked-in answer better than a logistic model refitted on the outcomes.