TL;DR
Can a small learned verifier pick better answers than majority voting when the token budget is fixed? On 48 held-out GSM8K problems it tied with voting. Adaptive early stopping saved 35% of generated tokens for about one point of accuracy, and the verifier’s high AUROC turned out to come from an answer-presence shortcut rather than checking the math.
- 54.2% vs 55.2%verifier best-of-N vs majority vote at a 1024-token cap
- −35%generated tokens with adaptive stopping, for about one point of accuracy
- 0.96 → 0.47verifier AUROC overall vs. among candidates that produced an answer
Abstract
Sampling several solutions and taking a majority vote (self-consistency) is a strong, cheap baseline for reasoning tasks. A verifier that scores each candidate should, in principle, do two things better: pick the right answer when the vote gets it wrong, and let sampling stop early once it is confident. VeriSearch tests whether either holds up under a hard cap on generated tokens, end to end on an 8 GB laptop GPU, with Qwen2.5-Math-1.5B-Instruct as the generator and a fully fine-tuned DistilRoBERTa verifier.
On 48 held-out GSM8K test problems, the verifier tied with voting. Adaptive early stopping did help, using 35% fewer generated tokens for about one point of accuracy. The verifier's 0.96 AUROC came mostly from a shortcut: almost every wrong training candidate never reached a final answer, so the verifier learned whether an answer was present, not whether it was right. Among candidates that did produce an answer, its AUROC was 0.47, no better than chance. That shortcut is the most useful finding of the pilot.
Method
- Generator: Qwen2.5-Math-1.5B-Instruct, run locally with Transformers. Every sampling call respects the remaining token budget.
- Verifier: DistilRoBERTa, fully fine-tuned to predict whether a candidate's final answer is correct from the question, the extracted answer and the reasoning.
- Policies: greedy, single sample, majority vote, verifier best-of-N, and adaptive stopping whose threshold is tuned on validation only and frozen before the test run.
- Evaluation: problem-level splits, one shared candidate pool per condition, paired bootstrap intervals, calibration metrics and a failure analysis of disagreements.
Pilot setup: 128 train / 32 validation / 48 test problems; up to four candidates per problem at temperature 0.8; 256 new tokens per trace; total caps of 256, 512 and 1024 tokens; sampling seeds 17 and 29.
Results
| Policy | Accuracy [95% CI] | Mean generated tokens | Mean candidates |
|---|---|---|---|
| Greedy | 35.4% [22.9, 50.0] | 237.9 | 1.00 |
| Majority vote | 55.2% [41.7, 67.7] | 953.5 | 4.00 |
| Verifier best-of-N | 54.2% [41.7, 66.7] | 953.5 | 4.00 |
| Adaptive early stopping | 53.1% [40.6, 65.6] | 617.0 | 2.49 |
| Oracle (analysis only) | 55.2% [41.7, 67.7] | 953.5 | 4.00 |
Table 1. Held-out GSM8K accuracy at a 1024-token cap, averaged over two sampling seeds. Intervals are 95% percentile bootstrap intervals over 2000 problem resamples. The oracle is an any-correct upper bound and cannot be deployed.
- The verifier didn't beat voting. At 1024 tokens it was one point behind; at 512 tokens it was one point ahead. Both gaps are within noise at this sample size.
- Adaptive stopping saved tokens. It used 35.3% fewer generated tokens than scoring the full pool and lost 1.04 points of accuracy.
The verifier learned a shortcut
In the training data, 335 of the 339 wrong candidates were ones that never reached a final answer. The verifier therefore mostly learned whether an answer was present at all. A trivial rule that scores 1 when an answer is extracted and 0 otherwise reaches 95.3% accuracy and 0.963 AUROC on the test traces, matching the learned verifier.
| Test traces | Count | Verifier AUROC |
|---|---|---|
| All candidates | 192 | 0.961 |
| Extracted and correct | 71 | 0.470 |
| Extracted and incorrect | 9 | |
| No extracted answer | 112 | – |
A high verifier score here does not mean the math was checked. With only nine wrong-but-present answers in the test set, the 0.47 figure is itself imprecise, but it shows the overall AUROC is not evidence of verification.
Limitations and next steps
Limitations
- Small test set: 48 problems, two sampling seeds and one verifier training seed, so intervals are wide.
- One generator, one dataset and short traces of up to 256 tokens each.
- Token savings count generated tokens only; verifier scoring, prompt prefill and model loading are excluded, so they are not wall-clock speedups.
- Labels check the final answer, not the reasoning, and pretraining contamination is not controlled.
What comes next
- Train the verifier on wrong answers that are present (hard negatives) to remove the shortcut.
- Run the full 1,319-problem GSM8K test set.
- Measure end-to-end latency, including the verifier's cost.
- Compare against a step-level (process) verifier.
This is a small, predeclared pilot, not a full benchmark evaluation.
Run it
Requires Python 3.11–3.13, uv and an NVIDIA GPU for generation and training. CPU tests need neither a GPU nor downloaded models.
git clone https://github.com/saidurpulok/verisearch && cd verisearch
uv sync --frozen --python 3.13
export HF_HOME="$PWD/.cache/huggingface"
uv run python scripts/download_assets.py --config configs/pilot.yaml
bash scripts/run_experiment.sh configs/pilot.yaml
Citation
@software{pulok2026verisearch,
author = {Pulok, Saidur Rahman},
title = {VeriSearch: Verifier-Guided Reasoning Under Fixed Inference Budgets},
year = {2026},
version = {0.1.0},
url = {https://github.com/saidurpulok/verisearch}
}
Builds on GSM8K, self-consistency and Qwen2.5-Math.