Test-time compute · Reasoning

VeriSearch

Verifier-guided reasoning under fixed inference budgets

Saidur Rahman PulokIndependent projectGSM8K · Qwen2.5-Math-1.5B2026

TL;DR

Can a small learned verifier pick better answers than majority voting when the token budget is fixed? On 48 held-out GSM8K problems it tied with voting. Adaptive early stopping saved 35% of generated tokens for about one point of accuracy, and the verifier’s high AUROC turned out to come from an answer-presence shortcut rather than checking the math.

  • 54.2% vs 55.2%verifier best-of-N vs majority vote at a 1024-token cap
  • −35%generated tokens with adaptive stopping, for about one point of accuracy
  • 0.96 → 0.47verifier AUROC overall vs. among candidates that produced an answer
Held-out answer accuracy versus generated-token cap for each selection policy
Figure 1. Held-out GSM8K accuracy against the generated-token cap per problem, averaged over two sampling seeds, with 95% bootstrap intervals. Majority vote, verifier best-of-N and adaptive stopping track each other closely and sit near the analysis-only oracle upper bound.

Abstract

Sampling several solutions and taking a majority vote (self-consistency) is a strong, cheap baseline for reasoning tasks. A verifier that scores each candidate should, in principle, do two things better: pick the right answer when the vote gets it wrong, and let sampling stop early once it is confident. VeriSearch tests whether either holds up under a hard cap on generated tokens, end to end on an 8 GB laptop GPU, with Qwen2.5-Math-1.5B-Instruct as the generator and a fully fine-tuned DistilRoBERTa verifier.

On 48 held-out GSM8K test problems, the verifier tied with voting. Adaptive early stopping did help, using 35% fewer generated tokens for about one point of accuracy. The verifier's 0.96 AUROC came mostly from a shortcut: almost every wrong training candidate never reached a final answer, so the verifier learned whether an answer was present, not whether it was right. Among candidates that did produce an answer, its AUROC was 0.47, no better than chance. That shortcut is the most useful finding of the pilot.

Method

GSM8Kproblem-level splits GeneratorQwen2.5-Math-1.5B Answer labelsexact numeric match VerifierDistilRoBERTa, fine-tuned Candidate poolsshared, token-capped Selection policiesvote, verifier, early stop Evaluationaccuracy, tokens, ECE
  • Generator: Qwen2.5-Math-1.5B-Instruct, run locally with Transformers. Every sampling call respects the remaining token budget.
  • Verifier: DistilRoBERTa, fully fine-tuned to predict whether a candidate's final answer is correct from the question, the extracted answer and the reasoning.
  • Policies: greedy, single sample, majority vote, verifier best-of-N, and adaptive stopping whose threshold is tuned on validation only and frozen before the test run.
  • Evaluation: problem-level splits, one shared candidate pool per condition, paired bootstrap intervals, calibration metrics and a failure analysis of disagreements.

Pilot setup: 128 train / 32 validation / 48 test problems; up to four candidates per problem at temperature 0.8; 256 new tokens per trace; total caps of 256, 512 and 1024 tokens; sampling seeds 17 and 29.

Results

PolicyAccuracy [95% CI]Mean generated tokensMean candidates
Greedy35.4% [22.9, 50.0]237.91.00
Majority vote55.2% [41.7, 67.7]953.54.00
Verifier best-of-N54.2% [41.7, 66.7]953.54.00
Adaptive early stopping53.1% [40.6, 65.6]617.02.49
Oracle (analysis only)55.2% [41.7, 67.7]953.54.00

Table 1. Held-out GSM8K accuracy at a 1024-token cap, averaged over two sampling seeds. Intervals are 95% percentile bootstrap intervals over 2000 problem resamples. The oracle is an any-correct upper bound and cannot be deployed.

Adaptive stopping accuracy versus consumed tokens
Figure 2. Adaptive stopping: accuracy against mean consumed tokens. The threshold was frozen on validation data before any test generation.
Accuracy versus number of candidates
Figure 3. Accuracy against the number of sampled candidates for each selection policy.

The verifier learned a shortcut

In the training data, 335 of the 339 wrong candidates were ones that never reached a final answer. The verifier therefore mostly learned whether an answer was present at all. A trivial rule that scores 1 when an answer is extracted and 0 otherwise reaches 95.3% accuracy and 0.963 AUROC on the test traces, matching the learned verifier.

Verifier score distributions for correct and incorrect candidates
Figure 4. Verifier scores are almost perfectly bimodal. The nine incorrect candidates with an extracted answer land in the same high-confidence bin as the correct ones.
Verifier reliability diagram
Figure 5. Reliability diagram. Calibration looks good overall (ECE 0.047, Brier 0.047), which hides the failure on answer-bearing candidates.
Test tracesCountVerifier AUROC
All candidates1920.961
  Extracted and correct710.470
  Extracted and incorrect9
  No extracted answer112–

A high verifier score here does not mean the math was checked. With only nine wrong-but-present answers in the test set, the 0.47 figure is itself imprecise, but it shows the overall AUROC is not evidence of verification.

Limitations and next steps

Limitations

  • Small test set: 48 problems, two sampling seeds and one verifier training seed, so intervals are wide.
  • One generator, one dataset and short traces of up to 256 tokens each.
  • Token savings count generated tokens only; verifier scoring, prompt prefill and model loading are excluded, so they are not wall-clock speedups.
  • Labels check the final answer, not the reasoning, and pretraining contamination is not controlled.

What comes next

  • Train the verifier on wrong answers that are present (hard negatives) to remove the shortcut.
  • Run the full 1,319-problem GSM8K test set.
  • Measure end-to-end latency, including the verifier's cost.
  • Compare against a step-level (process) verifier.

This is a small, predeclared pilot, not a full benchmark evaluation.

Run it

Requires Python 3.11–3.13, uv and an NVIDIA GPU for generation and training. CPU tests need neither a GPU nor downloaded models.

git clone https://github.com/saidurpulok/verisearch && cd verisearch
uv sync --frozen --python 3.13
export HF_HOME="$PWD/.cache/huggingface"
uv run python scripts/download_assets.py --config configs/pilot.yaml
bash scripts/run_experiment.sh configs/pilot.yaml

Citation

@software{pulok2026verisearch,
  author  = {Pulok, Saidur Rahman},
  title   = {VeriSearch: Verifier-Guided Reasoning Under Fixed Inference Budgets},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/saidurpulok/verisearch}
}

Builds on GSM8K, self-consistency and Qwen2.5-Math.