V-MATH and Veri-Math: Step-Level Verification for Enhancing Mathematical Reasoning in LLMs
Rugaba John Paul and Tang Shengbing*
Abstract
Recent advances in large language models (LLMs) have led to notable progress in automated problem solving; however, mathematical reasoning remains a persistent point of weakness, particularly in settings that demand long chains of logically dependent steps. Errors introduced early in a solution frequently propagate unnoticed, yet most existing benchmarks evaluate only the final answer and therefore provide little insight into where or why a model’s reasoning breaks down. This paper addresses this gap through two complementary
contributions: the V-MATH dataset and the Veri-Math step-level verification model. V-MATH comprises more than 260,000 problems ranging from elementary material to advanced university and competition topics. Each problem is accompanied by a detailed, stepwise solution annotated with explicit correctness labels ( Ꝋ), allowing the precise identification of logical failures within the solution process. Constructing a dataset at this scale highlights several challenges, including maintaining consistent standards for step granularity across disparate mathematical domains and preserving annotation quality in the presence of diverse problem structures and reasoning styles.
Building on this dataset, this paper introduces Veri-Math, a model trained to assess the validity of individual reasoning steps rather than final answers. The verifier employs a dual-objective training strategy that combines step classification with ranking, enabling it to discriminate between sound reasoning and subtle logical inconsistencies. Evaluations on GSM8K, MATH, MathPile, and PRM-800K demonstrate notable gains when existing language models are paired with Veri-Math: for example, LLaMA 3-8B achieves an average improvement of 10.9%, Qwen improves by 15.4%, and Gemma 2- 2B achieves a 19.9% increase on PRM-800K. On the V-MATH test set, the verifier attains 84.4% accuracy in step-correctness classification.
Taken together, V-MATH and Veri-Math establish a framework for examining mathematical reasoning at a finer level of detail than has previously been feasible. While the work exposes ongoing challenges such as the difficulty of detecting highly nuanced logical errors and the inherent complexity of constructing consistent step-level annotations across heterogeneous topics, it provides a foundation for building more reliable and transparent computational reasoning systems suited to use in education, scientific inquiry, and formal problem solving.
