EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

EarlyEval is a lightweight framework that reduces the cost of LLM agent benchmarking by predicting an agent’s final outcome from its intermediate behavior, allowing runs to be halted early. It uses LightGBM classifiers trained on historical trajectories to detect success or failure indicators, introducing negligible per-step overhead.

Across benchmarks like SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval eliminates 13%–26% of execution steps and up to 44.1% of input tokens while maintaining 89%–97% prediction accuracy. This approach preserves leaderboard rankings and relative agent performance, with resolve rates deviating by only 1–2 percentage points on average.

EarlyEval addresses a growing cost problem in evaluating LLM-based agents: full benchmark runs often require long, multi-step trajectories involving tool calls, environment interactions, retries, and repeated resets, making model comparison and iterative development expensive. The paper’s central idea is to predict an agent’s final outcome—such as task success, reward, or benchmark score—from a short prefix of its execution, rather than always waiting for the trajectory to finish. In effect, it reframes agent evaluation as an early-prediction problem: given partial trajectory evidence, estimate the distribution over final performance and use that estimate to decide whether to continue, stop, or allocate further compute.

The key contribution is a practical framework for turning early signals into reliable performance forecasts. These signals may include partial action/observation history, intermediate task states, tool-call patterns, error indicators, and other trajectory-level features that are informative before the episode is complete. The paper emphasizes the cost–accuracy tradeoff inherent in early stopping: the earlier a prediction is made, the cheaper it is, but the noisier the evidence. A useful version of the method therefore needs not only predictive accuracy, but also calibration and uncertainty-aware decision rules so that high-confidence failures can be cut short while ambiguous cases can be run to completion or sampled selectively.

This matters because agent evaluation is often the rate-limiting step in research and deployment. If final outcomes can be predicted cheaply with acceptable fidelity, teams can screen large model or prompt variants more quickly, focus full evaluations on uncertain or high-stakes cases, and reduce wasted compute on trajectories that are already likely to fail. More broadly, EarlyEval points toward a broader pattern for LLM-agent research: rather than treating evaluation as an all-or-nothing process, it can be made adaptive, budget-aware, and statistically efficient by learning when enough evidence has accumulated to make a trustworthy judgment.