Evaluation and Benchmarking of LLM Agents: A Survey

Evaluating multi-agent LLM systems requires distinct methodologies compared to traditional reinforcement learning (RL) approaches, primarily because LLM agents do not rely on predefined reward structures but instead coordinate through natural language, strategic reasoning, and decentralized problem-solving . Unlike RL-driven agents, which depend on fixed reward signals for coordination, LLM-based agents engage in dynamic interactions that necessitate new evaluation frameworks focused on collaborative efficiency, communication quality, and emergent behaviors . This divergence underscores the need for benchmarks specifically designed to assess multi-agent coordination in realistic, open-ended environments such as financial decision-making and AI research automation .

Recent surveys emphasize that current evaluation objectives for LLM agents span task completion, output quality, tool use, planning, memory retention, and safety, with a growing focus on multi-agent collaboration metrics like information sharing effectiveness, adaptive role switching, and reasoning ratings . Benchmarks such as MultiAgentBench (MARBLE) have been developed to evaluate these dynamics using milestone-based key performance indicators, assessing not only task success but also the quality of collaboration and competition among agents . These frameworks support various coordination protocols—including star, chain, tree, and graph topologies—and incorporate strategies like cognitive planning, which has been shown to improve milestone achievement rates by 3% .

Moreover, existing evaluations reveal a significant imbalance in the coverage of multi-agent failure modes: while 26 out of 32 identified papers measure miscoordination, only five address collusion, and most real-world AI threat models—such as cybersecurity, CBRN, and persuasion—remain underrepresented in current benchmarks . This gap highlights the importance of developing new, threat-informed evaluations to better understand risks in large-scale agent collaboration, particularly as multi-agent systems may play a central role in automating AI research by 2030 .

Dynamic, interactive benchmarks like WebArena, AssistantBench, and WorkArena simulate real-world complexity by incorporating long-horizon planning, user-agent interaction, and enterprise-level workflows, moving beyond static evaluation paradigms . These environments enable more realistic assessment of agent behavior, including robustness to interface changes and multimodal, multistep reasoning . The shift toward holistic, process-oriented evaluation—encompassing interaction modes, datasets, metrics computation, tooling, and contexts—reflects a broader effort to standardize and scale agent assessment for real-world deployment .

This survey provides a comprehensive review of the methodologies and challenges involved in evaluating Large Language Model (LLM) agents, with a specific focus on the limitations of applying traditional Reinforcement Learning (RL) paradigms to agentic systems. The authors argue that unlike standard RL environments, which rely on clearly defined reward functions and state transitions, multi-agent LLM systems operate in open-ended environments where rewards are often sparse, delayed, or subjective. Consequently, the paper explores the necessity of developing new evaluation frameworks that can assess complex behaviors such as tool use, long-term planning, and multi-agent collaboration—capabilities that static question-answering benchmarks fail to capture.

Key contributions of the work include a detailed taxonomy of current evaluation benchmarks categorized by environment type (e.g., web-based, social, or knowledge-intensive) and agent architecture. The survey highlights the difficulty of measuring "emergent" properties in multi-agent systems, where the collective intelligence differs significantly from individual agent capabilities. It further critiques existing approaches, such as LLM-based evaluators and human-in-the-loop feedback, discussing their respective biases and scalability issues. The authors advocate for holistic evaluation metrics that consider not just final task success, but also the efficiency, safety, and reasoning trace of the agent.

This research is critical for the advancement of reliable agentic AI, as it identifies the "evaluation gap" currently hindering the deployment of autonomous systems in high-stakes domains. By establishing that the lack of predefined rewards requires a fundamental shift in how we benchmark intelligence, the paper sets the agenda for future research in standardized agent testing. For practitioners and researchers, this survey serves as a foundational guide to understanding the current landscape of agent evaluation and the rigorous standards required to move beyond prototype demos towards robust, production-grade multi-agent systems.

# Evaluation and Benchmarking of LLM Agents: A Survey

This survey critiques traditional reinforcement learning (RL)-based evaluation methods for multi-agent large language model (LLM) systems, arguing that their reliance on predefined rewards is ill-suited for complex, emergent behaviors in LLM-driven agents. The paper highlights that standard RL benchmarks (e.g., reward-shaped tasks) fail to capture the unique challenges of multi-agent LLM interactions, such as dynamic goal alignment, emergent coordination, and adversarial or cooperative strategies. Instead, it advocates for novel evaluation frameworks that account for the open-ended, non-stationary nature of LLM agents, where rewards may not be predefined or may evolve during interaction.

Key contributions include a taxonomy of existing evaluation approaches, from rule-based metrics to human judgment, and a discussion of emerging methods like post-hoc reward modeling and self-play evaluation. The paper also stresses the importance of benchmarking generalization across tasks, adversarial robustness, and scalability—dimensions often overlooked in single-agent RL settings. This work is significant because it underscores the need for a paradigm shift in multi-agent LLM evaluation, bridging the gap between theoretical RL principles and the practical realities of deploying autonomous language agents in unpredictable environments. For researchers and practitioners, it serves as a call to develop more adaptive, context-aware evaluation protocols that reflect the true capabilities (and limitations) of LLM-driven systems.