Introduces MSI-Bench, a benchmark of short multi-party dialogues, to evaluate AI agents on multi-speaker voice interaction challenges absent from one-on-one settings.
MSI-Bench is a benchmark introduced to evaluate AI agents' ability to handle multi-speaker voice interactions, addressing challenges like speaker-scoped decision-making, memory, and reasoning that are absent in one-on-one settings. It consists of 1,152 test cases (576 English, 576 Mandarin) featuring short, multi-party, multi-turn audio scenes that conclude with an assistant-directed request, evaluated using atomic rubrics for grounding, authority, privacy, and constraint priority.
The benchmark assesses three capability families: multi-speaker memory, multi-speaker instruction following, and multi-speaker reasoning. Evaluation results indicate that current systems struggle with shared voice interaction, with the strongest configuration passing all rubrics on only 66.8% of English and 54.5% of Mandarin cases, highlighting significant gaps in handling interleaved group constraints and speaker-specific contexts.
MSI-Bench is a benchmark for evaluating AI agents in multi-speaker voice interaction, focusing on short multi-party dialogues rather than the more common one-on-one conversational setting. The material frames the core problem as a shift from dyadic assistant-style speech to group-oriented interaction, where an agent must operate amid multiple speakers, competing conversational roles, and interaction patterns that do not reduce cleanly to a single human–agent exchange.
Its key contribution is a structured testbed for measuring capabilities that are often absent from single-user voice benchmarks, such as tracking who is speaking, interpreting addressee and turn-taking dynamics, handling overlapping or fragmented speech, and maintaining coherent collaborative context across multiple participants. By using short dialogues, the benchmark appears designed to make these challenges tractable for controlled evaluation, allowing systematic assessment of how well agents reason about speaker identity, conversational state, and group coordination rather than merely generating appropriate responses in isolation.
This matters because collaborative AI agents are increasingly expected to participate in meetings, classrooms, shared workspaces, and other multi-party environments where voice interaction is central. A benchmark like MSI-Bench helps surface failure modes that one-on-one evaluation can miss, providing a more realistic basis for comparing and improving agents intended for group collaboration rather than private assistance.