SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training

Based on the provided search context, there is no mention of a system or paper titled "SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training."

The search results instead highlight the following related systems and concepts for LLM failure localization:

• Holmes: A system designed to accurately localize irregularities in LLM training by employing an enhanced abnormal operator detection model. It focuses on high accuracy, real-time detection, low overhead, and flexibility across various parallel training strategies.
• Mechanism-Driven Monitors: Approaches for the preemptive detection of LLM training instability, aiming to identify issues before they cause full job failures.
• Mycroft: A tool for tracing dependencies in collective communication to ensure reliable LLM training.
• General Challenges: The context notes that synchronization in LLM pre-training propagates rank-local stalls and numerical errors into job-wide symptoms, making root cause localization difficult. Existing diagnosis often relies on in-process monitors, but failures can still lead to significant throughput degradation and resource waste.

If you are looking for information on Holmes or other specific systems mentioned in the context, please clarify. Otherwise, the specific "SCOUT" system is not covered in the available data.

This material addresses a core observability gap in large-scale LLM pre-training: when a distributed trainer is tightly synchronized, a fault on one rank—such as a stall, slowdown, or numerical anomaly—can quickly become visible as a job-wide symptom, making it hard to identify the original source. The paper argues that common diagnosis approaches are insufficient in this setting. In-process monitors may stop reporting once the trainer blocks or terminates, post-mortem logs often preserve only the synchronized, downstream symptoms rather than the initiating rank-local event, and offline health checks can miss the specific workload and operating conditions that triggered the failure.

The proposed method, SCOUT, uses symmetric consensus outlier detection to localize failures while the pre-training job is still under realistic load. The key idea is that each participating rank contributes to a collective view of system health and progress, and deviations from the peer consensus can reveal which rank or component is behaving anomalously. Rather than relying on a privileged monitor or interpreting ambiguous global symptoms after the fact, SCOUT treats the synchronized job as a distributed ensemble and identifies the outlier whose local state diverges from the consensus baseline. This makes it possible to distinguish local stalls, performance degradation, and numerical errors from broadly shared symptoms.

The contribution matters because it targets one of the practical bottlenecks in running large pre-training jobs at scale: mean time to recovery. By localizing the origin of a failure closer to the point where it occurs, SCOUT can reduce the time spent triaging ambiguous logs, restarting healthy ranks, or reproducing failures offline. It also supports more automated operational responses, such as isolating faulty nodes, rescheduling work, or escalating specific hardware or numerical issues. In short, the paper offers a system-level approach to making distributed LLM training more diagnosable under the conditions where failures actually happen.