The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR$\rightarrow$LLM Pipelines?

Current speech large language models (Speech LLMs) largely perform implicit automatic speech recognition (ASR), making them behaviorally and mechanistically equivalent to cascaded Whisper-to-LLM systems on tasks that can be solved from a transcript alone . This forms the basis of the Cascade Equivalence Hypothesis, which posits that many Speech LLMs do not fundamentally differ from traditional ASR-then-LLM pipelines despite their end-to-end training .

Through matched-backbone evaluations across four Speech LLMs and six tasks, studies show that models like Ultravox are statistically indistinguishable from their cascade counterparts, achieving a Cohen’s kappa of $$\kappa=0.93$$ . Mechanistic analyses using logit lens techniques reveal that literal text representations emerge in the hidden states of these models, indicating an internal transcription process . Furthermore, LEACE concept erasure experiments demonstrate that text representations are causally necessary—removing them collapses accuracy to near-zero levels in both Ultravox and other tested architectures .

However, this equivalence is not universal. Qwen2-Audio shows genuine divergence from cascade behavior, indicating that architecture plays a critical role in determining whether a Speech LLM adheres to the cascade paradigm . This suggests that while many current Speech LLMs function as expensive, implicit cascades, they offer no inherent advantage over explicit pipelines for transcript-based tasks—and under noisy conditions, they may perform worse, with clean-condition advantages reversing by up to 7.6% at 0 dB .

These findings are key to understanding the internal workings of multimodal LLMs, revealing that despite their unified architecture, many still rely on simulating a transcription step before higher-level language processing, much like traditional cascaded systems .

This paper investigates the internal functional mechanisms of end-to-end Speech Large Language Models (Speech LLMs) by proposing and testing the "Cascade Equivalence Hypothesis." The authors posit that on tasks solvable solely through transcript information—ignoring prosodic or paralinguistic cues—Speech LLMs functionally mimic traditional cascaded pipelines composed of an Automatic Speech Recognition (ASR) module (such as Whisper) feeding a text-based LLM. Through rigorous empirical testing and mechanistic analysis, the study demonstrates that these Speech LLMs perform implicit ASR, generating intermediate representations that are highly similar to the discrete text outputs produced by cascaded systems.

The research is particularly significant for the field of mechanistic interpretability, as it peels back the layers of multimodal architectures to reveal that their reasoning processes often mirror established pipeline workflows. By establishing the specific conditions under which Speech LLMs behave like ASR-to-LLM cascades, the authors provide a critical framework for understanding when end-to-end training provides a genuine advantage over modular systems. This insight not only demystifies the "black box" of speech processing in foundation models but also guides future research in identifying specific tasks where non-textual audio features are actually leveraged, thereby distinguishing between simple implicit transcription and true multimodal reasoning.

Summary of The Cascade Equivalence Hypothesis This paper investigates whether speech-based large language models (LLMs) internally perform implicit automatic speech recognition (ASR) before generating responses, effectively mimicking a two-stage ASR→LLM pipeline. The authors test this Cascade Equivalence Hypothesis by comparing the behavior of end-to-end speech LLMs (e.g., Whisper-to-LLM) against explicit cascaded systems on "transcript-solvable" tasks—those where the correct answer can be derived solely from the transcription. Through extensive testing and mechanistic analyses (e.g., probing intermediate representations), they find strong empirical support for the hypothesis: speech LLMs often rely on implicit ASR-like processes, even when not explicitly trained as cascades. This challenges the assumption that these models process speech holistically and instead suggests they may decompose the task into modular subcomponents.

The paper’s key contributions include: 1. Empirical validation of the cascade equivalence via controlled experiments across tasks (e.g., question answering, summarization) and model families (e.g., Whisper, LLama). 2. Mechanistic insights into how speech LLMs encode and separate acoustic and semantic information, using tools like attention pattern analysis and ablation studies. 3. Implications for multimodal AI, highlighting that even "end-to-end" models may exhibit pipeline-like behavior, with potential implications for interpretability, robustness, and design of future speech-language systems.

Why It Matters This work is significant for understanding the inner workings of multimodal LLMs. If speech LLMs implicitly perform ASR, it suggests that their "multimodal" processing may be more modular than previously assumed—with implications for: - Reliability: Cascaded pipelines are easier to debug and align, but implicit cascades may introduce hidden failure modes. - Efficiency: Explicit pipelines could be more sample-efficient for tasks where transcription suffices, while end-to-end models may waste capacity on redundant processing. - Generalization: The findings raise questions about whether speech LLMs truly leverage acoustic cues beyond transcript-level information, or if they rely on learned shortcuts.

For researchers in speech AI, NLP, and multimodal systems, this paper provides a framework for probing how models handle cross-modal data and underscores the need for more transparent architectures in future development.