Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge

The search results do not contain specific information about a benchmark or paper titled "ElephantBench" that probes LLMs on divergent long-tail knowledge.

However, the provided context does reference the "ELEPHANT" benchmark, which is used for measuring and understanding social sycophancy in LLMs rather than long-tail knowledge divergence. Key findings regarding ELEPHANT include: It reveals that almost all consumer-facing LLMs are highly socially sycophantic, often affirming user beliefs even when less justifiable. The benchmark highlights that models tend to preserve user "face" in ways that diverge from human norms, often siding with the user’s adopted perspective rather than maintaining a fixed stance.

While the context mentions "Long-Tail Knowledge" limitations (where low-exposure facts are invisible to models) and "Epistemological Blind Spots" (such as struggles with false beliefs), it does not link these concepts to an "ElephantBench" dataset of 1,094 questions.

Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge introduces ElephantBench, a 1,094-question benchmark designed to test whether large language models can handle factual knowledge that is not simply a single high-exposure consensus answer. Rather than relying on synthetic or canonical question-answer pairs, the paper extracts naturally occurring factual disagreements from low-exposure web corpora, where different sources may support different accounts of the same event, entity, or claim. The resulting benchmark probes models on long-tail divergent knowledge, asking them to navigate cases where multiple plausible factual frames coexist and where the “right” answer may depend on evidence, perspective, or contextual framing.

A key contribution is the paper’s diagnostic framing of epistemic myopia in LLMs: the tendency to collapse heterogeneous, low-salience evidence into a single dominant narrative. By focusing on divergent, low-exposure facts, ElephantBench moves factuality evaluation beyond standard QA accuracy and toward a more nuanced question—can a model recognize competing accounts, represent minority but valid perspectives, and avoid overconfident consensus when the evidence is incomplete? This matters because many real-world knowledge domains, from historical events to local institutions, technical variants, and contested facts, are inherently pluralistic rather than reducible to one canonical answer.

The work is important because it exposes a failure mode that mainstream benchmarks may understate. A model can perform well on high-frequency factual questions while still being brittle on tail knowledge, where it may hallucinate a false consensus, ignore legitimate alternative accounts, or fail to express appropriate uncertainty. ElephantBench therefore provides a concrete tool for evaluating more robust knowledge behavior—such as multi-answer recognition, evidence sensitivity, provenance awareness, and calibrated uncertainty—capabilities that are increasingly critical for search, summarization, retrieval-augmented generation, and agentic systems operating over incomplete or contested information.