DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science

DARE-bench is a benchmark designed to evaluate large language models (LLMs) in complex, multi-step data science tasks, addressing two major gaps in existing evaluation frameworks: the lack of standardized, process-aware assessment of instruction adherence and modeling fidelity, and the scarcity of accurately labeled training data . It introduces a comprehensive set of 6,300 Kaggle-derived tasks, each with verifiable ground truth, enabling objective, reproducible, and fine-grained evaluation of both instruction fidelity and model performance across classification, regression, and time-series tasks . Unlike benchmarks that rely on human or model-based judges, DARE-bench ensures evaluation accuracy through deterministic validation, supporting agentic workflows and tool-augmented reasoning .

The benchmark provides large-scale training and evaluation datasets, facilitating the development of models with improved data science capabilities. Experiments show that fine-tuning on DARE-bench significantly enhances model performance—supervised fine-tuning increases Qwen3-32B's accuracy by 1.83×, while reinforcement learning improves Qwen3-4B by over 8×, underscoring its value as both an evaluation tool and a training resource . DARE-bench supports standardized assessment across key dimensions such as instruction fidelity (IF), modeling metrics (MM), and execution correctness, making it a robust framework for advancing LLMs in data science applications .

DARE-bench introduces a rigorous framework designed to evaluate the performance of Large Language Models (LLMs) specifically within the domain of complex data science. Unlike traditional benchmarks that often prioritize final output accuracy, DARE-bench focuses on "process-aware evaluation," assessing the model's ability to navigate the full lifecycle of data science tasks. The paper specifically addresses two critical dimensions: Modeling Fidelity, which measures the model's ability to accurately capture and represent underlying data patterns, and Instruction Fidelity, which evaluates how precisely the model adheres to complex, multi-step user directives. By moving beyond simple query-response metrics, this benchmark provides a more nuanced view of an LLM's capability to function as an autonomous agent in scientific and analytical workflows.

A key contribution of this work is the construction of a dataset that addresses the significant gap in labeled training data for process-oriented data science tasks. The authors curate a collection of problems that require genuine reasoning—ranging from data cleaning and preprocessing to model selection and interpretation—rather than mere code completion or memorization. This allows for a granular analysis of model performance, distinguishing between failures caused by hallucination and those caused by a lack of domain-specific reasoning. The benchmark establishes a new standard for assessing LLMs, providing the community with the tools necessary to diagnose and improve the reliability of models in high-stakes analytical environments.

The significance of DARE-bench lies in its potential to bridge the disconnect between general-purpose LLMs and the rigorous demands of professional data science. As organizations increasingly look to automate analytics and data exploration, the reliability of the process becomes just as critical as the accuracy of the result. By highlighting deficiencies in instruction following and modeling logic, DARE-bench directs research toward developing models that are not only knowledgeable but also trustworthy and precise collaborators in data-driven discovery. This evaluation framework is essential for advancing the state of the art in automated data analysis and ensuring that LLM applications in science are robust and reproducible.

# Summary of DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science

DARE-bench introduces a novel benchmark designed to evaluate the modeling and instruction fidelity of Large Language Models (LLMs) in complex data science tasks. The benchmark addresses a critical gap in current LLM evaluation frameworks, which often focus on simple task completion rather than assessing whether LLMs follow correct data science methodologies, produce reliable code, and generate interpretable insights. DARE-bench consists of rigorously designed tasks that require LLMs to perform end-to-end data science workflows—from data preprocessing and exploratory analysis to statistical modeling and result interpretation—while adhering to best practices.

A key contribution of DARE-bench is its process-aware evaluation, which goes beyond traditional accuracy metrics to assess whether LLMs follow logical, defensible steps in their reasoning. The benchmark also introduces labeled training data, enabling supervised fine-tuning of LLMs to improve their fidelity in data science applications. By benchmarking LLMs on real-world datasets and tasks, DARE-bench provides a more nuanced understanding of their capabilities and limitations in applied data science. This work is significant because it establishes a standardized way to measure not just the correctness of LLM outputs but also the transparency, reproducibility, and methodological rigor of their data science workflows—critical factors for real-world deployment in research and industry.

Why It Matters: As LLMs become integral to data science workflows, ensuring their outputs are both accurate and methodologically sound is paramount. DARE-bench fills a critical need by providing a framework to evaluate LLMs not just on task completion but on process fidelity, helping researchers and practitioners identify models that generate reliable, interpretable, and actionable insights. This benchmark could accelerate the development of more robust LLMs for data science, improve trust in AI-generated analyses, and set new standards for evaluating AI assistants in technical domains.