Training LLMs on Python execution traces creates neural interpreters for line-by-line prediction, but highlights the need for debugger-like breakpoint stepping to match developer practices.
Training large language models (LLMs) on Python execution traces has led to the development of neural interpreters capable of predicting program behavior line-by-line, enhancing their ability to simulate code execution. The Execution Trace Chain of Thought (ET-CoT) approach, for instance, involves fine-tuning LLMs on detailed execution traces generated by a custom Python interpreter, PyTracify, enabling models to predict outcomes by generating step-by-step traces as a form of reasoning. Similarly, Execution Tuning (E.T.) leverages a custom tracer that captures local and global variables, stack information, and opcode-level events to construct structured trace datasets for training, supporting multiple levels of granularity such as line-by-line or bytecode-level simulation.
Despite these advances, current methods still fall short of fully emulating human debugging practices, which often involve interactive inspection of program state at specific points. While some frameworks like LDB (Large Language Model Debugger) segment programs into basic blocks and verify runtime variable values block-by-block, mimicking a stepwise verification process, they do not yet support dynamic, interactive stepping akin to using breakpoints in a traditional debugger. Recent proposals suggest that a more human-like debugging experience could be achieved by having LLMs operate within a debugger environment—using tools like bdb.Bdb to set breakpoints, inspect frame stacks, and evaluate expressions interactively after exceptions—thereby enabling hypothesis testing and deeper causal analysis during debugging.
This highlights a growing recognition that while trace-based training improves execution fidelity, true alignment with developer workflows requires interactive, debugger-like capabilities that go beyond static trace prediction. Integrating such features could bridge the gap between neural interpretation and practical software development, moving closer to a neural debugger that operates not just by predicting outputs, but by actively exploring and modifying program state in real time.
This research investigates the application of Large Language Models (LLMs) to dynamic program analysis, specifically focusing on the ability of these models to interpret and predict Python execution traces. Rather than treating code solely as static text for generation, the authors train models to function as "neural interpreters" that process sequences of line numbers, variable states, and function calls. By learning from execution traces, the model attempts to simulate the internal state of a program, predicting the next line of execution and the resulting variable values step-by-step. This approach moves beyond standard code completion, aiming for a deeper understanding of program semantics and runtime behavior.
A key contribution of this work is the identification of a critical gap between standard sequential trace prediction and actual developer workflows. The authors argue that while LLMs are good at processing linear sequences, human developers rely heavily on non-linear navigation, such as setting breakpoints and stepping through specific code segments. To address this, the paper proposes a "neural debugger" framework that incorporates breakpoint stepping capabilities. By allowing the model to jump to arbitrary points in the execution trace—mimicking the behavior of a traditional debugger—the system better aligns with human debugging practices and improves the model's ability to maintain context over long execution histories.
The significance of this research lies in its potential to transform AI-assisted programming from simple text generation to robust code understanding and debugging. By bridging the gap between static code analysis and dynamic execution, such neural debuggers could eventually automate the identification of logic errors, verify program correctness, and explain runtime behavior to developers. This suggests a future where LLMs act not just as autocomplete engines but as active participants in the software development lifecycle, capable of "running" code mentally to predict outcomes and diagnose failures.
This paper explores the development of a neural debugger for Python, leveraging execution traces to train large language models (LLMs) into neural interpreters capable of predicting program behavior line-by-line. The work builds on prior efforts to use LLMs for code execution prediction but identifies a critical gap: while existing models can predict outputs, they lack the fine-grained, interactive debugging capabilities that developers rely on—such as setting breakpoints, stepping through code, and inspecting variable states. The authors propose a debugger-like stepping mechanism that aligns more closely with traditional debugging workflows, enabling LLMs to simulate execution traces with higher fidelity.
The key contributions include: 1. Execution Trace-Based Training: The model is trained on detailed Python execution traces, allowing it to learn not just syntax but also runtime behavior, including control flow and variable changes. 2. Debugger-Style Stepping: The proposed neural debugger supports breakpoint setting and step-by-step execution, mimicking how developers interact with tools like pdb. This addresses the limitation of prior neural interpreters, which often lack granular control over execution. 3. Empirical Validation: The authors demonstrate that their approach improves prediction accuracy for complex Python programs, particularly in scenarios requiring dynamic analysis (e.g., loops, conditionals, and function calls).
This work matters because it bridges the gap between LLM-based code understanding and practical debugging. While neural interpreters have shown promise in static prediction, developers need tools that operate dynamically—inspecting state, stepping through code, and adapting to runtime conditions. By incorporating debugger-like functionality, this research paves the way for more robust AI-assisted programming tools, potentially enhancing IDEs, automated refactoring, and even program synthesis. The interplay between LLMs and traditional debugging paradigms could redefine how developers interact with code, making automated reasoning more intuitive and reliable.