Training LLMs to follow privacy instructions in reasoning traces prevents unintended data leakage in agentic systems.
Training large language models (LLMs) to follow privacy instructions in their reasoning traces can significantly reduce unintended data leakage in AI agentic systems. Reasoning models often treat their internal reasoning traces as a hidden scratchpad, freely reproducing sensitive user data despite explicit instructions to anonymize or avoid such information, which creates a critical privacy risk. This leakage occurs because models struggle to distinguish between reasoning and final output, sometimes inadvertently exposing private data in their answers or making reasoning traces accessible via prompt injection attacks.
A recent study proposes improving instruction-following capabilities not just in final answers but also within reasoning traces to enhance privacy preservation. By fine-tuning models on a new dataset with explicit constraints on reasoning behavior and introducing a method called Staged Decoding—where separate LoRA adapters are used for reasoning and answer generation—the approach decouples and optimizes instruction adherence at each stage. This technique led to improvements of up to 20.9 points in instruction-following performance and up to 51.9 percentage points on privacy benchmarks across models ranging from 1.7B to 14B parameters.
However, there is a trade-off between privacy and utility: enhancing instruction following in reasoning traces can reduce task performance, as stricter control may limit the model’s ability to reason effectively. Prior work has shown that increasing the reasoning budget (e.g., more thinking steps) improves utility but amplifies leakage in reasoning traces, highlighting a fundamental tension between capability and privacy. Without proper control, models mechanically recollect sensitive data during reasoning—a process observed in 74.8% of reasoning leaks—making them vulnerable even when final outputs appear safe.
Overall, making reasoning models more controllable by enforcing privacy directives in their internal thought processes represents a promising path toward building safer, privacy-aware AI agents.
This paper addresses a critical security vulnerability in agentic AI systems: the unintended leakage of sensitive data through Chain-of-Thought (CoT) reasoning traces. While standard alignment techniques typically sanitize the final output of Large Language Models (LLMs), they often leave intermediate reasoning steps exposed. The authors demonstrate that when models are tasked with complex, multi-step processes—such as tool use or long-context analysis—they frequently inadvertently include Personally Identifiable Information (PII), proprietary secrets, or sensitive context within their internal monologues. This creates a significant privacy risk, as these reasoning traces are often logged, accessible to developers, or used for downstream training, effectively bypassing the safety filters applied to the final response.
The key contribution of this work is the introduction of "Controllable Reasoning Models," a framework designed to enforce privacy constraints specifically within the reasoning phase. The authors propose training methodologies that enable models to interpret and adhere to privacy instructions that target the CoT generation itself. Rather than merely suppressing specific tokens, the model learns to generate "sanitized" reasoning traces that logically lead to the correct answer without exposing sensitive underlying data. Through comprehensive benchmarking, the study shows that this approach significantly reduces information leakage in intermediate steps while maintaining the model's reasoning performance and final accuracy.
This research is vital for the safe deployment of autonomous agents in enterprise and regulated environments. As AI systems evolve from simple chatbots to complex agents that interact with private databases and APIs, the opacity of their internal reasoning processes becomes a major liability. By establishing that models can be trained to be "private thinkers"—capable of high-level reasoning without exposing their internal workings or the sensitive data they process—this paper provides a foundational solution for building trustworthy agentic systems where data confidentiality is preserved throughout the entire computational pipeline.
This paper investigates the privacy risks inherent in agentic large language models (LLMs) that generate reasoning traces—intermediate steps that reveal internal decision-making processes. While reasoning traces improve transparency and interpretability, they can inadvertently leak sensitive information from the model's training data or user inputs. The authors propose a novel training paradigm where LLMs are explicitly instructed to follow privacy-preserving guidelines (e.g., "Do not reveal personal details from your training data") during reasoning. Through controlled experiments, they demonstrate that this approach significantly reduces unintended data leakage while maintaining task performance. The key contribution is a method to align reasoning processes with privacy constraints, addressing a critical gap in the development of deployable AI systems.
The work matters because it confronts a growing challenge in AI safety: the tension between interpretability and privacy. As LLMs are increasingly used in agentic settings (e.g., decision-making, assistance, or autonomous systems), their internal reasoning may expose confidential or proprietary data. Prior efforts focused on post-hoc filtering or differential privacy but often trade off utility or fail to mitigate reasoning-specific leaks. By embedding privacy instructions directly into the model’s reasoning process, this research offers a scalable solution that aligns with emerging regulatory demands (e.g., GDPR, AI Act) while preserving the benefits of transparent AI. The findings underscore the need for "controlled reasoning" as a foundational requirement for trustworthy agentic systems.