Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views

Yes, recent research confirms that reallocating tokens from document repetition to auxiliary views significantly improves knowledge acquisition in Large Language Models (LLMs) under a fixed token budget.

Key findings from the study "Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views" include:

• Superior Performance: Replacing document repetition with auxiliary views (reformulations, analogies, and explanations) leads to better factual recall and inference capabilities than training on source documents alone or with simple paraphrases.
• Mechanistic Efficiency: Auxiliary views induce a distinct layer-wise bias and compression in the model's feed-forward networks, allowing for a more generalizable encoding of knowledge that requires less parameter movement than memorizing surface-level variants.
• Scalability: The benefits of auxiliary views grow with model size; while smaller models (e.g., 1B) see little advantage, larger models (7B–32B) show substantial gains in understanding and robustness against forgetting.
• Teacher Independence: The effectiveness of these views does not depend on the strength of the model generating them, suggesting they function as general data augmentation rather than distillation from a strong teacher.

This material investigates how large language models acquire knowledge during pre-training when the total token budget is fixed. Instead of treating repeated exposure to the same documents as the default way to reinforce learning, it compares a repetition-only training strategy with one that reallocates tokens toward auxiliary knowledge views—alternative presentations of the same underlying information, such as reformulations, related passages, or structurally different summaries. The central question is whether the model benefits more from seeing the same document multiple times or from encountering the same knowledge through multiple complementary surface forms.

The key finding is that, under a fixed token budget, shifting tokens from document repetition to auxiliary views can improve pre-training knowledge acquisition relative to repetition alone. This suggests that the value of repeated data is not purely a function of token count: identical redundancy may have diminishing returns, whereas diverse but semantically aligned evidence provides additional learning signals. In other words, the paper reframes repetition as a data-allocation choice, where the effectiveness of a token depends not only on whether it contains target knowledge, but also on how novel, complementary, or informative its representation is to the model.

This matters because it has direct implications for pre-training data curation, scaling, and compute efficiency. If auxiliary views are more token-efficient than blind repetition, then data pipelines can be designed to prioritize diverse reformulations and related evidence for high-value content, potentially improving factual coverage, robustness, and sample efficiency without increasing training compute. More broadly, the result challenges the assumption that more repetition is always better, and points toward more principled strategies for mixing, deduplicating, and curating pre-training corpora.