Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence

Recent research identifies Reinforcement Learning with Verifiable Rewards (RLVR) as the foundational paradigm for scaling Large Reasoning Models (LRMs), but extending this to open-ended and agentic tasks requires overcoming the scarcity of reliable, automated feedback. Key strategies to bypass human supervision and scale toward Artificial Superintelligence (ASI) include:

• Agentic Self-Learning (ASL): This framework replaces rigid rule-based rewards with a Generative Reward Model (GRM) that co-evolves with the policy. By coordinating task generation, policy execution, and evaluation in a closed loop, ASL enables scalable, open-domain agent training without human-curated datasets, though it requires continual GRM training to prevent reward hacking.
• Task Transformation (RLSVR/SpyRL): To apply RLVR to non-verifiable tasks like creative writing, models can transform open-ended problems into verifiable proxy environments. For example, SpyRL uses information-asymmetric self-play (resembling a social deduction game) to generate automatic, verifiable rewards based on agent interactions, allowing scalable self-improvement without external judges.
• Structure-Aware Optimization (SARL): Instead of relying on outcome correctness, SARL optimizes the topology of reasoning trajectories. By rewarding the structural coherence of intermediate thinking steps, it provides a label-free signal that works for both mathematical and open-ended reasoning, maintaining stable training dynamics and higher policy entropy.
• Self-Consistency and Data Scaling: Models can leverage their own self-consistency to infer correctness signals, enabling self-training without ground-truth labels. Additionally, scaling the quantity of agent-generated task data and using diverse training domains helps generalize reasoning capabilities beyond narrow, domain-specific behaviors.

These approaches collectively shift the bottleneck from human annotation to algorithm design and environment engineering, enabling LRMs to achieve continuous, self-supervised improvement in complex, open-ended scenarios.

Summary

This paper examines whether the scaling gains obtained by large reasoning models (LRMs) through reinforcement learning with verifiable rewards (RLVR) can be extended beyond closed, automatically checkable domains such as mathematics and code to open-ended and agentic tasks. It frames RLVR as a promising route toward reducing dependence on human supervision: when reward signals are cheap, dense, and reliable, post-training can convert broad pretraining capability into stronger multi-step reasoning. The central difficulty is that many high-value tasks—planning, tool use, research, software engineering, negotiation, or real-world interaction—lack concise ground-truth answers, have delayed or stochastic outcomes, and are vulnerable to reward hacking. The work therefore analyzes how to construct scalable reward structures that remain faithful to the intended objective as tasks become longer, more ambiguous, and more environment-dependent.

Its main contribution is a conceptual and methodological roadmap for extending RLVR to settings where verification is partial, procedural, or environment-mediated. The paper discusses mechanisms such as process-level rewards, self-verification, outcome-based rewards from executable or testable environments, verifier ensembles, automatic task generation, and curriculum strategies that move from tightly specified problems to broader agentic goals. A key insight is that the bottleneck for further scaling may no longer be merely data or compute, but the ability to generate valid training signals at scale while controlling for specification gaming, overfitting to narrow benchmarks, and degradation of general reasoning. In that sense, it positions reward design, evaluation, and self-improvement loops as core components of a path toward systems that can improve beyond the quality of human-provided supervision.

The significance of this material is that it addresses one of the key obstacles on the way to superintelligent AI: how to continue scaling reasoning capabilities once human labels and hand-crafted benchmarks become limiting. If RLVR can be generalized to open-ended and agentic domains, it offers a scalable alternative to purely preference-based RLHF and could enable models to learn from their own actions, environments, and verifiable consequences. At the same time, the paper highlights the safety and alignment stakes of such a path, because more autonomous reward generation and self-improvement increase the importance of robust verification, interpretability, and control. For a curated collection, this work is valuable as a bridge between current LRM training practice and the longer-term research agenda for scalable, self-improving reasoning systems.