arXiv:2609.24456v1 Announce Type: new Abstract: Distributed reinforcement learning (RL) scales training by parallelizing actors and learners around an Experience Buffer. As RL workloads grow, however, the buffer becomes more than a replay queue: it is the storage substrate of a large-capacity, latency-critical experience path that every iteration traverses to move, transform, sample, and batch ex
Conduit is a framework-agnostic runtime system that optimizes distributed reinforcement learning (RL) by treating the experience path as an independent, schedulable component. It introduces the Experience Data Plane (EDP), which decouples experience handling from framework execution logic by exposing three explicit control points: experience ingestion, experience placement, and experience delivery.
Key innovations include capacity-constrained, bandwidth-aware placement to distribute data across CPU/GPU memory tiers and latency-aware scheduling to minimize overhead. When integrated with RLlib, Conduit reduces exposed experience-path latency by up to 97% and end-to-end iteration latency by up to 38%, scaling efficiently to 1,024 GPUs while preserving RL algorithm semantics.
Conduit addresses a systems-level bottleneck in distributed reinforcement learning: as actor–learner parallelism scales, the experience buffer stops behaving like a simple replay queue and becomes the central data path of the training system. Every iteration must move experiences from actors, transform them into learner-ready representations, sample minibatches, and batch them for optimization. At large scale, this path is simultaneously high-capacity and latency-critical, so inefficiencies in experience movement, transformation, or sampling can directly throttle training throughput and resource utilization.
The paper’s key contribution is to treat this experience path as a first-class experience data plane rather than an incidental storage component. Conduit is presented as an architecture for coordinating the lifecycle of experiences—ingestion, transformation, buffering, sampling, and batching—across distributed actors and learners. By making these operations explicit and co-designed with the distributed RL workload, the system can better manage capacity, data movement, and the tight coupling between experience availability and learner progress, instead of treating the buffer as a passive FIFO store.
This matters because modern distributed RL systems are increasingly limited not by the model or optimizer alone, but by the pipeline that feeds them. If the experience path cannot keep pace with parallel actors and learners, expensive compute resources sit idle waiting for usable batches. Conduit’s framing provides a systems abstraction for attacking that problem: a dedicated data plane for experience handling that can improve scalability, reduce latency in the critical training loop, and make large-capacity RL training more predictable and efficient.