PSRL Documentation#
PSRL is an efficient asynchronous RL framework for LLM post-training. Developed by a joint team from PKU & SJTU and built on top of veRL, PSRL features efficient RDMA weight synchronization via parameter servers, fine-grained staleness control, and flexible rollout coordination to achieve up to 2.68x throughput improvement.
Project introduction, key features, and performance highlights.
Get PSRL running in minutes with a minimal DAPO training example.
Step-by-step guides for installation, configuration, and first training run.
Production-ready recipes for RLVR, Agentic RL, and more.
Understand the decoupled train/gen design and module interactions.
Deep-dive into staleness control, flexible rollout, and parameter server.
Key Features#
Push/Pull weights to/from CPU-side Parameter Server and P2P RDMA transfers.
Trajectory-level version binding with Reserve/Occupy/Consume protocol ensures data freshness without sacrificing throughput.
Partial rollout, redundant rollout, intelligent routing, and load-balanced migration work together to minimize idle time.
Native environment loops and SessionRouter/TITO support both integrated and black-box agents.
SMG cache-aware routing, vLLM GPU prefix cache, LMCache offload, and P2P transfer reduce re-prefill.
FSDP2 and Megatron training integrate with PPO, GRPO, DAPO, and generative reward workflows.