Performance Tuning

This guide collects the parameter choices that matter most for training throughput. It assumes you are already familiar with the config groups introduced in Configuration, and it focuses on how to set the knobs rather than repeating every field’s full reference entry.

Staleness

Config: psrl.staleness

psrl.staleness is the maximum version gap allowed between the rollout side and the training side. A larger staleness lets rollout run further ahead of training, which hides more of the rollout long tail behind ongoing training compute.

It is important to understand what staleness does not change. Rollout and training still overlap at the granularity of one staleness buffer against one staleness buffer. Raising staleness lets more buffers be in flight at once, it does not make that per-buffer overlap any finer grained. For finer-grained overlap within a single buffer, see the next section.

In practice, a value between 1 and 3 gets close to peak throughput while keeping the accuracy impact small.

psrl.staleness=2 \
psrl.staleness_buffer_entries=512

Because the training data is now generated by a policy that is a few versions behind the one being trained, pair a non-zero staleness with a rollout correction method that corrects for the resulting distribution shift. Token-level Importance Sampling (TIS) is the simplest effective choice. It reweights the training loss by the per-token probability ratio between the rollout policy and the current training policy.

algorithm.rollout_correction.rollout_is=token \
algorithm.rollout_correction.rollout_is_threshold=2.0 \
psrl.log_prob.enable_rollout_engine_log_prob=True

The third line is required for TIS. It makes the rollout engine return the token log-probabilities that TIS needs to compute the ratio.

See also

• Your Efficient RL Framework Secretly Brings You Off-Policy RL Training, on why staleness quietly turns on-policy RL into off-policy RL and how to correct for it.

• When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch and the veRL Rollout Correction reference, for the full family of correction methods (TIS, IcePop, rejection sampling) beyond the token-level example above.

• Fine-Grained Staleness Control, for the Reserve/Occupy/Consume protocol that implements staleness bounding.

Fine-Grain Overlap

Config: psrl.fine_grain_overlap.*

Staleness overlaps whole buffers. Fine-grain overlap goes one level deeper: training does not need to wait for an entire staleness buffer to fill before it starts working. It can start as soon as enough micro-batches or mini-batches of completed prompt groups have accumulated, so per-sample GPU work on the samples that have already arrived runs while rollout keeps generating the rest of the buffer.

psrl:
  fine_grain_overlap:
    granularity: mini_batch
    multiplier: 1
    overlap_scope: recompute

• granularity picks the chunk unit.

  • none disables the feature. Training waits for the full buffer, as before.

  • mini_batch releases one PPO mini-batch worth of prompt groups per chunk.

  • micro_batch releases ppo_micro_batch_size_per_gpu * dp_size samples per chunk, the finest granularity available.

• multiplier scales the base chunk size (chunk = base_unit * multiplier). If the resulting chunk would exceed the next level up (a micro-batch chunk bigger than one mini-batch, or a mini-batch chunk bigger than the full buffer), PSRL clamps the granularity up automatically.

• overlap_scope picks how much work runs inside each chunk.

  • recompute (default): only the per-sample forward stages (old log-prob, reference log-prob, values, reward) run per chunk. Advantage computation and the optimizer step still run once, on the full concatenated buffer, after every chunk has arrived. This is a safe starting point because the resulting math is identical to granularity: none for every advantage estimator.

  • pre_step: advantage computation and one optimizer step also run per chunk, so each mini-batch chunk becomes a complete PPO update on its own.

What cannot be combined, and why

• pre_step requires the effective granularity to be mini_batch. If a micro_batch chunk is small enough that it is not clamped up to mini_batch, combining it with pre_step raises a ValueError at startup. The reason is that stepping on a chunk smaller than a mini-batch would require true cross-chunk gradient accumulation, with the optimizer step deferred until a mini-batch boundary is reached. That needs new gradient-accumulation RPCs on the training worker and a shared loss denominator across chunks, neither of which is implemented yet. Use overlap_scope: recompute with micro_batch, or reduce multiplier so the chunk clamps up to mini_batch.

• pre_step requires ppo_epochs: 1, because a streaming, per-chunk update cannot revisit a chunk that has already been consumed by a previous epoch.

• micro_batch is incompatible with use_dynamic_bsz: True, because the chunk size is computed from a static ppo_micro_batch_size_per_gpu, and dynamic batching does not provide one.

• micro_batch combined with pre_step and actor.strategy: megatron is rejected even after clamping. Use fsdp2, or fall back to overlap_scope: recompute.

• Numerically, recompute is exact for every advantage estimator. pre_step is exact only with adv_estimator: grpo, because GRPO normalizes per prompt group, which matches the per-chunk scope naturally. With gae or reinforce_plus_plus, per-chunk whitening uses chunk-level statistics instead of full-buffer statistics, so the result is only approximate. Also note that with pre_step, the parameter-server weight push is deferred until the last chunk of the buffer finishes, so the training version only advances once the whole buffer has been consumed.

Example: recompute scope, safe starting point

fine_grain_overlap:
  granularity: mini_batch
  multiplier: 1
  overlap_scope: recompute

Example: pre_step scope with GRPO, maximum overlap

fine_grain_overlap:
  granularity: mini_batch
  multiplier: 1
  overlap_scope: pre_step
# also set:
# train_actor_rollout_ref.actor.ppo_epochs: 1
# algorithm.adv_estimator: grpo

Rollout Coordination

Config: psrl.rollout_coordination.*

Rollout coordination is made of five complementary pieces, partial rollout, redundant rollout, routing strategy, sync and migration strategy, and session strategy. Each one is documented in full, with its concept and every field, in Flexible Rollout Coordination and in the Rollout Coordination section of Configuration. This section only covers how to set them for throughput.

Baseline settings

Partial rollout can simply be turned on. It interrupts a generation at a version boundary instead of waiting for it to finish, which removes most of the rollout long tail with no downside for asynchronous training.

psrl.rollout_coordination.partial_rollout.enable=True

Redundant rollout changes the distribution of output lengths actually used for training, since it discards the slowest trajectories. Only turn it on when the tail latency problem is severe enough to justify that trade-off.

psrl.rollout_coordination.redundant_rollout.enable=False

Routing strategy needs to be chosen based on the workload, see the two profiles below. The shipped default balances by the number of in-flight requests on each instance.

psrl.rollout_coordination.routing_strategy.method=request_num_balance

admission_reject_on_waiting keeps an instance from receiving new requests once its own engine queue starts waiting, for example because its KV cache is full. This keeps the queueing at the router, where it can be redistributed, instead of stuck inside one instance.

psrl.rollout_coordination.routing_strategy.admission_reject_on_waiting=True

max_num_waiting_reqs_after_preemption controls a separate, later escape hatch. When an instance is forced to preempt requests and its waiting queue then grows past this threshold, the affected requests are looped back to the SMG router for global re-scheduling instead of staying queued on that vLLM instance. The shipped default of 1024 is high enough that this path is effectively never triggered.

psrl.rollout_coordination.routing_strategy.max_num_waiting_reqs_after_preemption=1024

For sync and migration, greedy is enough in most cases. It pulls new weights onto a rollout instance as soon as a new version is available, rather than trying to time the pull against workload indicators.

psrl.rollout_coordination.sync_and_mig_strategy.method=greedy

Migration can stay off by default.

psrl.rollout_coordination.sync_and_mig_strategy.mig.enable=False

Session strategy can also stay off by default. It is only relevant for agentic RL, see the agentic profile below.

psrl.rollout_coordination.session_strategy.thunder_agent.enable=False

Note

The settings above are the shipped defaults. A normal training run does not need to override any of them. The rest of this section covers two representative workloads that benefit from going beyond the defaults.

Reasoning workloads (single turn, short prefill, long decode)

A reasoning workload such as DAPO on math data issues one prefill per prompt and then decodes a long response. Prefix cache hits are not important here because there is little repeated prefix to exploit, so the bottleneck is keeping the decode load evenly spread across instances.

Keep request_num_balance as the routing method, and tighten max_num_waiting_reqs_after_preemption so that an instance which starts preempting requests immediately routes them back through SMG instead of continuing to hold them. Combine this with migration, so that an instance which has drifted far ahead of the others in load gets rebalanced.

psrl.rollout_coordination.routing_strategy.method=request_num_balance \
psrl.rollout_coordination.routing_strategy.max_num_waiting_reqs_after_preemption=0 \
psrl.rollout_coordination.sync_and_mig_strategy.mig.enable=True \
psrl.rollout_coordination.sync_and_mig_strategy.mig.indicator=request_num \
psrl.rollout_coordination.sync_and_mig_strategy.mig.threshold=5 \
psrl.rollout_coordination.sync_and_mig_strategy.mig.stop_indicator=request_num \
psrl.rollout_coordination.sync_and_mig_strategy.mig.stop_threshold=10

With this configuration, once the busiest instance has at least 5 times as many requests as the least busy one, and the busiest instance still has more than 10 requests, every request on the busiest instance is aborted and looped back to the router for a fresh routing decision. Migration requires psrl.status_collection.enable=True and psrl.rollout_coordination.partial_rollout.enable=True, both of which are already on by default.

When even more precise load balancing is needed than the request-count heuristic can give, throughput_optimal replaces it with a fitted cost model that estimates each instance’s marginal decoding throughput and routes each request to whichever instance gains the most from it.

psrl.rollout_coordination.routing_strategy.method=throughput_optimal \
psrl.rollout_coordination.routing_strategy.cost_model_path=${PSRL_PATH}/psrl/trainer/config/cost_model/qwen_7b.json \
psrl.rollout_coordination.routing_strategy.delta_throughput_threshold=0.5

This needs a cost model JSON fitted to the deployed model, GPU type, and parallelism degree. The repository ships a few examples under psrl/trainer/config/cost_model/ (qwen_7b.json, qwen_14b.json, qwen_32b.json, qwen_moe_30b.json), produced with analyze.py in that same directory. Re-fit the coefficients whenever the hardware or model changes, otherwise fall back to request_num_balance, which needs no calibration. throughput_optimal_with_budget extends this with a request_budget estimate of expected response length, useful when response lengths vary widely and should factor into the routing decision.

See also

The Cost Model section of Flexible Rollout Coordination for the throughput formula and calibration details, and examples/dapo_trainer/advanced_qwen2.5_7b_fsdp.sh for a working throughput_optimal setup.

Agentic workloads (multi turn, prefill dominated)

An agentic workload such as SWE-agent alternates repeatedly between prefill and decode, once per turn. Each decode segment is short, so the repeated prefill cost dominates the turn latency. Here prefix cache hits matter a lot, and routing has to balance cache locality against load at the same time. cache_aware_v1 scores both, since it is PSRL’s multi-tier cache-aware router that accounts for GPU-resident prefix hits and, when LMCache offload is enabled, off-GPU prefix hits as well.

psrl.rollout_coordination.routing_strategy.method=cache_aware_v1 \
psrl.rollout_coordination.routing_strategy.enable_trajectory_sticky=True

On top of that, enable session strategy with trajectory sticky routing. Session strategy currently supports the ThunderAgent scheduler, which reserves KV headroom on an instance for sessions that are away calling the environment. Trajectory sticky routing then makes sure the next turn of that trajectory comes back to the same instance, so the reserved prefix is the one that actually gets hit, maximizing prefix cache reuse across turns.

psrl.rollout_coordination.session_strategy.thunder_agent.enable=True \
psrl.rollout_coordination.session_strategy.thunder_agent.continue_scope=bucketed

continue_scope=bucketed is required whenever trajectory sticky routing is on, since a hung session must be readmitted on the same instance it already occupies. This mirrors the production configuration in examples/mini_swe/test_perf.sh.

See also

Router, SessionRouter, and TITO for SessionRouter, TITO session capture, and how hang/continue scheduling interacts with sticky routing.

LMCache

Config: psrl.lmcache.*

LMCache offloads KV cache blocks to CPU (or disk) once they age out of GPU memory, so a later request that shares a prefix can reuse them instead of re-prefilling. It is off by default. Turn it on when you need to push the prefix cache hit rate further than GPU-resident cache alone can provide, typically in long-sequence, multi-turn settings where reloading cached KV from CPU is clearly cheaper than recomputing it.

psrl.lmcache.enable=True \
psrl.lmcache.backend=cpu \
psrl.lmcache.offload_size_gb=100.0

Cache versioning: multi_version_kv and clear_on_weight_update

Every weight pull invalidates KV cache entries that were computed under the previous weights. PSRL has two ways to prevent stale KV from being reused after a pull. clear_on_weight_update wipes the entire offload backend right after each pull, which is simple and correct but discards every reusable prefix once per training step, GPU cache included. multi_version_kv instead tags every stored entry with the model version that produced it, so a request running under version N can never structurally hit an entry produced under version M, and stale entries are left to age out through ordinary LRU eviction as new-version entries fill the cache.

multi_version_kv is the shipped default, because under psrl.staleness > 0 different rollout instances legitimately sit at different model versions at the same time, and a partial rollout can even straddle a weight update mid-generation. Clearing the whole cache on every pull would throw away prefixes that other, still-behind instances could still use.

psrl.lmcache.multi_version_kv=True \
psrl.lmcache.clear_on_weight_update=False

Note that this only concerns the LMCache offload tier. vLLM’s own on-GPU prefix cache is always cleared on every weight pull, independent of these two flags.

P2P transfer

When the SMG router re-routes a request to an instance different from the one it was previously hinted to, it can trigger a direct CPU-to-CPU KV transfer over NIXL’s RDMA path instead of letting the new instance re-prefill from scratch. This needs P2P mode enabled on the LMCache side, and the matching switch on the routing side.

psrl.lmcache.enable_p2p=True \
psrl.lmcache.multi_version_kv=True \
psrl.lmcache.clear_on_weight_update=False \
psrl.rollout_coordination.routing_strategy.kv_transfer.enable=True \
psrl.rollout_coordination.routing_strategy.kv_transfer.transfer_mode=async

P2P transfer requires multi_version_kv=True and clear_on_weight_update=False, since the shared P2P backend has no clear operation of its own and relies entirely on version tags to keep a target instance from ever pulling in a stale entry.

Tip

Once offload is enabled, let the router account for off-GPU hits too by pairing method=cache_aware_v1 with a nonzero cache_aware_policy.lmcache_overlap_weight, for example 0.5.

See also

KV Cache Management for the full LMCache Controller architecture, and the LMCache section of Configuration for every individual switch.

Torch Memory Saver

Config: psrl.tms.*

Torch Memory Saver (TMS) manages GPU memory so that idle workers can release it and active workers can reclaim it, letting training and rollout share the same GPUs without both needing to be resident at once. It is off by default. Enable it when GPU memory is not the constraint and the priority instead is minimizing the latency of switching between training and rollout phases.

psrl.tms.range=all \
psrl.tms.enable_nixl=True \
psrl.tms.enable_cuda_graph=True

The reason enable_nixl speeds up the switch is that TMS keeps a worker’s virtual address range stable across a pause and resume cycle. Because the addresses do not move, NIXL does not need to acquire fresh buffer addresses and redo full registration from scratch when a worker wakes up, it can reuse its previous addresses and only refresh the physical pages and rkeys behind it. The trade-off is extra GPU memory overhead.

See also

Resource Elasticity for the full TMS lifecycle, and the TMS section of Configuration for every individual switch.