Fine-Grained Staleness Control#

Staleness control is PSRL’s mechanism for maintaining training quality in the face of asynchronous execution. It provides formal guarantees on how “stale” any training sample can be, while maximizing the throughput benefits of decoupled training and generation.

What is Staleness?#

In synchronous RL training, the policy used for generation is always identical to the policy being updated, staleness is zero. In asynchronous systems like PSRL, generation uses an older policy (version \(v\)) while training has moved ahead to version \(v + \eta\).

\[ \text{staleness} = \eta = v_{\text{train}} - v_{\text{gen}} \]

Where:

\(v_{\text{gen}}\) is the model version used to generate the trajectory.
\(v_{\text{train}}\) is the model version at the time the trajectory is consumed for training.
\(\eta\) is the number of training steps between generation and consumption.

psrl.staleness sets the maximum allowed staleness. This is the core knob for the throughput-freshness trade-off:

Setting	Meaning
`staleness=0`	Fully synchronous. Generation must complete before training can proceed.
`staleness=1`	1-step off-policy. Training may consume data generated by the previous version.
`staleness=2`	2-step off-policy. Higher throughput, slight convergence delay.
`staleness=3+`	Aggressive pipelining. Maximum throughput but requires correction (TIS/RS).

Trajectory Version Identifier (V_traj)#

Unlike systems that track staleness at the batch level, PSRL binds version information at the trajectory level. Each trajectory is tagged with a unique version identifier V_traj, the model version that was active when the trajectory’s generation started.

Why Trajectory-Level?#

Batch-level staleness is too coarse: a batch may contain trajectories generated across multiple model versions or consist of redundant trajectories that needs to be aborted. Simply use the amount of in-flight data for admission control is insufficient for more flexible rollout coordination.
Trajectory-level (V_traj) provides the right granularity: each trajectory is a self-contained unit generated by one policy version, and importance sampling correction can be applied precisely.

Assignment#

V_traj is assigned at the start of generation: when the rollout router reserves a buffer slot. Even if the model version changes mid-generation (due to a new training step), the trajectory retains its original V_traj, presenting its largest staleness.

Global Consistency Protocol#

Reserve/Occupy/Consume protocol operating on staleness buffers, the animation above is the running example used throughout this section.

The staleness system maintains a collection of versioned buffers, one per model version \(V_{buf}\). Each buffer corresponds to one training batch and consumes its entries when training advances from \(V_{buf}\) to \(V_{buf}+1\). Enforcing the staleness bound \(\eta\) reduces to placing every trajectory into a buffer that satisfies \(V_{traj} + \eta \geq V_{buf}\). Three primitives govern the lifecycle of buffer entries: Reserve, Occupy, and Consume.

1. Reserve (worst-case placement)#

The rollout router reserves a placeholder entry as soon as a trajectory is dispatched, before generation begins. Because the completion time of a trajectory is unknown in advance, Reserve always takes the worst-case admissible position: the latest buffer it could still legally land in without violating \(\eta\), i.e. \(V_{buf} = V_{traj} + \eta\).

In the animation above, at time \(T_1\) trajectories 1-6 are all assigned \(V_{traj} = V_1\). With \(\eta = 1\), their worst-case buffer is \(V_{buf} = V_2\), so all six placeholders are reserved there. This makes the staleness manager aware that 6 trajectories are in flight and must be placed no later than \(V_2\).

The entry transitions from EMPTY → RESERVED.
Because placement is the worst case, no later completion can violate \(\eta\).

2. Occupy (greedy placement)#

When a trajectory completes (generation done + reward computed), the staleness manager releases its worst-case reservation and re-places it greedily into the earliest available empty position. This frees up later buffers and lets earlier buffers fill up, and become READY, as fast as possible.

In the animation, at time \(T_1'\) trajectory 1 completes. Its placeholder in \(V_2\) is dropped, and the trajectory is occupied into the earliest open slot in \(V_1\). Buffer \(V_1\) moves closer to READY and can be consumed sooner.

The entry transitions from RESERVED → OCCUPIED.
When all entries in a buffer are OCCUPIED, the buffer becomes READY and is eligible for training.

Key invariant

Worst-case Reserve + greedy Occupy guarantees that the position where a trajectory is finally occupied is never later than the position originally reserved. Therefore the staleness bound \(\eta\) is never violated, regardless of how long any individual trajectory takes.

3. Consume#

The training worker consumes a READY buffer to perform one gradient step.

Only buffers in the READY state can be consumed.
After consumption, entries return to EMPTY and the buffer is released.
The training step produces a new model version \(V_{buf} + 1\), which becomes the version of the next buffer to fill.

Buffer States#

State	Meaning
`PENDING`	Buffer has empty slots and still accepts new `Reserve` calls.
`READY`	All entries are `OCCUPIED`, the buffer is eligible for training.
`READY_WITH_CAPACITY`	`READY` but with extra room for more entries (only seen when redundant rollout makes buffer size larger than batch size).
`STUCK`	Buffer is full but at least one entry is still `RESERVED`, both training on this buffer and further `Reserve` calls into it are blocked until those in-flight trajectories complete.

Compatibility with Flexible Rollout Coordination#

The protocol above is intentionally minimal, it only tracks V_traj and entry placements, and is by design compatible with every flexible-rollout technique introduced in Flexible Rollout Coordination.

Partial rollout and rollout migration#

Because Reserve / Occupy depend only on V_traj (not on how, where, or by which model version individual segments are produced), the protocol is agnostic to:

Partial rollout: different segments of a single trajectory generated under different model versions.
Rollout migration: different segments of a single trajectory generated on different rollout instances.

The only requirement is that every segment be generated at a model version \(\geq V_{traj}\) (i.e. never older than the pre-assigned version). Enforcing this requirement, and deciding when to trigger partial rollout / migration, is handled by the rollout coordinator and its strategies, see Partial Rollout and Migration Strategy for the relevant configuration.

Group sampling#

RL algorithms such as GRPO, DAPO and GSPO sample multiple trajectories per prompt and process them as a single group. The staleness buffer accommodates this by maintaining entries at group granularity:

Reserve / Occupy operate on whole groups, not individual trajectories.
A group entry can only be occupied when all trajectories in the group are complete.
The group’s recorded version is \(\min_{traj \in \mathcal{G}} V_{traj}\): the oldest member sets the tolerated staleness for the whole group.

Configuration is simply turning on psrl.rollout_n > 1:

psrl:
  rollout_n: 8   # >1 enables group sampling (GRPO/DAPO/GSPO style)

Redundancy and filtering#

The staleness buffer also accommodates dropping data, either passively through redundant over-provisioning, or proactively through filtering.

Redundancy (passive abort)#

The buffer supports two orthogonal levels of redundancy:

Batch-level: expand the number of buffer entries (B' > B).
Group-level: expand the number of trajectories per entry (N' > N).

Once a buffer reaches its target batch size, or an entry reaches its target group size, surplus trajectories are aborted. See Redundant Rollout for the full configuration.

Filtering (proactive abort)#

Filtering complements redundancy by explicitly dropping entries that do not benefit training. PSRL provides two mechanisms operating on entries in different states:

(a) On `OCCUPIED` entries: Dynamic Sampling Filter#

Once a group is fully occupied, it may turn out to carry no learning signal: most notably the DAPO scenario where all trajectories in the group receive the same reward, producing zero advantage. The Dynamic Sampling Filter drops such groups before they reach the policy update.

Mechanism: a group_post_process named dynamic_sampling_filter (implementation: psrl/utils/post_processor/group_post_process/filter.py) computes the standard deviation of algorithm.filter_groups.metric across the group and discards the group when it is 0.

Configuration (see examples/mini_swe/fsdp_qwen_7b_swe_smith.sh:220-222):

psrl:
  group_post_process:
    enable: true
    name: dynamic_sampling_filter
algorithm:
  filter_groups:
    metric: score   # or acc / seq_reward / seq_final_reward

(b) On `RESERVED` entries: Proactive Filter Strategy#

Some trajectories take disproportionately long to complete (e.g. a multi-turn agent stuck in a tool loop, or a generation that approaches max_response_length). While they remain RESERVED, their buffer cannot become READY, and once it enters the STUCK state training is blocked outright.

The proactive filter monitors STUCK buffers. When the number of remaining RESERVED entries drops to at most threshold, it aborts them according to the configured method:

Method	Behavior
`retry`	Abort the slow trajectory and re-dispatch the prompt to a fresh Agent Worker.
`truncate`	Force-complete the trajectory with its current partial output (treat as done).
`null`	Disabled, wait indefinitely for all trajectories to complete.

Configuration (defined in psrl/trainer/config/psrl/psrl.yaml):

psrl:
  proactive_filter_strategy:
    method: retry        # null | retry | truncate
    threshold: 4

Tip

Use retry for agentic RL where slow trajectories are likely stuck (infinite tool loops, partial failures). Use truncate for single-turn generation where the partial output still carries training value.