Review
From REINFORCE to one master equation, grouping policy-gradient methods by the ratio each one controls
Over the past year there has been a surge of new policy-gradient objectives for LLM reinforcement learning: GRPO, DAPO, GSPO and CISPO, then TIS, IcePop and MIS, then ScaleRL, DPPO, TRM, Score Centering and more. At first glance they look like a zoo of clipping rules and importance weights. Most of them, however, differ in one choice, namely which probability ratio they control and how. Some only limit how far the policy can move within a single update. Others correct for the gap between the inference engine that sampled the tokens and the trainer that learns from them, a gap that comes from mismatched kernels and from asynchronous rollouts. A few merge the two into a single ratio, and one avoids ratios altogether.
This post arranges these methods by the ratio they control. Section 1 introduces the three policies involved in LLM RL training, the two ratios between them, and the two ways a ratio can be clipped. Section 2 covers methods that control only the trust-region ratio \(r\), Section 3 those that handle \(r\) and the off-policy correction \(c\) separately, Section 4 those that merge them into a single ratio \(\rho\), and Section 5 Score Centering, which uses no ratio at all. Each method gets one objective and one gradient, and Section 6 puts all of them into a single master equation.
Three policies appear in LLM RL training, and keeping them apart is the key to everything that follows.
| Symbol | Name | What it is | Where its log-probs come from |
|---|---|---|---|
| \(\pi_\theta\) | current policy | The weights being updated; they change after every mini-batch step | Trainer forward pass, with gradient |
| \(\pi_{\mathrm{prox}}\) | proximal policy | A frozen snapshot of the trainer’s weights, taken when this batch’s updates began | Trainer forward pass, no gradient (an extra recompute) |
| \(\mu\) | behaviour policy | Whatever actually sampled the tokens: the inference engine (vLLM, SGLang) | Saved by the inference engine at generation time |
We follow decoupled PPO (Hilton et al., 2022) in calling \(\pi_{\mathrm{prox}}\) the proximal policy. The PPO and GRPO papers write it as \(\pi_{\theta_{old}}\) and call it the old policy.
In the original PPO, the proximal policy \(\pi_{\mathrm{prox}}\) and the behaviour policy \(\mu\) are the same, since you generate with \(\pi_{\mathrm{prox}}\) and then train on those samples. In LLM RL they differ for two reasons:
Because of this gap, the exact importance ratio between the current policy and the behaviour policy splits into two factors:
\[\rho_{i,t}=\frac{\pi_\theta(y_{i,t})}{\mu(y_{i,t})}=\underbrace{\frac{\pi_\theta(y_{i,t})}{\pi_{\mathrm{prox}}(y_{i,t})}}_{r_{i,t}:\ \text{trust region}}\cdot\underbrace{\frac{\pi_{\mathrm{prox}}(y_{i,t})}{\mu(y_{i,t})}}_{c_{i,t}:\ \text{off-policy correction}}\]The first factor, \(r\), measures how far the current update has moved away from the snapshot, and it is what a trust region constrains. The second, \(c\), measures how far the sampler was from the trainer snapshot. Multiplying each token’s gradient by \(c\) is importance sampling, and it corrects for the fact that the tokens were sampled by \(\mu\) rather than \(\pi_{\mathrm{prox}}\). The table below summarizes the difference.
| \(r=\pi_\theta/\pi_{\mathrm{prox}}\) | \(c=\pi_{\mathrm{prox}}/\mu\) | \(\rho=\pi_\theta/\mu=rc\) | |
|---|---|---|---|
| Role | Trust region: measures the step size of the updates on this batch | Off-policy correction: measures the policy mismatch between the sampler and the trainer | Full importance ratio: measures both at once, against the sampler |
| Caused by | Several updates per batch | Mismatch (kernels) and staleness (asynchronous lag) | Both |
The policy gradient pushes up each token’s log-probability, \(\nabla\log\pi_\theta\), in proportion to the advantage \(\hat A\) of its answer, and these ratios enter that gradient as an extra weight for importance sampling. When a ratio goes past a threshold, most methods bound it, which limits the step size and reduces the variance of the update. There are two ways to do this in the gradient, a masked ratio and a truncated ratio. We use \(r\) as the example and drop the indices and the sum over tokens. Here \(\mathrm{clip}(\cdot)\) bounds the ratio to a range, and each method chooses its own range, which can be symmetric, asymmetric or one-sided.
Masked ratio. The clip is applied to the ratio inside the objective. Past the threshold, \(\mathrm{clip}(r)\) is constant, so the token’s gradient is zeroed out.
Objective
\[J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\big[\mathrm{clip}(r)\,\hat A\big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\big[M\,r\,\hat A\,\nabla\log\pi_\theta\big]\]where \(M\) is 1 while \(r\) is inside the clip range and 0 once it leaves.
Truncated ratio. The clip is applied to a stop-gradient weight \(\mathrm{sg}(\cdot)\) on the log-likelihood. Without the clip, \(\mathrm{sg}(r)\,\hat A\log\pi_\theta\) has the same gradient as \(r\hat A\). With the clip, the token’s gradient is retained past the threshold, and only its weight is capped at the bound.
Objective
\[J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\big[\mathrm{sg}\big(\mathrm{clip}(r)\big)\,\hat A\,\log\pi_\theta\big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\big[\mathrm{clip}(r)\,\hat A\,\nabla\log\pi_\theta\big]\]The same two choices apply to the correction ratio \(c\) and to the full importance ratio \(\rho\).
The table below lists every method covered in this post, grouped by the ratio it controls, together with whether it masks or truncates that ratio.
| Ratio controlled | Method | Masked or truncated | Subsection |
|---|---|---|---|
| None (on-policy) | REINFORCE | – | 2.1 |
| \(r\) only (assumes \(c=1\)) | GRPO | Masked \(r\) | 2.2 |
| DAPO | Masked \(r\) | 2.3 | |
| GSPO | Masked \(r\) (sequence-level) | 2.4 | |
| CISPO | Truncated \(r\) | 2.5 | |
| \(r\) and \(c\) separately | Decoupled PPO, AReaL | Masked \(r\), unclipped \(c\) | 3.1 |
| Token-TIS | Masked \(r\), truncated \(c\) | 3.3 | |
| IcePop | Masked \(r\), masked \(c\) | 3.4 | |
| Seq-TIS | Masked \(r\), truncated \(c\) (sequence-level) | 3.5 | |
| Seq-MIS | Masked \(r\), masked \(c\) (sequence-level) | 3.6 | |
| Geo-MIS | Masked \(r\), masked \(c\) (sequence-level, geometric mean) | 3.7 | |
| One ratio \(\rho=\pi_\theta/\mu\) (no \(\pi_{\mathrm{prox}}\)) | ScaleRL | Masked \(\rho\) (DAPO, GSPO loss) or truncated \(\rho\) (CISPO loss) | 4.2 |
| PipelineRL | Truncated \(\rho\) | 4.3 | |
| SAO | Masked \(\rho\) | 4.4 | |
| TOPR | Truncated \(\rho\) (sequence-level, negative answers only) | 4.5 | |
| DPPO | Masked \(\rho\) (divergence test) | 4.6 | |
| TRM | Masked \(\rho\) (whole answer) | 4.7 | |
| No ratio: an additive correction | Score Centering | – | 5.2 |
ScaleRL appears in the one-ratio group because it plugs the DAPO, GSPO and CISPO losses into that setting, where those losses also act as off-policy corrections (Section 4.1).
We start from REINFORCE (Williams, 1992), the fully on-policy case, where answers are sampled from the current policy itself (\(y\sim\pi_\theta\)). Then \(\pi_\theta=\pi_{\mathrm{prox}}=\mu\) and every ratio is 1.
Objective
\[J(\theta)=\mathbb{E}_{y\sim\pi_\theta}\big[R(y)\big]\]Objective (stop-gradient form)
\[J(\theta)=\mathbb{E}_{y\sim\mathrm{sg}(\pi_\theta)}\Big[R(y)\sum_{t=1}^{|y|}\log\pi_\theta(y_t)\Big]\]Here \(\mathrm{sg}(\pi_\theta)\) means that the answers are sampled from the current policy but the sampling distribution is not differentiated. Both objectives yield the same gradient below.
Gradient (log-derivative trick, \(\nabla\pi_\theta=\pi_\theta\nabla\log\pi_\theta\))
\[\nabla J=\mathbb{E}_{y\sim\pi_\theta}\Big[R(y)\sum_{t=1}^{|y|}\nabla\log\pi_\theta(y_t)\Big]\]Every token of an answer is pushed up in proportion to that answer’s reward. Subtracting a baseline \(b\) that does not depend on the sampled \(y\) leaves the gradient unchanged, because \(\mathbb{E}_{y\sim\pi_\theta}[\nabla\log\pi_\theta(y)]=0\).
GRPO (Shao et al., 2024) samples \(G\) answers per prompt from the proximal policy \(\pi_{\mathrm{prox}}\) and assumes that the sampler is \(\pi_{\mathrm{prox}}\), so that \(\mu=\pi_{\mathrm{prox}}\) and \(c=1\). It may then take several updates on that batch, during which the current policy \(\pi_\theta\) drifts away from \(\pi_{\mathrm{prox}}\).1 The group-normalized advantage and the per-token ratio are
\[\hat A_i=\frac{R_i-\mathrm{mean}(R_{1:G})}{\mathrm{std}(R_{1:G})},\qquad r_{i,t}=\frac{\pi_\theta(y_{i,t})}{\pi_{\mathrm{prox}}(y_{i,t})}\]Objective2
\[J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\Big[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\min\big(r_{i,t}\hat A_i,\ \mathrm{clip}(r_{i,t},1-\epsilon,1+\epsilon)\,\hat A_i\big)\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\Big[\frac{1}{G}\sum_{i}\frac{1}{|y_i|}\sum_{t}M_{i,t}\,r_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]where \(M_{i,t}=0\) when the clip binds (\(\hat A_i\gt 0\) and \(r_{i,t}\gt 1+\epsilon\), or \(\hat A_i\lt 0\) and \(r_{i,t}\lt 1-\epsilon\)) and \(M_{i,t}=1\) otherwise. In the terms of Section 1.2, GRPO uses a masked ratio.
Why \(\min(\cdot,\mathrm{clip}(\cdot))\) rather than clip alone? With clip alone, the gradient would be zero whenever \(r\) leaves \([1-\epsilon,1+\epsilon]\), whichever direction the token moved in, so a token that moved the wrong way could never be pulled back. Taking the \(\min\) makes the clip one-sided, as the table below shows.
| \(r\gt 1+\epsilon\) (probability went up a lot) | \(r\lt 1-\epsilon\) (probability went down a lot) | |
|---|---|---|
| \(\hat A\gt 0\) (good answer) | Moved the right way, far enough: clipped, gradient 0 | Moved the wrong way: gradient flows, pushes it back up |
| \(\hat A\lt 0\) (bad answer) | Moved the wrong way: gradient flows, pushes it back down | Moved the right way, far enough: clipped, gradient 0 |
This construction comes from PPO (Schulman et al., 2017, Eq. 7), where the \(\min\) makes the objective a pessimistic lower bound on the unclipped objective \(r\hat A\). One gap remains. For \(\hat A\lt 0\) and \(r\gg1\) the term \(r\hat A\) is unbounded, which dual-clip PPO (Ye et al., 2020) addresses by bounding it below at \(c\hat A\) for a constant \(c\gt 1\) (\(c=3\) in their experiments). This gap matters more once the data is off-policy (Sections 3 and 4.5).
DAPO (Yu et al., 2025) keeps GRPO’s setting. It assumes \(\mu=\pi_{\mathrm{prox}}\), takes several mini-batch updates per batch, and uses the same advantage \(\hat A_i\) and ratio \(r_{i,t}\).
Objective
\[\begin{gathered}J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\Big[\frac{1}{\sum_{i}|y_i|}\sum_{i=1}^{G}\sum_{t=1}^{|y_i|}\min\big(r_{i,t}\hat A_i,\ \mathrm{clip}(r_{i,t},1-\epsilon_{low},1+\epsilon_{high})\,\hat A_i\big)\Big] \\[0.4em] \text{s.t. } 0\lt \#\{\text{correct } y_i\}\lt G\end{gathered}\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\Big[\frac{1}{\sum_i|y_i|}\sum_{i}\sum_{t}M_{i,t}\,r_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]with \(M_{i,t}=0\) when \(\hat A_i\gt 0\) and \(r_{i,t}\gt 1+\epsilon_{high}\), or \(\hat A_i\lt 0\) and \(r_{i,t}\lt 1-\epsilon_{low}\).
Compared with GRPO, DAPO makes four changes.
| Change | GRPO | DAPO | Why |
|---|---|---|---|
| Clip-higher | Symmetric, typically \(\epsilon=0.2\) | \(\epsilon_{low}=0.2\), \(\epsilon_{high}=0.28\) | A rare token (\(p=0.01\)) can only grow to 0.012 under \(1+\epsilon\); a looser upper bound lets low-probability tokens rise and slows entropy collapse |
| Token-level averaging | \(\frac1G\sum_i\frac1{\lvert y_i\rvert}\sum_t\) | \(\frac{1}{\sum_i\lvert y_i\rvert}\sum_i\sum_t\) | Every token counts equally, so long bad answers (rambling, repetition) get their full penalty |
| Dynamic sampling | All groups kept | Drop groups that are all right or all wrong; refill the batch | Those groups have \(\hat A=0\) and silently shrink the batch |
| Overlong shaping | None | Soft length penalty; truncated answers can be masked | A truncated answer may be sound but cut off, so a hard \(-1\) is noisy |
The last two change which data and rewards go into the loss, rather than the shape of the loss itself.
GSPO (Zheng et al., 2025) modifies GRPO (not DAPO) and keeps its setting, but replaces the per-token ratio with a single number per answer, the geometric mean of the token ratios:
\[s_i=\Big(\frac{\pi_\theta(y_i)}{\pi_{\mathrm{prox}}(y_i)}\Big)^{1/|y_i|}=\exp\Big(\frac{1}{|y_i|}\sum_{t}\log r_{i,t}\Big)\]Objective
\[J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\Big[\frac{1}{G}\sum_{i=1}^{G}\min\big(s_i\hat A_i,\ \mathrm{clip}(s_i,1-\epsilon_{low},1+\epsilon_{high})\,\hat A_i\big)\Big]\]Gradient (using \(\nabla s_i=s_i\cdot\frac{1}{\vert y_i\vert }\sum_t\nabla\log\pi_\theta(y_{i,t})\))
\[\nabla J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\Big[\frac{1}{G}\sum_{i}M_i\,s_i\,\hat A_i\,\frac{1}{|y_i|}\sum_{t}\nabla\log\pi_\theta(y_{i,t})\Big]\]The table below compares the two.
| GRPO | GSPO | |
|---|---|---|
| Ratio | Per token, \(r_{i,t}\) | Per answer, \(s_i\) |
| Weight on token \(t\)’s gradient | \(M_{i,t}r_{i,t}\), different for each token | \(M_is_i\), the same for every token of the answer |
| What the clip removes | Individual tokens | Whole answers |
| Clip range | Typically \(\epsilon=0.2\) | Tiny: \(\epsilon_{low}=3\times10^{-4}\), \(\epsilon_{high}=4\times10^{-4}\) |
The motivation is that a per-token ratio is a one-sample importance weight. It is noisy, and the noise accumulates over long answers. Since the reward is given per answer, GSPO argues that the ratio and the clip should be per answer too. This also stabilises MoE training, where changes in expert routing can jolt individual token ratios. A variant, GSPO-token, keeps the weight \(s_i\) but allows per-token advantages.
CISPO (MiniMax, 2025) uses the same setting as DAPO, with \(\mu=\pi_{\mathrm{prox}}\) assumed and token-level averaging, but it truncates the ratio instead of masking it (Section 1.2).
Objective
\[J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\Big[\frac{1}{\sum_i|y_i|}\sum_{i=1}^{G}\sum_{t=1}^{|y_i|}\mathrm{sg}\big(\mathrm{clip}(r_{i,t},1-\epsilon_{low},1+\epsilon_{high})\big)\,\hat A_i\,\log\pi_\theta(y_{i,t})\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\pi_{\mathrm{prox}}}\Big[\frac{1}{\sum_i|y_i|}\sum_{i}\sum_{t}\mathrm{clip}(r_{i,t},1-\epsilon_{low},1+\epsilon_{high})\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]The difference from DAPO shows up clearly if we look at the gradient for a few values of \(r\), using DAPO’s bounds \([0.8,1.28]\) for both methods.
| Case | DAPO (masked ratio) | CISPO (truncated ratio) |
|---|---|---|
| \(\hat A\gt 0\), \(r=1.1\) (inside the range) | \(1.1\,\hat A\,\nabla\log\pi_\theta\) | \(1.1\,\hat A\,\nabla\log\pi_\theta\) |
| \(\hat A\gt 0\), \(r=1.5\) (moved up past 1.28) | \(0\) (masked) | \(1.28\,\hat A\,\nabla\log\pi_\theta\) (truncated) |
| \(\hat A\gt 0\), \(r=0.6\) (moved the wrong way) | \(0.6\,\hat A\,\nabla\log\pi_\theta\) (the \(\min\) keeps \(r\)) | \(0.8\,\hat A\,\nabla\log\pi_\theta\) |
| \(\hat A\lt 0\), \(r=0.6\) (moved down past 0.8) | \(0\) (masked) | \(0.8\,\hat A\,\nabla\log\pi_\theta\) (truncated) |
| \(\hat A\lt 0\), \(r=3\) (a bad token went up a lot) | \(3\,\hat A\,\nabla\log\pi_\theta\) (unbounded) | \(1.28\,\hat A\,\nabla\log\pi_\theta\) (truncated) |
As a result, CISPO no longer has a hard trust region. A token that has moved far still gets a gradient, only with a bounded weight. M1’s unified formulation includes an optional mask \(M_{i,t}\); CISPO corresponds to \(M\equiv1\), and plugging in PPO’s mask recovers a PPO-style trust region. Note that CISPO still acts only on \(r\). As an aside, M1 also found a mismatch between training and inference probabilities and fixed it with an FP32 LM head, which ScaleRL later adopted.
Now suppose the answers are sampled by a behaviour policy \(\mu\) that may be stale or run on different kernels. The trainer recomputes the proximal policy \(\pi_{\mathrm{prox}}\) on those tokens and then takes several mini-batch updates, so the correction ratio \(c_{i,t}\) is no longer 1 (Section 1.1). Decoupled PPO handles this by clipping \(r\) as in GRPO and multiplying the result by \(c\).
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}c_{i,t}\,\min\big(r_{i,t}\hat A_i,\ \mathrm{clip}(r_{i,t},1-\epsilon,1+\epsilon)\,\hat A_i\big)\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\frac{1}{|y_i|}\sum_{t}c_{i,t}\,M_{i,t}\,r_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]The ratio \(c\) needs no gradient term, since neither \(\pi_{\mathrm{prox}}\) nor \(\mu\) depends on \(\theta\), and \(M_{i,t}\) is GRPO’s mask.
Where it comes from. Decoupled PPO (Hilton et al., 2022) separates the behaviour policy \(\mu\), which collected the data, from the proximal policy \(\pi_{\mathrm{prox}}\), which serves as the centre of the trust region. Their motivation was batch-size invariance and the ability to use stale data, and standard PPO is the special case \(c=1\). AReaL (Fu et al., 2025) applies the same idea to asynchronous LLM RL, with \(\mu\) the stale rollout weights and \(\pi_{\mathrm{prox}}\) the trainer’s recompute.
The raw correction \(c\) is usually close to 1 but heavy-tailed, so the methods in Sections 3.3–3.7 keep decoupled PPO’s structure but replace \(c_{i,t}\) with a tamed weight \(f(\cdot)\) to reduce the variance. Each of them corresponds to one choice of \(f\).
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}f(c_{i,t})\,\min\big(r_{i,t}\hat A_i,\ \mathrm{clip}(r_{i,t},1-\epsilon,1+\epsilon)\,\hat A_i\big)\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\frac{1}{|y_i|}\sum_{t}f(c_{i,t})\,M_{i,t}\,r_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]For sequence-level methods, \(f(c_{i,t})\) is replaced by \(f(c_i)\), with \(c_i=\prod_tc_{i,t}\), or by \(f(\bar c_i)\), with the geometric mean \(\bar c_i=c_i^{1/\vert y_i\vert }\), and the same weight then applies to every token of answer \(i\). The table below lists the choices.
| Method | Level | \(f\) | Typical setting | Source |
|---|---|---|---|---|
| Decoupled PPO | Token | \(c_{i,t}\) | – | Hilton et al. (2022); Fu et al. (2025) |
| Token-TIS | Token | \(\min(c_{i,t},C)\) | \(C=2\) | Yao et al. (2025) |
| IcePop | Token | \(c_{i,t}\,\mathbb{1}[\alpha\le c_{i,t}\le\beta]\) | \([0.5,5]\) | Ling Team (2025) |
| Seq-TIS | Sequence | \(\min(c_i,C)\) | \(C=2\) | Liu et al. (2025b) |
| Seq-MIS | Sequence | \(c_i\,\mathbb{1}[c_i\le C]\) | – | Liu et al. (2025b) |
| Geo-MIS | Sequence | mask if \(\bar c_i\) leaves a narrow band around 1 | e.g. \([0.9998,1.0002]\) (verl) | Li & Liu (2025) |
Token-TIS (Yao et al., 2025) applies truncated importance sampling to \(c\) at the token level.
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\frac{1}{|y_i|}\sum_{t}\min(c_{i,t},C)\,\min\big(r_{i,t}\hat A_i,\ \mathrm{clip}(r_{i,t})\,\hat A_i\big)\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\frac{1}{|y_i|}\sum_{t}\min(c_{i,t},C)\,M_{i,t}\,r_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]Every token is kept, and a large correction weight is capped at \(C\), typically \(C=2\). This is biased, since it under-weights tokens that \(\mu\) under-sampled, but the variance stays bounded.
IcePop, introduced with Ring-1T (Ling Team, 2025, Eqs. 1–2), instead applies a two-sided mask to \(c\) at the token level, on top of GRPO’s clip on \(r\).
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\frac{1}{|y_i|}\sum_{t}c_{i,t}\,\mathbb{1}[\alpha\le c_{i,t}\le\beta]\,\min\big(r_{i,t}\hat A_i,\ \mathrm{clip}(r_{i,t})\,\hat A_i\big)\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\frac{1}{|y_i|}\sum_{t}c_{i,t}\,\mathbb{1}[\alpha\le c_{i,t}\le\beta]\,M_{i,t}\,r_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]The defaults are \(\alpha=0.5\) and \(\beta=5\), which mask about 1–2 tokens per thousand. In their sensitivity test, \([0.5,2]\) destabilised training while \([0.4,5]\) remained stable.
Seq-TIS, from the MIS blog post “When speed kills stability” (Liu et al., 2025b), truncates the correction ratio of the whole answer, \(c_i=\prod_tc_{i,t}\).
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\min(c_i,C)\,\frac{1}{|y_i|}\sum_{t}\min\big(r_{i,t}\hat A_i,\ \mathrm{clip}(r_{i,t})\,\hat A_i\big)\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\min(c_i,C)\,\frac{1}{|y_i|}\sum_{t}M_{i,t}\,r_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]They use \(C=2\), and in their tool-use experiments Token-TIS collapsed while Seq-TIS stayed stable.
Seq-MIS, from the same blog post, masks instead of truncating. An answer whose correction ratio exceeds \(C\) is dropped entirely, so the mask is one-sided.
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}c_i\,\mathbb{1}[c_i\le C]\,\frac{1}{|y_i|}\sum_{t}\min\big(r_{i,t}\hat A_i,\ \mathrm{clip}(r_{i,t})\,\hat A_i\big)\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}c_i\,\mathbb{1}[c_i\le C]\,\frac{1}{|y_i|}\sum_{t}M_{i,t}\,r_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]In their experiments, Seq-MIS beat Seq-TIS in both training reward and test accuracy.
Since the product \(c_i\) blows up for long answers, Geo-MIS masks on the geometric mean \(\bar c_i=c_i^{1/\vert y_i\vert }\), which does not depend on length. It was introduced with the MIS authors’ verl 0.6 integration (noted in the blog’s October update) and written up by Li & Liu (2025). DeepSeek-V3.2 (DeepSeek-AI, 2025) uses a related but different sequence mask, which is one-sided and applies only to answers with negative advantage.
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\mathbb{1}[\bar c_i\in[1-\delta,1+\delta]]\,\frac{1}{|y_i|}\sum_{t}\min\big(r_{i,t}\hat A_i,\ \mathrm{clip}(r_{i,t})\,\hat A_i\big)\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac{1}{G}\sum_{i}\mathbb{1}[\bar c_i\in[1-\delta,1+\delta]]\,\frac{1}{|y_i|}\sum_{t}M_{i,t}\,r_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]The band is narrow, for example \([0.9998,1.0002]\) (verl’s guidance for the geometric level is an upper bound of 1.0002–1.001), because a geometric mean stays very close to 1, much like GSPO’s tiny clip.
In this setting, the behaviour policy \(\mu\) samples the answers and its log-probs are saved at generation time. The trainer does not recompute the proximal policy \(\pi_{\mathrm{prox}}\), and instead uses the merged ratio \(\rho_{i,t}=\pi_\theta/\mu=r_{i,t}c_{i,t}\) directly, for two reasons:
The objectives are obtained by taking a loss from Section 2 and replacing \(r\) by \(\rho\).
Objective
\[\begin{gathered}J_{\text{masked}}=\mathbb{E}_{y\sim\mu}\Big[\tfrac{1}{\sum_i|y_i|}\textstyle\sum_{i,t}\min\big(\rho_{i,t}\hat A_i,\ \mathrm{clip}(\rho_{i,t})\,\hat A_i\big)\Big], \\[0.6em] J_{\text{truncated}}=\mathbb{E}_{y\sim\mu}\Big[\tfrac{1}{\sum_i|y_i|}\textstyle\sum_{i,t}\mathrm{sg}\big(\min(\rho_{i,t},C)\big)\,\hat A_i\,\log\pi_\theta(y_{i,t})\Big]\end{gathered}\]Gradient
\[\text{masked: } M_{i,t}\,\rho_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t}),\qquad \text{truncated: } \min(\rho_{i,t},C)\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\]with \(M_{i,t}\) the PPO mask of Section 2.2, now applied to \(\rho_{i,t}\).
With a single ratio, one clip has to do both jobs. Since \(\rho=rc\) mixes the drift from updates with the sampler mismatch, the clip cannot tell the two apart, and mismatch alone can push \(\rho\) past the threshold before the policy has moved at all.
ScaleRL (Khatri et al., 2026) uses the token-level ratio \(\rho_{i,t}=\pi_\theta/\mu\) against the behaviour policy’s saved log-probs (their Eq. 2), and swaps the loss between DAPO, GSPO (with the sequence-level ratio \(\bar\rho_i=\big(\prod_t\rho_{i,t}\big)^{1/\vert y_i\vert }\)) and CISPO.
Objective (DAPO loss, masked; and CISPO loss, their Eq. 4, truncated)
\[J_{\text{DAPO}}=\mathbb{E}_{y\sim\mu}\Big[\tfrac{1}{\sum_i|y_i|}\textstyle\sum_{i,t}\min\big(\rho_{i,t}\hat A_i,\ \mathrm{clip}(\rho_{i,t},1-\epsilon^-,1+\epsilon^+)\,\hat A_i\big)\Big]\] \[J_{\text{CISPO}}=\mathbb{E}_{y\sim\mu}\Big[\tfrac{1}{\sum_i|y_i|}\textstyle\sum_{i,t}\mathrm{sg}\big(\min(\rho_{i,t},\epsilon_{max})\big)\,\hat A_i\,\log\pi_\theta(y_{i,t})\Big]\]Gradient
\[\nabla J_{\text{DAPO}}=\mathbb{E}_{y\sim\mu}\Big[\tfrac{1}{\sum_i|y_i|}\textstyle\sum_{i,t}M_{i,t}\,\rho_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\] \[\nabla J_{\text{CISPO}}=\mathbb{E}_{y\sim\mu}\Big[\tfrac{1}{\sum_i|y_i|}\textstyle\sum_{i,t}\min(\rho_{i,t},\epsilon_{max})\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]with \(M_{i,t}=0\) when \(\hat A_i\gt 0\) and \(\rho_{i,t}\gt 1+\epsilon^+\), or \(\hat A_i\lt 0\) and \(\rho_{i,t}\lt 1-\epsilon^-\).
After comparing the three losses, they chose CISPO for the final ScaleRL recipe, since GSPO and CISPO both outperformed DAPO by a large margin, and CISPO was marginally better than GSPO later in training while GSPO runs sometimes diverged.
In PipelineRL (Piché et al., 2026), generators stream continuously and new weights are pushed mid-generation while the KV cache stays stale. The behaviour policy \(\mu\) can therefore mix several policy versions within one answer, and there is no single \(\pi_{\mathrm{prox}}\). PipelineRL trains with truncated importance-sampling REINFORCE (their Eq. 5). They use the leave-one-out baseline \(\hat A_i=R_i-\frac{1}{G-1}\sum_{j\ne i}R_j\) over \(G=8\) answers per prompt and truncate the weight at \(C=1.02\).3
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac1G\sum_{i=1}^{G}\sum_{t}\mathrm{sg}\big(\min(\rho_{i,t},C)\big)\,\hat A_i\,\log\pi_\theta(y_{i,t})\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac1G\sum_{i=1}^{G}\sum_{t}\min(\rho_{i,t},C)\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]It uses a truncated weight with no PPO clip, so it acts as an off-policy correction rather than a trust region. ScaleRL’s PipelineRL-\(k\) caps the lag at \(k=8\) steps.
SAO (Hou et al., 2026) uses what it calls direct double-sided importance sampling (their Section 3.1). The ratio is computed directly against the behaviour policy’s saved log-probs, \(\rho_{i,t}=\pi_\theta/\mu\), and \(\pi_{\mathrm{prox}}\) is dropped explicitly because tracking mixed versions would require a history of checkpoints. The advantage comes from a value model with group size 1, so it is per token, \(\hat A_{i,t}\).
Objective (their Eq. 1, in our notation)
\[J=\mathbb{E}_{y\sim\mu}\Big[\textstyle\sum_{i,t}\mathrm{clip}(\rho_{i,t},1-\epsilon_\ell,1+\epsilon_h)\,\hat A_{i,t}\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\textstyle\sum_{i,t}M_{i,t}\,\rho_{i,t}\,\hat A_{i,t}\,\nabla\log\pi_\theta(y_{i,t})\Big]\]with \(M_{i,t}=0\) when \(\rho_{i,t}\gt 1+\epsilon_h\) or \(\rho_{i,t}\lt 1-\epsilon_\ell\).
Unlike DAPO’s mask, this one is two-sided and independent of the sign of \(\hat A\), and the paper describes it as restricting the trust region to that interval. The ranges are wide, \((0.7,6.0)\) for tool-integrated reasoning (\(\epsilon_\ell=0.3\), \(\epsilon_h=5\)) and \((0.2,4.0)\) for SWE (\(\epsilon_\ell=0.8\), \(\epsilon_h=3\)).
These one-ratio methods are analogous to the methods that control \(r\) only (Section 2) or \(r\) and \(c\) separately (Section 3). What differs is the ratio they act on.
| \(r\) only | \(r\) and \(c\) separately | One ratio \(\rho\) | |
|---|---|---|---|
| Masked ratio | GRPO, DAPO: \(M(r)\,r\) | IcePop: \(c\,\mathbb{1}[0.5\le c\le5]\), with a PPO clip on \(r\) | ScaleRL’s DAPO loss: \(M(\rho)\,\rho\); SAO: \(\rho\,\mathbb{1}[0.7\lt \rho\lt 6]\) |
| Truncated ratio | CISPO: \(\mathrm{clip}(r)\) | Token-TIS: \(\min(c,C)\), with a PPO clip on \(r\) | ScaleRL’s CISPO loss, PipelineRL: \(\min(\rho,C)\) |
The two-ratio methods clip \(r\) and tame \(c\) separately, whereas the one-ratio versions apply a single function to \(\rho=rc\) that does both jobs.
TOPR (Le Roux et al., 2025) also uses one ratio with no \(\pi_{\mathrm{prox}}\), but it is REINFORCE-shaped, with a stop-gradient weight, and has no PPO clip. It works at the sequence level with \(\rho_i=\prod_t\rho_{i,t}=\pi_\theta(y_i)/\mu(y_i)\).
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac1G\Big(\sum_{i:\,\hat A_i\ge0}\hat A_i\,\log\pi_\theta(y_i)\;+\sum_{i:\,\hat A_i\lt 0}\mathrm{sg}\big(\min(\rho_i,1)\big)\,\hat A_i\,\log\pi_\theta(y_i)\Big)\Big]\]Gradient
\[\begin{gathered}\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac1G\sum_i f_\pm(\rho_i)\,\hat A_i\sum_t\nabla\log\pi_\theta(y_{i,t})\Big], \\[0.6em] f_+(\rho)=1,\quad f_-(\rho)=\min(\rho,1)\end{gathered}\]Unlike the truncated methods above, TOPR treats positive and negative answers differently.
| Positive answers (\(\hat A\ge0\)) | Negative answers (\(\hat A\lt 0\)) | |
|---|---|---|
| Truncated (PipelineRL, ScaleRL’s CISPO) | \(\min(\rho,C)\) | \(\min(\rho,C)\) |
| TOPR | 1: no importance weight | \(\min(\rho,1)\) |
Positive answers get no importance weight, which amounts to SFT on good samples. This is biased but stable, since \(\log\pi\le0\) bounds how far a sample can be pushed up. Negative answers have no such bound (\(\log\pi\to-\infty\)), so their weight \(\min(\rho,1)\) shrinks once the model already finds the answer less likely than the sampler did, hence “tapered”. TOPR has no trust region.
DPPO (Qi et al., 2026) also uses one masked ratio. It keeps the PPO-shaped loss and changes only the test that decides the mask. Instead of asking whether the ratio has left \(1\pm\epsilon\), it asks whether the token’s probability has moved by more than \(\delta\).
Objective (their Eq. 12; the paper does not state a normaliser)
\[J=\mathbb{E}_{y\sim\mu}\Big[\sum_{i}\sum_{t}M_{i,t}\,\rho_{i,t}\,\hat A_i\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\sum_{i}\sum_{t}M_{i,t}\,\rho_{i,t}\,\hat A_i\,\nabla\log\pi_\theta(y_{i,t})\Big]\]with \(M_{i,t}=0\) when \(\hat A_i\gt 0\), \(\rho_{i,t}\gt 1\) and \(D_{i,t}\gt \delta\), or \(\hat A_i\lt 0\), \(\rho_{i,t}\lt 1\) and \(D_{i,t}\gt \delta\).
Here \(D_{i,t}\) compares \(\mu\) and \(\pi_\theta\) at that token, using either the binary TV distance \(\lvert\mu(y_{i,t})-\pi_\theta(y_{i,t})\rvert\) (\(\delta\approx0.15\)–\(0.2\)) or the binary KL over {sampled token, everything else} (\(\delta\approx0.05\)). The sign rule is PPO’s, and \(\rho\) is not truncated in their objective (their large-scale runs use a default cap of \(C=5\)). The advantage is \(R\) minus the group mean, without std normalisation, as proposed in Dr. GRPO (Liu et al., 2025a).
The reason for using a divergence is that the ratio misjudges how far a token has moved. A rare token going from 0.001 to 0.003 has ratio 3, so PPO clips it, yet it has barely moved (TV 0.002). A common token going from 0.99 to 0.80 has ratio 0.81, inside PPO’s \([0.8,1.2]\), yet it has moved a lot (TV 0.19).
TRM (Li et al., 2026), a follow-up to the MIS blog by overlapping authors, also uses one masked ratio, but it masks whole answers whose divergence from the sampler is too large at any token.
Objective (Eq. 33 in the ICML version; \(N\) is the full batch size, so rejected answers contribute zero)
\[J=\mathbb{E}_{y\sim\mu}\Big[\frac1N\sum_{i}M_i\sum_{t}\rho_{i,t}\,\hat A_{i}\Big]\]Gradient
\[\begin{gathered}\nabla J=\mathbb{E}_{y\sim\mu}\Big[\frac1N\sum_{i}M_i\sum_{t}\rho_{i,t}\,\hat A_{i}\,\nabla\log\pi_\theta(y_{i,t})\Big], \\[0.6em] M_i=\mathbb{1}\Big[\max_t\ \mathrm{KL}\big(\mu(\cdot|h_{i,t})\,\|\,\pi_\theta(\cdot|h_{i,t})\big)\le\delta\Big]\end{gathered}\]Here \(h_{i,t}=(x,y_{i,\lt t})\) is the prefix before token \(t\). An average variant uses \(\frac1T\sum_t\mathrm{KL}\le\delta_{avg}\) instead. Used alone, \(\delta=0.05\) for the max and \(\delta_{avg}=0.001\) for the average each stabilised training. The two can also be combined: the looser settings 0.1 and 0.002 each failed on their own but worked together.
The motivation is that the trust-region guarantee is governed by the maximum per-token KL, which is a sequence-level quantity, so whole answers are masked. A KL threshold also has the advantage of not depending on length.
Score Centering also works with samples from the behaviour policy \(\mu\) and no \(\pi_{\mathrm{prox}}\), but it takes a different route. At a prefix \(h_t=(x,y_{\lt t})\), let \(v\) range over every token in the vocabulary (about 152k), while \(y_t\) is the one token that was actually sampled. We write the score of token \(v\) and its expectation under the sampler as
\[\begin{gathered}s_v=\nabla\log\pi_\theta(v\mid h_t), \\[0.6em] \bar s=\sum_v\mu(v\mid h_t)\,s_v\ \ \text{(expected score under the sampler)}\end{gathered}\]A score averages to zero only under the distribution it was sampled from:
\[\mathbb{E}_{y\sim\pi_\theta}\big[\nabla\log\pi_\theta(y)\big]=\nabla\sum_y\pi_\theta(y)=0,\qquad\text{but}\qquad \mathbb{E}_{y\sim\mu}\big[\nabla\log\pi_\theta(y)\big]=\bar s\neq0.\]Under mismatch, the REINFORCE update at each prefix therefore splits into two parts:
\[\mathbb{E}_\mu[\hat A\,s_{y}]=\underbrace{\mathbb{E}_\mu[\hat A]\,\bar s}_{\text{drift}}+\underbrace{\mathrm{Cov}_\mu(\hat A,\,s_{y})}_{\text{signal}}.\]The term \(\bar s=\nabla\sum_v\mu(v)\log\pi_\theta(v)\) is exactly the gradient of SFT toward the sampler. Where the expected advantage is positive the trainer copies the sampler, and where it is negative the trainer is pushed away from it, in both cases regardless of which token earned the reward. Because the sampler is re-synced from the trainer, this bias compounds over training. Group centring makes the advantages sum to zero over a group but not at each prefix, so it does not remove the drift.
In logit space (\(z\) before the softmax), \(\nabla_z\log\pi_\theta(y)=e_y-\pi_\theta\), whose mean under \(\mu\) is \(\mu-\pi_\theta\). For example, with \(\pi_\theta=(0.5,0.5)\) and \(\mu=(0.6,0.4)\) the drift is \((+0.1,-0.1)\), so token 1 is pushed up simply because the sampler liked it more.
Importance sampling also cancels the drift exactly, since \(\mathbb{E}_\mu[\frac{\pi_\theta}{\mu}\nabla\log\pi_\theta]=0\). Raw IS has high variance, however, so almost every practical method clips or masks it, and then \(\mathbb{E}_\mu[\min(\rho,C)\,s_y]\neq0\). In other words, clipping reintroduces the drift.
Score Centering (Marek & Ryabinin, 2026) removes the drift directly. In the paper’s experiments it takes one SGD step per batch, with REINFORCE on group-centred rewards as the base loss.
Objective
\[J=\mathbb{E}_{y\sim\mu}\Big[\sum_{i,t}\hat A_i\Big(\log\pi_\theta(y_{i,t})-\sum_v\mathrm{sg}\big(\mu(v\mid h_{i,t})\big)\log\pi_\theta(v\mid h_{i,t})\Big)\Big]\]Gradient
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\sum_{i,t}\hat A_i\,\big(s_{y_{i,t}}-\bar s_{i,t}\big)\Big]\]The first term is REINFORCE. The second subtracts \(\bar s\), because \(\nabla\sum_v\mathrm{sg}(\mu_v)\log\pi_\theta(v)=\sum_v\mu_v s_v=\bar s\), which is exactly the drift, and this leaves only the part of the update that correlates with \(\hat A\). In logit space the score becomes \((e_y-\pi_\theta)-(\mu-\pi_\theta)=e_y-\mu\), which has zero mean under \(\mu\). The table below contrasts this with importance sampling.
| Importance sampling (Sections 3–4) | Score Centering | |
|---|---|---|
| How it gets zero mean | Reweights samples by \(\pi_\theta/\mu\) | Re-centres the score under \(\mu\) |
| The correction | Multiplicative and random: more variance, so clipping, which brings drift back | Additive and deterministic given the prefix: no ratio, clip or mask |
| Needs from \(\mu\) | The sampled token’s probability | The next-token distribution |
| Analogy | – | A baseline centres the reward (variance); SC centres the score (bias) |
Remark. An exact value baseline at each prefix would also cancel the drift in expectation, and Score Centering achieves the same effect without a critic.
Top-\(k\) approximation (their Section 4.1). Storing the full next-token distribution is expensive, so they log only the sampler’s top-\(k\) log-probs (\(k=128\)) and assume that the tail follows the trainer after rescaling. For \(v\) outside the top-\(k\), \(\mu(v)\approx\gamma\,\pi_\theta(v)\) with \(\gamma=\frac{1-\sum_{\text{top-}k}\mu(v)}{1-\sum_{\text{top-}k}\pi_\theta(v)}\). Since \(\sum_{\text{all }v}\pi_\theta(v)s_v=0\), the tail’s contribution can be rewritten using top-\(k\) terms only:
\[\begin{gathered}\bar s\approx\sum_{v\in\text{top-}k}\big(\mu_v-\gamma\,\pi_\theta(v)\big)\,s_v, \\[0.6em] J_t=\hat A\Big(\log\pi_\theta(y_t)-\sum_{v\in\text{top-}k}\mathrm{sg}\big(\mu_v-\gamma\,\pi_\theta(v)\big)\log\pi_\theta(v)\Big).\end{gathered}\]This stores roughly 1,000 times fewer entries per token (128 instead of about 152k), and \(k=32\) matched full score centring in every setting they tested.
Combining with IS (their Appendix A.2). With \(\rho(v)=\pi_\theta(v)/\mu(v)\), the gradient \(\hat A\big(f(\rho_{i,t})\,s_{y_{i,t}}-\sum_v\mu(v)f(\rho(v))\,s_v\big)\) has zero mean under \(\mu\) for any \(f\), so combining TIS with Score Centering keeps the bounded variance of TIS while restoring the zero mean. Under the most severe synthetic weight noise, only Score Centering and its combinations with TIS or MIS trained stably, and under the heaviest sampler quantisation Score Centering, alone or combined, performed best.
With \(r_{i,t}\), \(c_{i,t}\) and \(\rho_{i,t}\) defined as in Section 1.1 and \(s_v=\nabla\log\pi_\theta(v\mid h_{i,t})\), every method in this post can be written as
\[\nabla J=\mathbb{E}_{y\sim\mu}\Big[\sum_{i,t}\lambda_{i,t}\,\hat A_i\Big(\underbrace{f\cdot T}_{W_{i,t}}\;s_{y_{i,t}}\;-\;\kappa\sum_v\mu(v)\,W_v\,s_v\Big)\Big]\]where each knob plays one role:
The table below lists every method in this post as a choice of these knobs.
| Method | Level | \(T\) (trust region) | \(f\) (correction) | \(W=f\cdot T\) | \(\kappa\) |
|---|---|---|---|---|---|
| REINFORCE | Token | \(1\) | \(1\) | \(1\) | 0 |
| GRPO | Token | \(M(r_{i,t})\,r_{i,t}\), typically \(\epsilon=0.2\) | \(1\) | \(M(r_{i,t})\,r_{i,t}\) | 0 |
| DAPO | Token | \(M(r_{i,t})\,r_{i,t}\), \([0.8,1.28]\) | \(1\) | \(M(r_{i,t})\,r_{i,t}\) | 0 |
| GSPO | Sequence | \(M(s_i)\,s_i\) | \(1\) | \(M(s_i)\,s_i\) | 0 |
| CISPO | Token | \(\mathrm{clip}(r_{i,t})\) | \(1\) | \(\mathrm{clip}(r_{i,t})\) | 0 |
| Decoupled PPO / AReaL | Token | \(M(r_{i,t})\,r_{i,t}\) | \(c_{i,t}\) | \(c_{i,t}\,M(r_{i,t})\,r_{i,t}\) | 0 |
| Token-TIS | Token | \(M(r_{i,t})\,r_{i,t}\) | \(\min(c_{i,t},2)\) | \(\min(c_{i,t},2)\,M(r_{i,t})\,r_{i,t}\) | 0 |
| IcePop | Token | \(M(r_{i,t})\,r_{i,t}\) | \(c_{i,t}\mathbb{1}[0.5\le c_{i,t}\le5]\) | \(c_{i,t}\mathbb{1}[0.5\le c_{i,t}\le5]\,M(r_{i,t})\,r_{i,t}\) | 0 |
| Seq-TIS | Sequence | \(M(r_{i,t})\,r_{i,t}\) | \(\min(c_i,C)\) | \(\min(c_i,C)\,M(r_{i,t})\,r_{i,t}\) | 0 |
| Seq-MIS | Sequence | \(M(r_{i,t})\,r_{i,t}\) | \(c_i\mathbb{1}[c_i\le C]\) | \(c_i\mathbb{1}[c_i\le C]\,M(r_{i,t})\,r_{i,t}\) | 0 |
| Geo-MIS | Sequence | \(M(r_{i,t})\,r_{i,t}\) | \(\mathbb{1}[\bar c_i\text{ in band}]\) | \(\mathbb{1}[\bar c_i\text{ in band}]\,M(r_{i,t})\,r_{i,t}\) | 0 |
| ScaleRL (DAPO loss) | Token | – | \(M(\rho_{i,t})\,\rho_{i,t}\), \([0.8,1.26]\) | same as \(f\) | 0 |
| ScaleRL (GSPO loss) | Sequence | – | \(M(\bar\rho_i)\,\bar\rho_i\) | same as \(f\) | 0 |
| ScaleRL (CISPO loss) | Token | – | \(\min(\rho_{i,t},\epsilon_{max})\), 4–8 | same as \(f\) | 0 |
| DPPO | Token | – | \(M(\lvert\mu(y_{i,t})-\pi_\theta(y_{i,t})\rvert)\,\rho_{i,t}\) | same as \(f\) | 0 |
| TRM | Sequence | – | \(\mathbb{1}[\max_t\mathrm{KL}_{i,t}\le\delta]\,\rho_{i,t}\) | same as \(f\) | 0 |
| PipelineRL | Token | – | \(\min(\rho_{i,t},1.02)\) | same as \(f\) | 0 |
| SAO | Token | – | \(\rho_{i,t}\mathbb{1}[0.7\lt \rho_{i,t}\lt 6]\) | same as \(f\) | 0 |
| TOPR | Sequence | – | \(1\) if \(\hat A_i\ge0\); \(\min(\rho_i,1)\) if \(\hat A_i\lt 0\) | same as \(f\) | 0 |
| Score Centering | Token | \(1\) | \(1\) | \(1\) | 1 |
| TIS / MIS + SC | Token | – | \(\min(\rho_{i,t},C)\) or the MIS weight | same as \(f\) | 1 |
For the mixed rows (Seq-TIS, Seq-MIS, Geo-MIS), the sequence weight multiplies every token’s trust-region term.
Reading the table by group recovers the structure of this post:
How many updates per batch? PPO runs \(K\) epochs over mini-batches, so \(r\neq1\) after the first mini-batch. In the GRPO paper’s own experiments (DeepSeekMath, Section 4.2), there is a single update per exploration stage, so \(r=1\) and the clip is inactive. In practice, GRPO and DAPO implementations such as verl split the batch into mini-batches, and DAPO reports 16 gradient updates per rollout step. ScaleRL’s PPO-off-policy-\(k\) mode takes \(k\) updates per batch. Separately, \(r\) can differ from 1 even on the first update if the denominator uses the inference engine’s log-probs instead of a recomputed \(\pi_{\mathrm{prox}}\), which is the subject of Section 3. ↩
The original GRPO also subtracts a KL penalty \(\beta\,\mathrm{KL}(\pi_\theta\,\Vert \,\pi_{\mathrm{ref}})\) against a reference model \(\pi_{\mathrm{ref}}\), the initial SFT model (iterative GRPO resets it to the current policy at each outer iteration). It is estimated per token inside the token sum with an unbiased estimator, often called “k3”, \(\frac{\pi_{\mathrm{ref}}}{\pi_\theta}-\log\frac{\pi_{\mathrm{ref}}}{\pi_\theta}-1\). DAPO and most later recipes drop this term, so we omit it throughout. ↩
Their initial code used the group-mean baseline \(\hat A_i=R_i-\mathrm{mean}(R_{1:G})\) and \(C=5\). The defaults were updated in November 2025. ↩