← Infra · 2026-10-23
📖 DEEP DIVE · INFRA

RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding

Mingkang Liu, Weiming Meng, Peipei Jiao, Jiawei Zhang, Tianyuan Liu, Junyan Li, Lichen Pang
11 minread4.10score

Speculative decoding has a small draft model guess several tokens, then a large model verify them in one parallel pass; the more guesses accepted (longer acceptance length), the faster. Draft models are usually distilled so their output distribution matches the target, including low-probability tokens the target itself is unsure about. Hisense's authors argue that target is wrong: verification only cares whether the draft's tokens are accepted, so the draft only needs to rank the most likely few tokens correctly and need not reproduce the full distribution. They propose RCSD: in the EAGLE framework, start from a draft trained with HASS (a distillation method for draft models) and continue with DPO or GRPO, using contrast between sequences from the target alone and sequences that alternate target and draft. On Qwen3-1.7B and Llama-3.1-8B, acceptance length exceeds HASS on HumanEval, MT-Bench, GSM8K, and ShareGPT for draft steps 3–6; the paper reports a 35% average gain and 1.35× end-to-end speedup on ShareGPT-like loads.

1. Distillation target ≠ acceptance length

Distillation minimizes forward KL and forces the draft to spend capacity on every region of the target distribution, including the low-probability tail. The draft is orders of magnitude smaller and cannot fully match. The authors' claim: low KL need not mean high acceptance, and high KL can still accept well—what matters is whether the top few candidate tokens are ordered like the target.

2. Method

Both training variants start from a HASS-trained draft and keep it as a frozen reference.

DPO variant. For the same prompt, build a pair:

  • "Good" sequence: fully autoregressive from the target;
  • "Bad" sequence: the target emits 1 token, the draft continues for k tokens without verification, then repeat.

Standard DPO then raises the draft's probability of the good sequence relative to the reference and lowers the bad one. Tokens in the bad sequence that the target itself generated are masked out of the loss so the draft is not punished for matching the target.

GRPO variant. For the same prompt, generate a group: one sequence from the target alone, and several that alternate with different k (draft continues 1, 2, or 3 tokens). Each sequence's score is the sum of two normalized terms: the target's log probability of the sequence, and the sequence's acceptance rate. Train with GRPO's clipped objective plus a KL constraint.

RCSD framework (paper Figure 3). Left: during speculative decoding, verification splits outputs into accepted and early-rejected parts. Right: A uses DPO with target-only sequences as good and target–draft alternating sequences as bad; B uses GRPO and scores a group by acceptance rate. Bottom: after optimization the draft mass concentrates on the target's high-probability tokens.
RCSD framework (paper Figure 3). Left: during speculative decoding, verification splits outputs into accepted and early-rejected parts. Right: A uses DPO with target-only sequences as good and target–draft alternating sequences as bad; B uses GRPO and scores a group by acceptance rate. Bottom: after optimization the draft mass concentrates on the target's high-probability tokens.

3. Experiments

3.1 Acceptance length

This checks whether DPO/GRPO continued training makes more draft tokens accepted. Setup: Qwen3-1.7B and Llama-3.1-8B as targets, train on 90% of ShareGPT; test on HumanEval, MT-Bench, GSM8K, and the remaining 10% ShareGPT; draft steps 3–6.

Qwen3-1.7B at step 6 mean acceptance length; Llama-3.1-8B at step 6:

Both methods clearly beat HASS; GRPO is usually slightly better than DPO, with a small gap. Larger step sizes gain more—for example Qwen3-1.7B on GSM8K rises from +31% at step 3 to +49% at step 6. On Llama-3.1-8B the ShareGPT gain is largest, matching the training distribution.

3.2 Throughput

This checks whether longer acceptance becomes real speed. Setup: Llama-3.1-8B-Instruct, SGLang 5.5.0. The paper does not state the draft step for this plot; the acceptance lengths match the step-3 table above.

Throughput (top) and mean acceptance length (bottom) (paper Figure 2). Gray = no speculation; blue HASS; orange HASS + DPO; red HASS + GRPO.
Throughput (top) and mean acceptance length (bottom) (paper Figure 2). Gray = no speculation; blue HASS; orange HASS + DPO; red HASS + GRPO.

(token/s) On ShareGPT, GRPO rises from 142.3 to 176.6 token/s; the other three datasets gain less.

4. Limitations

  • The paper repeatedly claims "no quality impact" but gives no quality metrics. Speculative verification should make the target decide the output, but that is not separately checked.
  • The title mentions continual learning and domain-specific settings; the body has no continual-learning experiment and evaluates only general datasets.
  • Method text is inconsistent: Figure 3 left describes good/bad sequences from accepted versus early-rejected tokens, while the method section and the right panel use target-only versus target–draft alternating sequences.
  • Figure 2's caption says the baseline is SFT; the legend says HASS.
  • Throughput is measured only for one Llama-3.1-8B setup; training cost is unreported, and there is no comparison to non-distillation training methods.

5. Takeaways

If you already deploy an EAGLE/HASS-style draft, this paper suggests a low-cost next step: continue with one round of DPO on data from the target itself—no human labels or extra reward model. The GRPO variant is only slightly better and harder to implement. Without quality evals and training-cost numbers, check output consistency and training cost yourself before shipping.