PerSPL: Personalized Semi-supervised Preference Learning — Deep dive

Personalized reward models must score the same reply differently for different users. Training data are user-labeled preference pairs: under one prompt, which of two replies the user preferred. Labels are expensive—tens of pairs per user, maybe a dozen for a new user—while the model can generate as many unlabeled replies as needed. Duke and GWU's PerSPL turns those unlabeled replies into per-user training data: during training it first pushes each labeled pair's mean score toward 0 so scores under different prompts become comparable; then it fits a monotonic score→preference-probability curve per user, pseudo-labels only high-confidence unlabeled replies, and retrains iteratively with labeled data. On PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena, PerSPL leads average accuracy for both seen and new users; on PersonalLLM, new-user accuracy is 95.5% versus 94.0% for the strongest baseline LoRe.

1. Why unlabeled replies cannot be used directly

Reward models are often trained with Bradley–Terry (BT) loss: user $u$ prefers $y^{+}$ over $y^{-}$ under prompt $x$ ; the model scores them $r^{+}$ and $r^{-}$ ,
$L_{BT} = - lo g σ (r^{+} - r^{-})$
The loss only sees the gap. Adding 100 to both scores under the same prompt leaves the loss and within-prompt ranking unchanged.

That is fine for comparing two replies under one prompt, but it breaks for a single unlabeled reply. Pseudo-labeling "this user would like / dislike this reply" needs a threshold that works across prompts. Example from the paper: prompt a scores (101, 100); prompt b scores (1, 0). Within-prompt rankings are correct, but the chosen reply under b (score 1) is below the rejected reply under a (score 100)—no single threshold separates them. The authors call this score-coordinate indeterminacy: pairwise loss fixes gaps, not where each pair sits overall.

Existing semi-supervised reward models (SSRM, SSPO, …) learn one shared reward for all users and one global threshold, and cannot handle "the same reply has different labels for different users."

2. Method

2.1 Score centering

The fix is direct: since each pair's mean is unconstrained, add a term that pushes it toward 0,
$L_{re g} = E [(\frac{r ^{+} + r ^{-}}{2})^{2}]$
Expectation is over all labeled pairs of all users. In the example both pairs are pulled near (0.5, −0.5); within-prompt ranking is unchanged and cross-prompt comparison starts working. Main experiments use weight $λ_{r} = 0.01$ .

2.2 One calibration curve per user

With comparable scores, set each user's threshold. For user $u$ 's labeled pairs, mark chosen-reply scores as 1 and rejected as 0, and fit a nondecreasing isotonic regression $f_{u}$ that outputs the probability a score appears on the chosen side. The first point where $f_{u}$ reaches 0.5 is that user's threshold $δ_{u}$ .

For an unlabeled reply with score $r$ under user $u$ : label like if $f_{u} (r) > τ$ , dislike if $f_{u} (r) < 1 - τ$ , otherwise abstain. Main experiments use $τ = 0.6$ . Pseudo-label loss uses $δ_{u}$ as the boundary,
$L_{u n l ab e l} = - E [z lo g σ (r - δ_{u}) + (1 - z) lo g σ (δ_{u} - r)]$
$z$ is the pseudo-label. The same reply can land on different sides of different users' curves, so labels can differ.

2.3 Iterative self-training

Round 1 trains only on labeled data plus the centering term to get an initial model and per-user calibration curves. Each later round: re-pseudo-label the unlabeled pool with the current model, reinitialize the model, and train with $α L_{l ab e l} + (1 - α) L_{u n l ab e l} + λ_{r} L_{re g}$ (main $α = 0.7$ ). Calibration curves are fit on labeled data only.

Model structure matches baseline LoRe: on a frozen 4,096-d LM last-token representation, learn a set of low-rank linear bases and per-user mixture weights on the simplex. So PerSPL and LoRe differ only in training data and loss.

PerSPL flow. Top-left: each user's few labeled pairs (green chosen, red rejected). Bottom-left: shared unlabeled reply pool. Middle three rows: three users' score distributions and thresholds <span class= — PerSPL flow. Top-left: each user's few labeled pairs (green chosen, red rejected). Bottom-left: shared unlabeled reply pool. Middle three rows: three users' score distributions and thresholds $δ$ ; the same unlabeled replies (triangles) are pseudo-chosen, pseudo-rejected, or abstained (gray ?) per user. Bottom-right: how the pseudo-label set changes across iterations; the top arrow is each outer loop reinitializing the model, refitting curves, and rebuilding the pseudo set.

3. Experiments

Four benchmarks cover "are replies real" × "are users real": PersonalLLM is synthetic replies + simulated users; HelpSteer2 is human-rated replies + attribute-weighted simulated users; Reddit TLDR and Chatbot Arena are real user preferences. Seen users join joint training; new users adapt from a small support set. Metric is within-prompt pairwise accuracy, mean of five seeds.

Main results. This shows that adding unlabeled data raises accuracy under the same model structure. Seven baselines: non-personalized BT; GPO and VPL that take user info as input; PAL and LoRe that add small per-user modules; meta-learning MRM; and off-the-shelf Skywork V2.

PersonalLLM and HelpSteer2 use diversity 0.1; Chatbot Arena uses 25 pairs per seen user and 12 for new users. PerSPL leads every column mean. Versus same-structure LoRe, Reddit TLDR seen users rise from 65.3 to 70.7. Paired bootstrap against each setting's overall strongest baseline: seen-user and overall intervals never contain 0; new-user intervals exclude 0 in four of five settings.

2×2 ablation: centering vs unlabeled data. On HelpSteer2 and Reddit TLDR (two settings each), toggle centering ( $λ_{r} = 0$ ) and unlabeled training (one round only, $K=1$).

Centering alone changes seen-user accuracy within ±0.1. Adding unlabeled data without centering drops 2.9 to 6.5 points across four settings; with centering, adding unlabeled data gains 0.5 to 1.8. So centering itself barely lifts scores; its role is to turn unlabeled data from harmful into useful.

Left and right are the same model trained only on labeled data; the only difference is centering weight (0 left, 0.01 right). Green = chosen-reply score distribution; red = rejected. Left: the two distributions overlap heavily and sit between 5 and 25; right: they separate around 0 with much less overlap—only then does a single threshold make sense.

Label noise and annotation budget. On HelpSteer2 versus strongest baseline MRM. Randomly flipping preference labels from 0 to 0.5, both drop but PerSPL stays ahead—by 5.07 points at flip rate 0.4 and 6.69 at 0.5. Raising labeled pairs per seen user from 50 to 200, PerSPL stays 0.81–1.08 points ahead. Raising new-user support from 10 to 50 pairs widens the lead from 0.81 to 1.33.

Unlabeled replies from other generators. This asks whether the unlabeled pool still helps when it is not from the same generators as the labeled data. On PersonalLLM with 20% label flips fixed (to avoid accuracy saturation), raise the share of unlabeled replies from two unseen generators from 0 to 1. Lift over a labeled-only model falls from 3.21 to 0.66 points; all ten-seed paired intervals stay above 0.

4. Limitations

The authors note each user's calibration curve needs some labeled anchors, so the method is few-shot personalization only and cannot handle zero-shot new users. Distribution-shift experiments cover only the generators chosen in the paper.

Two more: every method sits on frozen LM representations and trains only the reward head—end-to-end fine-tuning of the reward model is untested; evaluation is pairwise reward accuracy, not downstream policy training or reply selection.

5. Practical takeaway

If you build personalized ranking or personalized reward models and have many unlabeled candidates, borrow PerSPL's two steps: add a term to pairwise loss that pushes each pair's mean score toward 0 and check that scores are comparable across requests; then fit isotonic regression per user and only pseudo-label high-confidence samples. The ablation shows the first step is the prerequisite—self-training without it drops points.