PERSONALIZATION · 2026-10-24
Personalization — Paper Daily, 2026-10-24
All three papers come from RecSys 2026 and address the gap between offline evaluation and online results. Google Discover Ads weights the conversion model's loss by advertiser bids so the offline metric tracks advertiser value; Amazon Music's GPBM redesigns position weights in off-policy evaluation of ranking policies; Intuit's open-source scikit-rec puts ranking, contextual bandits, uplift and two other recommendation paradigms behind one evaluation interface.
4.60
Google's Discover Ads team found that the offline Poisson Loss of its conversion model (pConvs) correlated only 0.1 with online advertiser value (Delivered Value: conversions per click times the advertiser's target cost per acquisition, tCPA, summed over clicks), so models that improved offline often did not gain once launched. The authors weight each click in the loss by its tCPA, and pair every low-traffic experiment with a retrained production model serving at the same traffic as its control. With tCPA weighting, the offline loss measures exactly the conversions advertisers are willing to pay for. Offline-to-online correlation rose to 0.65, and low-to-high-traffic experiment correlation rose from 0.53 to 0.93.
4.15
Recommendation teams often use a different library for ranking, contextual bandits, uplift and sequential recommendation, each with its own offline evaluation interface. Intuit's open-source scikit-rec splits business logic, scoring strategy and the underlying model into three layers (Recommender, Scorer, Estimator), and five recommendation paradigms share one evaluate() interface, which includes six off-policy evaluators such as IPS, SNIPS and DR whose results match Open Bandit Pipeline within floating-point error. On the same logged data, a bandit policy and a ranking model can be compared directly with the same evaluators. An internal version has powered 15 use cases in TurboTax and QuickBooks for more than three years under a sub-100 ms latency requirement.
3.90
Off-policy evaluation (OPE) uses an old policy's logs to estimate how a new ranking policy would perform. Common estimators either importance-weight each position separately, which has high variance, or assume click probability equals item relevance times the probability that the position is seen, which becomes biased when the position-bias curve is misestimated. INTERPOL compromises between the two with windows, but the windows must be set by hand and all positions inside a window are weighted equally. Amazon Music's GPBM learns a weight for every pair of logging position and target position, by minimizing an approximate upper bound on the estimation error. Because the weights are learned from data, the window size no longer has to be tuned by hand. In semi-synthetic experiments on Yahoo (29,921 queries) and MSLR, GPBM has lower MSE than all baselines in most settings.
RecSys 2026
Intuit