PERSONALIZATION · 2026-10-24
Personalization — Paper Daily, 2026-10-24
All 3 papers today come from RecSys 2026 and deal with offline evaluation that disagrees with online results. Google Discover ads weight the conversion model's loss by advertiser bids so offline metrics follow advertiser value. A paper from Amazon Music improves how old logs are used to evaluate a new ranking policy, relying less on an assumed chance that each position is seen. Intuit's open-source scikit-rec gives five kinds of recommenders one evaluation interface.
4.60
In the Google Discover ads team's model iteration, offline and online results did not line up. The correlation between the change in the conversion model's offline loss and online advertiser value was only 0.1, and many changes that looked better offline failed in the small-traffic A/B stage. This paper weights the offline loss by how much each click is worth to the advertiser, so that a loss decrease corresponds to a value increase. After the changes, the offline-to-online correlation rose to 0.65, and switching two models to the new objective raised online value by 4.7% in total.
4.15
When a recommendation team maintains ranking models, contextual bandits, uplift models and sequential recommenders at the same time, it usually ends up with one library for each, plus separate glue code for evaluation, each with its own interface. To compare a ranking policy against a bandit policy on the same logs, the team has to make sure both sides are evaluated the same way by hand. The open-source scikit-rec puts all of this in one library, and five recommendation paradigms share one evaluation interface, which includes six off-policy evaluators. Its IPS, SNIPS, DR and DM estimates differ from the Open Bandit Pipeline reference implementations by at most 4×10⁻¹⁶ relative. An internal version has run for 3+ years, powering 15 use cases.
3.90
When click logs from an old ranking policy are used to estimate a new policy's clicks, existing methods rely on hand-set rules for how clicks at each position are converted. GPBM learns a weight for every position from the data and is far more accurate when the assumed position bias is wrong. To predict clicks at one position of the new policy, the existing INTERPOL just pools, with equal weight, the logged clicks at positions inside a nearby window whose width must be set in advance. GPBM writes the new policy's click estimate at each position as a weighted sum of logged clicks at all positions, and picks the weights to minimise an upper bound on the error that covers both random noise and the bias from a wrong seen-probability curve. In clicks simulated from expert labels on Yahoo search data, when the assumed curve is too high, its mean squared error is more than 6 times lower than INTERPOL with the window tuned in hindsight.
RecSys 2026
Intuit