From Waste to Discovery: Recycling High-Scored Unexposed Candidates in Taobao Feed Recommendation

Ge Fan, Kai Meng, Huiping Chu, Yuning Jiang, Han Zhu, Bo Zheng
25 minread4.75scoreNo. 211-06 Personalization issue
Contents · 6 sections1A score was computed, but no impression followed2How an unshown video returns to recall3Why accumulate a month of candidates?4Which comparison establishes the online benefit?5A high score is a selection criterion, not a reliability guarantee6Start with the candidate logs
Rel 5Nov 4Prac 5Evid 5Time 5

Ranking already scores many candidates that never reach the screen. This work reuses the high-scoring part of those results to help users encounter additional categories. It accumulates these candidates per user over a month, keeps categories absent from recent exposure, and adds the results to recall. In Taobao feed experiments, new categories actually exposed to users increased by 3.47% and completion video views by 2.62%, both statistically significant. Total video views did not change significantly. The intervention changes how candidates are retained and reused; it requires no ranker retraining.

A score was computed, but no impression followed

A feed request passes through recall, ranking, and re-ranking. Far more items receive scores than can be displayed. An item can score well and still lose a slot because of limited positions or competition with other candidates. A later request will not necessarily retrieve it again. URED (Unexposed Recycling for Efficient Discovery), an additional recall channel, retains this part of the ranking output.

Consider an illustrative sequence: a gardening video receives a high score for a user but is not displayed. If a later request retrieves it again, the existing ranker and re-ranker still decide whether it is shown. Saving it does not reserve a display position. Nor does URED treat the missing impression as evidence that the user disliked it.

Whether this video qualifies as discovery content requires a separate check. In every online experiment, a discovery category is one that has not been exposed to that user in the previous 15 days. This definition measures expansion of recent exposure. It does not require a newly uploaded video or prove that the user has never been interested in the category.

The system therefore separates two decisions. Ranking scores select promising candidates; exposure history determines whether their categories expand what the user has recently seen. Selecting only by the second condition, by randomly injecting unfamiliar categories, does not establish that the resulting items suit the user.

How an unshown video returns to recall

The channel introduces no new predictive model, but it requires candidate logging, offline processing, storage, and a recall channel. It reuses candidates without removing the existing ranking and re-ranking computations in later requests. The original framework shows the existing recommendation path in green and the recycling path in blue.

URED Figure 1: The green path retains recall, ranking, and re-ranking. The blue path collects scored but unexposed candidates, aggregates them offline per user, and returns candidates from online storage. The separate exposure-log path supports category filtering. Returned items still pass through the original ranking stages.
URED Figure 1: The green path retains recall, ranking, and re-ranking. The blue path collects scored but unexposed candidates, aggregates them offline per user, and returns candidates from online storage. The separate exposure-log path supports category filtering. Returned items still pass through the original ranking stages.

After ranking, the system identifies scored but undisplayed items and retains the top 200 by score per session. It asynchronously records their categories, timestamps, and other fields in offline storage. These are candidates left unexposed after ranking, not previously displayed videos selected for another impression. Exposure logs are collected separately for the later discovery check.

Daily batch jobs combine each user's candidates from the preceding 30 days. They deduplicate by item ID, order by recency, and retain at most 10,000 items per user. The truncation here uses recency; it should not be read as directly comparing all raw scores from a month's separate ranking requests.

The pipeline then compares candidate categories with the user's exposed-category history. Using the experimental configuration, it keeps categories absent from the preceding 15 days; the window is configurable in the system. The gardening video in the example enters the discovery pool only if it passes this check. The filtered pool is stored in an online key-value index by user ID. A new request can retrieve up to 1,000 candidates, merge them with other recall channels, and submit the combined pool to the unchanged ranker and re-ranker.

Hoot saysOne video passes different checksClose ✕
Scores select candidates; exposure history defines discovery
Illustrative sequence: the same gardening video first meets the high-score and unexposed conditions.Gardening videohigh ranking scoreNot displayed this timeSave candidate and category
Illustrative sequence: the same gardening video first meets the high-score and unexposed conditions.Gardening videohigh ranking scoreNot displayed this timeSave candidate and category
1/3 · Current request: an unshown candidateIllustrative sequence: the same gardening video first meets the high-score and unexposed conditions.
The category must have no exposure in the previous 15 days to enter the discovery pool; otherwise the item is filtered here.Saved gardening videoGardening categorynot recently exposedEnter discovery pool
The category must have no exposure in the previous 15 days to enter the discovery pool; otherwise the item is filtered here.Saved gardening videoGardening categorynot recently exposedEnter discovery pool
2/3 · Offline: check category exposureThe category must have no exposure in the previous 15 days to enter the discovery pool; otherwise the item is filtered here.
Returning a candidate adds another opportunity to compete; the existing ranker and re-ranker still determine exposure.Look up user candidate poolMerge recall channelsOriginal rankingand re-ranking
Returning a candidate adds another opportunity to compete; the existing ranker and re-ranker still determine exposure.Look up user candidate poolMerge recall channelsOriginal rankingand re-ranking
3/3 · Next request: compete in ranking againReturning a candidate adds another opportunity to compete; the existing ranker and re-ranker still determine exposure.
1/3

Illustrative sequence, not an experimental sample. A high score without exposure enables retention; category filtering governs pool eligibility, and later ranking governs display.

SourcePaper Sections 2.1 and 3.1–3.4; 15-day criterion from Section 5

These limits control different quantities: collection per ranking session, accumulation time, stored items per user, and candidates returned to recall. The limit of 1,000 is not a promise to display 1,000 videos. No model is retrained, but the data pipeline and candidate-merge path still need operation and maintenance.

Why accumulate a month of candidates?

The supply of high-scoring discovery candidates from one request depends on the user's activity and the ranking output. A single day can leave too little supply. Raising the online retrieval quota cannot produce candidates that are missing from the pool.

The paper tests the accumulation window in concurrent online buckets. With a one-day window, discovered categories increased by 0.55%, which was not statistically significant. A 30-day window produced a 3.47% increase; extending it to 60 days reached only 3.51%. The authors chose one month as the operating point balancing benefit and storage.

URED Figure 2: The horizontal axis is the candidate-accumulation window; the vertical axis is the relative change in discovered categories against control. The curve flattens around 30 days. The positive one-day estimate is not statistically significant, so the plotted values are not all established gains.
URED Figure 2: The horizontal axis is the candidate-accumulation window; the vertical axis is the relative change in discovered categories against control. The curve flattens around 30 days. The positive one-day estimate is not statistically significant, so the plotted values are not all established gains.

The paper also reports candidate coverage: with a 30-day window, 88.3% of daily active users have a non-empty candidate pool, so this channel has candidates available for those users. This measures candidate availability, separately from the increase in categories actually exposed.

This explains the choice in the deployed system rather than prescribing a universal month-long window. The paper notes that longer windows add stale candidates; usable supply also depends on the number of qualifying candidates left by the existing ranker. The segment analysis is consistent with that limitation: more active users gained more discovery exposure, while low-activity users were essentially unchanged.

Which comparison establishes the online benefit?

The main experiment randomly assigns users to the baseline pipeline or the pipeline with the URED recall channel. The control bucket receives 10% of traffic and treatment buckets 5–10%; every reported experiment runs for more than a week. Significance is assessed with a Z-test. The following three selected values from Table 2 are relative changes against control, not percentage-point increases.

MetricRelative changeStatistical result
Discovered categories+3.47%Significant
Completion video views+2.62%Significant
Total video views+0.36%Not significant

The original table also reports a significant improvement in content diversity. The completion views and total views examined here answer different engagement questions. The experiment supports improved category discovery and completion viewing, not an established increase in total views. It also does not directly measure the formation of lasting new interests.

Table 3 tests whether the candidate source and accumulation window matter. One variant retains high-score filtering but uses a one-day window. Another keeps the injection quota but removes model-score filtering and randomly selects discovery items, including cold-start items never scored for that user. Previously shown items are still excluded. This is not restricted to the logged, already-scored candidates used by the full method.

Variant versus no-URED controlDiscovered categoriesTotal video views
High-scoring unexposed candidates, 30 days+3.47%, significant+0.36%, not significant
High-scoring unexposed candidates, 1 day+0.55%, not significant+0.02%, not significant
Random discovery candidates, including cold-start items+0.12%, not significant−0.17%, not significant

Only the full configuration achieves a statistically significant discovery increase in these settings. Random injection does not reproduce that benefit. Its negative total-view estimate is also not significant, so the result does not establish that random exploration harms viewing. The actionable distinction is that this experiment supports collecting high-scoring candidates over time, but does not support substituting random discovery injection.

A high score is a selection criterion, not a reliability guarantee

The authors' design assumption is that the existing ranker has enough personalized signal for some unexposed candidates, and high scores help select that region.

A high score alone does not prove low prediction error. URED's online results support this selection policy in the deployed system; they do not establish calibration for every high-scoring unexposed candidate. Users and categories poorly modeled by the existing ranker can still have weak or sparse pools. The authors attribute the negligible benefit for low-activity users to sparse candidate supply; the segment difference alone does not establish its cause.

A separate channel comparison is easy to overread. Discovery categories account for 85.9% of items exposed through URED, versus 6.2% across the platform. These are proportions of exposed content from different channels, not a claim that each user's discovery benefit increased more than tenfold. URED's valid-view rate, the fraction watched beyond a specified duration threshold, is 1.03 times the platform average. This channel-level composition and quality comparison measures something different from the randomized experiment's increase in discovered categories. The paper attributes the gap from 100% discovery content to lag between offline filtering and online serving.

Start with the candidate logs

URED has operated for more than six months in Taobao feed recommendation at tens-of-millions-of-daily-users scale. Online, it adds a per-user candidate lookup, with reported end-to-end serving overhead below 1%. Offline, it still accumulates, deduplicates, and filters a month of logs. The deployment uses spare capacity in existing batch infrastructure; that does not make processing and storage physically free.

An engineering trial can begin by measuring the category distribution, count, and age of high-scoring unexposed candidates. The purpose is to establish whether recently unseen categories have usable supply. Comparing nonempty-pool coverage and candidate counts across windows can then prevent a larger online quota from outgrowing that supply. A small recall-channel experiment can jointly track discovery, completion viewing, and total viewing, with significance reported separately. These are implementation suggestions derived from the mechanism, not additional experimental results.

The useful idea is to preserve candidates whose ranking cost has already been paid, but which received no impression, so they can compete again in later requests.

DimensionAssessment and reason
Problem importanceHigh: broader category exposure still requires candidate quality.
Method noveltyModerate: the contribution is candidate reuse and data flow rather than a new ranker.
EvidenceSubstantial: randomized online tests and source/window variants; total-view improvement is not significant.
Engineering costLow to moderate: no retraining, but per-user accumulation and online storage remain.
Worth trying?Conditional: most suitable when the existing ranker is reliable and leaves enough promising unshown candidates.

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…