Contents · 6 sections
1A score was computed, but no impression followed2How an unshown video returns to recall3Why accumulate a month of candidates?4Which comparison establishes the online benefit?5A high score is a selection criterion, not a reliability guarantee6Start with the candidate logsSHARE
Ranking already scores many candidates that never reach the screen. This work reuses the high-scoring part of those results to help users encounter additional categories. It accumulates these candidates per user over a month, keeps categories absent from recent exposure, and adds the results to recall. In Taobao feed experiments, new categories actually exposed to users increased by 3.47% and completion video views by 2.62%, both statistically significant. Total video views did not change significantly. The intervention changes how candidates are retained and reused; it requires no ranker retraining.
A score was computed, but no impression followed
A feed request passes through recall, ranking, and re-ranking. Far more items receive scores than can be displayed. An item can score well and still lose a slot because of limited positions or competition with other candidates. A later request will not necessarily retrieve it again. URED (Unexposed Recycling for Efficient Discovery), an additional recall channel, retains this part of the ranking output.
Consider an illustrative sequence: a gardening video receives a high score for a user but is not displayed. If a later request retrieves it again, the existing ranker and re-ranker still decide whether it is shown. Saving it does not reserve a display position. Nor does URED treat the missing impression as evidence that the user disliked it.
Whether this video qualifies as discovery content requires a separate check. In every online experiment, a discovery category is one that has not been exposed to that user in the previous 15 days. This definition measures expansion of recent exposure. It does not require a newly uploaded video or prove that the user has never been interested in the category.
The system therefore separates two decisions. Ranking scores select promising candidates; exposure history determines whether their categories expand what the user has recently seen. Selecting only by the second condition, by randomly injecting unfamiliar categories, does not establish that the resulting items suit the user.
How an unshown video returns to recall
The channel introduces no new predictive model, but it requires candidate logging, offline processing, storage, and a recall channel. It reuses candidates without removing the existing ranking and re-ranking computations in later requests. The original framework shows the existing recommendation path in green and the recycling path in blue.

After ranking, the system identifies scored but undisplayed items and retains the top 200 by score per session. It asynchronously records their categories, timestamps, and other fields in offline storage. These are candidates left unexposed after ranking, not previously displayed videos selected for another impression. Exposure logs are collected separately for the later discovery check.
Daily batch jobs combine each user's candidates from the preceding 30 days. They deduplicate by item ID, order by recency, and retain at most 10,000 items per user. The truncation here uses recency; it should not be read as directly comparing all raw scores from a month's separate ranking requests.
The pipeline then compares candidate categories with the user's exposed-category history. Using the experimental configuration, it keeps categories absent from the preceding 15 days; the window is configurable in the system. The gardening video in the example enters the discovery pool only if it passes this check. The filtered pool is stored in an online key-value index by user ID. A new request can retrieve up to 1,000 candidates, merge them with other recall channels, and submit the combined pool to the unchanged ranker and re-ranker. Hoot says
Illustrative sequence, not an experimental sample. A high score without exposure enables retention; category filtering governs pool eligibility, and later ranking governs display.
Source
Paper Sections 2.1 and 3.1–3.4; 15-day criterion from Section 5These limits control different quantities: collection per ranking session, accumulation time, stored items per user, and candidates returned to recall. The limit of 1,000 is not a promise to display 1,000 videos. No model is retrained, but the data pipeline and candidate-merge path still need operation and maintenance.
Why accumulate a month of candidates?
The supply of high-scoring discovery candidates from one request depends on the user's activity and the ranking output. A single day can leave too little supply. Raising the online retrieval quota cannot produce candidates that are missing from the pool.
The paper tests the accumulation window in concurrent online buckets. With a one-day window, discovered categories increased by 0.55%, which was not statistically significant. A 30-day window produced a 3.47% increase; extending it to 60 days reached only 3.51%. The authors chose one month as the operating point balancing benefit and storage.

The paper also reports candidate coverage: with a 30-day window, 88.3% of daily active users have a non-empty candidate pool, so this channel has candidates available for those users. This measures candidate availability, separately from the increase in categories actually exposed.
This explains the choice in the deployed system rather than prescribing a universal month-long window. The paper notes that longer windows add stale candidates; usable supply also depends on the number of qualifying candidates left by the existing ranker. The segment analysis is consistent with that limitation: more active users gained more discovery exposure, while low-activity users were essentially unchanged.
Which comparison establishes the online benefit?
The main experiment randomly assigns users to the baseline pipeline or the pipeline with the URED recall channel. The control bucket receives 10% of traffic and treatment buckets 5–10%; every reported experiment runs for more than a week. Significance is assessed with a Z-test. The following three selected values from Table 2 are relative changes against control, not percentage-point increases.
| Metric | Relative change | Statistical result |
|---|---|---|
| Discovered categories | +3.47% | Significant |
| Completion video views | +2.62% | Significant |
| Total video views | +0.36% | Not significant |
The original table also reports a significant improvement in content diversity. The completion views and total views examined here answer different engagement questions. The experiment supports improved category discovery and completion viewing, not an established increase in total views. It also does not directly measure the formation of lasting new interests.
Table 3 tests whether the candidate source and accumulation window matter. One variant retains high-score filtering but uses a one-day window. Another keeps the injection quota but removes model-score filtering and randomly selects discovery items, including cold-start items never scored for that user. Previously shown items are still excluded. This is not restricted to the logged, already-scored candidates used by the full method.
| Variant versus no-URED control | Discovered categories | Total video views |
|---|---|---|
| High-scoring unexposed candidates, 30 days | +3.47%, significant | +0.36%, not significant |
| High-scoring unexposed candidates, 1 day | +0.55%, not significant | +0.02%, not significant |
| Random discovery candidates, including cold-start items | +0.12%, not significant | −0.17%, not significant |
Only the full configuration achieves a statistically significant discovery increase in these settings. Random injection does not reproduce that benefit. Its negative total-view estimate is also not significant, so the result does not establish that random exploration harms viewing. The actionable distinction is that this experiment supports collecting high-scoring candidates over time, but does not support substituting random discovery injection.
A high score is a selection criterion, not a reliability guarantee
The authors' design assumption is that the existing ranker has enough personalized signal for some unexposed candidates, and high scores help select that region.
A high score alone does not prove low prediction error. URED's online results support this selection policy in the deployed system; they do not establish calibration for every high-scoring unexposed candidate. Users and categories poorly modeled by the existing ranker can still have weak or sparse pools. The authors attribute the negligible benefit for low-activity users to sparse candidate supply; the segment difference alone does not establish its cause.
A separate channel comparison is easy to overread. Discovery categories account for 85.9% of items exposed through URED, versus 6.2% across the platform. These are proportions of exposed content from different channels, not a claim that each user's discovery benefit increased more than tenfold. URED's valid-view rate, the fraction watched beyond a specified duration threshold, is 1.03 times the platform average. This channel-level composition and quality comparison measures something different from the randomized experiment's increase in discovered categories. The paper attributes the gap from 100% discovery content to lag between offline filtering and online serving.
Start with the candidate logs
URED has operated for more than six months in Taobao feed recommendation at tens-of-millions-of-daily-users scale. Online, it adds a per-user candidate lookup, with reported end-to-end serving overhead below 1%. Offline, it still accumulates, deduplicates, and filters a month of logs. The deployment uses spare capacity in existing batch infrastructure; that does not make processing and storage physically free.
An engineering trial can begin by measuring the category distribution, count, and age of high-scoring unexposed candidates. The purpose is to establish whether recently unseen categories have usable supply. Comparing nonempty-pool coverage and candidate counts across windows can then prevent a larger online quota from outgrowing that supply. A small recall-channel experiment can jointly track discovery, completion viewing, and total viewing, with significance reported separately. These are implementation suggestions derived from the mechanism, not additional experimental results.
The useful idea is to preserve candidates whose ranking cost has already been paid, but which received no impression, so they can compete again in later requests.
| Dimension | Assessment and reason |
|---|---|
| Problem importance | High: broader category exposure still requires candidate quality. |
| Method novelty | Moderate: the contribution is candidate reuse and data flow rather than a new ranker. |
| Evidence | Substantial: randomized online tests and source/window variants; total-view improvement is not significant. |
| Engineering cost | Low to moderate: no retraining, but per-user accumulation and online storage remain. |
| Worth trying? | Conditional: most suitable when the existing ranker is reliable and leaves enough promising unshown candidates. |
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…