ECHO: Efficient Cascade with Homogeneous Operators for Distillation-Guided 100K-Sequence Modeling

ECHO (Kuaishou) is a three-stage funnel: 100K raw behaviors → a user-side learned top-K (candidate-agnostic, computed outside the request path and cached per user) keeps 10K → each candidate runs GSU-style retrieval for 100 → target attention aggregates them for the ranker. Because the user-side selector never sees the candidate, it is trained by knowledge distillation from a target-attention teacher that sees the full history and the candidate, plus a boundary loss that pushes "teacher-preferred but not in top-K" events over the cut. Retrieval and the final attention reuse the teacher's relevance parameters; that is the "Homogeneous Operators" in the title. By the paper's analytical estimate, sequence-branch FLOPs are ~160× lower than STCA; GAUC on KuaiRand Comment improves 1.520% relative to the strongest baseline, and a 7-day online A/B test shows +0.613% app usage time.

1. Architecture: what each funnel stage does

This section shows how 100K events shrink to a size the ranker can afford.

Four terms used throughout:

• Target attention: attention with the candidate as query and history events as keys/values. It outputs a candidate-specific interest vector. DIN is the classic example.
• GSU / ESU: SIM's two-stage design. The General Search Unit retrieves a few relevant events with a light scorer. The Exact Search Unit runs full target attention over them.
• Top-K: keep the K highest-scoring positions.
• Knowledge distillation (KD): train a student to match a teacher's output distribution.

TFR stands for Target-aware Fine-grained Retriever, the stage-two retriever.

Two graphs train concurrently. The selector graph trains the teacher and DS; DS writes each user's top-K records to an intermediate store. The ranking graph reads those records and trains TFR, aggregation and the ranker.

Online requests only read the store. They never run DS or touch the 100K history.

DS keeps original records: item ID, timestamp, side features. C-Former and SequenceO1 compress history into latent tokens; each ECHO record stays individually matchable.

2. Walk-through: one request through three stages (illustrative numbers)

This section ties the table together with one example. All numbers here are illustrative, not from the paper.

User A has 80,000 events. During real-time training overnight, DS scores all 80,000 and stores the top 10,000. One of them is a fishing video A finished eight months ago.

At 9 a.m. A opens the feed. The ranker must score 500 candidates, one of which is a new fishing-gear video.

Stage one is already done. The ranker reads A's 10,000 records once; all 500 candidates share them.

Stage two: TFR uses the fishing-gear video as query, scores the 10,000 records and keeps the top 100. The eight-month-old fishing video scores high and is retrieved.

Stage three: target attention weights those 100 records. Fishing-related events get large weights, the interest vector goes into the ranking tower, which predicts click, like, comment and so on.

Had DS dropped that fishing record, stage two could never find it. That is why DS needs teacher supervision.

3. Title keywords mapped to modules

Cascade: the 100K → 10K → 100 → attention budgets. 10K bounds interest coverage; 100 bounds per-candidate compute.

Homogeneous Operators: TWIN shares relevance parameters between GSU and ESU. ECHO goes one step further: the teacher's learned query/key relevance parameters ${\Theta }_{rel}$ are exported to both TFR and the final target attention. All three use one scoring operator, so coarse retrieval and fine aggregation agree on what "relevant" means.

Distillation-Guided: DS is trained from the teacher (Section 4).

100K-Sequence Modeling: online raw history is capped at 100K events.

4. Training DS: scoring, distillation, boundary loss

This section solves stage one's core problem: DS cannot see the candidate but must pre-select events useful for candidates.

4.1 DS scoring

$$s_j^S=\frac{1}{H_s}\sum _{h=1}^{H_s}⟨W_{Q,h}^{S}e,W_{K,h}^S{z}_j^S⟩$$
Symbols: $s_j^S$ is the student's (DS) score for event $j$; $e$ is a user vector built only from profile features; $z_j^S$ encodes event $j$ (features plus timestamp); $W_{Q,h}^S$, $W_{K,h}^S$ are head-$h$ query/key projections; $H_s$ is the number of heads.

Intuition: attention scoring with the user as query. No candidate is involved, so one set of scores serves every candidate and can be cached. DS keeps the top $K$ positions, ${\mathcal{I}}^K$; in production $K=10,000$.

4.2 Teacher scoring

$$s_{v,j}^T=\frac{1}{H_t}\sum _{h=1}^{H_t}⟨W_{Q,h}^T{f}_T(c,x_v),W_{K,h}^T{z}_j^T⟩$$
Symbols: $s_{v,j}^T$ is the teacher's score for event $j$ given candidate $v$; $f_T(c,x_v)$ encodes context $c$ and candidate features $x_v$ into a query; $z_j^T$ is the teacher's own event encoding (not shared with DS).

Intuition: a standard full-history target attention. It knows the candidate, so its scores say which history truly matters for that candidate.

4.3 Distillation loss

$${\mathcal{L}}_{KD}=-\frac{T^2}{|\mathcal{B}|}\sum _{b\in \mathcal{B}}\sum _{j=1}^{L_b}\mathrm{sg}\left[p_{b,j}^T\right]\log p_{b,j}^S$$
Symbols: $\mathcal{B}$ is a batch and $b$ one user–candidate sample; $L_b$ is that user's history length; $p_{b,j}^T$, $p_{b,j}^S$ are teacher and student softmax probabilities over history positions at temperature $T$; $\mathrm{sg}$ is stop-gradient, so the student loss never updates the teacher.

Intuition: soft-label cross-entropy. A user has several training candidates, each with its own teacher distribution, but the student has one score vector, so it fits what those distributions share. It learns to rank events useful for most candidates first.

4.4 Boundary loss

Even after distribution matching, a few teacher-preferred events can sit outside the student's top-K. The boundary loss targets them.

Three steps. Compute a sharper teacher distribution ${\tilde{p}}^T$ with temperature $T_m<T$. Outside the top-K, take the $n$ positions with the highest teacher probability, $j^{+}$ (missed positives); inside it, take the $n$ lowest, $j^{-}$ (placeholder negatives). Pair them by rank:
$${\omega }_{b,i}=\mathrm{sg}\left[\max \right({\tilde{p}}_{b,j_{b,i}^{+}}^T-{\tilde{p}}_{b,j_{b,i}^{-}}^T,0\left)\right]$$$${\mathcal{L}}_{aux}=\frac{{\sum }_b{\sum }_i{\omega }_{b,i}\mathrm{softplus}((s_{b,j_{b,i}^{-}}^S-s_{b,j_{b,i}^{+}}^S+\delta )/T_{rank})}{{\sum }_b{\sum }_i{\omega }_{b,i}+ϵ}$$
Symbols: $i$ indexes pairs; ${\omega }_{b,i}$ is the pair weight, the teacher's preference gap (clipped at 0); $s^S$ is the student score; $\delta$ is the desired margin; $T_{rank}$ scales the loss; $ϵ$ avoids division by zero.

Intuition: a weighted pairwise ranking loss acting only at the top-K boundary. It pushes "should be in" events up and "taking a slot without mattering" events down, harder when the teacher's preference gap is larger. Pairs and weights are detached, so no gradient flows through the discrete top-K.

4.5 Full objective

$${\mathcal{L}}_{sel}=\alpha {\mathcal{L}}_{pred}+\beta {\mathcal{L}}_{KD}+\gamma {\mathcal{L}}_{aux}$$
Symbols: ${\mathcal{L}}_{pred}$ is the teacher's multi-task prediction loss; $\alpha$, $\beta$, $\gamma$ are weights.

Intuition: ${\mathcal{L}}_{pred}$ updates only the teacher, keeping it current on what is relevant; ${\mathcal{L}}_{KD}$ and ${\mathcal{L}}_{aux}$ update only DS. Both train live in the same stream; the teacher is not frozen. The ranking graph uses its own ${\mathcal{L}}_{rank}$.

5. Deployment: three engineering optimizations

Common-feature sharing. Samples of one user in a batch load the long history into GPU memory once.

Compact record encoding. Eight int64 fields cost 64 bytes per record, 640 KB at 10K records. ECHO packs each field into a fixed width sized to its value range.

Move the key projection to the query side. Standard target attention projects every history event into a key, so online cost or key-cache size grows with history length. ECHO uses:
$$⟨q{W}_Q,h{W}_K⟩=⟨(q{W}_Q)W_K^{\top },h⟩$$
Symbols: $q$ is the candidate-side query vector; $h$ is one history event vector; $W_Q$, $W_K$ are the projection matrices.

Intuition: multiply $W_K$ into the query once per request; history vectors enter the dot product directly, with no per-event projection and no key cache.

6. Experiments

Baselines. DIN: target attention over the latest 100 events. SIM: GSU/ESU retrieval. TWIN: SIM with shared GSU/ESU parameters. STCA: end-to-end stacked target-aware cross-attention (trained at ~2K, extrapolated to 10K). TWIN V2: clusters the latest 50K events into ~10K units. SequenceO1: a fixed 100-token sketch. GAUC is per-user AUC, weighted-averaged.

1.520% is relative: $(0.6347-0.6252)/0.6252\approx 1.52\%$; the absolute gap is 0.0095. The ablation removes both losses together, so the boundary loss alone is not isolated.

7. Limitations

The 100K → 10K cut is hard. Events DS drops cannot be recovered by TFR; the authors note extremely long-tail candidates may suffer.

DS updates with real-time training and store refresh, so new interests enter after a delay.

Retention quality for new users, interest shifts and new candidate types is not evaluated separately. The industrial data is not public.

8. Practical takeaways

Add a user-side top-K before the GSU and compare it with recency and random at the same storage budget.

Monitor retention quality offline with teacher top-100 recall and attention-mass coverage.

Reuse an existing model as teacher and export its query/key parameters to both GSU and ESU.

Use the key-projection identity on its own in any online target attention.

Define refresh metrics: delay for new events to enter top-K, model/store version compatibility, rollback.

Source: https://arxiv.org/html/2610.28569v1 . Numbers and formulas are from the paper; walk-through numbers are illustrative; practical advice is our analysis.