Task-Blind No MORE: Multi-Task Information Flow in Unified Ranking Backbones

39 minread-scoreDeep dives
Contents · 11 sections1What problem is it actually solving?2Pipeline overview: anchors run from start to finish3Input side: behavior-combination encoding and anchor initialization4Task-aware sequence reading: each task with its own attention5Selective semantic mixing: a mask keeps tasks from bleeding into each other6Task enhancing: features rescale the private anchors7Deployment: sharing user-side compute per request8Experiments: ahead on all seven tasks, but by small margins9Limitations as I see them10How a multi-task ranking team could try it11Verdict

Teams building multi-task ranking models should read this paper, because it reopens a design choice most people take for granted: at which step should the tasks split apart? Today's popular unified backbones put the behavior sequence and the other features into one stackable model, but tasks only part ways at the final prediction towers, so every earlier layer serves "the average need of all tasks". Momo's MORE (Multi-task cO-evolving Ranking modEl) in one sentence: put a set of small vectors that are updated layer by layer inside the backbone, some shared by all tasks and one per task, so that every task reads the history and mixes features its own way from the first layer. Offline it beats three unified backbones on all seven tasks; for example the relative GAUC gain on Greet is 1.58% versus 1.44% for OneTrans, and deep chat goes from 0.11% to 0.33%. Compute per example is 1.85G, lower than the reference model's 1.92G, and a two-week online test raised usage duration by 3.0%.

What problem is it actually solving?

Start with an illustrative scenario. In a nearby-posts feed, a user viewed 200 posts last week, opened 40, liked 10 of them, greeted 3 posters, and chatted past five rounds with 1 of them. For a new post we now need to predict at once "will they open it", "will they like it", "will they greet the poster", and "will they have a deep chat".

Task (illustrative)What in the history matters most
ClickWhat they saw recently, whether the cover catches the eye
CommentContent depth, whether the topic suits them
GreetWhat kind of people they are interested in
Deep chat (at least five rounds)What kind of people they actually kept talking to

The same history, four different things to read. A unified backbone first squeezes the history and features into one representation with a single shared reading, then hands it to each task's tower. The paper names three consequences. First, sequence reading and feature crossing never know which task they serve. Second, every task's preferences are compressed into one representation space before any split, so gating towers such as MMoE or PLE can only re-weight a few already-compressed basis vectors; task-specific information that was never formed upstream cannot be recovered. Third, going deeper or wider mostly enlarges shared capacity, so multi-task quality grows slowly with scale.

This shows the issue is not weak task towers but a late split: as long as the backbone never distinguishes tasks, more parameters mainly strengthen the shared part.

Figure 1 from the paper: information flow. Left, a task-blind backbone where task information only appears in the final towers; right, MORE, where task information runs from the input through every module.
Figure 1 from the paper: information flow. Left, a task-blind backbone where task information only appears in the final towers; right, MORE, where task information runs from the input through every module.

Pipeline overview: anchors run from start to finish

MORE's core concept is the "anchor token". In plain words, it is a small vector dedicated to carrying task information; anchors sit alongside the feature tokens, pass through every layer, get updated at each one, and are finally handed to the prediction towers. There are two kinds: shared anchors hold what the tasks have in common, and private anchors, one per task, hold that task's own preferences.

StageInputWhat it doesOutput
Input encodingBehavior sequence, user profile, candidate post attributes, contextEach sequence position becomes a token; non-sequential features become 15 tokensSequence matrix S, feature matrix F
Anchor constructionMean of S, all of FAn MLP produces 8 shared and 9 private anchors; each private anchor adds its task's learnable prior vector17 anchors
MORE blocks (2 by default)S, F, anchorsTask-aware sequence reading, masked feature mixing, per-task enhancement of private anchorsUpdated S, F, anchors
PredictionPrivate anchor t, shared anchors, FOne tower per taskEach task's probability
Figure 2 from the paper: MORE overview. The middle shows the stacked MORE blocks, the lower left is task-aware sequence reading, the right is selective semantic mixing (with the task-boundary mask), and the upper left is task enhancing.
Figure 2 from the paper: MORE overview. The middle shows the stacked MORE blocks, the lower left is task-aware sequence reading, the right is selective semantic mixing (with the task-boundary mask), and the upper left is task enhancing.

The training objective is a weighted sum of per-task binary cross-entropy.

Input side: behavior-combination encoding and anchor initialization

Why encode behavior combinations? If the sequence only records "which post was seen" and not "what was done to it", the private anchors later cannot tell which history matters for their task. MORE writes the behaviors that happened on each history item as a length-T 0/1 string, gives each combination an ID, gives the ID a vector, and adds it to the item vector and a time-position vector to form the token: .

A hand-worked example: with only click, like, and deep chat, there are strings. A post that was opened and liked but not chatted about is 110, one ID; opened only is 100, another ID. So "opened and liked" and "opened only" become two different tokens, and the model does not have to assemble the combination from several behavior fields itself. Removing this step costs 0.08% average GAUC in the ablation.

The anchors' initial values are also computed from the input rather than being a fixed set of parameters. The mean of the sequence tokens (256 dims) is concatenated with the 15 feature tokens flattened (15 × 256 = 3840 dims), 4096 dims in total, passed through an MLP that outputs 17 × 256 dims, and split into 8 shared and 9 private anchors. Each private anchor then adds a learnable vector for its task, so tasks differ from the start. This means the same user facing different candidates already starts from different anchors. Switching to context-free initialization costs 0.10% on average and 0.17% on deep chat.

Task-aware sequence reading: each task with its own attention

Why this step? A shared reading assigns only one set of importance scores to the history items, while click and deep chat care about different items.

Each MORE block first updates the sequence with a SwiGLU layer and projects it into attention keys and values. The anchors then act as queries reading the sequence: shared anchors use a shared cross-attention, and each private anchor uses its own task's independent attention parameters. So the 9 private anchors read the same history and each get a different weighted result.

The block also keeps a "sequence register": a 256-dim vector that starts at 0. At every layer, the updated sequence of that layer is pooled, passed through an MLP, added in, and layer-normalized. The text places this step "at the end of each block" but then says anchors "then" read the register, so the two descriptions disagree on order; following the equation order (Eq. 7 before Eq. 8), my reading is that each layer updates the register first and anchors read the updated version, so even the first layer does not read all zeros. After reading, each anchor has its own independent SwiGLU gating layer that reads only the register, not the anchor itself, and its output is added directly to the anchor. Because each anchor's gate has different parameters, the same register yields different content for different anchors. According to the paper, the register keeps sequence information across layers, because the sequence itself changes at every layer.

In this step the sequence update does not look at the candidate post; it depends only on the user. This matters for the deployment optimization later.

The ablation shows that if private anchors share attention parameters instead, average GAUC drops 0.15%; removing the register drops 0.05% on average, mostly on deep chat (0.13%). This shows per-task sequence reading matters more than the register, which mainly helps deep chat, the task that relies on long-term information.

Selective semantic mixing: a mask keeps tasks from bleeding into each other

This is the most important and hardest step to follow. After reading the sequence, the anchors have not yet talked to the non-sequential features. If all tokens were mixed indiscriminately, private anchors would dilute one another, and the click signal would seep into the deep-chat anchor.

MORE borrows RankMixer's token mixing. The 15 feature tokens, 8 shared anchors, and 9 private anchors are stacked into one matrix of tokens, each 256-dim. The 256 dims are cut into 32 segments of dims. So segment i holds the i-th segment of all 32 tokens, a 32 × 8 block.

For segment i, a small MLP reads this 32 × 8 block and scores each of the 32 tokens. The scores are multiplied element-wise by row i of the mask matrix to give 32 weights, and where the mask is 0 the weight is 0. Each token's 8-dim content in segment i is multiplied by its weight, so a zero weight turns the whole piece into 0. My reading: the cross-token exchange comes from RankMixer-style rearrangement. When transposed back, position i concatenates these 32 weighted 8-dim pieces end to end, exactly 32 × 8 = 256 dims, as its new vector. So wherever row i of the mask has a 0, that token's content cannot reach position i. The paper's own wording for this step is just "transposed back to recover the token layout".

The mask rule: feature tokens and shared anchors are visible to every position, and each private anchor is visible only to itself. A hand-computed count of the 1s in the 32 × 32 mask: the first 23 columns (15 features plus 8 shared anchors) are all 1, giving 23 × 32 = 736; each of the last 9 columns has a single 1 on the diagonal, giving 9; 745 in total, about 73% of 1024. So each private anchor's position sees the 23 common tokens plus itself and none of the other 8 private anchors, and feature and shared-anchor positions see no private anchors either.

After mixing come two per-token feed-forward layers with residuals and normalization, and the tokens are split back into features, shared anchors, and private anchors. The feed-forward layers compute within each token only, so they carry no information across the boundary.

There is one easy-to-miss detail. By Eq. (11), the MLP scoring segment i reads segment i of all 32 tokens, including every private anchor, and the mask is multiplied in only after scoring. My reading: the private anchors' contents therefore affect the scores, and so the weights other positions receive. So "strictly invisible" holds in the formulas only for the content being moved.

In the ablation, removing the mask costs 0.21% on average and 0.34% on the average of like, comment, and greet, the largest of the six components. The paper's explanation: without the mask, gradients from high-frequency tasks such as click dominate the shared mixing weights and dilute the low-frequency interaction and social tasks. This shows that guarding task boundaries in the mixing stage matters more than giving each task extra parameters in the reading or enhancing stage.

Task enhancing: features rescale the private anchors

Mixing applies the same operation to all tokens, but private anchors, as the representations finally handed to the towers, need more task-specific information. This step borrows the FiLM idea: each task has an independent MLP that reads the 3840-dim concatenation of the 15 feature tokens and outputs two 256-dim vectors and ; the private anchor is scaled and shifted per dimension as , added back to its original value, and normalized.

A hand-computed example on one dimension: original value 0.5, 1.2, -0.1; after modulation 0.5 × 1.2 - 0.1 = 0.5, adding the original gives 1.0, then normalization. If another task has 0.2 and 0, the same dimension gives 0.1 + 0.5 = 0.6. This means the same candidate features can push two tasks' anchors in different directions. Removing this step costs 0.14% on average.

Finally, task t's tower reads private anchor t, the shared anchors, and the feature tokens. The paper trains 13 tasks but has only 9 private anchors, which it says correspond to the "core tasks"; what the towers of the other 4 tasks read is not stated in the paper.

Deployment: sharing user-side compute per request

A ranking request usually has thousands of candidates. Running a forward pass per candidate repeats the same user's sequence encoding thousands of times. In MORE, the sequence path depends only on the user, not the candidate: each layer's sequence update, keys and values, and the register are all candidate-independent, and differences between candidates show up only in the feature tokens and anchors. So the candidates of one request can be merged into one sample, the sequence computed once and broadcast to all candidates; at inference, candidates are split into mini-batches, and user-side compute is shared within each mini-batch.

Online P99 scoring latency drops about 30% versus per-candidate serving, with no change to the model structure.

Experiments: ahead on all seven tasks, but by small margins

The data comes from Momo's Nearby Feed online logs: 90 days for training and the following day for evaluation, with GAUC as the metric. The default model has 2 layers and 256 dims. The table lists selected baselines; numbers are relative GAUC changes versus Transformer+RankMixer.

MethodClickLikeCommentGreetDeep chatParamsCompute per example
Transformer+RankMixer (reference)0.78630.68920.69790.72790.844477.85M1.92G
STCA+0.13%+0.66%+0.38%-0.01%-0.07%88.21M2.36G
MixFormer+0.29%+1.13%+0.89%+1.25%+0.15%98.68M4.65G
HyFormer+0.43%+0.89%+1.09%+1.32%+0.08%153M2.16G
OneTrans+0.38%+1.15%+0.80%+1.44%+0.11%108.65M1.73G
MORE+0.44%+1.47%+1.26%+1.58%+0.33%132M1.85G

The three unified backbones show very similar gain profiles across tasks, because they all separate tasks only in the towers. MORE pulls ahead most on like, greet, and deep chat, while on click it beats HyFormer by just 0.01 percentage points. This shows task awareness mainly helps the low-frequency interaction and social tasks; high-frequency click already dominates the shared representation, so its gain is small.

Removing all six components in the ablation falls back to a unified backbone with only multiple task towers and costs 0.33% on average. The individual losses sum to 0.73%, more than 0.33% (my calculation from the table), so the components overlap and cannot simply be added up.

Figure 3 from the paper: scaling, with the y-axis showing average GAUC gain over the online model. (a) by parameter count; (b) by forward compute per example. Blue is MORE, orange is the task-agnostic unified backbone.
Figure 3 from the paper: scaling, with the y-axis showing average GAUC gain over the online model. (a) by parameter count; (b) by forward compute per example. Blue is MORE, orange is the task-agnostic unified backbone.

The scaling study grows parameters from about 0.1B to about 0.9B, against a HyFormer-style unified backbone without the task-aware components. Read from the figure, the gap is about 0.05 percentage points at the smallest size, and at about 0.9B MORE is around 1.68% versus about 1.48% for the baseline, widening to about 0.2 points (my reading of the figure). This shows the task-aware gain grows with scale and is not a small-model fluke.

Online, a two-week A/B test on Nearby Feed against the production DLRM-style ranking model was positive and significant (p < 0.05) on all six metrics:

MetricRelative lift
Usage duration+3.0%
Click+2.8%
Like+2.0%
Comment+3.6%
Greet+2.5%
Deep chat+2.0%

This shows that on real traffic the gain covers browsing, interaction, and social engagement together, and the model is fully deployed.

Limitations as I see them

First, the online gain cannot all be credited to task awareness. The online control is an older DLRM-style model, not a unified backbone. From the scaling figure, a task-agnostic unified backbone of the same size gets most of the offline gain too, with the two only about 0.05 to 0.1 percentage points apart near the default size (my reading of the figure). So how much of the 3.0% duration lift comes from switching to a unified backbone and how much from the anchors was not measured separately in the experiments.

Second, offline margins are small and come without variance. Click is only 0.01 points above HyFormer, and MORE's 132M parameters are about 21% more than OneTrans's 108.65M (my calculation from the table). The tables show no variance over repeated runs, so there is no way to judge whether the small offline gaps are stable.

Third, the number of tasks is tied to the structure. The mixing layer needs 256 dims divisible by the token count, currently 32 tokens with 8 dims per segment. My reading: adding a task adds a private anchor, so the token count and segment size both change, and tasks cannot be hot-swapped. The paper also lists dynamic task configuration as future work. On top of that there is only one scenario and one private dataset, and only 7 of the 13 training tasks are reported.

How a multi-task ranking team could try it

  1. First confirm the problem exists: look at per-task offline metrics on the current unified backbone. If low-frequency tasks (follow, deep interaction) clearly lag click, the shared representation may be biased toward high-frequency tasks.
  2. Start with the smallest change, masked mixing: keep the current backbone, add shared anchors and one private anchor per task, and add the task-boundary mask only in the token-mixing layer. The ablation shows this is the single component with the largest gain.
  3. Then add per-task sequence attention, check whether the low-frequency gains keep widening, and record the added compute.
  4. Then do per-request sharing: first confirm the sequence path does not depend on the candidate at all, then merge a request's candidates into one sample and compare P99 latency.
  5. Run two online controls: one against the old model and one against a unified backbone without anchors, so that the gains from "switching backbone" and from "task awareness" can be separated.

Verdict

DimensionAssessment
Importance of the problemHigh. Multi-task ranking is nearly standard in industry, and where tasks should split is a practical question
Novelty of the methodMedium. The parts come from RankMixer, FiLM, and HSTU; the novelty is using anchors to chain them into a task channel running through the backbone
Experimental evidenceMedium. Ahead on every offline task, clear ablations, full online deployment; but a single scenario, no variance, and the online control is not a unified backbone
Engineering costMedium. Compute is on par with existing unified backbones, masks and anchors are simple to implement, and per-request sharing requires reorganizing samples
Worth trying?Yes. Teams that already run a unified backbone and see low-frequency tasks lag can first add only masked mixing as a cheap test

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Related papersMore from Paper Daily

  1. Personalization · 2026-11-03Lightweight Ranking Heads: Accelerating Multi-Task Experimentation in Production Recommender SystemsThe bigger a multi-task ranking model gets, the slower it becomes to add a new prediction target. YouTube's answer keeps the new target out of the main model and makes it a small prediction head on top of the shared layers (the paper calls it a Light Head). It only reads the shared layers' output, sends no gradient down, and is reset at the start of every training run, and every ranking model rea…
  2. Personalization · 2026-10-23Generative Recommenders Forget the Long Tail: Restoring Memorization in Retrieval and RankingGenerative recommenders use beam search to emit item-ID tokens one by one, and tail items have low generation probability and are often pruned in the first one or two steps: on Amazon Industrial & Scientific, MiniOneRec's 50 candidates cover only 11.94% of tail targets. CAPE does not retrain the model; at inference it counts which items co-occurred with the user's history in the training data, re…
  3. Personalization · 2026-07-22Uncertainty as Remedy: Mitigating Satisfaction Label Bias in Short Video Multi-Objective Ensemble RankingDeep dive: Uncertainty as Remedy: Mitigating Satisfaction Label Bias in Short Video Multi-Objective Ensemble Ranking.
  4. Personalization · 2026-11-12SeqFold: Timestamp-Addressable Compressed States for User Action Sequence BackbonesDownstream search and recommendation models all want to reuse the output of a frozen long-sequence user model, but the full multi-layer hidden state for one user at one point in time averages several million bytes, too much to store or ship. Coupang's SeqFold compresses each layer's hidden states into a fixed number of interest slots: a learned gate decides how much each past action puts into eac…
  5. Personalization · 2026-11-02Read More, Reason Deeper: Latent Workspaces for Long-Sequence RecommendationThis Meta paper gives long-sequence ranking a new structure in which reading more history and stacking more layers become two separately tunable knobs: with 10K history events, its NE beats HSTU by 0.19% while using 69.6% less forward compute. Current models tie the two together. A sequence encoder pays for every extra event in every layer, while models that first select relevant events select on…
  6. Personalization · 2026-11-03BrowseFlow: Sparse Event Flow Matching for Generative User SimulationShown the same feed, the same user might scroll to item 20 and click two things this time, and leave at item 8 the next; the log keeps only one of these. BrowseFlow argues that a user simulator should learn the distribution of such outcomes rather than copy that one log. It splits a browsing session into two things, how deep the user scrolls and at which positions they click or like, and generate…

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…