CADET: Context-Conditioned Ads CTR Prediction With a Decoder-Only Transformer

35 minread-scoreDeep dives
Contents · 10 sections1What problem is it solving?2The whole picture first3Handling position: one output head per position bucket4Why hide the last hour during training5Time encoding and self-gating6Scoring hundreds of candidates within 50 ms7Experiments: where the gains come from8Where I think it falls short9How a recommendation or ads team could try it10Verdict

Ads ranking teams should read this. It does not invent a new attention design. Its value is that it spells out the practical problems you hit when you bring a generative recommender into ads: the ad position is only known after scoring, online sequences lag behind, training is unstable, and hundreds of candidates must be scored within 50 ms. CADET (Context-Conditioned Ads Decoder-Only Transformer) in one sentence: the user's history is laid out as one sequence of alternating "impression, then action" tokens, a decoder-only Transformer predicts the click for every impression, and post-scoring information such as position is never an input; instead each position bucket gets its own output head. Key numbers: online, against the production model LiRank, CTR rises 11.04% and CPC falls 10.9%; offline AUC is 0.8554, versus at most 0.8505 for the HSTU baselines; removing the per-position heads costs 0.61% relative AUC, the largest drop of any ablation.

What problem is it solving?

The setting is Sponsored Updates, the ads in LinkedIn's homefeed. The production model LiRank combines two models: DCNv2 for feature crossing, and TransAct (a Transformer module that encodes the user's behavior sequence into a vector) for history. Each impression is an independent sample with more than 100 hand-built features. The authors want to replace all of it with one decoder-only Transformer, treating ranking as sequence prediction the way HSTU does.

The hard part is not model capacity but a few constraints specific to ads. The clearest one is position. An illustrative example:

IllustrativeAt position 2At position 8
Ad A, true click rate2.0%0.8%
Ad B, true click rate1.6%0.7%

The same ad gets more clicks when it is placed higher. The auction multiplies predicted CTR by the bid to decide who goes on top, yet predicted CTR itself depends on where the ad ends up. This is the paper's "chicken-and-egg" problem: position is unknown at scoring time because it is decided after scoring. The usual fixes are to feed position as a feature during training and fill in a default value online, or to train a separate debiasing model. The first makes training inputs differ from serving inputs; the second adds another model to maintain.

The second constraint is that online sequences lag. A user sees an ad at 10:00, clicks another one at 10:20, and is scored again at 10:30; the 10:20 click may not be in the online user sequence yet. If offline training can see it, the model learns to rely on information it will not have online, offline metrics look too good, and the model does worse after launch. The third constraint is training stability, and the fourth is inference cost: each request scores hundreds of candidates under a p99 latency budget of 50 ms.

The whole picture first

CADET overall architecture. The bottom row is the input sequence: static user features M, then each impression's features I alternating with that impression's context and action (C, A). The middle is N stacked CADET Transformer blocks. The output at each impression goes into a context-conditioned decoding block, and each computes its own loss.
CADET overall architecture. The bottom row is the input sequence: static user features M, then each impression's features I alternating with that impression's context and action (C, A). The middle is N stacked CADET Transformer blocks. The output at each impression goes into a context-conditioned decoding block, and each computes its own loss.

The pipeline has five steps:

StepWhat it doesOutput
1. Build the sequenceStatic user features, then impression features and context+action in time orderOne sequence of up to 4096 tokens
2. Encode8 Transformer layers with self-gated attention, RoPE driven by timestampsOne hidden state per impression
3. Per-bucket headsOne MLP head per position bucket, each outputs the click logit for that bucketK predictions per impression (K=2 online)
4. Training maskDo not attend to same-session events from the last hourMatches what is visible online
5. Online scoringHundreds of candidates appended to the user sequence, all scored in one forward passK CTRs per candidate; the auction picks by position

The key division of labor: position only selects which head's prediction is used; it is never an input for the current impression. Now the parts one by one.

Handling position: one output head per position bucket

The authors split positions into K buckets by historical click rate. Online they use K=2: positions 1–4 form one bucket and positions 5 and beyond form the other. The shared Transformer output feeds K small MLPs, and head k outputs the click logit "if this ad lands in bucket k".

During training, each impression's real position is known. If an impression was actually shown at position 3, only the bucket-1 head's output enters the cross-entropy; the bucket-2 head does not take part for that sample. The loss is:

Here is the bucket impression t actually landed in, is that head's output, and is whether it was clicked. The intuition: each head is trained only on samples from its own bucket, so it learns "the CTR in this bucket".

Online, one forward pass computes both heads. With the illustrative numbers above, ad A gets two estimates: about 2.0% for positions 1–4 and about 0.8% for positions 5 and beyond. When the auction fills the top four slots it uses the first value; for later slots it uses the second. There is no need to rank once and then re-score by the resulting positions.

Why not just feed position as an input feature? Because it does not exist at serving time, and filling in a default creates a training/serving mismatch. With per-bucket heads, the current impression's position only picks the head that computes the loss; the model's inputs never contain it. Positions of past impressions are known, so the authors put them in the (C, A) slot paired with the action. The model can still learn patterns such as "this user often clicks at lower positions" without leaking the current impression's position.

The context-conditioned decoding block. The Transformer output for impression t goes into K CTR heads and J auxiliary heads; Context t on the left (the real position) only enters the output-head router, which decides which CTR head contributes to the loss.
The context-conditioned decoding block. The Transformer output for impression t goes into K CTR heads and J auxiliary heads; Context t on the left (the real position) only enters the output-head router, which decides which CTR head contributes to the loss.

The auxiliary heads on the right predict click type, landing-page dwell time and impression duration. They are added to the loss only during training as a regularizer and are not computed online. The total loss is a weighted sum of the CTR loss and the auxiliary losses; the paper does not give the weights.

Why hide the last hour during training

The 10:20 click example shows the problem. The authors add a time-based attention mask in training: impression i cannot see any event whose timestamp falls within one hour before it.

Here is the minimum visibility delay the online system guarantees, one hour in the experiments. Positions set to minus infinity get zero weight after softmax, so the model cannot see them. The paper stresses that "session" here is not a hard boundary, only "events inside the delay window".

The ablation for this design is done carefully. The validation set is always masked at one hour, simulating real serving; only training sets the delay to zero, letting the model see every earlier event. AUC drops 0.31% relative. In other words, the extra hour of information seen in training does not help under serving conditions; it makes the model worse, because the model learns to depend on a signal it cannot get online. Many teams evaluate such leaks with a validation set that leaks too, and see an inflated offline gain; this setup avoids that trap.

Time encoding and self-gating

Standard RoPE rotates queries and keys by position index in the sequence. Two adjacent impressions might be 30 seconds apart or three weeks apart, and index-based encoding cannot tell the difference. CADET replaces the index with the Unix timestamp. The rotation angle of the i-th two-dimensional component is:

is the maximum lookback window of the sequence (1 year), sets how far the slowest component rotates for a full one-year gap (), and sets how fast the fastest component is (). Attention between two events depends only on their time difference, and closer events differ less in rotation. Removing the time RoPE costs 0.13% relative AUC.

One thing here does not match the paper's wording. With these settings, the fastest component rotates about radians over one year, about 0.16 radians over one day, and only about 0.007 radians over one hour (my calculation from the paper's formula). So this setting mainly separates "days to months"; differences within an hour produce almost no rotation, which is some distance from the abstract's "seconds to months". The authors do note that ordering within a dense burst of events is kept by the causal mask, not by RoPE.

Self-gating is there for training stability. There are two places with gates, shown below. The first sits before attention: a linear layer plus sigmoid computes a coefficient between 0 and 1 for each dimension and multiplies it back onto the vector, suppressing noisy dimensions. Illustrative: if a dimension's gate input is −2, the sigmoid gives about 0.12, so only about one eighth of that value remains. The second acts separately on the projected Q and K, limiting dot-product magnitude so that a few tokens cannot absorb most of the attention, the effect usually called an attention sink.

The self-gated attention module. Green is the gating unit: the input passes a dense layer and a sigmoid, then is multiplied element-wise with the input. One gate acts on the input tokens before Group LN; two more act on the K and Q projections.
The self-gated attention module. Green is the gating unit: the input passes a dense layer and a sigmoid, then is multiplied element-wise with the input. One gate acts on the input tokens before Group LN; two more act on the K and Q projections.

Self-gating has no AUC number. The paper only describes training curves over several runs: without gating, AUC drops sharply early in training; with the first gate only, the drops are fewer and smaller; with both, training is most stable. For a production model this matters more than a few hundredths of AUC: a model that may blow up on any retrain cannot be shipped.

Scoring hundreds of candidates within 50 ms

Each online request scores hundreds of candidate ads. CADET appends all candidates to the same user sequence and uses a special mask so that each candidate sees only the user history and none of the other candidates; one forward pass scores them all. The authors wrote a custom FlashAttention kernel that skips fully masked blocks, so attention cost goes from to , where L is the history length and N the number of candidates.

An illustrative calculation: with L=4096 and N=500, the first is about 21.1 million pairs and the second about 10.4 million, half as many. More importantly, the second is linear in N: doubling the candidates only grows the L×N term. Measured, the kernel is 3 times faster than the standard fused multi-head attention kernel (FMHA), reaching 330 requests per second per GPU at p99 under 50 ms. The appendix stresses that this kernel is used only for serving; training and offline evaluation use stock FlexAttention (PyTorch's attention interface that supports custom masks), so the accuracy results do not depend on it.

On the training side there are two pieces of data engineering. First, moving from "one sample per impression with 100+ features" to "one sequence per user with 12 high-value signals per token" cut data volume by 98% with no loss of accuracy. Second, packing: instead of padding every sequence to the same length, all tokens in a batch are concatenated into one contiguous buffer with the start and end of each sequence recorded. User sequence lengths are heavily right-skewed and most are short, so dropping the padding makes the effective batch about 4 times larger. The authors note that the total token count per batch must be fixed, or torch.compile and training stability both suffer.

Experiments: where the gains come from

Offline experiments train on one year of ad click data and validate on the last day. The model has 8 layers, 4 heads, hidden size 352 and sequence length 4096, which is not large.

ModelAUC
HSTU-Base0.8497
HSTU-Large0.8505
CADET-HSTU (no training mask)0.8535
CADET-Transformer (no training mask)0.8534
Full CADET0.8554

Start with the two HSTU sizes: going from Base to Large adds only 0.0008, so on this data simply enlarging the backbone helps little. Then the two CADET variants: swapping the backbone to HSTU while keeping everything else changes AUC by only 0.0001 against the default Transformer backbone. This shows that the gains come from the per-position heads, time encoding and gating, not from the choice of backbone. Neither variant uses the training mask, because Meta's released HSTU Triton kernel does not support that mask. Full CADET with the mask adds about 0.002 more.

AblationAUC change relative to full CADET
Remove per-position heads−0.61%
Remove time RoPE−0.13%
Remove training mask−0.31%

The per-position heads contribute the most, which fits the problem: position has a large effect on click rate, so a model that gives different estimates per position ranks better.

The online A/B test ran on homefeed Sponsored Updates against LiRank, with CADET fully replacing both DCNv2 and TransAct. CTR rose 11.04%, statistically significant. Revenue was roughly flat and CPC fell 10.9%. Read these together: most LinkedIn advertisers use auto-bidding and usually spend their full budget, so total spend stays the same; the same budget buys more clicks, so the cost per click goes down. This means the gain landed in advertiser ROI rather than in short-term platform revenue.

Where I think it falls short

First, a link is missing between offline and online. The paper gives no offline AUC for LiRank: the offline table compares only with HSTU, and the online test compares only with LiRank. Readers cannot tell how much of the 11.04% comes from the architecture and how much from sequence-format data and longer history. Traffic share and duration of the online test are not disclosed either.

Second, the two tables do not quite agree. The ablation says removing the training mask lowers AUC by 0.31%, which from 0.8554 would be about 0.8527; the comparison table gives 0.8534 for CADET-Transformer without the mask, only about 0.23% lower. The gap is small, but it suggests the two sets of experiments used different settings, and the paper does not explain this.

Third, position has only two buckets, and position is the only post-scoring signal handled. K=2 was chosen to make auction integration easy. Card format, whether the ad is in the first screen and similar factors also affect clicks, and the authors leave modeling several post-scoring signals together for future work. The auction multiplies pCTR by the bid, so calibration directly affects pricing, yet the paper reports no calibration for either head.

How a recommendation or ads team could try it

  1. Start with the training mask alone. Set a time window equal to your real online sequence delay, mask events inside it during training, and always mask the validation set. This is the smallest change and directly shows whether your current model relies on information it cannot see online.
  2. Then try per-position heads. Remove the position feature from your ranking model's inputs, split positions into 2–3 buckets by historical CTR, and give each its own head. Beyond AUC, check calibration per bucket and make sure each bucket has enough samples.
  3. If you run a sequence model, replace RoPE's index with timestamps and work out the parameters for your own business: to separate hour-level gaps, the fastest component must rotate noticeably within an hour.
  4. Only then consider a full replacement. First compare AUC against the production model on the same offline data to confirm the architectural gain, then run an A/B test watching CTR, CPC, revenue and p99 together.

Verdict

In one sentence: bringing generative recommenders into ads is hard not because of the backbone, but because you must get "information unknown at scoring time" and "information not visible online" right.

DimensionAssessment
Problem importanceHigh. Position dependence and training/serving mismatch exist in every ads ranking system
NoveltyMedium. Each part has prior work, but the combination and the per-position heads are very practical
EvidenceMedium. +11.04% online is strong, but there is no offline comparison with the production model and no numbers for gating
Engineering costMedium-high. The full version needs a sequence data pipeline, packing and a custom attention kernel
Worth tryingYes. The training mask and per-position heads can be tried on their own at low cost

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Related papersMore from Paper Daily

  1. Personalization · 2026-11-04Numerical Embeddings for CTR Prediction: A Taxonomy, Unified Benchmark, and Open-Source FuxiCTR ExtensionThere are a dozen or so ways for a CTR model to turn numerical features such as prices and counts into vectors, and in one fair comparison no single method wins everywhere. This used to be hard to compare, because papers differ in preprocessing and training pipelines. This paper puts 18 of them into one codebase with the same tuning budget and compares them across several datasets and backbones.…
  2. Personalization · 2026-10-29Rethinking Positional Encoding and Interaction Granularity for CTR PredictionIn unified Transformer CTR models like OneTrans, the authors tried six positional encodings on Kuaishou's KuaiRand data, and none beat using no positional encoding at all. Building on this, the Wuhan University and Tencent paper proposes TRIP (the authors' name for the whole method). Each behavior encodes both its rank in the sequence and how long before this prediction it happened. Behaviors are…
  3. Personalization · 2026-10-28Towards a Unified Framework of Optimal Input-Side Dimensional Configuration for Deep CTR PredictionThere are three common ways to trim the input side of a CTR model: drop low-quality training samples, switch off useless features, and prune useless embedding dimensions. Earlier methods do one at a time. This paper from Shenzhen University and collaborators argues that all three are the same operation, multiplying one embedding tensor by a mask at different granularities. They also interact: onc…
  4. Personalization · 2026-10-23Progressive Learning from Content to Relevance to Personalization for CTR PredictionTaobao search CTR models mainly remember historical clicks through item IDs, so the same item with a different main image is almost indistinguishable to the model. FineSID turns each item image into 30 discrete codes across four VAR tokenizer scales from 1×1 to 4×4, tokenizes titles and OCR text by field, and looks them all up like ordinary ID features; the shared network is first trained on quer…
  5. Personalization · 2026-10-21FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR PredictionCTR self-distillation can be replaced with a computable disagreement penalty. FLMNet keeps gates on clean inputs and distills independently trained towers into one serving model. The distilled model reaches 0.8169 AUC on Criteo.
  6. Personalization · 2026-08-21Dual-Stream MLP is All You Need for CTR PredictionDeep dive: Dual-Stream MLP is All You Need for CTR Prediction.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…