Improving Ads Ranking Models with Auxiliary Supervision: Techniques and Insights

Ad CVR models are trained on conversion events such as purchases and installs, but conversions are rare, and about half of the training samples in this Meta model have no conversion label. The paper fills the gap with other signals: how long a user looked at the ad before clicking, add-to-cart events, the semantics of the ad content and similar signals all become auxiliary tasks trained alongside the main task, at no extra online inference cost. Auxiliary signals are often noisy. The most effective fix is called LaCo, and it works on yes-or-no signals, such as whether the user added the item to the cart. LaCo combines several such signals with AND, OR and XOR, picks the combination that best matches real conversions, above all the one with the fewest false positives, and trains on it as a new label. It lowers offline NE by 0.25%. After these changes shipped together with other improvements, advertiser conversion volume rose with statistical significance in A/B tests. For teams doing conversion prediction, making auxiliary labels accurate pays off more than piling on more auxiliary tasks.

Teams doing conversion prediction should spend ten minutes on it. It also describes a few changes to the multi-task structure, but its focus is a more practical question: when the main label is badly missing, which signals should serve as auxiliary labels, and how to clean them first. The core idea in one sentence: the accuracy of an auxiliary label matters more than the number of them, so it is better to merge several signals into one label with few false positives than to give each signal its own task. The paper is short, and its evidence is one offline table and one sentence about the launch, so read it as a summary of experience rather than a rigorous controlled study.

1. What problem is it really solving?

A pCVR model predicts whether a user who clicked an ad will purchase or install within the attribution window. Each training sample is one clicked ad impression, and the main label says whether that click led to a conversion.

The trouble is the main label. Conversions are rare to begin with, and for many users no conversion result comes back at all because of privacy choices or attribution limits. In the paper's baseline model, about 50% of training samples have no conversion label. The model can learn "who converts" only from the other half, so the supervision is thin.

After a click, though, users leave plenty of other traces. Here is an e-commerce ad as a demonstration (illustrative):

A and B both lack a conversion label, so the main task learns nothing from these two samples. Yet A stayed 45 seconds and added to cart while B left after 3 seconds, so their purchase intent clearly differs. Auxiliary signals are meant to supply exactly this information.

How are these signals used? The paper uses multi-task learning. The bottom of the model is shared, and on top, besides the main task that predicts conversion, there are a few small fully connected heads, each predicting one auxiliary label such as "stayed longer than 10 seconds". During training, the auxiliary heads' losses also flow back into the shared part, so the shared representation learns more about purchase intent. At online inference only the main task's output is used, and the auxiliary heads take no part in scoring. Dwell time and add-to-cart happen after the click, so they serve only as training labels, not as input features, and the model never sees them at inference.

2. Which signals can serve as auxiliary labels?

The paper sorts usable signals into four kinds; three of them are covered below. The first is behavioral signals, actions the user takes on the ad: how long they looked before clicking, funnel events such as "add to cart" reported by third-party tracking, and how long they stayed in the web browser before returning to the app (the dwell time).

The second is semantic signals, judged from the content of the ad and the session. Examples are that the product category browsed in the session matches the ad's category, or that the session URL contains a word like "confirmation" that usually means an order was completed.

The third is synthetic signals, which create data from user-advertiser interactions in other settings. The paper's example is for a pCTR model: building training data from users' views of an advertiser's video content. These labels are less trustworthy than the main label, so they serve only as auxiliary supervision, never as the main label.

3. How are the signals wired into the model?

The most direct wiring depends on the signal type. Probability or yes/no signals get a binary classification auxiliary task. Numeric signals such as dwell time get a regression task, and some signals work better after a log transform. The paper notes that regression auxiliary tasks are especially effective when the main task is itself a regression. On top of this, the paper gives three further techniques.

Why split dwell time into a few bins?

A binary "stayed longer than 10 seconds" label gives a user who stayed a dozen seconds and one who stayed several minutes the same label, so the model cannot tell how strong their intent is. So linearized labels cut a continuous or ordinal signal into a few bins and turn the task into multi-class classification. The paper's example splits dwell time into three bins: under 10 seconds, 10 seconds to 1 minute, and over 1 minute.

The bin boundaries are hard to set. The paper's experience is that quantile boundaries with 2 to 4 bins work well. Boundaries can also differ by ad type: e-commerce users take longer before buying, so their boundaries can be longer; lead-generation ads convert quickly, so their boundaries can be shorter.

Several signals can also be merged into one multi-class label. Order them by how close they are to conversion: "conversion" is closest, then "dwell longer than 10 seconds", then "HTML content loaded". Each sample's class is the first signal in this order that holds. User A in the table above has no conversion record but stayed 45 seconds, so A's class is "dwell longer than 10 seconds". User C purchased, so C's class is "conversion".

Why add gating?

Some auxiliary labels exist only on part of the samples. Dwell time is recorded only when the click opens a web browser; an install ad that opens an app store has no such value. Treating the missing value as a negative teaches the model that "these people were not interested", which hurts results. Gating gives the auxiliary task a condition and computes its loss only on samples that meet it; on the other samples this head does not train.

Split architecture: cutting one conversion window into two parts

The third technique changes the main task itself. When the conversion window is 7 days after the click, it can be split into two sub-tasks: converting within 0 to 1 day, and converting within 1 to 7 days. The paper requires the sub-tasks to be mutually exclusive and to cover the whole 7-day window together, so the two probabilities add up to the probability of converting within 7 days. For example, if a sample's prediction is 0.03 for 0 to 1 day and 0.01 for 1 to 7 days (illustrative), together that is 0.04 within 7 days.

The sum of the two sub-task predictions is then ensembled with the original main-task prediction to give the final prediction. Figure 2 of the paper labels this step as a combination with learned weights. Note that the two split sub-tasks take part in the final score, unlike the auxiliary heads above that exist only during training.

4. What to do when the signals are noisy: LaCo

Auxiliary signals are noisy by nature. A long stay does not mean a purchase, and a loaded HTML page means even less. The paper sorts the noise into three kinds: logging errors, which outlier filtering can remove; missing labels, which gating handles; and weak labels, meaning that signals farther from conversion are less accurate. Moving down the funnel, signals become more precise but cover fewer samples.

With many signals, giving each one its own auxiliary task can make things worse, because the tasks' gradients interfere with each other. LaCo (Label Combination) takes a different route. At the label-design stage, before training starts, it merges several signals into one cleaner label and uses only that label for an auxiliary task.

How LaCo picks a combination

Each LaCo candidate label is a logical expression: pick a few of the available signals and join them with AND, OR, XOR, or the negated forms of OR and AND, for example "add to cart" AND "dwell longer than 10 seconds". Each candidate label is measured against the real conversion label to get its precision and recall, which are merged into the score below, and the highest-scoring candidate is kept:
$$F_{\beta }=(1+{\beta }^2)\cdot \frac{P\cdot R}{{\beta }^{2}P+R}$$
Here P is precision and R is recall. The smaller $\beta$ is, the more the score weighs precision; at $\beta =1$ both count equally.

A hand-checkable example (illustrative): there are just two signals, "add to cart" and "dwell longer than 10 seconds", and assume both are recorded on every sample. Requiring both (AND) gives precision 0.9 and recall 0.3; requiring either (OR) gives precision 0.5 and recall 0.8. With $\beta =0.02$, AND scores about 0.90 and OR about 0.50, so AND is chosen. With $\beta =1$, AND scores 0.45 and OR about 0.62, so the choice flips to OR. So $\beta$ really decides whether you want an accurate label with low coverage or a high-coverage label with more false positives.

The conversion label is used here only to pick the combination offline. Once chosen, the combined label becomes a training target; at online scoring the model sees neither the conversion label nor any of these auxiliary signals. Hoot says▾

Hoot saysThe conversion label appears only when picking the combination offline, not at online scoringClose ✕

The same signals play different roles in three stages: offline, the conversion label scores each candidate combination and the one with fewer false positives is chosen; in training, that combined label teaches only one auxiliary head; at online scoring the model outputs only the conversion prediction, and neither the conversion label nor the auxiliary signals take part.

1. At β=0.02, AND: precision 0.9 and recall 0.3 give a score of about 0.90.
2. OR: precision 0.5 and recall 0.8 give about 0.50. With a very small β the score is almost only precision, so AND is chosen.

Illustrative: the same two-signal example as the text, assuming both signals are recorded on every sample; precision and recall are illustrative, and β=0.02 is the best value in the paper. Source: the worked example in section 4 of the dive; the split into three stages follows sections I and IV of the paper.

Precision matters more than recall

On one conversion prediction model the paper tried $\beta$ from 0.001 to 10. The best was $\beta =0.02$, where the combined label had 91% precision and moderate recall; every setting with $\beta \ge 1$ did worse. This shows that for auxiliary labels, fewer false positives are worth more than covering more samples. The authors' practice is to hold precision near 90% first and then raise recall as far as possible under that constraint. The best $\beta$ varies with the model and data, so each team needs to tune it.

Noise-robust training methods

Besides LaCo, which works on the label, the paper also tried two methods that change training directly. One is DivideMix, and the other is Tri-Training. The paper says both helped on its pCVR model, and Table I reports them together as one row.

5. Results: each technique improves NE by 0.10% to 0.25%

The paper's offline metric is NE, where lower is better. The table below gives each technique's NE when added to the baseline model on its own. Every row is compared with the same baseline; the rows do not stack:

LaCo gives the largest improvement, about 1.7 times that of the runner-up, linearized labels. This matches the paper's claim that merging several noisy signals into one accurate label works better than giving each signal its own task. The table has no row for "one auxiliary task per signal", however, so that comparison is visible only in the authors' description.

These numbers are tiny in absolute terms: NE differs by only 0.0005 to 0.0013. The authors say that in their system, offline gains of this size have historically correlated with measurable online gains. At launch, these techniques shipped together with other improvements, and advertiser conversion volume rose with statistical significance in A/B tests, but the contribution of each technique cannot be separated.

6. Where I think it falls short

First, the evidence is thin. All results are one table, one model and four numbers, with no error bars and no run-to-run variation. A gain of 0.0005 NE could fall within the noise in another system.

Second, the online effect cannot be attributed. The paper says only that conversion volume rose with statistical significance after a combined launch, without giving the size, and other improvements shipped in the same batch. To learn what LaCo alone is worth, you have to run your own experiment.

Third, $\beta =0.02$ was tuned on a single conversion model, and the authors themselves say the best value varies by model. In a business with a very different conversion rate, the 90% precision rule of thumb may not hold, and $\beta$ needs a fresh sweep.

7. How a recommendation or search team could try it

1. Start by listing the post-click signals you have, such as dwell, add to cart, page load and visits to key pages, and compute each signal's precision and recall against conversion on samples that have a conversion label. This step needs no model training, and offline counting alone already shows which signals are worth using.
2. Use the LaCo approach to enumerate logical combinations of these signals offline, and at first look only at the highest-recall combination among those with precision near 90%. Wire it in as one auxiliary task and compare it offline with the "one auxiliary task per signal" version to see which gives the lower NE or logloss.
3. Add gating for signals that exist on only part of the samples, then try linearized labels with dwell time cut into 2 to 4 bins. Change one thing at a time to confirm whether each brings its own gain.
4. Move to an A/B test once the offline results hold. Auxiliary heads do not score online, so inference cost stays essentially the same; the split architecture changes the final score and should be evaluated separately.

8. Verdict

The main takeaway: when the main label is badly missing, effort spent making auxiliary labels accurate pays off more than adding more auxiliary tasks.