FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction

FLMNet replaces repeated masked predictions with a computable disagreement penalty, then distills the agreement of independently trained models into one serving tower. The paper separates two sources of variation: masks within a model and training seeds across models.

1. Problem and motivation

CTR models combine categorical user, item, and context features. Possible feature combinations greatly outnumber training examples, so the model must generalize beyond observed combinations. Dropout exposes it to incomplete views during training. Self-distillation additionally encourages these views to agree with an ensemble prediction.

The open question is what that extra objective buys. Is random sampling essential, or is it a noisy way of applying a useful regularizer? Repeated forward passes also make training more expensive, even when only one network is served.

2. Method

The derivation pools logits, the scores before conversion to click probabilities. Averaging logits and averaging probabilities are different operations. The exact result applies to the specified pooling objective, rather than every loss described as self-distillation.

As the number of masks tends to infinity, the objective decomposes into label loss at the mean logit and a penalty on disagreement. That penalty is a Jensen gap: the expected value of a convex function minus its value at the expected input. This is an identity about the objective, not a guarantee about optimization trajectories or test accuracy.

FLMNet makes the required moments tractable by keeping its gates off the masked path. Gates read clean input embeddings. If gates instead read masked intermediate representations, later factors inherit earlier randomness and independence assumptions fail.

Training still approximates the true disagreement gap. It replaces the mask distribution with a Gaussian of matching variance and uses three-point quadrature. Variance truncation is a separate approximation examined in the paper. One unmasked forward pass supplies the prediction and statistics; computing their gradients still costs work.

The second stage addresses seed disagreement. Independently initialized and trained towers remain different even after mask disagreement is regularized. Their pooled scores become a teacher for a single student. This saves serving cost but requires extra offline training runs.

3. Main contribution

The useful distinction is between within-run regularization and across-run distillation. Their gains largely add in the experiments. The architecture also exposes a concrete condition to audit: whether random masking reaches the gates that control feature interactions.

Single-pass training should not be confused with free training. Moment computation, differentiation, teacher training, and student fitting all belong in a cost comparison.

4. Results

The main comparison covers five benchmarks and seventeen established models. Selected advertising results are test-mean AUC values; higher is better.

CF uses the closed-form penalty approximation; KD adds cross-seed distillation. On Criteo, each step contributes. Same-backbone regularizer comparisons help separate the loss effect from architecture changes.

The timing intervention is especially informative. Turning the penalty on slows the growth of the train–validation gap; turning it off accelerates overfitting. Applying it from the first epoch can also slow early learning.

That intervention uses one dataset and a constant learning rate. It supports the proposed mechanism without establishing a universal activation epoch.

5. Limitations

The authors identify three boundaries: moment formulas require the stated architectural conditions, training uses a Gaussian surrogate, and the recoverable cross-seed gain is known empirically rather than in closed form.

Evaluation details matter too. Avazu uses a disclosed re-splitting and retraining protocol. Some distilled students share teacher sets, so they are not independent replications. Public benchmark AUC gains do not establish an online CTR lift.

An additional engineering implication is refresh cost. A frequently updated service must budget for teacher runs, score caching, and student training, alongside its single-tower serving savings.

6. Practical implications

Audit the pooling rule and the gate inputs before adapting the formula to an existing CTR model. Hold the backbone, embedding sizes, and training budget fixed when comparing no penalty, sampling-based regularization, and the deterministic approximation.

Report paired-seed differences and train–validation curves. Evaluate cross-seed distillation as a separate budget decision, with calibration, latency, and refresh interval alongside AUC.

Sources: Sections 3–5, the conclusion, and Appendix B. All figures are original assets from the paper’s arXiv HTML.