Predictable Scaling: Efficient Model Exploration via Scaling-Law Prediction

A smaller ranking model is useful for experimentation only if it still identifies which changes are better. Meta's Predictable Scaling makes that requirement explicit and tests it against known full-scale results. The compute savings concern model-development experiments, rather than a claim that a compressed production model preserves every business metric.

1. Motivation: experimentation becomes a bottleneck

Recommendation teams repeatedly change features, interaction layers, and data sampling. Each idea may require a costly full training run before its value becomes clear. This verification cost limits how many ideas a team can explore.

Reducing data or network size seems straightforward, but ranking models contain coupled embedding tables, interaction networks, history encoders, and task heads. Shrinking one component can change the bottleneck and reverse the relative value of two candidate changes.

The paper therefore predicts each candidate's improvement relative to its baseline rather than predicting absolute final loss. This matches the development decision: which changes deserve further compute?

2. Method: choose the proxy and test its distortions

Each candidate and baseline receive the same scaling treatment and are trained as a pair. Their relative metric gap is the proxy observation. Without that pairing, scaling effects could be confused with the candidate's actual contribution.

The main metric is normalized entropy, which scales prediction cross-entropy by the entropy of the overall positive rate. Lower is better; a negative relative gap means improvement. These gaps are not CTR percentage-point changes.

Scaling functions reduce samples, embedding-table rows, embedding width, input features, hidden dimensions, or layer depth. They may be composed, but their interactions can create more distortion than either component alone. Historical candidates with known full-scale outcomes provide a benchmark for selecting configurations.

The simplest predictor uses the proxy gap directly. Linear regression adjusts approximately affine relationships. An empirical Gaussian process learns the mean and covariance structure of historical learning curves and extrapolates a proxy's convergence behavior while estimating uncertainty.

This distinction matters: a flexible predictor cannot recover information destroyed by an unsuitable scaling rule. Predictions with excessive relative uncertainty are rejected rather than treated as dependable screening results.

The framework checks stable convergence, transferable gaps, and preservation of improvement direction. Under a compute budget, it evaluates feasible proxy configurations against historical results using cross-validation. Ranking fidelity and sign reversals complement average error when deciding which configuration can screen new hypotheses.

3. Contribution: evaluate proxies as decision tools

The framework connects proxy design, prediction accuracy, and development decisions. It replaces a universal shrink ratio with evidence that a particular configuration works for a particular architecture.

Repeated baseline training provides a noise floor. Differences below that scale may be indistinguishable from run-to-run variability. The paper separately measures ordering among actionable changes whose full-scale effects exceed the noise floor.

A low mean absolute error alone is insufficient. Predictions clustered near zero can have a small error while offering little information about which idea is better. Ordering matters when a team must decide which candidate receives the next expensive run.

4. Evidence: reduced experiment cost, with architecture dependence

The industrial evaluation includes a task-specific feed CTR/CVR model and a multi-task foundation model. Public Criteo experiments examine several common recommendation architectures to test behavior beyond the internal systems.

Errors are expressed in units of the repeated-run noise scale. The task-specific improvement is indicative and has substantial standard error. Some foundation-model heads benefit more, consistent with a larger proxy-to-full-scale mismatch that curve fitting can address.

Reported training-compute reductions are 88% and 79% for the two industrial models. The accounting includes trainer count, processed samples, and throughput. Lower proxy throughput can coexist with lower total consumption when devices and samples decrease sufficiently.

Perfect ordering is reported for actionable industrial hypotheses above the noise floor. It does not apply to every candidate, architecture, or public experiment.

Public results expose the boundary. Different architectures prefer different scaling configurations. Embedding-width reduction alone performs poorly at preserving rankings, and the best DCN-v2 configuration still provides only moderate ranking correlation. Low prediction error and reliable candidate ordering can diverge.

5. Limits: fitting cannot repair lost structure

The authors discuss tightly coupled feature-interaction networks as a difficult case. Shrinking dimensions can remove the interactions responsible for a candidate's gain. A Gaussian process may correct predictable bias but cannot undo that structural loss.

Data subsampling also depends on the original scale. A fraction of a huge internal dataset may retain sufficient signal, while the same fraction of a smaller public dataset may not. There is no universal optimal reduction ratio.

The industrial target is primarily offline normalized entropy. Extending the framework to additional online metrics is future work. Development-compute savings cannot be converted into equivalent CTR or revenue gains from these experiments.

An operational implication is that the historical benchmark must remain representative. A new backbone or feature pipeline can invalidate an old proxy configuration. The fallback to full training should be enforced in the workflow whenever uncertainty or fidelity checks fail.

6. Practical implications

Maintain a paired-experiment ledger containing the baseline, candidate change, proxy configuration, final metrics, learning curves, and actual compute. Relative changes are more useful for this purpose than isolated final scores.

Replay screening against held-out historical changes. Measure missed useful candidates and incorrect improvement signs, rather than only average prediction error. For community-search ranking, separately examine how click-prediction effects relate to relevance metrics.

Allocate compute between exploration and confirmation. A validated proxy can cheaply explore many ideas; close calls, contradictory rankings, and uncertain predictions return to full-scale evaluation. The practical objective is more credible discoveries within a fixed experiment budget.

Source: https://arxiv.org/html/2610.23453v1 . Methods and numerical results come from the main paper and appendices. The workflow illustration and practical suggestions are editorial analysis.