Contents · 5 sections
1What problem does it actually solve?2Method3Results4Limitations5What to borrowSHARE
Payment platforms process enormous numbers of transactions every day, and fraud control, risk and marketing all want to "read" a user's spending habits. But labelled fraud cases are extremely rare, and purely supervised models easily memorise accidental feature combinations. This paper from WeChat Pay, Shanghai Jiao Tong University and others carries over the large-model recipe of "first learn to predict the next word on unlabelled data" to transaction sequences. Offline, they train PANTHER (Pattern AttentioN Transformer with Hybrid User ProfilER) on 5.3 billion transactions to predict the next transaction, which yields a representation vector for every user; online, a light classifier combines these precomputed vectors with the current transaction to score it. In a 10-day full-traffic online A/B test, fraud caught among the riskiest 0.1% of transactions rose by 38.6%, and among the riskiest 0.01% by 109.5%, at a cost of only 5 extra milliseconds.
It is worth reading for teams doing transaction risk, user-sequence modelling, or anyone trying to fit "pretraining plus a light downstream head" into a low-latency production system. Most of the evidence comes from internal WeChat Pay data; the gains on public datasets are much smaller, so read the two parts separately.
What problem does it actually solve?
The paper argues that supervised risk models at billion-user scale have four old problems: too few positive examples to cover feature combinations; high-cardinality categorical features that invite memorising spurious co-occurrences; latency budgets that force long histories to be truncated; and static user vectors that do badly on new users.
Transaction sequences also differ from text. Text has grammar; transactions do not, but they have strong cycles: the morning commute every day, leisure every weekend, fixed bills every quarter. Standard Transformer self-attention looks at events one by one, and the authors argue it is poor at catching such local, recurring regularities, which also get diluted in long sequences.
Method

PANTHER splits the computation into an offline part and an online part so that the heavy work does not add to online latency. Offline, a profile token is placed in front of each user's transaction sequence and the model is trained to predict the next transaction, producing a user profile vector and a predicted distribution over "what the next transaction is likely to be". Online, the classifier takes four inputs: the context of the current transaction, the last 100 transactions (encoded with TextCNN), the precomputed user profile vector, and how far this transaction deviates from the model's prediction. The deviation is computed as the model's predicted probability for the actual transaction, minus the mean of the model's predicted probabilities over all possible transactions for this user, divided by their standard deviation; users whose spending is naturally scattered have a larger standard deviation, so their deviation score is not inflated.
Transactions first have to become tokens the model can read. A transaction combines payment channel, amount range, merchant category and merchant risk level into one token, for example "credit card, 50 to 100, fuel, low risk". There are about 2 million combinations, far too sparse, so the authors keep the 60,000 most frequent; they cover more than 96% of historical transactions, which shows the truncation loses little, and every other combination becomes "unknown".
To recognise recurring spending patterns, the authors add a module (called SPRM, Sequence Pattern Recognition Module) in parallel with self-attention and add its output to the self-attention output. It has two layers:
- First, convolutions with different gaps scan the sequence: a gap of 1 looks at neighbouring transactions, a gap of 2 skips every other one and can pick up regularities that repeat every few transactions, even when occasional transactions sit in between.
- Then there are m learnable vectors, each standing for a typical spending pattern such as "weekend leisure" or "weekday commute"; the paper calls them prototypes, with m=64 in practice. Each short stretch of transactions attends over these 64 vectors using the convolution output, finds which patterns it most resembles, and is represented by a mix of them.
Because the number of prototypes is fixed and far smaller than the sequence length, this step's cost grows linearly with sequence length rather than quadratically as in self-attention.

Finally, the user profile vector combines the user's static attributes with transaction history. The authors try two uses: as the first token of the sequence, or as a user-specific positional encoding so that every transaction in the sequence carries the user's background. The paper gives this use only by name and result; exactly how it combines with each transaction's representation is something to settle when reproducing it. On top of that, contrastive learning pulls together users with similar attributes (age within 2 years, same region, and so on) and pushes others apart. The final objective is a weighted mix of the next-transaction loss and the contrastive loss.
Results
Pretraining uses 5.3 billion transactions from 38 million users over 6 months. For next-transaction prediction, with a plain Transformer as reference, the cumulative effect of the components is:
| Configuration | HR@1 | vs Transformer |
|---|---|---|
| Pattern module only | 0.2243 | +14.9% |
| Add profile vector as the first token | 0.2301 | +17.9% |
| Profile vector as positional encoding instead (replaces the row above) | 0.2351 | +20.4% |
| Add contrastive learning on top of the positional-encoding use | 0.2452 | +25.6% |
The pattern module contributes the most on its own. The paper's text says using the profile as a positional encoding adds 5.5 points over the pattern module alone, so that row swaps the usage on top of the first row rather than stacking both uses; the positional-encoding use beats putting the profile first.

The online fraud baseline is DeepFM with handcrafted features plus TextCNN. The recall gain shrinks as the threshold loosens: +109.5% at the strictest 0.01%, +38.6% at 0.1%, +12.1% at 1%. The advantage is concentrated on the rarest, highest-risk transactions, exactly the cases where personalised vectors and deviation scores make the difference.
Gains on public datasets are clearly smaller. All methods use the same depth, learning rate and batch size:
| Dataset | Type | PANTHER HR@1 vs Transformer |
|---|---|---|
| MBD-mini | Bank transactions | +2.95% |
| CCT (Credit Card Transactions) | Synthetic card transactions | +15.2% |
| MovieLens-1M | Movie recommendation | +21.8% |
| Yelp | Business recommendation | +9.17% |
Two more downstream results: adding the pretrained vectors to a DCN recommender on Yelp lifts NDCG@5 by 29.6%; adding them to a TabBERT-MLP risk model on the synthetic card dataset CCT (Credit Card Transactions) lifts recall by 19.9%. For merchant risk, each merchant's customers are scanned one by one, and the method above gives a deviation score for "this user transacting with this merchant"; the value at a fixed quantile of these scores becomes the merchant's risk score (the quantile is a tunable hyperparameter, to be chosen on your own data), some statistics of the scores are added, and XGBoost makes the call. Among merchants flagged high-risk, 76% truly were, against 50% for the previous method.
Limitations
- The most persuasive numbers (25.6%, 38.6%, 109.5%) all come from internal data that outsiders cannot reproduce; on public MBD-mini the gain is only about 3%.
- CCT is synthetic, with 2,000 users, far from real payment scale.
- The online baseline is DeepFM, not a strong model that also uses long sequences, so it is hard to separate how much of the gain comes simply from using a longer history.
- Only the 60,000 most common transaction combinations are kept and rare ones are merged into "unknown", yet fraud often happens precisely in rare combinations.
What to borrow
The easiest piece to copy is the two-stage setup: train a sequence model offline to predict the next action, store the user vector and the next-step prediction, and add only a light classifier online, which adds very little latency. Multi-attribute events such as transactions can be combined into tokens by attribute and the vocabulary truncated by frequency. For behaviour sequences with clear cycles, a module of gapped convolutions plus a small set of learnable pattern vectors can run in parallel with self-attention at linear cost. The gap between the predicted distribution and the actual action is itself a very useful anomaly feature that can be added to an existing risk model on its own.
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Related papersMore from Paper Daily
Personalization · 2026-10-22KuaFu: Compressing Long User Behavior into Understanding at Billion ScaleKuaFu avoids repeatedly encoding the same content across users and profiling tasks. It caches compact item representations, assembles them chronologically and uses staged training to preserve evidence. Tencent reports a 1.37% overall GMV increase against a long-term holdout.
Personalization · 2026-07-13UCGRec: User-Centric Graph Learning for LLM-based Sequential RecommendationDeep dive: UCGRec: User-Centric Graph Learning for LLM-based Sequential Recommendation.
Personalization · 2026-11-11Query- and Candidate-Aware Long User Sequence Modeling for Sponsored Product Ranking in SearchWhen sponsored search ranking uses long-term behavior, the usual approach searches history for actions similar to the candidate product, which can miss history highly relevant to the current query but only loosely tied to the candidate. Beyond recent behavior, Wayfair retrieves one batch of long-term actions by candidate and another by query, then feeds them to a Transformer with two context toke…
Personalization · 2026-10-20LSNR: Learned Sparse News Recommendations with Attentive User ModelingEncode articles separately, then learn which ones matter. LSNR combines weighted sparse article vectors with a user-level semantic expansion, improving MIND NDCG@10 by about 11% over STAR.
Personalization · 2026-08-03UxSID: Semantic-Aware User Interests Modeling for Ultra-Long SequenceDeep dive: UxSID: Semantic-Aware User Interests Modeling for Ultra-Long Sequence.
Personalization · 2026-11-03BrowseFlow: Sparse Event Flow Matching for Generative User SimulationShown the same feed, the same user might scroll to item 20 and click two things this time, and leave at item 8 the next; the log keeps only one of these. BrowseFlow argues that a user simulator should learn the distribution of such outcomes rather than copy that one log. It splits a browsing session into two things, how deep the user scrolls and at which positions they click or like, and generate…
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…