Auditing Representation-Space Fraud Detection: A Bayesian IRL Framework for Calibrated Uncertainty and Its Limits

Shilling detection almost always reports a single AUROC, and that number cannot say when the detector can be trusted. This paper builds an auditing framework called VEIL to check exactly that: it learns a score on frozen user embeddings to separate real accounts from injected fake ones, and it also reports how much confidence the detector has in its own calls. With it, fakes from classic injection to GAN-generated, style-varied accounts are all caught, at AUROC 0.936 to 1.000. A "copy a real user" attack changes everything. In this attack each fake account copies every movie a real user rated and then adds 1 to 3 target movies to promote. On MovieLens-1M, detection AUROC falls to 0.48–0.51, the same as a coin flip.

Yet these fake accounts are not hard to catch. Take the raw rating records, compute how much each account's rated items overlap with every other account's, and use the largest overlap as the score: AUROC reaches 0.92–0.97. The problem is the embedding. LightGCN sees only which items were rated and then smooths similar accounts together through graph propagation, so the raw-data signal of being a near-duplicate of one specific real user is lost. Recommender anti-fraud teams should read this, because embedding-based detection is common practice and this blind spot cannot be seen from AUROC alone. The core idea in one sentence: a signal missing from the representation cannot be recovered by any classifier or uncertainty estimate built on it.

What problem is it actually solving?

Shilling detection has to pick out the fake accounts that are injected into a recommender to rate a target item highly. The paper starts from one observation: almost every detection paper reports only an AUROC, and that number does not say when the detector can be trusted.

A small example shows why the copy attack is hard. The table is illustrative:

G's movies are picked at random and resemble no real user, so G is easy to catch. F differs from U by one movie, so the Jaccard of their rated movies is 5/6, about 0.83 (illustrative). If a detector can see only "which movies were rated", F looks much like an ordinary real user. Comparing the raw records directly, though, still shows that F and U overlap heavily. The paper calls this attack donor-mimicry and the copied real user the donor; below I call them "the copy attack" and "the donor".

The authors order a set of attacks from easy to hard. The easiest is classic random, average and bandwagon injection. Next come GAN-generated fake accounts (AUSH). Then AUSH with fake accounts made more human (below, "style variation"). Here each fake's number of ratings is drawn from real users' distribution, and filler items (non-target items rated to look human) are picked at random weighted by popularity and preference. Target-item ratings are jittered between 4 and 5, and each fake gets its own rating bias and noise. The hardest rung adds the copy attack on top of style variation: each fake copies one real user's rated items and their ratings, and splices in 1 to 3 target movies.

How VEIL is built: the overview first

The pipeline has four steps. First, an encoder computes an embedding for every account (64-dimensional for LightGCN), which is then frozen. There are two encoders. One is LightGCN, trained only with the BPR ranking loss on the poisoned user–item graph; it never sees a "who is fake" label, so it is the honest test. Every account, including those later held out for testing, is a node of this graph; after the encoder is trained, an embedding is taken for every account, and only then are the accounts split into training and test sets. The other is Sup-GNN, trained directly on the fake-account labels; its embedding already encodes the answer, so it is only a positive control, and good results there are expected.

The other three steps all work on these frozen embeddings. During training, the authors pair one real account with one fake account at a time and learn a linear score $r(x)={\theta }^{\top }\varphi (x)$, where $\varphi (x)$ is account x's embedding and $\theta$ is the scoring direction to learn. The goal is for the score gap within each pair to reliably tell which account is real and which is fake, with probability:
$$P(x^{+}\succ x^{-})=\sigma (\alpha \cdot {\theta }^{\top }(\varphi (x^{+})-\varphi (x^{-})))$$
$\sigma$ is the sigmoid, and $\alpha$ is a confidence factor that says how much to amplify the score gap. Intuitively, the larger the gap between two accounts, the more certain the model is about which is which. With $\alpha =1$ and a gap of 2, the probability is $\sigma (2)\approx 0.88$ (illustrative). The authors note that without sequential decisions this is essentially Bayesian logistic regression on embedding differences. At detection time there is no pairing, and each account is scored on its own: take a point estimate of $\theta$, and the score is the projection of the account's embedding onto that direction, ${\theta }^{\top }\varphi (x)$. A one-dimensional logistic curve with just a slope and an intercept, fitted with training-split labels, then turns the score into a "probability of being fake", which the ECE and reliability diagrams below use. Labels of the held-out test accounts are used only at evaluation.

VEIL has three components, and only the first two are covered below. The third, Component C, is a correctness check the authors run with the true labels; it cannot be deployed, and this piece does not go into it.

• Component A: $\alpha$ is fixed at 1 and the posterior distribution of $\theta$ is learned. The wider it is, the more ambiguous the data is about which direction to score in.
• Component B: on top of A, $\alpha$ is also learned, and each embedding dimension gets its own learned prior on how important it is, which pushes unimportant dimensions toward zero weight. The learned $\alpha$ is the audit signal: how much of the score gap the model is willing to trust.

Why does the copy attack blind the detector?

The authors trace the cause to two functions in the LightGCN code. The graph builder turns every user–item rating into one unweighted edge, whether the rating was 1 or 5. The BPR loss takes only item IDs and never reads a rating. So a LightGCN embedding reflects only which items an account rated, plus, through graph propagation, who else rated those items; rating values never get in. Hoot says▾

Hoot saysOne copy-attack fake account, in the raw data and in the LightGCN embeddingClose ✕

In the raw data the near-duplicate is obvious; LightGCN keeps only which items were rated, and propagation pulls the fake toward its donor, so the signal is gone from the embedding.

Raw rating records: item overlap

Real user (donor) rated

A B C D E

Copying fake rated

A B C D E + target item

Shared items ÷ all items of both

5 ÷ 6

Overlap (illustrative)

≈0.83

Why this catches it: real users are rarely this alike, so "highest overlap with anyone else" works as a score. In the paper the median overlap between a fake and its donor is 0.96.

LightGCN embedding: the signal is smoothed away

Rating values

not in the graph, not in BPR

Items the fake links to

almost the donor's set

After propagation

fake pulled toward donor

Mean distance to donor

1.98 (MF without propagation: 3.09)

LightGCN 1.98MF 3.09

The detector sees only the embedding: here the fake sits right next to a real user, and no scorer can pull them apart.

The left column is illustrative: a real user rated 5 items, and the copying fake rated the same 5 plus 1 target item. The distances on the right are the paper's measurement on the same injected data. Source: The graph builder and BPR description, the median overlap of 0.96, and the distances 1.98 and 3.09 come from the paper's code trace and distance measurement of the LightGCN blind spot; the left-column overlap is an illustrative hand calculation.

This explains the pattern on the attack ladder. The rating-value part of style variation is invisible to LightGCN, so detection only slips from 1.000 to about 0.94 rather than collapsing. The copy attack is the first one that also matches real users on which items were rated, so it alone drives detection to chance.

The authors then state this as a general result (Proposition 1). Suppose the embedding used for detection depends on only part of an account's information, which the paper calls the "structural signature". For LightGCN, the structural signature is the deduplicated item and co-occurrence structure of an account's ratings, with rating values discarded entirely. If fake and real accounts have exactly the same distribution of structural signatures, then any detector score built on the representation, however complex and however trained, gets AUROC exactly 0.5. An analogy: an appraiser who sees only black-and-white photos cannot tell apart two garments that differ only in colour, and a better appraiser does not help, because the colour was lost when the photo was taken. Here "taking a black-and-white photo" maps to LightGCN looking only at which items were rated and never reading the ratings, and "colour" maps to the discarded rating values. The copy attack also copies almost the whole item set of its donor, so on the information LightGCN can see, fake and real accounts look nearly the same. The copy attack satisfies this only approximately: the fake accounts share the 1 to 3 target movies, which leaves a small residual signal, so on the two MovieLens-1M copy-attack configurations AUROC lies between 0.48 and 0.54 rather than exactly 0.5 every time.

The authors also measure this directly. The attack generator did not record who each donor was, so the authors recovered donors by item overlap: the median overlap is 0.96, and 81 of 91 fake accounts have a confident donor match. On the same poisoned data, they then compare each fake account's embedding distance to its donor. LightGCN places fake accounts closer to their donors: mean Euclidean distance 1.98, versus 3.09 for the version without graph propagation (zero layers, equivalent to BPR matrix factorization, MF). This fits the explanation that propagation smooths fake accounts toward their donors, though the authors note that one configuration is not enough to prove causation.

The detector is confidently wrong

AUROC alone cannot show whether a detector still knows its limits when it fails. The authors run every configuration with 8 random seeds and report both AUROC and ECE.

The numbers are 8-seed means. Every configuration with AUROC of at least 0.94 has ECE of at most 0.09. Every configuration below 0.88 has ECE between 0.18 and 0.37. This means on hard attacks the detector is not just wrong, it is confidently wrong.

Component B's learned confidence $\alpha$ also does not follow "confident when easy, hesitant when hard". On the LightGCN attack ladder, the two easiest steps have the lowest $\alpha$ (0.386 and 0.329), the highest is the hardest step that is still separable (0.688), and the collapsed copy attack falls back to the middle (0.551). The authors explicitly label this inverted U as a post-hoc reading: $\alpha$ may track how much usable score-gap signal remains, not difficulty in the AUROC sense. For readers, this means the learned confidence cannot be used directly as a switch for whether a detection is trustworthy.

Does the conclusion change with dataset, encoder or detection method?

The authors rerun the same attack recipe on two more datasets, both without AUSH. MovieLens-1M is also rerun with this recipe so the rows line up.

The collapse varies a lot. MovieLens-1M falls almost to chance, Musical Instruments drops by about 0.18, and CiaoDVD barely drops. CiaoDVD's case is mostly because classic injection starts at only 0.877, leaving little room to fall. The authors tried catalog size and ratings per user as explanations, and neither matches.

Changing the encoder can flip the ranking too. On the MovieLens-1M copy attack, MF is more robust than LightGCN (0.632 vs. 0.481–0.494). On CiaoDVD it reverses, and LightGCN is more robust (0.846–0.848 vs. 0.764 under the copy attack). On Musical Instruments the authors first fixed the attack generator: the original always put target items last with evenly spaced timestamps, and the sequential model SASRec exploited that position flaw to reach above 0.9997, which is not real skill. After the fix, each target item takes the position and time of one donor item it replaces. The four encoders rank MF 0.940, SASRec 0.901, mean pooling without position information 0.874, and LightGCN 0.800. LightGCN's 0.800 here comes from the corrected generator; the 0.805 in the table above came from the original one. LightGCN is the only encoder here that does graph propagation and it comes last, which matches the earlier distance measurement on propagation smoothing fakes toward donors.

The same holds for the detection method. The authors compare against classic rating-statistics features (for example, how far an account's ratings deviate from each item's average) plus logistic regression. On MovieLens-1M both collapse: VEIL falls from 1.00 to 0.53, and the classic method falls from 0.982 to 0.584. On CiaoDVD the classic method falls more, from 0.840 to 0.617, while VEIL barely drops. This shows the three dimensions interact, and a ranking found along one may not hold along another.

Limitations as I see them

First, only MovieLens-1M has the full attack ladder; the other two datasets ran only the easiest and hardest steps, and the cause of their different collapse is still unknown. CiaoDVD starts at only 0.877 under classic injection, so its "resisting the copy attack" deserves a discount.

Second, there are too few comparisons. The only other detection method is the classic rating-statistics one, with no recent learned shilling detector. The 8 seeds behind $\alpha$ vary only inference randomness, and each configuration's encoder was trained once, so the inverted U is not yet solid.

How a recommender team could try this

1. Start by adding the cheapest raw overlap check to the existing detection flow: for each account, compute the largest Jaccard between its rated item set and any other account's. Scoring uses no labels; labels are used only when computing AUROC offline. In the paper it reaches AUROC 0.92–0.97 on the copy attack, and it can be computed with chunked sparse matrix products. It is largely ineffective on classic injections, so it complements rather than replaces the existing detector.
2. Build a red-team set of copy attacks: copy real users' full behaviour, then add 1 to 3 target items, keeping the original order and time gaps, or a sequential model will score falsely high on the position flaw.
3. Report AUROC and ECE separately for each attack type. If ECE rises clearly under one attack type, the detector is confidently wrong there, and the overall AUROC alone is not enough to ship.
4. Check what information the detector's representation drops, for example whether the encoder reads rating values and whether it propagates over a graph. If the dropped information is exactly what an attacker can match, add features that do not depend on that representation.

Conclusion

The key lesson: what a detector can see sets the most it can catch. A detection result that reports only AUROC, without auditing calibration or checking other datasets and encoders, may be one attack design away from a coin flip.