RecGroupMemory: Feedback-Aware Cohort Memory for E-Commerce Recommendation

Wendong Zhang, Weipeng Huang, Zhixin Ma, Bin Zhang, Dimin Wang
36 minread4.05scoreNo. 111-09 Generative issue
Contents · 9 sections1The gap the method fills2Overall flow3How feedback is matched to a record4What may change and what may not5Validation, repair and fallback6Online experiments7Limitations8How a recommendation or search team could try it9Verdict
Rel 5Nov 3Prac 4Evid 4Time 4

This paper is worth a recommender engineer's time for its engineering pattern, not for the +1.78% headline. Suppose an LLM has written a "group profile and advice" record that is served online. The next day, orders and browsing outcomes arrive. Can the LLM revise its advice without also quietly rewriting the cohort definition and metrics that the record is supposed to document? RecGroupMemory's answer is to have the prompt tell the LLM to revise only decision content: Profile, Advise and the structured advice that goes with them. Code then checks that seven fields, including the cohort definition, observation window and aggregated metrics, stay identical to the original record, and a failed check falls back to that record. In the online tests, adding memory without feedback revision raised transaction users by 1.58% over a no-memory baseline, and adding feedback revision raised them by 1.78%. Each figure is measured against its own no-memory baseline, so they cannot be subtracted, and the paper has not shown how much the feedback revision itself contributes.

The gap the method fills

Start with what already exists. In this system an online orchestration LLM plans the recommendation workflow for each request. Retrieval, ranking and re-ranking are still done by existing components. In the baseline B, the orchestration LLM sees only the request context.

RecGroupMemory first gives it cohort memory (the paper's term). For a group of users (delimited by the segment definition in the record) and a time window, behavior, experiment results and context observations are summarized into one record. Two pieces of text in the record are read by the orchestration LLM. Profile summarizes the cohort's current funnel performance. Advise gives adjustments for particular objectives, with reasons and rollback conditions. The record itself is a model-generated JSON response. Each update changes this record, not any model parameters.

The gap appears after the record is written. Transaction and browsing outcomes arrive a day later and may show that yesterday's advice was wrong. The simplest fix is to have the LLM regenerate the whole record. The paper worries about exactly this step: regeneration can also change the cohort definition, the observation window or the metric values. Then the advice may look better only because the described population or numbers changed. The other half of RecGroupMemory targets this update step: once feedback arrives, the LLM revises the interpretation of the observations, and code checks that the observations themselves were not changed.

A running illustration for the rest of the article (all numbers illustrative): each cohort memory record has an ID (Memory ID) that feedback uses to find it. One record, m-17, has an advice strength of +1 for item-page views (IPV) and 0 for CVR. The next day's feedback shows IPV 3% above the reference traffic (how the reference is chosen comes in the next section), but the transaction-user metric is missing.

Overall flow

RecGroupMemory Figure 1: multi-source signals pass through the Evidence Extractor into Shallow Memory; the Memory Builder creates records and the Memory Refiner revises them with previous-day feedback; after publication only Profile and Advise are injected into the online orchestration LLM, and online logs flow back to Evidence Logs.
RecGroupMemory Figure 1: multi-source signals pass through the Evidence Extractor into Shallow Memory; the Memory Builder creates records and the Memory Refiner revises them with previous-day feedback; after publication only Profile and Advise are injected into the online orchestration LLM, and online logs flow back to Evidence Logs.

Figure 1 breaks into four steps. The Evidence Extractor organizes behavior, experiment results, profile and context signals into event, experiment and context memory inside Shallow Memory, the storage layer for these records. The Memory Builder combines the observations in these three memory types into one cohort memory record, which holds fields such as the cohort, observation window and aggregated metrics, plus the initial Profile and Advise (the figure also shows a knowledge component supporting both the Builder and the Refiner). The Memory Refiner revises Profile and Advise when feedback is available. After publication, Shallow Memory keeps Profile, Advise and Evidence, but only Profile and Advise are injected into the orchestration LLM. Online logs return to Evidence Logs, are matched to each memory by Memory ID, and become the next round's feedback. Online planning reads the records published to Shallow Memory, but the paper does not describe how a request is matched to a cohort record, or which version of it the orchestration LLM receives.

The division of labor is that the Builder writes the record from the evidence and the Refiner only rewrites it once feedback arrives. For how the Builder turns evidence into Profile and Advise, the paper gives only the inputs and outputs.

How feedback is matched to a record

Not every record goes to the Refiner. Revision is run by a feedback node: aligning inputs, checking eligibility, validation and the database write are program steps in this node, and the Memory Refiner is only the step that calls the LLM to produce a new record. Records are stored in date partitions. C_T is the current record read from date partition T, the object to be revised, and F_{T-1} is the feedback from the previous day (T−1). The paper does not say whether C_T is rebuilt by the Builder that day or carries over the previous revision; it only says C_T is the starting point. The feedback node first left-joins the two by Memory ID. The paper itself cautions that linking by Memory ID does not establish that the feedback was caused by this advice. Revision starts only if exactly one feedback row matches, three flags set by upstream steps all pass (the source lists only their names: profile validity, baseline matching and feedback eligibility; the judgments are made upstream and the feedback node reads the results), and the source record itself is valid. Otherwise the record does not enter the Refiner and keeps C_T. Eligibility does not mean every objective has enough evidence; the prompt has to account for missing metrics when choosing an adjustment direction.

Hoot saysIllustration: three records through the eligibility checkClose ✕
Eligible for revision is not the same as every objective being strengthenable: eligibility is per record, evidence is per objective.
Record A has no feedback row after the Memory ID left join and keeps C_TC_T: record ALeft-join feedbackno matching rowSkips the Refinerkeeps C_T
Record A has no feedback row after the Memory ID left join and keeps C_TC_T: record ALeft-join feedbackno matching rowSkips the Refinerkeeps C_T
1/4 · Record A: no feedback row matchesAfter the left join, A has no feedback row, so the exactly-one-row condition fails; A never reaches the Refiner and keeps C_T.
Record B matches two feedback rows, not exactly one, and keeps C_TC_T: record BLeft-join feedbacktwo rows matchSkips the Refinerkeeps C_T
Record B matches two feedback rows, not exactly one, and keeps C_TC_T: record BLeft-join feedbacktwo rows matchSkips the Refinerkeeps C_T
2/4 · Record B: two rows matchB matches two feedback rows, which is not exactly one either, so it also keeps C_T. An extra row disqualifies it just as a missing one does.
Record C has exactly one feedback row, all three upstream flags and the source record are valid, and it enters the RefinerC_T: record CExactly one rowthree flags passEligiblegoes to the Refiner
Record C has exactly one feedback row, all three upstream flags and the source record are valid, and it enters the RefinerC_T: record CExactly one rowthree flags passEligiblegoes to the Refiner
3/4 · Record C: one row, all three flags passC matches exactly one feedback row; the upstream flags for profile validity, baseline matching and feedback eligibility all pass and the source record is valid, so only now does it go to the Refiner.
Record C is revised, but transaction users are missing, so the prompt does not allow strengthening CVR adviceFeedbackbrowsing present, transactions missingRefiner revises as usualCVR advicecannot be strengthened
Record C is revised, but transaction users are missing, so the prompt does not allow strengthening CVR adviceFeedbackbrowsing present, transactions missingRefiner revises as usualCVR advicecannot be strengthened
4/4 · Record C is eligible, but a transaction metric is missingC's feedback has a browsing value but transaction users are missing. The record is still revised, but the prompt has to handle the gap: under the conversion-prioritized prompt, CVR advice cannot be strengthened.
1/4

Records A, B, C and their feedback are illustrative, not samples from the paper.

Sourcepaper §2 feedback revision: exactly one matching row, three upstream flags, valid source record; eligibility does not guarantee evidence for every objective; the conversion-prioritized prompt's CVR condition

The feedback is a relative change. For Memory ID i and metric q, δ = q_i / q_base(i) − 1, where q_base(i) is the same metric over all traffic with the same profile in the platform's first-guess recommendation feed. The numerator q_i is the metric value for this memory; the paper does not specify which traffic it covers or how online logs are aggregated by Memory ID. The prompt uses this signed number directly. Zero means no observed change. A missing value means no evidence and must not be treated as neutral or positive feedback.

For an eligible record, an optional historical record H_T can also be retrieved from Shallow Memory for the same cohort (not necessarily from the previous day's partition), to compare with earlier advice. It is background only: historical numbers cannot replace current observations or override new feedback, and revision proceeds without it if retrieval fails. C_T is always the baseline for revision.

What may change and what may not

RecGroupMemory Figure 2: revision runs in three stages. Inputs are aligned and checked for eligibility; the Refiner produces a candidate U_T; a valid candidate U_T is accepted and an invalid one gets one repair; candidates that stay invalid, handled model failures and ineligible records keep C_T; both accepted and kept records are written to database partition T.
RecGroupMemory Figure 2: revision runs in three stages. Inputs are aligned and checked for eligibility; the Refiner produces a candidate U_T; a valid candidate U_T is accepted and an invalid one gets one repair; candidates that stay invalid, handled model failures and ineligible records keep C_T; both accepted and kept records are written to database partition T.

The Refiner returns a complete JSON response, not a patch. The third stage of Figure 2 shows how it is constrained. Seven values are fixed: schema version, cohort, recommender version, memory scope, segment definition, observation window and aggregated metrics. Profile and Advise must be nonempty strings. The prompt permits changes to their text, the structured advice, profile confidence, uncertainty notes and change notes. The validator does not keep a complete whitelist of editable fields: it enforces only the seven fixed values and the required structure, so "revise only the interpretation" relies partly on the prompt, and what code actually guards is those seven values.

Advice strengths are integers in {−2, −1, 0, 1, 2} for four objectives: CTR, IPV, CVR and GMV. The intended update is

Here a_q is the current strength in C_T (zero for legacy records without structured advice), Δ_q is this round's adjustment: up one level, down one level, or none. Under this intended update, each objective moves at most one level per round. The stored value is a', not Δ. A strength already at ±2 that is pushed further outward is stored unchanged; the rationale text then helps show that an adjustment was requested. The "revert to previous" action likewise keeps the strength in C_T; it does not restore the value from the historical record H_T. These strengths express how strongly to advise, not calibrated probabilities or ranking-score multipliers; the revision node itself produces no ranking weights. The orchestration LLM reads only the Profile and Advise text; the paper does not describe which workflow decisions it changes after reading them, or whether and how the strengths in the structured advice affect downstream retrieval and ranking.

Back to the illustration: m-17 has IPV strength +1 and the IPV feedback is +3% against the reference traffic. If the Refiner decides to add one level, IPV becomes +2. CVR is different. The paper's conversion-prioritized prompt allows strengthening CVR advice only when the relative changes in both transaction users and order count are present and positive; good browsing numbers cannot substitute. Here the transaction-user value is missing, so the CVR advice stays conservative. The record is still revisable: Profile reports the browsing response, Advise keeps a conservative conversion position, and the missing outcome goes into the uncertainty notes. The prompt also separates judging the previous strategy from choosing the next adjustment: keeping the current advice when evidence is insufficient does not mean the strategy is judged a success.

Objective priorities are configurable: the conversion-prioritized prompt above emphasizes CVR and suppresses GMV adjustment, while a browsing-prioritized prompt emphasizes IPV with transaction safeguards. The authors stress that Equation (1) is the intended update, not a rule enforced by code in every configuration. For strengths, the validator checks only structure, integer type and the −2 to 2 range. Only the browsing-prioritized configuration also checks "direction-to-strength consistency". The source gives this check only the name "direction-to-strength consistency"; from context, "direction" is presumably whether Advise wants to strengthen, weaken or keep an objective, while strength is the stored integer. Constraints such as "be conservative" depend mainly on the prompt.

Hoot saysIllustration: a candidate that jumps from 0 to +2Close ✕
"At most one level per round" is the prompt's intent, not a validator rule: an integer that jumps two levels still passes the strength check.

What the prompt and Eq. (1) intend

Per-round change
at most one level up or down per objective
0 → +2
violates the intent: two levels
Weak evidence
stay conservative, enforced by the prompt

What the validator checks

Fixed values
all seven must equal C_T: pass
Strength
integer within −2 to 2: +2 passes
0 → +2
not rejected for jumping two levels

Direction-to-strength consistency is checked only in the browsing-prioritized configuration.

Suppose that under the conversion-prioritized configuration the Refiner moves one objective's strength from 0 straight to +2, with all fixed values and required structure intact. Left: what Equation (1) intends; right: what the validator actually checks. Numbers are illustrative.

Sourcepaper Eq. (1); §2 validation checks advice structure, integer type and range, and only the browsing configuration also checks direction-to-strength consistency

Validation, repair and fallback

After the Refiner produces a candidate, the feedback node validates it. If validation fails, the model gets one repair attempt: it is asked to restore valid JSON and the fixed values while keeping its conclusions and advice text. U_T denotes the candidate after at most one repair. The final output O_T follows this rule:

e_T=1 means the record is eligible and its source is valid. 𝒱 checks that the seven fixed values equal those in C_T and that required fields are present. If the repaired candidate still fails, C_T is kept; invalid source records skip generation, and a handled model failure also keeps C_T. For these record-level cases every record gets an output and coverage does not drop; a failure of the partition write itself is not covered (see Limitations). An accepted update replaces the response and model metadata, while the source evidence columns are kept.

This guards "the observations were not changed", not "the advice was changed correctly". Validation compares fixed values and structure. It cannot tell whether the repaired text is semantically equivalent, and passing does not mean the advice changed or outcomes improved. Fallback likewise shows only that no record was lost, not that the original advice was optimal.

Before processing, source IDs must be nonempty and unique, and after writing the job checks that input and output have the same record count and ID set. Writing to the database partition and publishing to Shallow Memory are separate stages, and Figure 2 ends at the database write.

Online experiments

The users are low-purchasing-power Taobao users selected by the business team. Baseline B is an orchestration LLM planning from request context alone. E1 compares B with M, which adds Builder-generated Profile and Advise without feedback revision. E2 compares its own no-memory baseline with F, which uses the same interface and enables the Refiner. Each experiment reports changes against its own no-memory control, so the two columns do not share a baseline.

MetricM/B (E1)F/B (E2)
Transaction UV+1.58%+1.78%
Orders+0.80%+2.05%
PCVR+1.60%+1.95%
Exposure UV+0.14%+0.02%
Click UV−0.42%+0.04%
IPV UV−0.50%+0.17%

PCVR (Table 1's own metric label, listed alongside transaction users and orders) is kept as written; this article does not guess whether it is a predicted or an observed conversion rate, and transaction users and orders are unaffected by that ambiguity. Transaction users, orders and PCVR are positive in both runs, while exposure users move only +0.14% and +0.02%, so the positive changes are concentrated at the transaction end and exposure and browsing volumes stay close to baseline. In M, click and IPV users slip slightly (−0.42%, −0.50%) while transactions rise; in F, those two edge up (+0.04%, +0.17%). These are observations from the table, and the cause was not tested separately.

The most important caveat: F's order gain (+2.05%) is larger than M's (+0.80%), but they are measured against different controls, and the paper itself says the results do not directly estimate the incremental benefit of adding feedback to M.

Limitations

First, Table 1 gives only relative changes, with no sample sizes or confidence intervals, so values like +0.02% or +0.04% cannot be judged against noise.

Second, the experiments compare complete workflow configurations, not individual safeguards. The authors say the deployment artifacts corresponding to the experiments are unavailable, so gains cannot be attributed to the seven fixed values or the single repair. Showing that would take feedback ablations under matched conditions, traces linking the memory served online to its revisions, and runtime measurements.

Third, historical retrieval is not tied to a date or content hash, so Memory ID alone cannot identify the version used online. Fallback works per record, and a failed partition write can still leave partial output. Natural-language memory over small cohorts also needs access controls, retention limits and aggregation limits.

How a recommendation or search team could try it

  1. As the paper's discussion recommends, log record eligibility, validation acceptance, whether the advice actually changed, and downstream metric change as four separate events. In my view this is the cheapest step and the precondition for knowing whether updates work.
  2. For any LLM-generated summary record, list the fields that must not change (cohort definition, time window, metric values), check them for equality after each rewrite, and fall back on failure. This needs no live traffic.
  3. Make feedback revision a separate switch: one arm uses memory without feedback, the other adds feedback revision, both against the same no-memory baseline. That yields the increment the paper did not measure.
  4. Mark missing feedback explicitly, and state which advice may be strengthened only when which metrics are present and positive.

Verdict

RecGroupMemory splits "the LLM revises advice" into two layers, immutable evidence and revisable interpretation, with validation and fallback as the safety net. That pattern is directly reusable. The experiments show a small positive change at the transaction end for the whole memory pipeline (without confidence intervals), and the separate contribution of feedback revision is not measured.

DimensionAssessment
Problem importanceMedium-high. LLM-generated memory must be updated from feedback without contaminating the recorded observations
NoveltyMedium. The contribution is the fixed-value validation and fallback, not a new model
EvidenceMedium-low. Online results exist, but no confidence intervals, and the feedback increment cannot be separated
Engineering costMedium (my own estimate, not a figure from the paper). Besides date partitions, Memory ID alignment and validation logic, both the Builder and the Refiner call an LLM
Worth tryingStart with fixed-value validation plus fallback, then test feedback revision on its own

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Related papersMore from Paper Daily

  1. Generative · 2026-11-17RecMem: Recurrent Memory Compression for Long-Sequence Generative RecommendationOnce a generative recommender feeds a user's long history straight into the model, compute and memory grow with history length, and long interaction sequences quickly become slow and expensive. The authors compress history segment by segment into a fixed number of memory slots, and the decoder uses a fixed-length context that no longer expands as the history grows. On a long-sequence benchmark fr…
  2. Generative · 2026-11-02UGR: Unified Generative Ranking for Heterogeneous E-commerce RecommendationTaobao's generative ranking framework UGR has one generative model directly produce the screen the user finally sees. The roughly one thousand organic, ad and other-business candidates of a request go to the model together, and it generates the final list item by item, instead of each business ranking its own items and a blending stage stitching them together. That stitching step is the old probl…
  3. Generative · 2026-10-29KL-MOPD: Discrepancy-Aware Multi-Objective On-Policy Distillation for Generative RecommendationWhen click and value teachers are distilled together into a generative recommender, equal per-token weights are unfair. On TencentGR the student's KL to the value teacher at the first token is 3.13 times its KL to the click teacher, and the harder teacher switches to the click teacher on the last two tokens. KL-MOPD from Zhejiang University and Alibaba weights each teacher per token by its pre-tr…
  4. Personalization · 2026-10-30Multi-Branch Cooperation Networks for Enhanced Click-Through Rate Prediction in Large-Scale E-Commerce SearchRanking teams should read this one, because it points out an often overlooked waste in multi-branch models: the branches are good at different samples, yet never learn from each other in training. The Pailitao team at Taobao (search by photo) makes three branches correct each other during training. On a given sample, if one branch gets the click right and another gets it wrong, the right branch's…

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…