StaRSE: Compact Russian Sentence Embeddings with a Sign-Coded Static Encoder

Boris Malashenko, Ivan Jarsky, Egor Soldatov, Vladislav Luzhnov, Sviatoslav Stumpf, Vladislav Shmatkov, Viacheslav Shalamov, Andrey Stankevich, Valeria Efimova
34 minread4.10scoreNo. 411-12 AI Search issue
Contents · 5 sections1How a Russian sentence becomes a vector: look up signs, multiply by a length, average2Where the table and the positive pairs come from3Let the loss see the signs during training48 points above static models, 2.6 points behind the smallest Transformer5BM25 is stronger on RuMTEB retrieval and long truncations, StaRSE on short truncations
Rel 4Nov 3Prac 5Evid 4Time 5

StaRSE squeezes Russian sentence embeddings into an 11.3 MiB table: each token stores only 512 signs and one length, a lookup-and-average yields the sentence vector, and it scores 8 points above every static model it is compared with. On the RuMTEB retrieval mean and on long-document retrieval with long truncations, though, BM25 with the same tokenizer still beats it, and StaRSE is ahead only when long documents are truncated short.

Start with a scenario. A Russian knowledge base needs semantic search. A user types «Как заварить зелёный чай» (how to brew green tea), and the base contains the sentence «Зелёный чай заваривают водой 80 градусов» (green tea is brewed with 80-degree water). The system encodes the query and every stored sentence into vectors, then finds the nearest sentences by cosine similarity. These two Russian sentences are illustrative, and the rest of the article keeps using them.

The hard part is the encoding step. With a Transformer sentence encoder, the query needs one forward pass, and every sentence in the base needs its own forward pass too. When embeddings must be computed on CPU, refreshed often, served under a tight latency budget, or produced for a very large corpus, these repeated forward passes become the bottleneck.

The usual cheap alternative is a static model: every word or token has one fixed vector in a table, the sentence vector is the average of those vectors followed by normalization, and inference is just a lookup. The same example shows its gap. «как» (how) appears in almost every question, yet it enters the average with the same weight as «чай» (tea), so any two questions that start with «как» get pulled together. Averaging also throws away word order and context: «заварить» has the same vector in every sentence. The real question is therefore whether a static model can deliver useful sentence-vector quality at a much lower cost.

StaRSE answers in two halves. During training, it starts from the input embeddings of ruBERT-base and runs contrastive learning on Russian sentence pairs. At deployment, it keeps only a "sign table": 120 thousand tokens, each row holding 512 signs plus one length. On RuMTEB (the Russian version of MTEB, 23 tasks), its mean is 51.16, which is 7.98 points above the strongest static baseline and only 2.60 points below ruBERT-tiny-turbo, the smallest contextual encoder in the comparison. This means it occupies a new spot between static models and Transformers while still costing only a lookup: one sentence takes 0.08 ms on CPU on average, versus 1.43 ms for ruBERT-tiny-turbo.

How a Russian sentence becomes a vector: look up signs, multiply by a length, average

At deployment a sentence goes through only four steps: tokenize, look up, average with norm (vector length) weights, and normalize. One row of the sign table looks like this (illustrative):

token "чай"  → sign bits: + + − + − − + … (512 bits)   norm ρ = 2.0
token "как"  → sign bits: + − − − + + − … (512 bits)   norm ρ = 0.5

The sign bits record whether the token's direction is positive or negative in each dimension. The norm ρ is the length of the token's original vector, and it decides how loudly the token speaks in the average. At lookup time each dimension becomes +1 or −1, is divided by √512, and is multiplied by ρ. All tokens of the query are processed this way and averaged, and a final L2 normalization yields a sentence vector that is still real-valued.

Here is the same computation in 4 dimensions (illustrative, √4 = 2). «чай» has signs (+,+,−,+) and ρ = 2, so it contributes (1, 1, −1, 1). «как» has signs (+,−,−,−) and ρ = 0.5, so it contributes (0.25, −0.25, −0.25, −0.25). Their average is (0.625, 0.375, −0.625, 0.375), and the direction mostly follows «чай». This shows that the norm decides how much each token weighs in the average: in the illustration the function word «как» has a small ρ, so it gets pushed down.

Why not a simpler design? The first simpler option is to store signs only and drop the norms. In the example both tokens then count as 0.5, and the average is (0.5, 0, −0.5, 0). Because the two tokens weigh the same, the second and fourth dimensions cancel, and «как» pulls as hard as «чай».

On the real model, dropping the norms costs 8.42 RuMTEB points, which means it falls back to the level of ordinary static models.

The second, even cheaper option keeps pooling over this sign table as usual and then also signs the final sentence vector; some earlier retrieval systems compressed sentence vectors this way. In the example, (0.625, 0.375, −0.625, 0.375) becomes (+,+,−,+), and the difference between 0.625 and 0.375 is gone. The paper tested this on the final sign table: keeping the norms but signing the pooled sentence vector costs 3.84 points; dropping the norms and also signing the sentence vector costs 12.10 points. StaRSE therefore signs only the token directions and keeps the sentence vector real-valued.

The norm is what makes this table work: it is a single number per row, and removing it hurts far more than signing the sentence vector.

The size of the table can be worked out by hand: 120 thousand tokens times 512 bits is about 7.3 MiB, and adding one norm per token lands in the same range as the measured 11.3 MiB package (my estimate). The same table with one 32-bit float per dimension would take more than 200 MiB (my estimate).

Where the table and the positive pairs come from

The sign table is produced in four stages, and the first three exist only during training:

StepInputOperationOutputTraining only?
1 InitializationruBERT-base tokenizer (120,138 tokens) and its input embedding matrixToken-level PCA to 512 dimensions, drop the few most dominant directions, then downweight by frequencyReal-valued initial tableYes
2 Building positivesCultura Russian web documents, mMARCO, ru-HNP, back-translated pairsFilter adjacent sentences by lemma overlap, then sample them with the other three pair sources in a source-balanced wayTraining pairsYes
3 Continuous trainingInitial table + pairsBidirectional InfoNCE, 20,000 steps followed by another 10,000Real-valued table E⁽⁰⁾Yes (later used as the frozen teacher)
4 Sign-projected post-trainingE⁽⁰⁾ + Cultura pairsCompute the loss through the sign table in the forward pass, plus a term that pulls toward the teacherSign table: 512 bits + 1 norm per rowNo, it is the only thing deployed

Both operations in step 1 target averaging. Dropping the top principal directions (the paper calls this ABTT, keeping only the components after the dominant ones) matters because nearly every token shares these directions, and they would dominate cosine similarity. The frequency downweighting uses SIF (a smoothed weighting where more common tokens get smaller weights), so that frequent words such as «как» weigh less in the average.

Step 2 answers a practical problem: Russian has few large, reliably labeled sentence-pair collections. StaRSE's main positives are two consecutive sentences from the same document. Adjacent sentences are not necessarily paraphrases, so the authors treat them only as weak positives and use a cheap lexical filter to remove sentence-splitting errors and unrelated jumps.

The filter is the Dice overlap: each sentence is reduced to its lemmas with stopwords removed, overlap = 2 × shared lemmas ÷ total lemmas of both sentences, and a pair is kept only at 0.20 or above. Illustration: «Зелёный чай заваривают водой 80 градусов» and the next sentence «Слишком горячая вода делает чай горьким» (water that is too hot makes tea bitter) each have 5 lemmas and share two, «чай» and «вода». The overlap is 2×2÷10 = 0.4, so the pair is kept. Paired with a sentence from another document that shares no lemma, the overlap is 0 and the pair is dropped.

The filter does select more similar pairs. Using USER2-small as an external encoder that plays no part in training, the authors sampled 30 thousand pairs of each kind and computed the median cosine: 0.573 for random cross-document pairs, 0.726 for unfiltered adjacent sentences, and 0.820 after filtering. This shows that adjacent sentences are much more similar than random pairs, and the filter pushes them a step further.

Why not simply distill an existing strong encoder? The paper tried this in a set of single runs with a smaller budget: static students distilled from two Russian sentence encoders (BERTA and USER2-small) reach only 42.0 and 44.2 RuMTEB, contrastive learning on adjacent sentences reaches 48.2, and adding the lemma filter plus the other sources reaches 49.9. The two groups use different budgets and a single run each, so they only show that the contrastive route works better in this setting.

Let the loss see the signs during training

The laziest option is to train the real-valued table E⁽⁰⁾ and then take signs. The problem is that the loss has never seen the signed table, so nothing compensates for the information lost when signing.

StaRSE's final stage moves the signing into the forward pass. The optimizer still updates a real-valued table E, but every forward pass first splits each row into a length ρ and a direction u, then uses ρ × sign(u)/√512 in place of the original row to compute sentence vectors and the loss. sign itself has no gradient, so the authors use a straight-through estimator: backpropagation treats it as the identity, gradients flow back to E as usual, and both direction and length get updated.

The loss in this stage has two terms. One is the same bidirectional InfoNCE as before: a batch holds k sentence pairs, each sentence must score its own partner above the other k−1 sentences, and this is computed in both directions. The other term pulls the sign table toward the frozen E⁽⁰⁾: within the same batch, the teacher produces a softmax similarity distribution for every row and every column, and the student's distributions are matched to it with KL divergence. Intuitively, the teacher tells the student which sentences in this batch should be closer to which.

The paper compared several routes to a sign table in the same 512-dimensional architecture:

  • Sign directly after contrastive training (no post-training): 49.33
  • Continue ordinary contrastive training, then sign: 50.09 (+0.76)
  • Sign-projected post-training (final recipe): 51.16 (+1.83)

This shows that extra training alone explains only 0.76 points, and the remaining 1.07 points appear on the route where the loss sees the signs; that route also switches to Cultura pairs only and adds the teacher term, so these factors cannot be separated. The final recipe's hyperparameters were chosen by an Optuna sweep on the full RuMTEB mean, and only one seed was run, so these 1.83 points are optimistic.

8 points above static models, 2.6 points behind the smallest Transformer

Here are the main results side by side (RuMTEB columns are task-family means; size and single-sentence CPU latency come from the paper's Table 4):

ModelTypeRuMTEB meanClassificationClusteringRetrievalSTSSize MiBCPU ms/sentence
GigaEmbeddingsContextual74.1676.6667.4881.4175.756580 (est.)773.61
ruBERT-tiny-turboContextual53.7655.0849.6248.8271.7355.7 (est.)1.43
StaRSEStatic51.1656.8151.8039.0962.1711.30.08
Potion multilingualStatic43.1850.1241.3922.0359.005070.07
fastText wiki+lentaStatic42.8551.8742.5218.7855.5318530.05

Against static models, StaRSE's mean is 7.98 points higher, its package is more than an order of magnitude smaller, and its single-sentence latency is comparable. It leads in every task family except pair classification (TERRa, an entailment task where the static models are essentially tied). This shows that, still within lookup-and-average, a different training recipe can push the score up a long way while making the table much smaller.

Against ruBERT-tiny-turbo, the mean gap is only 2.60 points, and StaRSE is slightly higher on classification and clustering; the gap is concentrated in retrieval (9.73 lower), reranking, and STS. On CPU it takes 0.08 ms versus 1.43 ms per sentence, roughly 18 times faster. Back to the knowledge base: encoding an illustrative 10-million-sentence corpus once on a single core would take about 13 minutes versus 4 hours at these two latencies.

Keep two measurement conventions in mind. Contextual model sizes are estimates of parameter count times 2 bytes, while static model sizes are measured packages. Latency covers encoding one sentence, including tokenization, pooling, and normalization, but not indexing or search. The 51.16 itself comes from hyperparameters selected on RuMTEB with a single seed, so it is not a held-out estimate. The distance to large models also remains wide: GigaEmbeddings is 23 points above StaRSE.

RuMTEB mean versus single-sentence encoding latency (log-scale horizontal axis): static models on the left are measured on CPU, contextual models on the right on GPU; marker area shows encoder size, and the star is StaRSE
RuMTEB mean versus single-sentence encoding latency (log-scale horizontal axis): static models on the left are measured on CPU, contextual models on the right on GPU; marker area shows encoder size, and the star is StaRSE

The two sides of the plot use different devices, so only the rough positions are meaningful: StaRSE sits clearly above the other static models and slightly below ruBERT-tiny-turbo.

BM25 is stronger on RuMTEB retrieval and long truncations, StaRSE on short truncations

The abstract's claim of being stronger than static baselines does not mean being stronger than lexical retrieval. In the same table the paper adds several lexical rankers that use exactly StaRSE's ruBERT WordPiece tokenizer, with no stemming and fixed parameters.

The paper's Table 1: RuMTEB scores over 23 tasks grouped by task family, in three blocks for contextual encoders, static models, and lexical rankers; lexical rankers have only the reranking and retrieval columns
The paper's Table 1: RuMTEB scores over 23 tasks grouped by task family, in three blocks for contextual encoders, static models, and lexical rankers; lexical rankers have only the reranking and retrieval columns

BM25 scores 46.45 on retrieval and StaRSE 39.09, which is 7.36 points lower; on reranking it is 42.76 versus 41.88, again 0.88 lower. This shows that on RuMTEB retrieval tasks, StaRSE on its own falls short of a BM25 with fixed parameters.

The long-document results split into two parts. MLDR-RU is a Russian long-document retrieval set, and the authors use 10 thousand documents and 200 queries. A static encoder runs no self-attention, so a long document can be streamed through lookups into the same running average, with no context-window limit. StaRSE's nDCG@10 rises from 32.11 when each document keeps only 128 tokens to 52.01 when it keeps 4,096 tokens, which shows that the extra document text is actually used. At truncations from 128 to 2,048 tokens StaRSE stays above BM25, for example 32.11 versus 28.34 at 128 tokens; BM25 only jumps to 68.12 at 4,096 tokens.

At 8,192 tokens StaRSE scores 51.66, almost the same as the small contextual encoder USER2-small, although the latter's number is copied from its model card. In the same setting BM25 scores 69.97, which means that once truncation reaches 4,096 tokens or more, BM25 overtakes StaRSE by a wide margin. Fusing the two reaches 73.07, but the fusion weight was chosen on the same test set, so this number is descriptive only.

Mean pooling also has a boundary set by its structure. «Собака укусила человека» (the dog bit the man) and «Человека укусила собака» (the man was bitten by the dog) consist of the same tokens, so lookup-and-average gives exactly the same vector. StaRSE gives up word order and context by design, and the authors use this to explain why it has no advantage on the entailment task TERRa.

On the RuMTEB retrieval mean and on long MLDR-RU truncations (4,096 tokens and above), BM25 with fixed parameters scores higher than StaRSE; at truncations from 128 to 2,048 tokens, StaRSE scores higher.

The authors view the two as complementary, and leave for future work the question of which queries and domains they complement each other on.

Back to the opening judgment: an 11.3 MiB table that stores only signs and lengths does lift Russian static sentence vectors by 8 points and brings them within 2.6 points of the smallest Transformer. But its main score comes from hyperparameters selected on RuMTEB, and on RuMTEB retrieval and long-truncation MLDR-RU it is still below BM25 with the same tokenizer and fixed parameters. Moving to another language would require revalidating the tokenizer, filtering threshold, and data mix.

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…