EmbeddingGemma 2: Unified and Efficient Multimodal Representations

To search text, images, video and audio with one embedding model, the usual route is to transcribe audio and caption images first, then hand everything to a text model. That costs extra compute and throws away tone of voice and visual detail. Google's EmbeddingGemma 2 is a 744M-parameter model that maps these inputs directly into one vector space, and each modality encoder can be loaded only when needed. Finding text passages from a spoken question shows the difference best: the same model transcribing speech to text first reaches an MRR@10 of 57.59, while reading the audio directly reaches 69.54. A small multimodal model need not give up text and code; its code retrieval score is almost ten points above its predecessor's, from 68.76 to 78.68.

Teams building on-device or low-cost retrieval should read this paper, because it gives a reproducible training recipe for a small multimodal embedding model and backs each key choice with an ablation: distill first, finish with contrastive learning, train the vision encoder too, and average the final checkpoint with an early one. Quantized to int4, the full multimodal model peaks at 567MB of memory on a Pixel 11 Pro, and 191MB for text only. The caveats: English text scores dip slightly below the predecessor, int4 quantization hurts images and speech much more than text, and video gets worse when the vision token budget is raised.

What problem is it solving?

Multimodal retrieval today follows roughly three routes.

EmbeddingGemma 2 aims for a fourth option: stay under 1B parameters yet still embed any modality, and any mix of them, into one space.

A small example (illustrative). A phone gallery assistant must answer the spoken request "that video of me playing guitar". The library holds a guitar video, a guitar photo and a note titled "guitar tuner".

Pipeline overview

Look at the row of coloured squares in the middle: tokens from different modalities are interleaved in their original order, so a single input can be "some text, then an image, then a voice clip".

Images are pooled to a token budget and wrapped in start and end markers, video is processed frame by frame up to 32 frames, and audio is cut into 40 ms chunks. Tasks are signalled by prefixes such as task: search result | query: {text}.

Design 1: drop the per-layer lookup table

Why it is needed: Gemma 4 injects Per-Layer Embeddings (PLE) into every layer, the sum of a projection of the main token embedding and a vocabulary lookup table. Image and audio soft tokens have no vocabulary ids, so they cannot use that table, and the table costs memory.

How it works: EmbeddingGemma 2 keeps only the projection, so every position gets per-layer input computed from its own embedding, whatever its modality.

A tiny example (illustrative): an input of two words plus 280 image tokens. Before, the two words got the lookup and the 280 image tokens only the projection; now all 282 positions get the same kind of per-layer input. Hoot says▾

Hoot saysOne text-plus-image input: where each layer's extra input comes from, before and afterClose ✕

Before, text took projection + lookup while image tokens took projection only; without the table, all 282 positions get it from their own embedding.

Gemma 4: projection + lookup

2 text positions

projection + lookup

280 image positions

projection only, no entry

Overall

mixed sources, extra memory

A whole per-layer table also sits in memory.

EmbeddingGemma 2: projection only

2 text positions

projection

280 image positions

projection

Overall

all 282 positions alike

Audio tokens work the same: any position with an embedding gets per-layer input.

Illustrative: 2 text positions plus 280 image positions; the lookup needs a vocabulary id, which image tokens lack. Source: PLE description and illustrative example in Design 1

This shows the lookup table does not help multimodal retrieval; removing it saves memory and slightly improves video and visual documents.

Design 2: distill first, finish with contrastive learning

Why it is needed: a small model trained contrastively from scratch struggles to learn the structure a large model has, but pure distillation only approaches the teacher's space, which was not built for a tiny student, 768 dimensions or Matryoshka Representation Learning (MRL) truncation.

How it works: four stages, about 2.9T tokens in total.

The contrastive loss is ${\mathcal{L}}_C=-\frac{1}{B}{\sum }_i\log \frac{Z_i^P}{Z_i^P+Z_i^{HN}+Z_i^{IB}+Z_i^{STN}}$, where $Z^{STN}$ covers same-tower negatives: a query's similarity to the other queries in the batch, plus its positive document's similarity to the other positive documents. They stop queries and documents from collapsing into two disjoint subspaces.

This shows distillation plateaus. Switching to the full finetuning recipe (contrastive loss plus the spread-out regularizer) adds over three points; the comparison does not test the two parts separately. The merge step uses a single run instead of several finetuning jobs: tasks that overfit late recover from the early checkpoint, while tasks that gained keep gaining. In a reproduction, the early checkpoint can be picked on a validation set, before overfitting tasks start to drop.

Design 3: train the vision encoder, tune the token budget per use

Keeping the vision encoder trainable gives 52.27 on MMEB-v2 against 48.18 when frozen. This shows freezing costs about four points, which the saved training compute does not justify.

This shows bigger budgets help documents such as PDFs and slides most, over four points from 280 to 560, while video frames drop from 48.51 to 46.52 past 280, so budgets should be set per content type.

Main results

Find the star, then the shading: code and image show the largest shaded areas, so that is where it leads most below 1B; audio has a small area, meaning it roughly ties the best model of its size.

This shows its strengths sit in code, images and documents: on code it beats every sub-4B model and comes within 2 points of a leader 28 times larger, while text roughly matches the predecessor and English slips.

This shows skipping ASR is not just cheaper but worth about 12 points, and even at 256 dimensions (66.76) it beats its own ASR route (57.59) by over 9 points, though not Gemini Embedding 2 via ASR (70.40). After int4 per-channel quantization, MTEB English stays at 68.49 while MIEB(lite) drops from 60.17 to 56.71 and speech retrieval from 69.54 to 63.92, though the paper used different evaluation harnesses for these.

Limitations

First, English text pays for multimodality: MTEB English falls from 69.67 to 68.46. Teams serving mostly English text should measure that 1.2-point drop on their own data first.

Second, video is the weakest link: in the vision-token budget experiment, video Hit@1 stays between 46 and 49 and loses about 2 points when the budget doubles past 280.

Third, the audio encoder was frozen. One sign: a linear probe for speaker gender on IEMOCAP reaches 66.40%, so the information is in the vectors; but clustering VoxPopuli recordings directly by gender is among the weakest audio results in appendix Table 17. The information is there but not arranged in a shape clustering can use.

How a search or recommendation team could try it

1. Load only the 271M text part, swap it for the current text embedder, and compare offline on English, Chinese and code queries, watching English closely.
2. For voice search, route spoken queries two ways, through current ASR plus text and straight into the audio encoder, and compare MRR@10 and latency.
3. For image or document libraries, test 140, 280 and 560 vision tokens separately for documents and for video.
4. Truncate to 256 dimensions with MRL and weigh index size against recall before choosing the serving dimension.
5. For on-device use, evaluate the int4 per-channel build per modality, with separate bars for images and speech.
6. To finetune your own, copy the order: distill from a large model, finish with contrastive learning with same-tower negatives, then average the final and an earlier checkpoint.

Verdict

The most complete sub-1B multimodal embedding recipe so far, leading its size class in code, image and speech retrieval, at the price of slightly weaker English text and weak video.