AutoR&D-Engineer refines Embedding Gemma 2 into two compact models for stronger document retrieval: a research version and a production-supported version. We evaluate both against the pretrained model on MTEB and internal document collections. The internal comparison also includes two independent checkpoints.
The same checkpoints, evaluated across 41 embedding tasks, including 10 retrieval tasks.
Mean nDCG@10 · 10 retrieval tasks · higher is better
Scores use a 0–100 scale. We use the same task suite and scoring definitions, with each model’s input conventions and context limit. These local measurements compare evaluated checkpoints; they are not official leaderboard ranks.
Five models, compared across technical, multilingual, and support-document retrieval.
| Model / checkpoint | Type | Size | Honda synthetic | Multilingual synthetic | Disney support | ||||
|---|---|---|---|---|---|---|---|---|---|
| MRR | Recall@10 | MRR | Recall@10 | MRR | Hit@1 | R@10 | |||
| SFR Embedding MistralIndependent checkpoint · 7 B | Independent checkpoint | 7 B | 0.7118 | 0.9132 | 0.7558 | 0.9558 | 0.8409 | 0.7711 | 0.4155 |
| Embedding Gemma 300MIndependent checkpoint · 300 M | Independent checkpoint | 300 M | 0.7852 | 0.9488 | 0.8625 | 0.9958 | 0.8887 | 0.8313 | 0.4392 |
| Embedding Gemma 2Baseline · 271 M | Baseline | 271 M | 0.7515 | 0.9385 | 0.8641 | 0.9933 | 0.8622 | 0.8012 | 0.4132 |
| sfr-embedding-rnd_RResearch · 272.6 M | Research | 272.6 M | 0.7910(+0.0395) | 0.9620(+0.0234) | 0.8777(+0.0136) | 0.9967(+0.0033) | 0.9153(+0.0531) | 0.8735(+0.0723) | 0.4617(+0.0485) |
| sfr-embedding-rnd_PProduction · 272.6 M | Production | 272.6 M | 0.7939(+0.0423) | 0.9610(+0.0224) | 0.8770(+0.0129) | 0.9958(+0.0025) | 0.9109(+0.0487) | 0.8614(+0.0602) | 0.4262(+0.0129) |
Scores on a 0–1 scale. Parentheses show R/P changes from Embedding Gemma 2, calculated before rounding; bold marks the best score in each column.
MRR · Hit@1 · R@10 = Recall@10
R leads on six of the seven metrics.Its strongest gains over Embedding Gemma 2 appear on Disney support: MRR rises from 0.8622 to 0.9153, and Hit@1 from 0.8012 to 0.8735.
P achieves the highest Honda MRR.The production-data variant reaches 0.7939, while improving over Embedding Gemma 2 on every metric shown here.
The research loop starts with a pretrained embedding model, develops training ideas, and evaluates the resulting checkpoints. These two variants combine supervised retrieval with teacher-guided ranking, under different data and representation choices.
The research version combines retrieval cross-entropy with Nemotron ranking distillation along both rows and columns of the query–passage score matrix. A learned projection after native pooling produces 2,048-dimensional embeddings.
The production-supported version uses retrieval supervision and Nemotron teacher scores, with 768-dimensional output. Its fine-tuning data includes HotpotQA, Natural Questions, FEVER, and SQuAD v2.
These results compare complete training recipes. The R model’s wider vectors change its storage footprint relative to the 768-dimensional base model. The P data policy describes new fine-tuning, rather than the base model’s pretraining data.
Setup, metrics, and interpretation.
We use MTEB 1.38.61, the pinned 41-task MTEB (eng, v2) suite, restored title/body formatting, and per-task seed 42. MTEB mean averages the main scores across all tasks; retrieval averages nDCG@10 across its ten tasks.
All three models in the public comparison use an 8,192-token context limit. Native prompts and pooling conventions are retained.
We use 512-token inputs, BF16 encoders, a batch size of 16, and FP32-normalized vectors. Queries and collections remain fixed across models; exhaustive cosine ranking determines the results.
MRR averages the reciprocal rank of the first relevant item. Hit@1 measures whether the first result is relevant. Recall@10 measures the fraction of relevant items found in the top ten. Honda and multilingual collections use supplied chunk-level relevance labels; Disney support uses document-level relevance proxies.
Each named model uses the same checkpoint across its public and internal scores. R corresponds to the row-and-column-distillation checkpoint; P corresponds to the production-data Nemotron checkpoint. Parameter counts follow the supplied model comparison. Output dimensions are 2,048 for R and 768 for P.
See the EmbeddingGemma 2 model card and the MTEB project.