How Embedding Models Are Trained

Open your phone and type "best pizza near me" into Google. In under 300 milliseconds you get results that include "top-rated Italian restaurants nearby" — a page that shares zerokeywords with your query. Spotify somehow knows that because you liked one lo-fi hip-hop track, you'll probably enjoy an ambient jazz album you've never heard of. ChatGPT can answer a follow-up question about a paragraph it read three messages ago.

All of this is powered by the same invisible machinery: embeddings — a way of converting messy, ambiguous human language into precise lists of numbers that machines can compare, search, and reason over. Every word, sentence, or document gets its own set of coordinates in a vast, high-dimensional space — and the distance between those coordinates is the distance between their meanings.

The idea is disarmingly simple. Instead of asking "do these two texts share the same words?", embeddings let us ask "do these two texts meanthe same thing?" That single shift — from lexical matching to semantic matching — is what makes modern search, recommendations, and RAG pipelines work. And to understand why, we need to see how these numbers are built.

The Representation Problem

Before embeddings existed, engineers represented words the only way they could: one-hot encoding. Give each word in your vocabulary a unique index. "cat" becomes [1, 0, 0, 0, 0]. "dog" becomes [0, 1, 0, 0, 0]. Simple — but catastrophically limited.

The problem is geometric. In a one-hot space, every pair of words is exactly the same distance apart. "cat" is just as far from "kitten" as it is from "refrigerator." A vocabulary of 50,000 words creates 50,000-dimensional vectors that are all mutually orthogonal — they carry identity information and literally zero meaning. No machine learning model can learn that "cat" and "kitten" are related from vectors that are indistinguishable in distance from "cat" and "tax audit."

Dense embeddings flip this completely. Instead of one giant sparse vector per word, you learn a compact vector (say, 5 dimensions) where every single number encodes a fragment of meaning. And when you compare two of these compact vectors, the math actually reflects the semantic relationship between the words.

How Similarity Is Measured — Visually

The central operation in all of embedding-land is cosine similarity. It answers one question: how much do two vectors point in the same direction? Forget the magnitude (length) of each vector — we only care about the angle between them.

There is a critical distinction worth internalizing early: embeddings are not compression. Compression preserves information — you can decompress and recover the original. Embeddings are a lossy semantic projection. You cannot reconstruct the original text from its embedding vector. What you can do is compare two embedding vectors and instantly know how semantically similar the source texts are. That property — fast, numeric semantic comparison — is the entire value proposition.

One more thing about dimensionality. Each dimension in a dense embedding captures a tiny sliver of meaning — think of it as one axis of the semantic coordinate system. 256 dimensions capture word-level relationships well. 1,536 dimensions handle paragraph-level nuance. 3,072 dimensions (used by OpenAI's largest model) can distinguish subtle differences in tone, intent, and domain. More dimensions means more expressive power, but also more storage, more compute, and slower search. That trade-off sits at the core of every production embedding decision.

So now we know what embeddings are and why they matter. The natural next question: how do we actually learn these vectors? That story begins in 2013, with a Google researcher named Tomas Mikolov and a model called Word2Vec.

In 2013, a researcher at Google named Tomas Mikolov published a paper that quietly reshaped the entire field of natural language processing. The model was called Word2Vec, and its central insight was almost embarrassingly simple: you can learn what a word means by looking at the words that surround it.

This is called the distributional hypothesis — the idea that words appearing in similar contexts tend to have similar meanings. "Dog" and "puppy" both show up near words like "bark," "leash," and "walk." So even without anyone ever telling the model that dogs and puppies are related, it figures it out purely from co-occurrence patterns in text.

Word2Vec works by training a tiny neural network on a deceptively simple task: given one word, predict the words around it. The trick is that after training on billions of word pairs, the weights inside that network become the embedding vectors. The network is just scaffolding — what survives is the learned representation.

Two Architectures, One Idea

Word2Vec comes in two flavors. Skip-gram takes the center word and tries to predict every context word around it. CBOW (Continuous Bag of Words) does the reverse — it takes all the context words and predicts the center. Skip-gram learns better representations for rare words because each word gets multiple training signals. CBOW trains faster because it averages context into a single prediction.

Negative Sampling — Making It Scale

There is a practical problem with the architecture above. The output layer computes a softmax over the entire vocabulary — 50,000 to 500,000 words. For every single training pair, the model must compute a score for every word in the vocabulary and normalize them into a probability distribution. That is brutally expensive.

The solution is negative sampling: instead of predicting the right word out of 500,000 options, we reframe the task as a series of binary yes/no questions. For the real pair ("fox", "brown"), we ask: "did these words actually appear together?" (answer: yes). Then we randomly sample a few words that did notappear near "fox" — say "pizza", "laptop", "ocean" — and ask the same question (answer: no). Now the model just needs to separate real pairs from fake ones, which reduces computation per step from O(500K) to O(k), where k is typically 5–15.

Word2Vec has a fundamental limitation that every practitioner should understand: it assigns one vector per word, regardless of context. The word "bank" gets the exact same embedding whether the sentence is "I sat on the river bank" or "I deposited money at the bank." This is the polysemy problem, and it's the reason production systems today don't use vanilla Word2Vec.

But Word2Vec's legacy is not its specific architecture — it's the proof that self-supervised prediction tasks on raw text produce useful representations. Every modern embedding model, from BERT to OpenAI's text-embedding-3-large, still uses this core principle. The prediction task evolved (masked language modeling, contrastive learning). The architecture evolved (transformers replaced shallow networks). But the foundational idea — that predicting context teaches meaning — remains unchanged.

And those learned vectors had one more surprise in store — a property so striking that it became the most famous result in all of embedding research: you could do arithmetic with meaning.

When Word2Vec was first published, the embedding vectors it produced had a property so striking that it became the single most cited result in all of NLP: you could do arithmetic with meaning.

Take the vector for "king." Subtract the vector for "man." Add the vector for "woman." The result lands almost exactly on the vector for "queen." Written as an equation: v(king) − v(man) + v(woman) ≈ v(queen). The operation works because the directionfrom "man" to "king" — the geometric representation of the concept "royalty applied to male" — is nearly identical to the direction from "woman" to "queen." Subtracting "man" removes the male component, adding "woman" inserts the female component, and what remains is royalty. The four words form a parallelogram in vector space.

This is not a hand-tuned trick. It emerges spontaneously from the training process. Because "king" and "queen" appear in nearly identical grammatical contexts ("ruled," "decreed," "inherited the throne"), their vectors share most of their dimensions. The few dimensions that differ are exactly the ones that encode gender — the same dimensions that separate "man" from "woman." Linear algebra does the rest.

When Arithmetic Goes Wrong

The analogy trick works beautifully for clean semantic relationships — gender, verb tense, country-capital. But it reveals something troubling when you push it into social territory. Compute "doctor − man + woman" and you shouldget "doctor" — it's a gender-neutral profession. But older embeddings trained on web text returned "nurse." The model had learned the statistical bias of its training corpus: in the texts it read, doctors were disproportionately described as male and nurses as female.

This is not a bug in the algorithm. It is a faithful reflection of the data. Embeddings do not encode truth — they encode the patterns present in whatever text they were trained on. If the corpus is biased (and all internet-scale corpora are), the embeddings will be biased. Production systems must audit for these patterns, apply debiasing techniques, or choose models specifically trained to mitigate them.

One final note: the analogy benchmark is a useful sanity check, but a terrible evaluation metric for production systems. In practice, nobody cares whether king−man+woman=queen. What matters is whether your embedding model returns the right documents for a query, classifies text correctly, or clusters topics coherently. Always evaluate on your actual downstream task, not on toy analogies. Models that ace the analogy test can fail spectacularly on domain-specific retrieval if your domain was underrepresented in the training data.

But Word2Vec had one more fundamental limitation beyond bias — one that motivated the entire next generation of embedding models. It assigns one vector per word, regardless of context. The word "bank" gets the same vector in "river bank" and "investment bank." Modern contrastive learning fixes this completely.

Word2Vec proved that self-supervised prediction on raw text produces useful representations. But it had a fundamental ceiling: it assigns one vector per word, and it only captures word-level relationships. What if we want to compare entire sentences? What if the query "How do I reset my password?" needs to match the document "Steps to change your account password" — a passage that shares almost no keywords?

This is where contrastive learning enters the picture. It is the training paradigm behind every modern sentence embedding model — from sentence-transformers and OpenAI's text-embedding-3-large to Cohere's embed-v3 and Google's Gecko. Instead of predicting neighboring words, contrastive learning directly optimizes the property we actually need: semantically similar texts should have nearby embeddings, and dissimilar texts should be far apart.

The setup is elegant. You start with an anchor text. You pair it with one positive (a text that means the same thing) and several negatives(texts that don't). The training objective: push the anchor's embedding toward the positive and away from the negatives. After millions of such updates, the model has learned a geometric space where proximity equals semantic similarity.

The InfoNCE Loss Function

The training objective that makes all of this work is called InfoNCE(Information Noise-Contrastive Estimation). It is essentially a softmax cross-entropy loss where the "correct class" is the positive pair. The model computes a similarity score between the anchor and every candidate (the positive + all negatives), divides by a temperature parameter τ, applies softmax, and tries to maximize the probability assigned to the positive.

Hard Negatives — The Secret Weapon

Here is one of the most important practical insights in all of embedding engineering: the quality of your negatives matters more than the architecture of your model. A random negative like "Best pizza in New York" is trivially easy for the model to distinguish from "How to deploy to AWS." It learns nothing useful from that pair. The gradient is tiny, the update is meaningless.

A hard negative, on the other hand, is a text that is topically related but semantically different — like "How to deploy to Google Cloud" when the anchor is "How to deploy to AWS." These force the model to learn fine-grained distinctions: same topic, different platform. The gradient is large, the update is meaningful, and the resulting embeddings are dramatically more discriminative.

Contrastive learning solved the context problem that plagued Word2Vec. Because the model processes entire sentencesthrough a transformer encoder before comparing them, the same word gets different embeddings depending on its surrounding context. "Bank" in a financial document and "bank" in a nature article produce different vectors. This is the power of contextual embeddings — and it's why modern sentence-transformers dominate every retrieval benchmark.

But there is still one more practical challenge that production teams face every day: the dimensionality trade-off. Higher-dimensional embeddings capture more nuance, but they cost more to store, index, and search. What if you could train one model whose embeddings work at multiple dimensions — 256 for fast search, 1024 for high-precision tasks — without retraining? That is exactly what Matryoshka embeddings achieve.

Every embedding model produces a fixed-size vector — 768 dimensions for most open-source models, 1,536 or 3,072 for OpenAI's largest. More dimensions capture more nuance, but they cost more to store, index, and search. At 100 million documents, the difference between 256-D and 1,536-D is tens of gigabytes of vector storage and an order of magnitude in search latency.

In practice, teams face an uncomfortable choice: use the full-dimension embedding and pay the compute bill, or retrain a smaller model and accept lower quality. Matryoshka Representation Learning (MRL) eliminates this trade-off entirely.

Named after Russian nesting dolls, a Matryoshka embedding is trained so that the first d dimensions of the full vector form a valid, high-quality embedding on their own. The first 64 dimensions capture the broad strokes — topic, domain, general sentiment. The first 256 dimensions capture most of the semantic nuance. The full 1,536 dimensions capture everything. And because every prefix is valid, you can choose your quality/cost trade-off at query time — without retraining anything.

How MRL Training Works

The training trick is surprisingly simple. Instead of computing the contrastive loss only on the full 1,536-D embedding, you compute it simultaneously at multiple truncation levels: the first 64, 128, 256, 512, and 1,536 dimensions. The total loss is a weighted sum of all these partial losses. Because every level shares the same model parameters, the gradient from the 64-D loss forces the first 64 dimensions to be maximally informative on their own, while the full-dimension loss ensures the complete vector misses nothing.

There is one honest trade-off to acknowledge: Matryoshka embeddings sacrifice roughly 2–5% quality at the full dimension compared to a model trained exclusively at that size. If your system always uses the full 1,536-D vector and never truncates, a standard contrastive model will be marginally better. But the moment you need flexibility — and production systems almost always do — MRL pays for itself many times over.

The production pattern is a two-stage search. First stage: use 256-D truncated vectors to search 100 million documents in ~20ms. This returns the top 100 candidates at 92% quality. Second stage: re-score those 100 candidates using the full 1,536-D vectors in ~1ms. Total latency: 21ms, with the precision of full-dimension search. This two-stage pattern — fast coarse retrieval followed by precise re-ranking — is how every serious vector search system works at scale, from Google to Spotify to Notion.

Now we have all the building blocks: embeddings, training objectives, and efficiency techniques. The final question is the most practical one: how do you actually use these embeddings in a production application?

Embeddings are not an end product. They are infrastructure — the invisible layer that powers semantic search, recommendation engines, clustering pipelines, classification systems, anomaly detection, and most critically, Retrieval-Augmented Generation (RAG). If you build anything that needs to understand the meaning of text rather than just its keywords, you are using embeddings, whether you realize it or not.

Understanding how embeddings are trained — everything we have covered so far — gives you a diagnostic superpower. When your RAG pipeline returns irrelevant documents, the root cause is almost always traceable to the embedding layer: wrong model for your domain, wrong chunking strategy, or a distributional mismatch between your data and the model's training corpus. Knowing what went wrong inside the model tells you exactly how to fix it.

Semantic Search in Action

The query "How do I change my password?" contains none of the keywords in the document titled "Password Reset Guide." A traditional keyword search (BM25) would miss it entirely. But the embedding model maps both to nearby vectors — because they mean the same thing — and the cosine similarity is 0.94. This is the difference between searching for words and searching for meaning.

Choosing an Embedding Model

The model you choose sets the quality ceiling for your entire retrieval system. No amount of re-ranking, prompt engineering, or chunking optimization can compensate for a fundamentally wrong embedding model. The key dimensions: retrieval quality (MTEB benchmark), cost per million tokens, dimensionality (storage + latency), context length, and whether it supports Matryoshka truncation for flexible dimensionality.

Where Production Systems Break

Three failure modes account for the vast majority of RAG quality issues in production:

And one final, practical note: always build an evaluation set. Before you tune any parameter — model, chunk size, overlap, re-ranker — create 50–100 query/answer pairs that represent your actual workload. Measure retrieval precision and recall objectively. Without this, you are tuning blindly, and every change is a coin flip. With it, every experiment is a data point, and your system improves monotonically.

You now have the complete picture: from one-hot encodings to Word2Vec, from contrastive learning to Matryoshka embeddings, from vector arithmetic to production RAG. The next time you build a retrieval system, you will understand not just what to do — but why it works.