From a sentence to a vector

A help center contains a passage titled “Charging.” A visitor asks, “My device will not turn on.” The wording differs, but the passage might still help. How could a search system compare more than the words they share?

One approach gives each text a numerical representation: an embedding. A vector is an ordered list of numbers. An encoder is the function that produces that vector from an input. Text-retrieval models can encode questions and passages this way. Dense Passage Retrieval is one concrete example.

Let $x$ be an input text, $f$ its encoder, and $e$ its embedding. If the encoder produces $d$ coordinates, we write:

Here $\mathbb{R}^{d}$ means “a list of $d$ real numbers.” Fixed-length means that this encoder's representation has that coordinate count; it does not mean every input sentence has the same number of words.

Read the list as a place

We will use a small help center throughout. For the battery passage B, invent the vector $[0.8,0.2]$. These are hand-authored 2D toy coordinates, not values produced by an actual text model.

The first coordinate moves 0.8 units along the first axis; the second moves 0.2 units along the second. You can draw the result as a point or as an arrow from the origin to that point. Both depict the same vector.

The axes are coordinate 1 and coordinate 2—not “battery” and “display.” A dimension is not automatically a named human concept. Indeed, rotating an entire vector space together can change its coordinates without changing its distances or dot products.

A representation is not an ID

A document ID tells us which record to fetch. Its numeric difference from another ID need not tell us anything useful. Embedding coordinates instead participate in a comparison rule: we can ask about their directions, lengths, or distances.

That is the promise, not yet the explanation. Our invented coordinates make geometry easy to inspect. In a learned system, what gives the coordinates a useful arrangement?

Teach a relation, not a dictionary of coordinates

Imagine training examples that pair “My device will not turn on” with a useful charging passage. Other passages compete for the same query. The training task can reward the encoder for assigning the useful pair a higher score.

This is the idea behind the objective in Dense Passage Retrieval: encourage the relevant passage to score above selected negative passages. A negative here is a competing training example, not a negative numerical coordinate.

The pull/push picture is schematic. It describes a desired relation, not a measured training trajectory. An objective does not guarantee that every update improves every example, or that the trained model will rank every future query correctly.

Three things called an embedding

“Embedding” can refer to different stages. Keep these three apart:

1. Token lookup. A token ID selects a row from an embedding table. With the table fixed, the same ID retrieves the same row. See the embedding-table definition.
2. Contextual token states. Further processing lets each token's representation depend on surrounding tokens. In BERT, this uses bidirectional Transformer processing. A word may occupy several tokens.
3. One vector for a text. An aggregation step, often called pooling, can combine token states. Sentence-BERT studies pooling together with sentence-level training. Pooling alone is not a guarantee of useful similarity.

Using the map is not redrawing it

After training, we can keep an encoder's parameters fixed and run a new text through it. That is inference: computing a representation with the learned function. Encoding a query need not update the function.

The design implication is important: ask similar for which task? Matching a useful answer, sharing a topic, and expressing the same claim are different goals. Next, we need the numerical rule that turns two vectors into a score.

Start with the arrow

Let $q=[q_1,q_2]$ be a query vector and $p=[p_1,p_2]$ a passage vector. Their coordinates have no physical units here. A vector's norm is its length from the origin:

The dot product multiplies matching coordinates and adds the results. It is sensitive to both direction and magnitude:

Direction, product, or endpoint distance?

Cosine similarity divides the dot product by both lengths. For nonzero vectors, it compares direction; higher is more aligned. Its range is −1 to 1, not a probability or confidence scale. See the metric definitions.

Euclidean distance measures the gap between the arrow endpoints. Lower is nearer:

Our rankings use higher cosine, higher dot product, or lower Euclidean distance. These are different instructions, not interchangeable names for “relevance.”

Keep the documents fixed. Change the question.

This lab uses hand-authored 2D toy vectors. No model reads the passage text or embeds a typed query. The controls directly set the query arrow's angle and length; the five document vectors stay fixed.

First change only query length. Positive rescaling leaves cosine unchanged. It multiplies every dot score by the same factor, so dot-product order stays unchanged. Euclidean order can change because the query endpoint moves.

Now reset and switch from cosine to dot product. At $q=[1,0]$, A has cosine 1 and dot product 1. E has cosine 0.8 and dot product 2.4. A wins cosine; E wins dot. Here the difference comes from the candidates' different magnitudes. E's length was stipulated—not inferred from its passage length.

When the rules agree

Normalize a nonzero vector by dividing it by its norm. If every compared vector has unit length, cosine and dot product coincide, and their rankings agree with Euclidean distance:

The condition matters: normalize all compared vectors, not just the query. A metric choice only makes sense alongside the representation it is meant to compare.

Prepare the passages; then receive a query

Return to the help center. We have five passages, each with an ID and source text. A retrieval system can prepare their embeddings ahead of time. When a visitor asks a question, it computes a query embedding and compares it with those prepared vectors.

Let $f_D$ encode a document passage $d_i$, giving $p_i=f_D(d_i)$. Let $f_Q$ encode the incoming query text $x$, giving $q=f_Q(x)$. The two paths must produce compatible representations, but need not have identical weights: DPR jointly trains separate query and passage encoders.

Equal coordinate counts are not enough. Two maps can each have 100 coordinates while assigning different meanings to their directions. Use a suitable encoder or deliberately compatible pair, together with its intended preprocessing and scoring rule.

Score → rank → inspect

The score produces an ordering of candidate passages. Each result still needs its record ID so that the system can return the original text, title, and source—not merely a list of coordinates.

In our toy example, assign the query $q=[1,0]$. With cosine, passage A comes first, and its record gives us “Connect the charger to restore power.” With raw dot product, E comes first instead. This is the geometry from the previous chapter, not a measured search result for the visitor's wording.

Ranking first does not establish that a passage is correct, current, or sufficient to solve the visitor's problem. A similarity score compares representations; it does not check the passage's claims. DPR likewise separates retrieval from the downstream step of extracting an answer.

We now have a complete path: text becomes a vector, a rule compares vectors, and a rank selects candidates to read. What can that path—and its attractive geometry—leave out?

A high score is not a verdict

“The charger is compatible” and “The charger is not compatible” discuss almost the same topic but make opposite claims. We have not measured either sentence's embedding here. The point is about the task: related subject matter is not the same as agreement, and agreement is not the same as truth.

A good retrieval evaluation therefore needs representative questions and judgments about which passages actually help. Geometry can generate candidates; the returned text and its provenance remain essential evidence.

A picture can hide a difference

Our lab is genuinely two-dimensional. To draw a higher-dimensional vector as a point in a 2D plot, we need a mapping to two coordinates. Consider the deliberately simple projection that drops the third coordinate:

The original points $[1,0,0]$ and $[1,0,4]$ are 4 units apart. Both become $[1,0]$ on the page, so their displayed distance is zero. This counterexample does not condemn every projection. It shows why a 2D picture is not automatically evidence about original-space distances.

Try a fresh pair

Take $q=[1,0]$, $U=[0.6,0.8]$, and $V=[3,4]$. Which candidate wins under cosine, dot product, and Euclidean distance? What changes if you normalize every vector first?

Take the map—not the illusion

Keep three questions: What was this representation trained to preserve? Which comparison rule does it expect? Does the retrieved source actually help?

An embedding gives you coordinates for a task. The coordinates make comparison possible; the task, scoring rule, and evidence make the comparison meaningful.