↗Embeddings
Chapter 1 of 5
Notes & sources
Contents ≡
Contents / Chapter 1

01

A place in space

An embedding turns an input into coordinates. The useful part is what those coordinates let us compare.

From a sentence to a vector

A help center contains a passage titled “Charging.” A visitor asks, “My device will not turn on.” The wording differs, but the passage might still help. How could a search system compare more than the words they share?

One approach gives each text a numerical representation: an embedding. A vector is an ordered list of numbers. An encoder is the function that produces that vector from an input. Text-retrieval models can encode questions and passages this way. Dense Passage Retrieval is one concrete example.

Let xx be an input text, ff its encoder, and ee its embedding. If the encoder produces dd coordinates, we write:

e=f(x)∈Rde=f(x)\in\mathbb{R}^{d}

Here Rd\mathbb{R}^{d} means “a list of dd real numbers.” Fixed-length means that this encoder's representation has that coordinate count; it does not mean every input sentence has the same number of words.

Read the list as a place

We will use a small help center throughout. For the battery passage B, invent the vector [0.8,0.2][0.8,0.2]. These are hand-authored 2D toy coordinates, not values produced by an actual text model.

FIG 001 From a passage to a place
From a passage to a placeThe toy passage is represented by vector [0.8,0.2], whose coordinates locate point B. The actual numbers are hand-authored, not encoder output.“Improve battery life…”a short passageencoder[0.8, 0.2]B0.80.2x₁x₂From a passage to a placeThe toy passage is represented by vector [0.8,0.2], whose coordinates locate point B. The actual numbers are hand-authored, not encoder output.“Improve battery life…”passage → encoder → [0.8, 0.2]B0.80.2x₁x₂

Hand-authored 2D example. The coordinate list and the point are two views of the same numbers; x₁ and x₂ are not named semantic features.

The first coordinate moves 0.8 units along the first axis; the second moves 0.2 units along the second. You can draw the result as a point or as an arrow from the origin to that point. Both depict the same vector.

The axes are coordinate 1 and coordinate 2—not “battery” and “display.” A dimension is not automatically a named human concept. Indeed, rotating an entire vector space together can change its coordinates without changing its distances or dot products.

A representation is not an ID

A document ID tells us which record to fetch. Its numeric difference from another ID need not tell us anything useful. Embedding coordinates instead participate in a comparison rule: we can ask about their directions, lengths, or distances.

That is the promise, not yet the explanation. Our invented coordinates make geometry easy to inspect. In a learned system, what gives the coordinates a useful arrangement?