↗Embeddings
Chapter 2 of 5
Notes & sources
Contents ≡
Contents / Chapter 2

02

Learning the map

Training gives a representation a purpose. It does not label every axis with a concept.

Teach a relation, not a dictionary of coordinates

Imagine training examples that pair “My device will not turn on” with a useful charging passage. Other passages compete for the same query. The training task can reward the encoder for assigning the useful pair a higher score.

This is the idea behind the objective in Dense Passage Retrieval: encourage the relevant passage to score above selected negative passages. A negative here is a competing training example, not a negative numerical coordinate.

FIG 002 A task teaches a useful neighborhood
A task teaches a useful neighborhoodA schematic retrieval objective scores a useful passage above selected competitors. Arrows express the objective, not measured point trajectories or guaranteed updates.queryuseful matchcompeting matchraise relative scorelower relative scoreA task teaches a useful neighborhoodA schematic retrieval objective scores a useful passage above selected competitors. Arrows express the objective, not measured point trajectories or guaranteed updates.queryuseful matchcompeting matchraise scorelower score

Schematic learning objective, not a recorded training run. A useful match and a competitor are defined by the training task; the arrows are not literal trajectories every optimizer must follow.

The pull/push picture is schematic. It describes a desired relation, not a measured training trajectory. An objective does not guarantee that every update improves every example, or that the trained model will rank every future query correctly.

Three things called an embedding

“Embedding” can refer to different stages. Keep these three apart:

  1. Token lookup. A token ID selects a row from an embedding table. With the table fixed, the same ID retrieves the same row. See the embedding-table definition.
  2. Contextual token states. Further processing lets each token's representation depend on surrounding tokens. In BERT, this uses bidirectional Transformer processing. A word may occupy several tokens.
  3. One vector for a text. An aggregation step, often called pooling, can combine token states. Sentence-BERT studies pooling together with sentence-level training. Pooling alone is not a guarantee of useful similarity.
Another way to learn: predict nearby words

Skip-gram learns word representations by predicting surrounding words from a target word. This is one historical method, not a universal recipe. Sharing a context is not the same as being a synonym.

Using the map is not redrawing it

After training, we can keep an encoder's parameters fixed and run a new text through it. That is inference: computing a representation with the learned function. Encoding a query need not update the function.

The design implication is important: ask similar for which task? Matching a useful answer, sharing a topic, and expressing the same claim are different goals. Next, we need the numerical rule that turns two vectors into a score.