mean pooling does: averages all token embeddings to get a sentence embedding

How can we summarize a whole sentence's meaning with just one number?

Imagine you're trying to describe the mood of a conversation with just one word. Each person's words are like individual ingredients in a recipe.

To get a single summary, you could average the mood of each person's words. This way, you get a general mood for the whole conversation.

Example

If Alice said she's happy (3), Bob said he's sad (1), and Carol said she's neutral (2), the average mood is (3+1+2)/3 = 2.

Remember this

Averaging all the mood scores gives us a sentence embedding that represents the overall mood.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

mean pooling often outperforms [CLS] for sentence similarity tasks

Mean pooling captures overall sentence meaning better than [CLS] token embedding

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

Graph neural network

Graph pooling reduces graphs to single vectors for graph-level prediction

1536-dim OpenAI text-embedding-3-large is used for: semantic search and RAG

Used for semantic search, RAG, and enhancing language models' understanding

the embedding layer does: maps discrete token IDs to dense learned vectors

Embeddings convert token IDs to dense vectors for neural network processing

Swipe through 100 ML concepts daily