300-dim word2vec encodes: trained on word co-occurrence with skip-gram window

Ever wondered how computers understand words?

Imagine you're trying to teach your smart speaker to recognize different ways to say "I will go to the store."

The speaker uses a special method to link words that mean the same thing based on how they're used together.

Example

If you say "I will go to the store," the speaker learns to connect it with "I will shop."

Remember this

Word2vec helps computers grasp language nuances by finding patterns in word usage.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

384-dim all-MiniLM-L6-v2 optimizes: fast sentence similarity with 6 layers

All-MiniLM-L6-v2 optimizes fast sentence similarity with 6 layers

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

the vocabulary size matters: larger vocab = shorter sequences but more parameters

Larger vocab leads to shorter sequences but more parameters

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

768-dim BERT embeddings capture: bidirectional context from masked language modeling

768-dim BERT embeddings capture bidirectional context from masked language modeling

1536-dim OpenAI text-embedding-3-large is used for: semantic search and RAG

Used for semantic search, RAG, and enhancing language models' understanding

Swipe through 100 ML concepts daily