the embedding layer does: maps discrete token IDs to dense learned vectors

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

Transformer (deep learning)

Transformers use multi-head attention for contextualizing tokens

the compute-optimal training ratio is: roughly 20 tokens per parameter

How can we train AI efficiently without wasting resources?

mean pooling often outperforms [CLS] for sentence similarity tasks

Mean pooling captures overall sentence meaning better than [CLS] token embedding

autoencoders learn the data manifold

Autoencoders compress data manifold by forcing information through a bottleneck layer, learning efficient representations

768-dim BERT embeddings capture: bidirectional context from masked language modeling

768-dim BERT embeddings capture bidirectional context from masked language modeling