What if we could turn raw scores into probabilities?
Image: RepRapPro, CC BY 3.0, via Wikimedia Commons
to write a fused softmax kernel in Triton: load row, compute max, subtract, exp, sum, divide
What if we could turn raw scores into probabilities?
Imagine you're at a restaurant with friends, deciding who gets the last slice of pizza. Everyone orders a slice, but there's only one left. You need a fair way to decide who gets it.
Think of each person's score as how much they want the pizza. We want to know the chances each person has of getting it. We'll do this by finding the highest score, making everyone's score smaller by subtracting it, and then turning these smaller scores into a chance by using a special math trick called softmax.
Example
If Alice scored 5, Bob scored 3, and Charlie scored 2, we first find the highest score (5), then subtract it from each score: Alice gets 0, Bob gets 2, Charlie gets 3. Now we use softmax to turn these numbers into probabilities: Alice gets 0, Bob gets 0.67, Charlie gets 1.
Remember this
The softmax formula turns raw scores into probabilities, ensuring everyone gets a fair chance based on their interest.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
Softmax function
Softmax converts real numbers into a probability distribution
to write a vector addition kernel in Triton: load blocks, add, store
```
tl.dot does in Triton: block-level matrix multiply using tensor cores
tl.dot performs block-level matrix multiplication using tensor cores in Triton
a Triton kernel is
Can your phone run faster with a different brain?
fused kernels do
Why wait for a computer to finish one task before starting another?
tl.load and tl.store do in Triton: read/write tensors from/to GPU global memory
`tl.load` reads tensors from GPU memory; `tl.store` writes tensors to GPU memory
Swipe through 100 ML concepts daily
Open Pocket Polymath