Glossary of probability and statistics

Ever wonder why your average score on quizzes improves with more quizzes?

Imagine you're tracking your daily coffee consumption to see if it affects your productivity. You record how many cups you drink each day and whether you feel more productive.

As you keep track of more days, your average coffee consumption and productivity levels start to resemble the true average for everyone who drinks coffee, not just you.

Example

Over 30 days, you drink an average of 3 cups a day and feel moderately productive. Over 300 days, your average consumption might be closer to 3 cups a day, with productivity levels stabilizing around the same point.

Remember this

The Law of Large Numbers (LLN) explains why your sample mean (average coffee consumption, productivity) gets closer to the population mean (true average coffee consumption, productivity) as you record more data.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

Law of large numbers

Law of large numbers: X̄_ n → μ as n → ∞ with probability 1

Central limit theorem

Central limit theorem states that sample means converge to normal distribution as sample size increases

the vocabulary size matters: larger vocab = shorter sequences but more parameters

Larger vocab leads to shorter sequences but more parameters

importance sampling does: reweights samples from proposal to estimate target expectation

Importance sampling estimates target expectations using samples from a different distribution

Neural scaling law

Chinchilla scaling law: optimal model size scales linearly with compute budget

Effect size

Cohen's D benchmarks: 0.2 = small, 0.5 = medium, 0.8 = large effect

Swipe through 100 ML concepts daily

Open Pocket Polymath