Machine Learning

self-attention: Attention(Q,K,V) = softmax(QK^T/√d_k)V

Ever wondered how computers understand what's important in a sentence?

Cosine similarity

Cosine similarity formula: cos(θ) = (A · B) / (||A|| ||B||)

Matrix norm

L1 norm of a vector is the sum of absolute values of its components

Normalization (machine learning)

L2 normalization equation: x_i' = x_i / ||x||_2

Softmax function

Softmax converts real numbers into a probability distribution

Cross-entropy

Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))

Bayes' theorem

Bayes' theorem formula: P(A|B) = [P(B|A) * P(A)] / P(B)

Gradient descent

Gradient descent weight update equation: w := w - α * ∇J(w)

Mean squared error

Mean squared error (MSE) formula: MSE = (1/n) * Σ(y_i - ŷ_i)²

Activation function

Tanh activation function equation: tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))

ReLU and Leaky ReLU

Why do computers sometimes struggle with simple decisions?

Mutual information

Mutual information formula: I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y)))

Jensen–Shannon divergence

Jensen-Shannon divergence formula: D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M))

Dot product

Dot product = sum of products of corresponding entries

Euclidean distance

Euclidean distance formula: √((x2 - x1)² + (y2 - y1)²)

Distance transform

Manhattan distance formula: |x1 - x2| + |y1 - y2|

Minkowski spacetime

Minkowski distance formula: D = (Σ |x_i - y_i|^p)^(1/p)

Mahalanobis distance

Mahalanobis distance formula: D² = (x - μ)'Σ^(-1)(x - μ)

Rotation matrix

Determinant of a 2x2 matrix: ad - bc

Standard deviation

Standard deviation (σ) is the square root of variance

Covariance matrix

Covariance formula: Cov(X, Y) = E[(X - E[X])(Y - E[Y])]

Pearson correlation coefficient

Pearson correlation coefficient formula: r = Σ[(xi - x̄)(yi - ȳ)] / [√(Σ(xi - x̄)²) * √(Σ(yi - ȳ)²)]

Regression analysis

Linear regression equation: ŷ = β0 + β1X

Logistic regression

Logistic regression probability formula: P(Y=1) = 1 / (1 + exp(-z))

Batch normalization

Batch normalization formula: Y = (X - μ) / σ * γ + β

Adam optimizer weight update with m and v terms

Ever wondered how we can predict complex financial markets with high accuracy?

Learning to rank

Learning rate cosine annealing formula: learning_rate = learning_rate_initial * 0.5 * (1 + cos(pi * epoch / total_epochs))

Dropout (neural networks)

Dropout randomly sets neuron inputs/outputs to zero during training

Precision and recall

Precision = Relevant retrieved instances / All retrieved instances

TF-IDF scoring

TF-IDF = (Term Frequency) * (Inverse Document Frequency)

PageRank

PageRank formula: PR(A) = (1-d) + d Σ(PR(C)/L(C))

Discrete Fourier transform

Discrete Fourier Transform (DFT) equation: X[k] = Σ(n=0 to N-1) x[n] * e^(-j*2π*k*n/N)

Taylor series

Taylor series formula: f(x) = f(a) + f'(a)(x-a) + f''(a)(x-a)²/2! +

Chain rule

Chain rule formula: h'(x) = z'(y(x)) * y'(x)

Euler's identity

Euler's identity: e^(iπ) + 1 = 0

Quadratic equation

Quadratic equation standard form: ax² + bx + c = 0

Binomial coefficient

Binomial coefficient formula: (n choose k) = n! / (k!(n-k)!)

Poisson distribution

Poisson distribution formula: P(k; λ) = (λ^k * e^(-λ)) / k!

Normal distribution

Normal distribution PDF formula

Conditional probability

P(A|B) = P(A ∩ B) / P(B)

Expected value

Expected value formula: E[X] = Σ [x * P(x)]

Hessian matrix

The Hessian matrix is denoted by H or ∇²

Lagrangian L(x,λ) = f(x) - λg(x)

How do we find the path an object naturally takes?

convolution (f * g)(t) = ∫f(τ)g(t-τ)dτ

How do different shapes combine to create new patterns?

Write the Bellman equation for reinforcement learning

Predicting tomorrow's weather with today's clues

Stochastic gradient descent

Policy Gradient Theorem Equation

Variational autoencoder

ELBO formula in variational inference

Write the reparameterization trick z = μ + σ⊙ε

How can you predict the weather without getting wet?

Write the contrastive loss function for SimCLR

Contrastive loss function: L = (1/2N) Σ [max(0, margin - y_i * (z_i - z_j))² + max(0, y_i * (z_i - z_j) - margin)²]

Write the triplet loss formula: max(d(a,p) - d(a,n) + margin, 0)

Can you tell friends apart even if they've changed a lot over time?

BLEU

Ever wondered how computers know if a translation makes sense?

Perplexity

Perplexity = 2^H

Write the attention score formula before softmax: e_ij = a(s_i, h_j)

How do we know what's important in a sentence?

Write the multi-head attention formula: MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O

Ever wondered how machines understand the importance of words in a sentence?

the L1 unit ball is a diamond shape and the L2 unit ball is a circle

Why does a ball look different in various dimensions?

Regularization (mathematics)

L1 regularization results in sparse solutions

the Lp norm ball shape changes as p goes from 1 to 2 to infinity

How does the shape of a ball change as we measure distance differently?

Norm (mathematics)

L∞ norm equals max absolute value

Chebyshev distance

Chebyshev distance is named after Pafnuty Chebyshev

The elastic net combines L1 and L2: λ₁|w| + λ₂w² gives both sparsity and stability

Why do we sometimes need both a scalpel and a hammer in surgery?

the L1 norm is not differentiable at zero

Absolute value's kink makes it non-differentiable at zero

Proximal gradient methods for learning

Why can't we always find the best path in a maze?

LASSO uses L1 to do feature selection by driving coefficients to exactly zero

Why do some numbers disappear when solving complex problems?

Ridge regression uses L2 to shrink coefficients without eliminating them

Why do some roads get smoother and straighter over time?

CUDA

CUDA enables parallel computation on GPUs

a Triton kernel is

Can your phone run faster with a different brain?

Triton differs from CUDA

Why does a super-fast computer sometimes run slower than a regular one?

Thread block (CUDA programming)

Thread blocks can contain up to 1024 threads as of March 2010

Dynamic random-access memory

DRAM requires periodic refreshing to maintain data integrity

2024–present global memory supply shortage

Global DRAM shortage began in 2024

Flashbulb memory

Flashbulb memories are vivid but not always accurate

warp divergence kills performance

Warp divergence causes threads to execute non-uniformly, leading to idle cycles and reduced throughput

occupancy means in GPU programming

Occupancy = Active Warps / Max Warps

Overdrawn at the Memory Bank

Overdrawn at the Memory Bank was shot on videotape due to budget constraints

Matrix multiplication algorithm

Ever wondered how computers speed up multiplying huge numbers?

Attention (machine learning)

Why do we sometimes zoom in faster on a scene?

fused kernels do

Why wait for a computer to finish one task before starting another?

kernel fusion reduces memory bandwidth bottleneck

Can you imagine faster video games without waiting for loading screens?

tensor cores are

Why can computers crunch numbers faster than humans?

tensor cores do 4x4 matrix multiply in one clock cycle

Tensor cores perform 4x4 matrix multiply using optimized GEMM (General Matrix Multiply) instructions in one clock cycle

Tesla Model Y

Tesla Model Y is the world's best-selling electric vehicle in 2023

arithmetic intensity is

Arithmetic intensity = FLOPs / Bytes accessed

most transformer operations are memory-bound, not compute-bound

Why do computers sometimes get tired?

NVIDIA's A100 has: 80GB HBM2e, 2TB/s bandwidth, 312 TFLOPS FP16

NVIDIA's A100 features: 80GB HBM2e, 2TB/s bandwidth, 312 TFLOPS FP16

NVIDIA's H100 has: 80GB HBM3, 3.35TB/s bandwidth, 990 TFLOPS FP16

NVIDIA H100 features: 80GB HBM3, 3.35TB/s bandwidth, 990 TFLOPS FP16

quantization to INT8 doubles throughput

Quantization to INT8 doubles throughput because tensor cores process INT8 2x faster

GPTQ quantization does

Post-training quantization using second-order information for model compression

AWQ does differently

AWQ selectively retains weights crucial for model performance, unlike traditional quantization

KV-cache reduces redundant computation in autoregressive generation

KV-cache stores previously computed outputs to avoid redundant calculations in autoregressive models

paged attention (vLLM) improves serving throughput

Paged attention (vLLM) improves serving throughput by reducing latency through non-contiguous KV-cache pages, enabling faster data retrieval

continuous batching does

How can you mix a cocktail perfectly every time without stirring?

Masking (behavior)

Can you not see what's right in front of you?

Transformer (deep learning)

Transformers use multi-head attention for contextualizing tokens

transformers use LayerNorm not BatchNorm

LayerNorm normalizes across all features, accommodating variable-length sequences unlike BatchNorm, which relies on fixed-size batches

Pre-LN

Why is normalizing data like tuning instruments before a concert?

Pre-LN transformers are easier to train

Pre-LN transformers use residual connections, allowing gradients to flow more smoothly during backpropagation

rotary position embeddings (RoPE) do

RoPE encodes relative position by applying rotation matrices to input features

RoPE encodes position: multiply Q,K by rotation matrix R(θ_i) at each position

How does a robot arm rotate smoothly?

Her Alibi

Bruce Beresford directed Her Alibi

ALiBi allows length extrapolation better than learned position embeddings

ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences

grouped query attention (GQA) does

GQA shares KV heads across multiple Q heads for efficient parameter usage

GQA reduces KV-cache memory by the group factor

Ever wondered how websites stay fresh in search results?

multi-query attention (MQA) is

Multi-query attention (MQA) with shared KV head: Q heads share a single KV head for efficient parameter usage

Attention Is All You Need

"Attention Is All You Need" introduced the transformer architecture in 2017

Mixture of experts

Mixture of experts (MoE) divides problem space into homogeneous regions

MoE models have more parameters but similar compute cost

MoE models distribute parameters across k experts, reducing active experts' compute cost

Graduate Aptitude Test in Engineering

GATE exam assesses engineering and science undergraduate subjects for postgraduate admissions in India

load balancing loss is needed in MoE

Can one expert handle all tasks perfectly?

Alex Lora Cercos

Alex Lora is a Spanish film director

LoRA (machine learning)

LoRA uses r << d for efficient adaptation

QLoRA adds

Ever wondered how computers can understand and mimic human speech?

2024 in hip-hop

LoRA rank r controls model capacity and parameters

Surface tension

High surface tension in water

the volume of a unit ball approaches zero as dimensions increase

How does a coastline's length change with different measuring sticks?

List of unsolved problems in mathematics

Why do random points in high dimensions seem to be evenly spaced?

the curse of dimensionality makes nearest neighbor search unreliable

Why can't we find our friends easily as we move to a city with more and more neighborhoods?

cosine similarity works better than Euclidean distance in high dimensions

Cosine similarity measures orientation, not magnitude, making it more robust to irrelevant dimensions in high-dimensional spaces

the Johnson-Lindenstrauss lemma says

Can we shrink big data without losing important details?

random projection to O(log n/ε²) dimensions preserves pairwise distances within 1±ε

Can you shrink a 3D object into 2D without losing its shape?

Locality-sensitive hashing

Locality-sensitive hashing (LSH) hashes similar items into the same buckets

Hierarchical navigable small world

HNSW is an efficient ANN search algorithm

IVF (Inverted File Index) does

Ever searched for a word in a massive book collection?

Vector quantization

How can you store a huge library of books in a tiny closet?

Manifold hypothesis

High-dimensional data lies on lower-dimensional manifolds

autoencoders learn the data manifold

Autoencoders compress data manifold by forcing information through a bottleneck layer, learning efficient representations

t-SNE preserves local structure

Can we see the hidden patterns in a cloud of data points?

UMAP is faster than t-SNE

UMAP is faster due to approximate nearest neighbors and cross-entropy optimization

Intrinsic dimension

Intrinsic dimension M satisfies 0 ≤ M ≤ N

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

How do we measure uncertainty in everyday decisions?

Entropy (information theory)

Entropy of a fair coin is 1 bit

A fair die has entropy of log₂(6) ≈ 2.58 bits

How much information do you need to guess a die roll?

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Ever wondered how well we can guess the outcome of a random event?

KL divergence is always ≥ 0 and equals 0 only when P = Q exactly

Why can't we just compare two things directly?

Kullback–Leibler divergence

KL divergence is not symmetric: D_KL(P||Q) ≠ D_KL(Q||P)

Channel capacity

Shannon's channel capacity: C = B log₂(1 + S/N) bits per second

Huffman coding

Huffman coding is an entropy-optimal prefix code for lossless data compression

Shannon's source coding theorem: you can't compress below entropy

Can you squeeze endless text into fewer bits without losing anything?

Rate-distortion theory: minimum bits to represent data within distortion D

How many bits do we need to perfectly copy a song?

Chebyshev's inequality

Chebyshev's inequality limits the probability of deviation from the mean

Binary search

How fast can you find a word in a dictionary?

Best, worst and average case

Quicksort's average time complexity is O(n log n)

merge sort: O(n log n) always

Ever wondered why sorting your music library takes ages?

Hash table

Hash table lookup: O(1) average time complexity

Computational complexity of matrix multiplication

O(n³) naive matrix multiplication

Graph (abstract data type)

Time complexity of BFS and DFS: O(V + E)

Dijkstra's algorithm

Dijkstra's algorithm time complexity: O((V+E) log V)

O(n log n) is the lower bound for comparison-based sorting

Ever wonder why sorting can't be faster than a certain point?

amortized O(1) means

Why does a slow day at the bank not mean every transaction is slow?

Overlapping subproblems

Ever calculated a huge Fibonacci sequence by hand?

Master theorem (analysis of algorithms)

Master theorem solves T(n) = aT(n/b) + f(n) recurrences

P

P vs NP asks if every problem whose solution is quickly verifiable can also be quickly solved

P versus NP problem

P vs NP problem determines if problems verifiable in polynomial time are also solvable in polynomial time

Halting problem

Alan Turing proved the halting problem is undecidable

Kolmogorov complexity

Kolmogorov complexity is uncomputable

approximation algorithms guarantee: solution within factor α of optimal

Can we always find the best solution quickly?

Randomized algorithm

Randomized algorithms use random bits for expected polynomial time

the A* algorithm does: BFS with heuristic f(n) = g(n) + h(n)

How do GPS systems find the best route when driving?

consistent hashing does: minimizes remapping when nodes join/leave

How can we efficiently share resources without constant reorganization?

SGD with momentum escapes local minima better than vanilla SGD

Ever felt stuck on a hill, unable to find the way down?

the momentum term does: v_t = βv_{t-1} + ∇L, accumulates gradient direction

Momentum term accelerates convergence in the gradient direction

Adam combines momentum and RMSprop: adapts per-parameter learning rates

Ever wondered how a car adjusts its speed for smooth turns?

the β₁ and β₂ hyperparameters control in Adam

β₁ controls the exponential decay rate of the first moment estimates; β₂ controls the exponential decay rate of the second moment estimates in Adam optimizer

Adam has bias correction: divides by (1-β^t) in early steps

Why do we sometimes need to fix mistakes in computer decisions?

AdaGrad does: divides learning rate by sqrt of sum of squared gradients

How do we avoid overshooting in learning?

AdaGrad's learning rate decays to zero

Why does a car's speed drop when it goes uphill?

RMSprop fixes about AdaGrad: uses exponential moving average instead of sum

RMSprop uses an exponentially decaying average of squared gradients, unlike AdaGrad's cumulative sum

learning rate warmup does: starts small to avoid early training instability

Why does starting big in learning sometimes lead to chaos?

cosine annealing does: lr = lr_min + 0.5(lr_max - lr_min)(1 + cos(πt/T))

Ever wondered how a computer decides what's more important when sorting through tons of data?

gradient clipping does: caps gradient norm to prevent exploding gradients

How do deep learning networks avoid getting stuck or going haywire during training?

second-order methods (Newton's) converge faster but are expensive: O(n³) per step

Second-order methods converge faster due to quadratic convergence but are expensive due to O(n³) per iteration

Fisher information

Fisher information measures information about unknown parameters

natural gradient descent does: preconditions with inverse Fisher matrix

Ever wondered how to climb a steep mountain without measuring every step?

LAMB optimizer does: layer-wise adaptive learning rates for large batch training

LAMB optimizer adjusts learning rates layer-wise for large batch training

mixed precision training does: forward in FP16, accumulate gradients in FP32

Ever wished your phone's camera could take stunning photos even in low light?

gradient accumulation simulates larger batch sizes without more memory

Can you train a machine like you do with a computer?

gradient checkpointing trades: recomputes activations to save memory

Gradient checkpointing trades off computation time for memory savings by recomputing activations

Knowledge distillation

Knowledge distillation transfers knowledge from a large model to a smaller one without loss of validity

soft targets carry more information than hard labels: they encode class similarities

Why do some learning methods need to explore more than others?

label smoothing does: replaces one-hot [0,0,1,0] with [0.025, 0.025, 0.925, 0.025]

How can a computer learn without being told exactly what to do?

Project-based learning

Why do students sometimes struggle with complex topics?

data augmentation does for generalization: artificially expands training set

How can you teach a computer to see better?

mixup does: trains on convex combinations of pairs: x̃=λx_i+(1-λ)x_j

Ever wondered how to blend two colors perfectly?

cutmix does: replaces a patch of one image with a patch from another

Cutmix replaces a patch of one image with a patch from another image

the lottery ticket hypothesis says: sparse subnetworks can match full network performance

Can a small part of a puzzle fit perfectly into its place by chance?

structured pruning removes: entire filters or attention heads, not individual weights

Why do we sometimes remove parts of a neural network to improve its performance?

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

300-dim word2vec encodes: trained on word co-occurrence with skip-gram window

Ever wondered how computers understand words?

768-dim BERT embeddings capture: bidirectional context from masked language modeling

768-dim BERT embeddings capture bidirectional context from masked language modeling

1536-dim OpenAI text-embedding-3-large is used for: semantic search and RAG

Used for semantic search, RAG, and enhancing language models' understanding

384-dim all-MiniLM-L6-v2 optimizes: fast sentence similarity with 6 layers

All-MiniLM-L6-v2 optimizes fast sentence similarity with 6 layers

mean pooling does: averages all token embeddings to get a sentence embedding

How can we summarize a whole sentence's meaning with just one number?

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

mean pooling often outperforms [CLS] for sentence similarity tasks

Mean pooling captures overall sentence meaning better than [CLS] token embedding

Matryoshka embeddings are: trained to be useful at multiple truncated dimensions

Ever wondered how a simple doll can teach us about nested complexities?

Contrastive Language–Image Pre-training

CLIP embeds images and text into a shared space using contrastive learning

the embedding layer does: maps discrete token IDs to dense learned vectors

Embeddings convert token IDs to dense vectors for neural network processing

cosine similarity is preferred over dot product for normalized embeddings

Why do we need a special way to measure similarity in high-dimensional spaces?

P-value

A p-value < 0.05 means: if H₀ is true, this result has <5% probability

Binomial proportion confidence interval

Binomial proportion confidence interval estimates success probability

Effect size

Cohen's D benchmarks: 0.2 = small, 0.5 = medium, 0.8 = large effect

Central limit theorem

Central limit theorem states that sample means converge to normal distribution as sample size increases

Law of large numbers

Law of large numbers: X̄_ n → μ as n → ∞ with probability 1

the Bonferroni correction does: divides α by number of tests

Bonferroni correction adjusts significance level by dividing α by the number of tests

Resampling (statistics)

Bootstrapping samples with replacement to estimate distributions

Maximum a posteriori estimation

Maximum a posteriori (MAP) estimate maximizes the posterior density

Expectation–maximization algorithm

EM algorithm iteratively maximizes likelihood estimates with latent variables

Markov chain Monte Carlo

MCMC samples from complex posterior distributions

Metropolis–Hastings algorithm

Metropolis-Hastings algorithm samples from difficult distributions

Gibbs sampling

Gibbs sampling samples each variable conditioned on all others

Conjugate prior

Conjugate priors simplify Bayesian updating

Beta-binomial distribution

Beta distribution is conjugate to binomial likelihood

the multivariate Gaussian is parameterized by: mean vector μ and covariance matrix Σ

Ever wondered how weather patterns or stock market trends can be predicted with surprising accuracy?

Wishart distribution

Wishart distribution is a generalization of the gamma distribution to multiple dimensions

the Dirichlet distribution does: distribution over probability simplices

How do we predict the likelihood of various outcomes in uncertain situations?

L1 vs L2 regularization: L1 gives sparsity (feature selection), L2 gives small weights

L1 regularization: L1 = L2 + sparsity; L2 regularization: L2 = L1 + small weights

Adam vs SGD: Adam adapts per-parameter rates, SGD often generalizes better with tuning

Adam adjusts learning rates per-parameter, SGD generalizes better with tuning

Batch norm vs layer norm: BN across batch, LN across features

Batch norm (BN) normalizes across batch, layer norm (LN) normalizes across features; LN handles variable-length sequences

GRU vs LSTM: GRU uses 2 gates and is faster, LSTM uses 3 gates and captures longer dependencies

GRU: 2 gates, faster; LSTM: 3 gates, longer dependencies

Encoder vs decoder: encoder sees all tokens bidirectionally, decoder sees only past tokens

Encoder: Sees all tokens bidirectionally; Decoder: Sees only past tokens

Float16 vs bfloat16: bfloat16 has same exponent range as float32, less precision but more stable

bfloat16: same exponent range as float32, less precision but more stable

Bias vs variance: high bias = underfitting, high variance = overfitting

Can a perfect fit to past data predict future events?

Boosting (machine learning)

Boosting reduces bias in ML models

PCA vs t-SNE: PCA preserves global variance linearly, t-SNE preserves local structure nonlinearly

How can we teach computers to understand what we like?

Greedy vs dynamic programming: greedy makes locally optimal choices, DP considers all subproblems

Greedy: locally optimal choices; DP: considers all subproblems

BFS vs DFS: BFS finds shortest path in unweighted graphs, DFS uses less memory

BFS finds shortest path in unweighted graphs; DFS uses less memory

TCP vs UDP: TCP guarantees delivery order, UDP is faster but unreliable

TCP guarantees delivery order, UDP is faster but unreliable

SQL vs NoSQL: SQL enforces schema and ACID, NoSQL offers flexibility and horizontal scaling

Ever wondered how databases handle massive data across the globe?

REST vs GraphQL: REST has fixed endpoints, GraphQL lets clients request specific fields

REST uses fixed endpoints; GraphQL allows clients to request specific fields

Containers vs VMs: containers share the host kernel and are lighter, VMs have full OS isolation

Why can't you run a Windows app on a Mac?

Git merge vs rebase: merge preserves history as-is, rebase linearizes it

Ever wondered how to clean up messy code history?

GPTQ vs AWQ: GPTQ uses Hessian-based quantization, AWQ preserves activation-important weights

GPTQ applies Hessian-based quantization, AWQ retains weights crucial for activations

LoRA vs full fine-tuning: LoRA trains rank-r adapters (~0.1% params), full FT updates everything

LoRA trains rank-r adapters (~0.1% params), full FT updates everything

Top-k vs top-p sampling: top-k fixes candidate count, top-p fixes cumulative probability mass

Top-k sampling fixes candidate count; top-p sampling fixes cumulative probability mass

Greedy vs beam search decoding: greedy picks best token, beam maintains k candidates

Ever wondered why Google Search sometimes shows you the top results first?

BLEU vs ROUGE: BLEU measures precision of n-grams, ROUGE measures recall

BLEU measures precision of n-grams, ROUGE measures recall

List of algorithms

Cosine similarity measures the angle between vectors, not their magnitude

Euclidean geometry

Euclidean distance measures absolute position in space

to use F1 score: when classes are imbalanced and both FP and FN matter

Why might a sports team need a different score to judge their performance?

to use AUC-ROC: comparing classifiers across all thresholds

Why can't we always trust a yes-or-no answer?

Global Forest Change dataset

Global Forest Change dataset covers 2000-2024

Noise-induced hearing loss

Noise-induced hearing loss (NIHL) is a hearing impairment resulting from exposure to loud sound

to use random forests: when you want a strong baseline with minimal hyperparameter tuning

Why not always use the best single tree for predictions?

to use XGBoost: for tabular data where you want the best possible performance

Why do some people always seem to win at chess?

to use a CNN: for data with spatial structure like images or time series

Why can't we just feed all data into one big neural net?

to use an RNN/LSTM: for sequential data where order matters (mostly replaced by transformers)

Why do we remember stories better when they have a clear beginning, middle, and end?

Philosophy of language

Philosophy of language studies language's nature and its relationship with users and the world

to normalize features: when features have different scales and you use distance-based methods

Why do some things need to be adjusted to compare fairly?

to standardize: when you need zero mean and unit variance for gradient-based optimization

Why do we need to make data uniform before training a model?

Dimensionality reduction

Dimensionality reduction transforms high-dimensional data into low-dimensional space while preserving meaningful properties

to use log-transform: when data is right-skewed or spans multiple orders of magnitude

Why can't we just add numbers to compare incomes?

the dot product measures alignment: it equals |a||b|cos(θ)

Why do vectors sometimes "agree" with each other?

Principal component analysis

Eigenvectors point along maximum variance

the determinant tells you about volume scaling under a linear transformation

The determinant of a matrix representing a linear transformation indicates the factor by which volumes are scaled

orthogonal matrices preserve distances: O^T O = I means no stretching or squashing

How can you stretch or squash a square without changing its shape?

the trace equals the sum of eigenvalues: tr(A) = Σλ_i

How can a vector stay the same after a transformation?

Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

weight initialization matters: Xavier/He init keeps activation variance ≈ 1 across layers

Why does starting a video game with random settings affect gameplay?

Gradient

Gradient points uphill in the direction of steepest increase of f

Convex optimization

Convex functions have only one global minimum

non-convex loss landscapes are hard: many local minima and saddle points

Why do some hills have more tricky paths than others?

saddle points are more common than local minima in high dimensions

Why do some mountains have flat tops instead of peaks or valleys?

dropout works as regularization: it approximates an ensemble of subnetworks

Why does turning off neurons randomly help a brainy computer learn better?

temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform

How does adjusting T affect the certainty of choices?

log-probabilities are used instead of probabilities: avoids numerical underflow

Why can't we just add up tiny chances over time?

cross-entropy equals negative log-likelihood for classification

Why does knowing the wrong probability help us measure information loss?

sinusoidal position encoding works: each dimension has a different frequency

Sinusoidal position encoding assigns unique frequencies to each dimension, enabling the model to distinguish positions effectively

Load balancing (computing)

Load balancing distributes tasks efficiently across resources

database sharding does: splits data across machines by a partition key

Why can't you just split a huge library into smaller ones?

CAP theorem states: you can have at most 2 of consistency, availability, partition tolerance

Ever wondered why you can't always get the latest news instantly?

ACID

ACID guarantees data validity in transactions

eventual consistency means: all replicas converge to the same state given enough time

Ever wonder why your favorite online store always shows you the same price for a product, even if you see different prices elsewhere?

a message queue decouples: producer and consumer can operate at different speeds

Ever wondered why a kitchen blender and a coffee maker can work at different speeds?

Features new to Windows XP

Windows XP introduced connection pooling

a CDN does: caches content at edge locations close to users

Ever wondered why YouTube videos load instantly even in remote areas?

Domain Name System

DNS translates domain names to IP addresses

consistent hashing solves: minimizes key redistribution when servers are added/removed

Consistent hashing minimizes key redistribution when servers are added/removed

Design of the FAT file system

Write-ahead log (WAL) ensures crash recovery by logging changes before applying them

B-trees optimize: disk-based sorted data with O(log n) reads per query

B-trees optimize disk-based sorted data with O(log n) reads per query

LSM trees optimize: write-heavy workloads by buffering writes in memory

LSM trees buffer writes in memory for write-heavy workloads

Bloom filter

Bloom filters check if an element is possibly in a set with high probability, avoiding false negatives

Reinforcement learning from human feedback

RLHF optimizes a reward model trained on human preference pairs

DPO simplifies: removes the explicit reward model, trains directly on preferences

DeFi removes intermediaries like banks

constitutional AI does: model critiques and revises its own outputs using principles

Constitutional AI critiques and revises outputs using principles

RAG does: retrieves relevant documents before generating to reduce hallucination

RAG reduces AI hallucinations

Reasoning model

RLMs excel in logic, math, and programming tasks

Prompt engineering

The GenAI model learns tasks from examples in the prompt

instruction tuning does: fine-tunes on (instruction, response) pairs

Fine-tuning adapts pre-trained models to new tasks

BPE tokenization does: iteratively merges the most frequent byte pairs

BPE tokenizes text by merging frequent byte pairs

SentencePiece does differently from BPE: operates on raw text including whitespace

SentencePiece tokenizes text without pre-tokenization, preserving whitespace

Large language model

LLMs can generate, summarize, translate, and analyze text in many contexts

Glossary of poker terms

Context window limit refers to the maximum number of tokens a model can process at once

RoPE's advantage is: supports length extrapolation beyond training context length

RoPE (Relative Position Encoding) advantage: supports length extrapolation beyond training context length

ring attention does: distributes long sequences across multiple devices

"Attention Is All You Need" introduced the transformer architecture in 2017

Neural scaling law

Chinchilla scaling law: optimal model size scales linearly with compute budget

the compute-optimal training ratio is: roughly 20 tokens per parameter

Compute-optimal training ratio: roughly 20 tokens per parameter

Retrieval-augmented generation

RAG enables LLMs to access new information without retraining

function calling enables: LLMs can invoke external tools and APIs

LLMs can invoke external tools and APIs

the system prompt does: sets persistent behavior instructions for the conversation

Prompt engineering shapes GenAI outputs

a Triton @triton.jit decorator does: compiles a Python function into a GPU kernel

@triton.jit decorator compiles Python function into a GPU kernel

tl.load and tl.store do in Triton: read/write tensors from/to GPU global memory

`tl.load` reads tensors from GPU memory; `tl.store` writes tensors to GPU memory

BLOCK_SIZE means in Triton: the tile size each program instance processes

Block size refers to the minimal unit of data for block ciphers

tl.program_id(0) returns: the index of the current parallel block

tl.program_id(0) returns: the index of the current parallel block

tl.arange(0, BLOCK_SIZE) creates: a range of indices within the current block

Python's annual release cycle

tl.dot does in Triton: block-level matrix multiply using tensor cores

tl.dot performs block-level matrix multiplication using tensor cores in Triton

Triton auto-tunes BLOCK_SIZE: different sizes optimize for different hardware

Rate-distortion optimization balances video quality and file size

tl.where(mask, x, 0) does: conditional select to handle boundary conditions

`tl.where(mask, x, 0) = x if mask else 0`

to write a vector addition kernel in Triton: load blocks, add, store

```

to write a fused softmax kernel in Triton: load row, compute max, subtract, exp, sum, divide

Softmax function converts real numbers to a probability distribution

Parallel Thread Execution

PTX is an intermediate GPU instruction set used in Nvidia's CUDA

SASS is: the actual machine code that runs on NVIDIA GPU hardware

SASS: compiled machine code executing on NVIDIA GPU hardware

nvcc does: NVIDIA's CUDA compiler that produces PTX and SASS

NVCC compiles CUDA code into PTX and SASS

__syncthreads() does in CUDA: synchronizes all threads within a block

__syncthreads() synchronizes all threads within a block

List of gay characters in animation

AtomicAdd adds values to shared or global memory atomically

cooperative groups enable in CUDA: flexible thread synchronization patterns

Cooperative groups enable flexible thread synchronization patterns in CUDA

CPU cache

L1/L2 cache hierarchy reduces global memory latency

register pressure means: too many variables per thread reduces occupancy

Too many variables per thread reduces occupancy

loop unrolling does: trades code size for reduced loop overhead

Loop unrolling optimizes execution speed by reducing loop control instructions

instruction-level parallelism (ILP) achieves: multiple operations per clock cycle

ILP = average number of instructions per clock cycle

LU decomposition

LU decomposition factors a matrix as the product of a lower triangular matrix and an upper triangular matrix

Cholesky decomposition

Cholesky decomposition factors A = LLᵀ for symmetric positive definite matrices

QR decomposition

QR decomposition factors A = QR

Ordinary least squares

OLS minimizes squared differences

the condition number κ(A) measures: sensitivity of Ax=b to perturbations

Condition number κ(A) measures sensitivity of Ax=b to perturbations

ill-conditioned matrices cause numerical instability: small input changes → large output changes

Ill-conditioned matrices lead to numerical instability

iterative methods (CG, GMRES) do: solve Ax=b without explicitly inverting A

Iterative methods solve Ax=b without explicitly inverting A

Eigenvalues and eigenvectors

Eigenvectors are unchanged in direction by a linear transformation

the Gram-Schmidt process does: orthogonalizes a set of vectors

Gram-Schmidt orthogonalizes vectors in Rⁿ

Invertible matrix

Rank-nullity theorem: rank(A) + nullity(A) = n

Cayley–Hamilton theorem

Cayley-Hamilton theorem states every matrix satisfies its own characteristic equation

Moment generating function

Moment generating function uniquely determines a distribution

Characteristic function (probability theory)

Characteristic function φ(t) = E[e^(itX)] is the Fourier transform of the PDF

Chebyshev's inequality says: P(|X-μ| ≥ kσ) ≤ 1/k²

Chebyshev's inequality states P(|X-μ| ≥ kσ) ≤ 1/k²

Hoeffding's inequality

Hoeffding's inequality bounds tail probability for sums of bounded random variables

Chernoff bound

Chernoff bounds provide exponentially tight concentration inequalities

the union bound says: P(A∪B) ≤ P(A) + P(B)

The union bound states: P(A∪B) ≤ P(A) + P(B)

Local martingale

E[X_{n+1}|X_1,...,X_n] = X_n

the optional stopping theorem says about martingales and stopping times

Martingale: E[X_{n+1} | X_1, X_2, ..., X_n] = X_n

Sufficient statistic

Sufficiency captures all information about θ in the data

Minimum-variance unbiased estimator

MVUE achieves lower variance than any other unbiased estimator

Monte Carlo method

Delta method approximates variance of g(X) ≈ [g'(μ)]² Var(X)

message passing does in GNNs: each node aggregates features from its neighbors

Nodes in GNNs aggregate features from neighbors

GCN (Graph Convolutional Network) does: spectral convolution approximated by neighbor averaging

GNNs use pairwise message passing for node representation updates

GraphSAGE does: samples and aggregates a fixed-size neighborhood

GraphSAGE samples and aggregates a fixed-size neighborhood

GAT (Graph Attention Network) adds: learned attention weights between neighbors

GATs use learned attention weights between neighbors

the over-smoothing problem is in GNNs: deep GNNs make all node features converge

Over-smoothing in GNNs causes node features to converge

Graph neural network

Graph pooling reduces graphs to single vectors for graph-level prediction

Functor

Functors map between categories preserving composition and identity

Monad (functional programming)

Monads are a type constructor with two operations: return and bind

Curry–Howard correspondence

Proofs are programs, types are propositions

Yoneda lemma

The Yoneda lemma embeds a locally small category into a functor category

Lambda calculus

Lambda calculus represents data using only functions

Nyquist–Shannon sampling theorem

Sample at ≥ 2× the highest frequency to avoid aliasing

aliasing is: high frequencies masquerading as low frequencies due to undersampling

Aliasing occurs when sampling frequency is less than twice the highest frequency component (f_s < 2f_max)

a low-pass filter does: removes frequencies above a cutoff, keeps slow-varying signal

Low-pass filter removes frequencies above a cutoff

AI content watermarking

AI content watermarking embeds imperceptible signals

mel-frequency cepstral coefficients (MFCCs) capture: speech features on a perceptual scale

Mel-frequency cepstral coefficients (MFCCs) represent sound on a perceptual scale

the mel scale is: a nonlinear frequency scale that models human pitch perception

Mel scale: a nonlinear frequency scale modeling human pitch perception

Short-time Fourier transform

STFT divides a signal into shorter segments for analysis

wavelets provide over Fourier: both time and frequency localization

Wavelets provide both time and frequency localization, unlike Fourier transforms which offer only frequency localization

Physics-informed neural networks

Neural ODEs model continuous-time dynamics with a neural network as the derivative

Euler method

Euler method approximates ODE solution with y_{n+1} = y_n + h·f(y_n)

Finite element method

Runge-Kutta method improves Euler by providing higher-order accuracy with k₁,k₂,k₃,k₄

Lyapunov exponents measure: rate of divergence of nearby trajectories in a dynamical system

Lyapunov exponent quantifies divergence rate: e^(λt)|δ₀| ≈ |δ(t)|

Dynamical system

A fixed point is where dx/dt = 0

Chaos theory

Butterfly effect demonstrates sensitive dependence on initial conditions

Hamming distance

Hamming distance measures the number of differing positions between two strings

a parity check bit does: detects single-bit errors by making total 1s even/odd

Parity bit makes total 1s even/odd

Error detection and correction

Reed-Solomon codes correct burst errors in data transmission and storage

Error correction code

Turbo codes achieve near-Shannon-limit error correction with iterative decoding

Low-density parity-check code

LDPC codes revolutionized coding theory with significant performance improvements

Quantum superposition

A qubit exists in superposition of |0⟩ and |1⟩

quantum entanglement means: measuring one qubit instantly determines the other's state

Quantum entanglement instantly links particles' states

Quantum logic gate

Hadamard gate puts a qubit into equal superposition of |0⟩ and |1⟩

Shor's algorithm

Shor's algorithm factors integers in polynomial time on a quantum computer

Quantum computing

Quantum computers can solve certain problems exponentially faster than classical computers

Dependent type

Dependent types depend on values, not just types

Sum of angles of a triangle

Sum of angles in a triangle equals 180 degrees

Product type

Product types combine two types into a single structure

parametric polymorphism does: a function works for any type T without knowing what T is

Parametric polymorphism allows code to work with any type T

the Y combinator does: enables recursion in languages without named functions

Y Combinator launched over 5,000 companies

continuation-passing style (CPS) does: makes control flow explicit via callbacks

Continuations pass control explicitly via callbacks

Memory hierarchy

Memory hierarchy levels: registers → L1 → L2 → L3 → RAM → SSD → HDD (each ~10× slower)

branch prediction does: guesses which way an if-statement will go to keep the pipeline full

Branch predictors guess the outcome of conditional jumps to keep the pipeline full

Single instruction, multiple data

SIMD processes multiple data elements simultaneously

Delay-line memory

CPU speed grows faster than memory speed

HBM (High Bandwidth Memory) provides: stacked DRAM with much higher bandwidth than DDR

HBM provides stacked DRAM with much higher bandwidth than DDR

NVLink provides: high-bandwidth GPU-to-GPU interconnect (900 GB/s on H100)

NVLink provides: high-bandwidth GPU-to-GPU interconnect (900 GB/s on H100)

PCIe bandwidth limits: ~64 GB/s for PCIe 5.0 x16, bottleneck for CPU-GPU transfer

PCIe 5.0 x16 bandwidth limit ~64 GB/s, bottleneck for CPU-GPU transfer

Von Neumann architecture

CPU must fetch both data and instructions from memory

Closed set

A closed set contains all its boundary points

Sigma-additive set function

A σ-additive set function maintains additivity for countably infinite sets

Lebesgue integral

Lebesgue integral generalizes Riemann by integrating over more complex domains

Lebesgue measure

Lebesgue measure assigns zero to countable sets

Normed vector space

A Banach space is a complete normed vector space

Inner product space

Inner product space generalizes Euclidean geometry

Riesz representation theorem

Riesz representation theorem connects Hilbert spaces with continuous dual spaces

Spectral theorem

Spectral theorem applies to normal operators on Hilbert spaces

Manifold

A manifold locally resembles Rⁿ

Tangent space

Tangent space at a point represents all possible velocity vectors

Riemannian manifold

Riemannian manifolds generalize Euclidean space concepts like distance and curvature

Geodesics on an ellipsoid

Geodesics are the shortest paths on a curved surface

Curvature

Curvature measures the angular rate of change of the direction of the tangent line per unit distance along the curve

parallel transport does: moves vectors along a curve while preserving their properties

Parallel transport preserves vector properties along curves

Nash equilibrium

Nash equilibrium: no unilateral gain

a dominant strategy is: optimal regardless of what other players do

A dominant strategy maximizes payoff irrespective of opponents' actions

Prisoner's dilemma

Prisoner's dilemma illustrates how individual rationality can lead to collectively worse outcomes

Zero-sum game

Zero-sum game: one player's gain equals another's loss

the minimax theorem says: in zero-sum games, there's a saddle point strategy

Minimax theorem guarantees a saddle point strategy in zero-sum games

mechanism design does: designs rules so rational agents produce desired outcomes

Mechanism design constructs rules for desired outcomes

the revelation principle says: any mechanism can be converted to a truthful one

Tragedy of the commons leads to resource depletion

Causal model

Causal models use DAGs to represent causal relationships

the do-calculus does: computes interventional probabilities from observational data

Do-calculus computes interventional probabilities from observational data

an instrumental variable does: isolates causal effect when you can't randomize

Instrumental variables (IV) isolate causal effects when randomization isn't possible

Controlling for a variable

Confounders influence both treatment and outcome

the back-door criterion identifies: sufficient adjustment sets for causal estimation

Causal models use formal notation like DAGs for causal inference

Race and intelligence

IQ test performance differences between racial groups have decreased over time

Regression discontinuity design

RDD uses a sharp threshold for treatment assignment

rejection sampling does: samples from target by accepting/rejecting proposals

Rejection sampling generates observations from a target distribution

importance sampling does: reweights samples from proposal to estimate target expectation

Importance sampling estimates target expectations using samples from a different distribution

Reparameterization trick

Reparameterization trick enables differentiable sampling for VAE training

Langevin dynamics does: adds noise to gradient descent to sample from a distribution

Langevin dynamics uses stochastic differential equations

score matching does: learns the gradient of the log-density without normalizing

Propensity score matching reduces bias in treatment effect estimates

denoising score matching does: learns to denoise, which equals learning the score

Propensity score matching (PSM) reduces bias in treatment effect estimates

Stable Diffusion

Stable Diffusion generates images from text descriptions

Diffusion model

q(x_t|x_{t-1}) adds Gaussian noise at each step

the reverse process learns: p_θ(x_{t-1}|x_t)

Ancestral reconstruction extrapolates characteristics back in time

classifier-free guidance does: interpolates between conditional and unconditional generation

"Classifies samples as either conditioned or unconditioned, guiding generation towards desired outcomes."

DDPM stands for: Denoising Diffusion Probabilistic Model

DDPM stands for Denoising Diffusion Probabilistic Model

DDIM does: deterministic sampling for faster generation with fewer steps

DDIM accelerates image generation by deterministically sampling intermediate steps

BPE tokenization does: iteratively merges the most frequent adjacent byte pairs

BPE tokenization merges frequent adjacent byte pairs iteratively

WordPiece tokenization does: similar to BPE but uses likelihood instead of frequency

WordPiece tokenization splits words into subwords based on token likelihood rather than frequency

Unigram tokenization does: starts with large vocabulary and prunes using EM

Unigram tokenization starts with a large vocabulary and prunes using EM

subword tokenization solves: handles rare words by breaking into known pieces

Subword tokenization solves rare word handling by breaking into known pieces

the vocabulary size matters: larger vocab = shorter sequences but more parameters

Larger vocab leads to shorter sequences but more parameters

the tokenizer's special tokens do: [CLS], [SEP], [PAD], [MASK] have specific roles

[CLS] marks the start of input, [SEP] denotes separation, [PAD] fills space, [MASK] hides words for prediction

Phi coefficient

Matthews correlation coefficient (MCC) measures balanced metric even with class imbalance

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

Cross-entropy loss penalizes confident wrong predictions more heavily

calibration means: a model predicting 80% should be correct 80% of the time

A model predicting 80% should be correct 80% of the time

expected calibration error (ECE) measures: gap between confidence and accuracy

Expected Calibration Error (ECE) measures the gap between predicted confidence levels and actual accuracy

Brier score

Brier score measures mean squared error of probability predictions

NDCG measures: ranking quality with graded relevance scores

NDCG measures ranking quality with graded relevance scores

MAP (mean average precision) measures: area under the precision-recall curve averaged across queries

MAP measures the area under the precision-recall curve averaged across queries

word error rate (WER) measures: edit distance between predicted and reference transcriptions

WER measures the percentage of errors in transcription

Fréchet inception distance

Fréchet inception distance (FID) compares image distributions

IS (Inception Score) measures: diversity and quality of generated images

The Inception Score (IS) measures diversity and quality of generated images

Arm architecture family

ARM processors are the most widely used family of instruction set architectures

torch.compile does in PyTorch 2.0: traces and optimizes the computation graph

torch.compile optimizes computation graph by tracing and compiling it for efficiency

XLA does for TensorFlow/JAX: compiles computation graphs for TPU/GPU execution

XLA accelerates TensorFlow models

operator fusion does at the compiler level: merges adjacent ops to reduce memory traffic

Compiler optimizations merge adjacent operations to reduce memory traffic

Tracing

Tracing records operations, scripting parses Python

MLIR (software)

MLIR was publicly released as part of LLVM in 2019

the ONNX format does: standardizes model representation for cross-framework deployment

ONNX standardizes machine learning model representation

TensorRT does: NVIDIA's inference optimizer that quantizes and fuses operations

TensorRT optimizes deep learning inference by quantizing and fusing operations for NVIDIA GPUs

Outline of databases

Ever wonder how your online bank keeps your money safe during transactions?

Travelling salesman problem

Can we always find the shortest path visiting all cities?

General-purpose computing on graphics processing units

Did you know your computer can do more than just compute numbers?

Divergence

How does air flow change volume?

Hahn–Banach theorem

Can you always stretch a stretchable fabric to cover a larger table?

Receiver operating characteristic

Ever wondered how doctors decide if a test really finds cancer?

Glossary of engineering: A–L

Can you hear colors?

Loop nest optimization

Can speeding up your computer make tasks quicker?

Glossary of artificial intelligence

Can computers learn to mimic human brain functions?

List of free and open-source software packages

Can you run AI on your phone?

Filtration (mathematics)

How can we predict future events with uncertainty?

Bayesian inference

Ever wondered how doctors update diagnoses as new symptoms arise?

Viterbi semiring

How do we find the best path in a maze of choices?

Entropy in thermodynamics and information theory

Ever wondered how computers decide what's important in a message?

Convolutional neural network

Can a neural network learn too well?

Matrix (mathematics)

Ever wondered how computers can predict your favorite songs?

Machine learning

Can we teach computers to learn like humans?

Quantum key distribution

Can secret messages be intercepted without anyone noticing?

Open set

How can we understand closeness without measuring distance?

Markov's inequality

Ever wondered how math can predict unlikely events?

Evaluation of machine translation

Can we truly measure how good a machine translation is?

Torus

Ever wondered how a doughnut's shape is mathematically understood?

Glossary of probability and statistics

Ever wonder why your average score on quizzes improves with more quizzes?

Generative adversarial network

Ever wondered how computers can create new images that look real?