Reparameterization trick

Reparameterization trick enables differentiable sampling for VAE training

The reparameterization trick allows for gradients to be computed through random variables, which is crucial for optimizing models with stochastic elements. It was developed in the 1980s and later applied to variational autoencoders in 2013.

Example

In a VAE, the trick enables the computation of gradients for the latent variable sampling process, allowing for efficient training of the model.

Remember this

This technique is essential for training VAEs as it enables the use of stochastic gradient descent and reduces the variance of estimators.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

LAMB optimizer does: layer-wise adaptive learning rates for large batch training

LAMB optimizer adjusts learning rates layer-wise for large batch training

importance sampling does: reweights samples from proposal to estimate target expectation

Importance sampling estimates target expectations using samples from a different distribution

to standardize: when you need zero mean and unit variance for gradient-based optimization

Why do we need to make data uniform before training a model?

MoE models have more parameters but similar compute cost

MoE models distribute parameters across k experts, reducing active experts' compute cost

data augmentation does for generalization: artificially expands training set

How can you teach a computer to see better?

Swipe through 100 ML concepts daily

Open Pocket Polymath