RLHF optimizes a reward model trained on human preference pairs
Reinforcement learning from human feedback
RLHF optimizes a reward model trained on human preference pairs
Reinforcement learning from human feedback (RLHF) is a technique that aligns an intelligent agent with human preferences by training a reward model. This reward model is initially trained in a supervised manner to predict the quality of responses based on human rankings. Once trained, it serves as a reward function to guide the optimization of an agent's policy.
Example
In natural language processing, RLHF can be used to train conversational agents by having human annotators rank responses, and then using those rankings to train a reward model that helps improve the agent's conversation skills.
Remember this
Understanding RLHF is crucial for developing AI systems that better align with human values and preferences.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
DPO simplifies: removes the explicit reward model, trains directly on preferences
DeFi removes intermediaries like banks
MoE models have more parameters but similar compute cost
MoE models distribute parameters across k experts, reducing active experts' compute cost
Write the Bellman equation for reinforcement learning
Predicting tomorrow's weather with today's clues
score matching does: learns the gradient of the log-density without normalizing
Propensity score matching reduces bias in treatment effect estimates
batch size affects generalization: larger batches find sharper minima
Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates
soft targets carry more information than hard labels: they encode class similarities
Why do some learning methods need to explore more than others?
Swipe through 100 ML concepts daily
Open Pocket Polymath