Reinforcement learning, or RL, is a way to learn decisions from actions and their consequences. Instead of receiving a correct answer for every situation, a learner uses rewards to improve a strategy over time.
The idea sounds familiar if you have ever accidentally rewarded a cat for shouting at a cupboard. The mathematical version is more precise, but the household lesson is surprisingly relevant: be careful what behavior your reward encourages.
Table of Contents
- What is reinforcement learning?
- The learning loop
- The key concepts
- The main algorithm families
- Applications and examples
- How RL differs from other approaches
- The difficult parts
- What to ask about an RL system
- FAQ
What Is Reinforcement Learning?
An RL agent learns a policy: a rule for choosing actions. The objective is usually to maximize expected cumulative reward, rather than chase only the next positive score. OpenAI’s Spinning Up introduction provides the formal vocabulary behind this idea.
Imagine a little delivery robot in a simulated building. It can move, wait or turn. Successful deliveries earn a reward; wasting time or colliding with obstacles can carry a penalty. Training aims to find a strategy that does well across the task, not just one that looks busy.
This is an illustration, not a recipe for deploying an actual robot. The reward, available observations, allowed actions and safety boundaries would all need careful design.
How Reinforcement Learning Works: The Loop
- The agent receives an observation.
- Its policy chooses an action.
- The environment changes, producing a new observation and a reward.
- The training algorithm uses experience to update what the agent has learned.
- The process continues, with evaluation to check whether performance improves.
Updates need not happen after every individual action. Training may use batches or revisit stored experience. Nor is improvement guaranteed: repeatedly doing a badly specified task can make an agent very good at the wrong thing.
Exploration and Exploitation
Exploration means trying alternatives to learn about them. Exploitation means using what already appears to work. A learner that never tries another route may miss a shortcut; one that constantly wanders may never deliver the parcel.
Epsilon-greedy is one simple approach: sometimes choose randomly, otherwise use the highest-valued action. It is not the rule every RL system uses. Other methods encourage exploration through a stochastic policy, uncertainty estimates or additional objectives.
Immediate Rewards and Future Returns
A discount factor can reduce the weight of rewards further in the future. Other formulations use undiscounted finite episodes or average reward. The general issue is how to value consequences over time, not that every agent must prefer an immediate prize.
In our delivery example, moving toward a charging station may look unproductive now but make the next delivery possible. That is the kind of trade-off a cumulative objective can represent.
Agents, Environments, Policies and Values
- Agent: the decision-making learner.
- Environment: the system it interacts with.
- State: the underlying situation.
- Observation: the information the agent receives, which may be incomplete.
- Policy: how actions are selected, sometimes probabilistically.
- Value: an estimate of future return from a state or a state-action pair.
A robot camera cannot reveal everything behind a closed door. That is why state and observation should not be treated as synonyms in every problem. Some decisions require a history of observations, rather than only the latest frame.
The Main Algorithm Families
Q-Learning and Deep Q-Networks
Q-learning estimates the value of taking an action in a state and then following an optimal policy. Small problems can use a table; a Deep Q-Network uses a neural network to approximate those values.
The 2013 Atari DQN paper tested seven games, reporting performance above human experts on three. It should not be confused with later, larger Atari evaluations.
Policy Gradients and Actor-Critic Methods
Policy-gradient methods adjust the policy directly. Actor-critic methods combine a policy—the actor—with value estimates used to guide learning—the critic. PPO is a prominent example of policy optimization, but choosing it does not remove the need for tuning and evaluation.
Model-Based Learning
A model-based method uses a representation of how the environment changes to support planning. That model may be supplied or learned. Model-free methods learn policies or values without requiring that kind of explicit dynamics model for planning.
DeepMind’s comparison of AlphaZero and MuZero makes the distinction concrete. AlphaZero used known game rules during search. MuZero learned a model of quantities relevant to planning; it did not need to reconstruct every visual detail of the environment.
Where RL Has Been Useful
Games and Control
Games offer repeatable environments where an agent can collect experience without breaking physical equipment. Their clear rules and outcomes make them useful testing grounds, though success in a game does not guarantee success in a messy real-world task.
Robotics and industrial control are important application areas. Simulation can help collect experience, but differences between a simulator and actual hardware remain a challenge. RL is one approach among several, not the sole method behind modern robots.
For a concrete control example, DeepMind’s 2018 cooling report describes automated action selection with safety checks, operator oversight and the ability to revert to conventional control. It reported average cooling-efficiency improvements around 30% in that deployment. That is not a claim of cutting an entire data centre’s electricity consumption by 30%.
Language Models and Human Feedback
The 2022 InstructGPT paper describes a process involving supervised demonstrations, human comparisons, a learned reward model and reinforcement learning. The aim was to improve how well responses followed users’ intentions.
This is a classic RLHF pattern. It is not the only way to train a helpful assistant. Direct Preference Optimization, introduced in 2023, offers a different way to learn from preference data without a separate conventional RL training loop.
Neither method guarantees truth or safety. Preferences and reward signals are imperfect, and language-model training usually involves several stages. Using an assistant also does not necessarily mean it is updating its model weights during your conversation.
Recommendations
Recommendation problems can be framed as sequential decisions, but a recommendation feed is not automatically an RL system. Bandit methods, supervised ranking and other approaches can also be involved. Claims about a particular service need evidence about that service.
RL Can Learn From Existing Data Too
Supervised learning commonly learns from target outputs; unsupervised learning seeks structure without those targets. RL focuses on decisions and returns. These approaches can be combined rather than placed in sealed boxes.
Crucially, RL does not always generate fresh experience while training. Offline reinforcement learning learns from previously collected interactions without additional online data collection.
That creates a difficult question: how reliable are the estimates for actions poorly represented in the existing data? A dataset can be large and still tell you little about the decision you want to make.
The Difficult Parts
Reward Hacking
In OpenAI’s CoastRunners example from 2016, an agent repeatedly collected targets to score points rather than completing the race. It was exploiting the difference between the scoring rule and the intended task.
Our imaginary delivery robot could make a similar mistake if we rewarded “parcel picked up” generously but forgot to require arrival. We might receive an excellent pickup specialist and a building full of undelivered boxes.
Data, Credit and Stability
Some tasks require enormous amounts of experience. Delayed rewards make it difficult to determine which earlier decisions deserve credit. Training can also be sensitive to settings and random seeds, so one impressive run is not enough to establish reliability.
A reward curve rising during training is encouraging. It does not by itself show that the agent will handle new situations, avoid hidden costs or respect constraints omitted from the reward.
What to Ask About an RL System
Before being impressed by the label, ask what earns reward, where the experience comes from, how the policy is evaluated and what prevents unsafe actions. Also ask whether a simpler method would solve the problem.
More training is not a substitute for a useful objective. If the cat has learned that shouting opens the cupboard, extending the experiment may primarily improve its vocal stamina.
FAQ
Does reinforcement learning require a neural network?
No. Tabular and other non-neural methods are useful in suitable settings. Neural networks help approximate policies or values in large problems; continuous actions alone do not make deep learning mandatory.
Does RL need live trial and error?
Not always. Offline RL can learn from recorded interactions. The experience still concerns actions and outcomes, but it need not be collected by the learner during training.
Is RLHF the same as all assistant training?
No. It is one family of methods involving human feedback. Demonstrations, preference optimization and other training stages can also shape behavior.
How long does training take?
There is no universal duration. It depends on the problem, algorithm, available data, simulator or hardware speed, and the standard of performance required.
Does maximizing reward mean the system understands our goal?
Not necessarily. A numerical objective can leave out things people care about. Testing the resulting behavior matters as much as watching the score improve.
