Loading Curriculum...

Artificial Intelligence (AI) Reinforcement Learning

DeepRL

Lesson 9/9

Reinforcement Learning

Deep Reinforcement Learning (Deep RL)

Deep RL is RL with function approximation (usually neural networks). It’s powerful—but can be unstable. This lesson gives you the mental model: what changes when tables become networks, and why replay/targets/advantages exist.

Deep Reinforcement Learning

The big picture

Deep RL combines three moving parts: (1) function approximation, (2) bootstrapping, (3) exploration. Stability tricks exist to control their interactions.

Value-based: learn \(Q(s,a)\), act greedily. (DQN-style)
Policy-based: learn \(\pi(a|s)\) directly. (REINFORCE-style)
Actor-Critic: learn both, using a critic baseline/advantage for low-variance gradients.
Stability: replay buffers, target networks, normalization, entropy, clipping (PPO) and more.

Deep Reinforcement Learning

Why Deep RL can be unstable

With function approximation, small update errors can generalize across many states. Bootstrapping (targets that depend on the model) can amplify mistakes.

TD target: y = r + γ · maxₐ' Q(s',a') Update: Q(s,a) ← Q(s,a) + α · (y − Q(s,a))

Replay randomizes data to reduce correlation. Target nets slow down target drift.Advantages/baselines reduce policy-gradient variance.

Interactive sandbox

Same environment, three learning styles

This is a 2-action bandit (no state). The goal is to pick action 1 more often because it has a higher average reward. Use this to feel the difference between Q-learning, REINFORCE, and Actor-Critic updates.

Status

Steps

Avg reward

Action counts

a0: 0 | a1: 0

Policy / values

π(a)

π(a0)=0.5 | π(a1)=0.5

Q (value-based)

Q(a0)=0 | Q(a1)=0

V / baseline (critic)

V≈0 | b≈0

Last update

Press Step to start.

Auto speed

6/s

γ (shown for notation)

The bandit has no next state, so γ doesn’t affect targets here—this slider is just to keep the symbols consistent with RL formulas.

Algorithm knobs

Policy lr (απ)

0.15

Q lr (αQ)

0.25

Only used in Q-learning mode.

ε exploration

0.2

Only used in Q-learning mode.

Critic lr (αV)

0.2

Only used in Actor-Critic mode.

Baseline (variance reduction)

In REINFORCE/Actor-Critic, subtracting a baseline turns reward into an advantage signal, reducing gradient variance.

Baseline lr (β)

0.1

Baseline value

b ≈ 0

In this sandbox, baseline is a moving average of rewards.

Connection to real Deep RL: replace Q/logits with neural networks, replace the bandit with an MDP, and the exact same update structure appears—just with (a) bootstrapped targets and (b) far more correlated data.

Deep Reinforcement Learning

Where to go next in this topic

Use these ideas as a map while reading the other lessons in Reinforcement Learning.

Bellman backups

Deep RL builds on Bellman expectation/optimality (targets and bootstrapping).

DQN

Value-based deep RL: replay buffer + target network + ε-greedy exploration.

Policy Gradient, Actor-Critic, PPO

Policy optimization deep RL: advantages, entropy, clipping/trust regions, and stability via constrained updates.

Loading Lab...

Reinforcement Learning

Deep Reinforcement Learning (Deep RL)

Deep Reinforcement Learning

The big picture

Deep RL combines three moving parts: (1) function approximation, (2) bootstrapping, (3) exploration. Stability tricks exist to control their interactions.

Value-based: learn \(Q(s,a)\), act greedily. (DQN-style)
Policy-based: learn \(\pi(a|s)\) directly. (REINFORCE-style)
Actor-Critic: learn both, using a critic baseline/advantage for low-variance gradients.
Stability: replay buffers, target networks, normalization, entropy, clipping (PPO) and more.

Deep Reinforcement Learning

Why Deep RL can be unstable

With function approximation, small update errors can generalize across many states. Bootstrapping (targets that depend on the model) can amplify mistakes.

TD target: y = r + γ · maxₐ' Q(s',a') Update: Q(s,a) ← Q(s,a) + α · (y − Q(s,a))

Replay randomizes data to reduce correlation. Target nets slow down target drift.Advantages/baselines reduce policy-gradient variance.

Interactive sandbox

Same environment, three learning styles

Status

Steps

Avg reward

Action counts

a0: 0 | a1: 0

Policy / values

π(a)

π(a0)=0.5 | π(a1)=0.5

Q (value-based)

Q(a0)=0 | Q(a1)=0

V / baseline (critic)

V≈0 | b≈0

Last update

Press Step to start.

Auto speed

6/s

γ (shown for notation)

The bandit has no next state, so γ doesn’t affect targets here—this slider is just to keep the symbols consistent with RL formulas.

Algorithm knobs

Policy lr (απ)

0.15

Q lr (αQ)

0.25

Only used in Q-learning mode.

ε exploration

0.2

Only used in Q-learning mode.

Critic lr (αV)

0.2

Only used in Actor-Critic mode.

Baseline (variance reduction)

In REINFORCE/Actor-Critic, subtracting a baseline turns reward into an advantage signal, reducing gradient variance.

Baseline lr (β)

0.1

Baseline value

b ≈ 0

In this sandbox, baseline is a moving average of rewards.

Deep Reinforcement Learning

Where to go next in this topic

Use these ideas as a map while reading the other lessons in Reinforcement Learning.

Bellman backups

Deep RL builds on Bellman expectation/optimality (targets and bootstrapping).

DQN

Value-based deep RL: replay buffer + target network + ε-greedy exploration.

Policy Gradient, Actor-Critic, PPO

Policy optimization deep RL: advantages, entropy, clipping/trust regions, and stability via constrained updates.