Loading Curriculum...

Reinforcement Learning

Deep Q-Network (DQN)

DQN replaces the tabular Q-table with a function approximator (a “Q-network”), then stabilizes learning with experience replay and a target network. This page simulates that loop step-by-step.

Core idea

DQN is just Q-learning + a Q-network, plus two stability tricks.

Q(s,a) ← Q(s,a) + α · ( y − Q(s,a) ) y = r + γ · maxₐ' Q_target(s', a') (if terminal: y = r)

Replay buffer: learn from random past transitions to break correlations.
Target network: keep a delayed copy of the network to make the target \(y\) less volatile.

Simulation controls

Play, step, and tune hyperparameters.

Speed

8/s

Discount γ

0.95

Exploration ε

0.25

Learning rate α

0.2

Batch

Replay cap

200

Train every

1 step(s)

Target sync

25 grad steps

Environment

1D corridor

Start at state 0. Each move costs -1. Reaching goal state 5 gives +10 then terminates.

Episode

Steps

Avg return

π: ←

π: →

π: ←

s5G

π: →

Online network Q

Q(←) 0.017 | Q(→) 0.011

Q(←) 0.009 | Q(→) -0.013

Q(←) -0.019 | Q(→) -0.018

Q(←) -0.004 | Q(→) 0.004

Q(←) 0.02 | Q(→) 0.02

Q(←) -0.002 | Q(→) 0.007

Target network Q̂

Q̂(←) 0.018 | Q̂(→) -0.008

Q̂(←) -0.011 | Q̂(→) -0.02

Q̂(←) -0.015 | Q̂(→) -0.002

Q̂(←) 0.015 | Q̂(→) 0.019

Q̂(←) 0.012 | Q̂(→) 0.012

Q̂(←) -0.018 | Q̂(→) -0.012

Target updates are delayed: every 25 gradient steps, copy online weights into the target network.

One update, explained

TD target and TD error

Replay size

Replay buffer (latest)

No transitions yet. Press Step.

Math panel

Press Step to generate the first transition.

Training may be skipped until the replay buffer is large enough.

Important: In real DQN, the Q-network is a deep neural net and the update is done via backprop. This demo uses a one-hot linear model so you can clearly see what DQN’s data flow is doing.

Loading Lab...

DeepQNetwork

DeepQNetwork

Deep Q-Network (DQN)

Core idea

Simulation controls

1D corridor

TD target and TD error

Deep Q-Network (DQN)

Core idea

Simulation controls

1D corridor

TD target and TD error