Loading Curriculum...

Artificial Intelligence (AI) Reinforcement Learning

BellmanEqualtion

Lesson 2/9

Bellman Equation

The Bellman equation is the core “backup” identity behind dynamic programming and modern reinforcement learning. In this simulation, you’ll watch values update step-by-step as the equation propagates information through an MDP.

Bellman optimality equation

V(s) = maxₐ Σₛ' P(s'|s,a) · [ R(s,a,s') + γ V(s') ]

Read it as: value = immediate reward + discounted next value, averaged over what happens next. The only difference is whether you average over a policy (expectation) or take the best action (optimality).

Simulation controls

Value iteration (synchronous backups)

Iteration

Discount γ

0.9

Lower γ makes the agent more “short-sighted”; higher γ propagates terminal rewards farther back.

Auto speed

6/s

The MDP (a tiny corridor)

A → B → C → D → T

Each step costs -1. From D, going right reaches terminal with reward +10. Value iteration uses the Bellman equation to push that information backward.

Tip: In optimality mode, a “max” chooses the best action. You should see values propagate leftward quickly because “go right” becomes optimal everywhere once the terminal payoff dominates.

One backup, explained

Updating state B

New value

-1

Compute Q(left)

1 · ( -1 + 0.9·V(A)=0.9·0 ) = -1

Q(left) = -1

Compute Q(right)

1 · ( -1 + 0.9·V(C)=0.9·0 ) = -1

Q(right) = -1

Combine into V(B)

V(B) = max(Q(left), Q(right)) = max(-1, -1) = -1

Greedy action is →.

What to notice

Bootstrapping: values update using other values (the \(γV(s')\) term), so information flows through the state graph over iterations.
Expectation vs optimality: switch modes to see how the backup changes from “average under π” to “take the best action.”
γ controls horizon: with smaller γ, terminal reward matters less far away; with larger γ it propagates farther back.
Q-values are just intermediate: they make the max/average explicit, and are what Q-learning-style methods learn directly.

Loading Lab...