Loading Curriculum...

Artificial Intelligence (AI) Reinforcement Learning

MarkovDecisionProcess

Lesson 1/9

Reinforcement Learning

Markov Decision Process (MDP)

An MDP is the mathematical model behind most reinforcement learning: states, actions, transition probabilities, and rewards. The “Markov” assumption says the future depends only on the current state and action.

Definition

An MDP is a 5-tuple (S, A, P, R, γ).

S = set of states A = set of actions P(s'|s,a) = transition probabilities R(s,a,s') = reward function γ ∈ [0,1) = discount factor

In this page we’ll use a tiny MDP with three non-terminal states (A,B,C) and one terminal state. You can change a transition probability, set a policy, roll out trajectories, and run policy evaluation.

Bellman expectation (policy evaluation)

Given a fixed policy π, you can compute V^π by repeatedly applying the Bellman backup.

V^π(s) = Σₐ π(a|s) Σₛ' P(s'|s,a) [ R(s,a,s') + γ V^π(s') ]

iterations: 0

Interactive MDP

Roll out a trajectory

Press Step to sample action from π and then sample next state from P(·|s,a). Terminal ends the episode; the next Step starts a new episode at A.

●

V = 0

Has actions: left/right

V = 0

Has actions: left/right

V = 0

Has actions: left/right

Terminal

V = 0

Terminal: no actions, V=0

Episode

Steps

Episode return

Avg return (recent)

Last transition

Press Step to sample.

Policy π(a|s)

State A

left 0.5 · right 0.5

State B

left 0.5 · right 0.5

State C

left 0.5 · right 0.5

Edit a transition probability

Change how “slippery” action left in state A is: it either stays in A or accidentally moves to B.

P(A | A,left)

0.7

A,left: → A with p=0.7, r=-1 → B with p=0.3, r=-1

Discount γ

0.9

Auto speed

5/s

MDP intuition: a policy π turns an MDP into a Markov reward process. The Bellman expectation backup above is simply “expected immediate reward + discounted expected next value” under that policy.

Try these experiments

Small changes here explain big behaviors in real RL.

Set π(right|C)=1 to reach terminal quickly, then press “Policy eval step” repeatedly and watch V(C), V(B), V(A) rise.
Decrease γ: terminal reward matters less far away, so V(A) shrinks relative to V(C).
Increase P(A|A,left): action left in A becomes “sticky” and trajectories take longer, lowering expected returns.

Loading Lab...

Reinforcement Learning

Markov Decision Process (MDP)

Definition

An MDP is a 5-tuple (S, A, P, R, γ).

S = set of states A = set of actions P(s'|s,a) = transition probabilities R(s,a,s') = reward function γ ∈ [0,1) = discount factor

Bellman expectation (policy evaluation)

Given a fixed policy π, you can compute V^π by repeatedly applying the Bellman backup.

V^π(s) = Σₐ π(a|s) Σₛ' P(s'|s,a) [ R(s,a,s') + γ V^π(s') ]

iterations: 0

Interactive MDP

Roll out a trajectory

Press Step to sample action from π and then sample next state from P(·|s,a). Terminal ends the episode; the next Step starts a new episode at A.

●

V = 0

Has actions: left/right

V = 0

Has actions: left/right

V = 0

Has actions: left/right

Terminal

V = 0

Terminal: no actions, V=0

Episode

Steps

Episode return

Avg return (recent)

Last transition

Press Step to sample.

Policy π(a|s)

State A

left 0.5 · right 0.5

State B

left 0.5 · right 0.5

State C

left 0.5 · right 0.5

Edit a transition probability

Change how “slippery” action left in state A is: it either stays in A or accidentally moves to B.

P(A | A,left)

0.7

A,left: → A with p=0.7, r=-1 → B with p=0.3, r=-1

Discount γ

0.9

Auto speed

5/s

Try these experiments

Small changes here explain big behaviors in real RL.

Set π(right|C)=1 to reach terminal quickly, then press “Policy eval step” repeatedly and watch V(C), V(B), V(A) rise.
Decrease γ: terminal reward matters less far away, so V(A) shrinks relative to V(C).
Increase P(A|A,left): action left in A becomes “sticky” and trajectories take longer, lowering expected returns.