Policy Gradient (REINFORCE)

Policy gradients optimize a policy directly: increase the probability of actions that lead to high return. This page shows REINFORCE on a noisy bandit so you can see variance and why we use baselines and entropy.

The update rule

REINFORCE pushes logits in the direction of ∇ log π(a).

θ ← θ + α · (G − b) · ∇_θ log π_θ(a) For softmax: ∂/∂logit_k log π(a) = 1[k=a] − π(k)

G is return (here: reward of the bandit pull).
b is a baseline (e.g. a value function) that reduces variance without changing the expected gradient.
Entropy bonus keeps the policy from collapsing too early.

Controls

Step through updates or run continuously.

Auto speed

6/s

Learning rate α

0.15

Entropy bonus η

Baseline

b (moving average)

Baseline lr β

0.1

Bandit environment

Two actions, noisy rewards

Action 1 has higher expected reward, but both are noisy. The policy must learn to shift probability mass toward the better action.

Policy

logits

l0=0 | l1=0

probabilities

π(a0)=0.5 | π(a1)=0.5

entropy H(π)

0.693

Training stats

Steps

Avg reward

Action counts

a0: 0 | a1: 0

One update, explained

Press Step to generate an update.

Try this: turn Baseline OFF and watch the policy wobble more (high-variance updates). Then turn it back ON. Add a small entropy bonus to keep exploration early.

Loading Lab...

PolicyGradient

PolicyGradient

Policy Gradient (REINFORCE)

The update rule

Controls

Two actions, noisy rewards

Policy Gradient (REINFORCE)

The update rule

Controls

Two actions, noisy rewards