Loading Curriculum...

Proximal Policy Optimization (PPO)

PPO stabilizes policy gradient methods by clipping parameter update ratios. Below is an interactive reinforcement learning agent training sandbox using clipped surrogate objective updates to master a dynamic running obstacle course.

Policy Probabilities

JUMP (Low Obstacles)50.0%

SLIDE (High Obstacles)30.0%

RUN (Wide Obstacles)20.0%

Clipping Hyperparameter (ε)0.20

Environment: Running CourseDecision area: x < 250

Episodes: 0Avg Performance: 15

Run

LOW BARRIER

Update Ratio r_t(θ)1.000UNCLIPPED

KL Divergence0.0000Policy Drift metric

Surrogate Objective1.00L_CLIP objective value

Understanding Clipped Surrogate Objectives

Standard policy gradient updates are prone to destructively large changes. PPO limits policy updates by defining a clipped probability ratio objective:

L^CLIP(θ) = Ê_t [ min( r_t(θ)Â_t , clip(r_t(θ), 1 - ε, 1 + ε)Â_t ) ]
Where r_t(θ) = π_θ(a_t | s_t) / π_θ_old(a_t | s_t) is the probability ratio between the new and old policy parameters.

If the updated policy deviates too far from the old values (outside the boundary of [1 - ε, 1 + ε]), the gradient is truncated (clipped). This ensures safe, progressive policy iterations, allowing stable updates without catastrophic policy failure.

Loading Lab...