Loading Curriculum...

Artificial Intelligence (AI) Reinforcement Learning

ActorCritic

Lesson 7/9

Reinforcement Learning

Actor-Critic Architecture

Actor-Critic is a hybrid reinforcement learning architecture that combines the strengths of both policy-based (Actor) and value-based (Critic) methods to achieve stable, sample-efficient learning in complex environments.

Policy vs Value-Based

Policy-based RL: Modifies policies directly to maximize return. Great for high-dimensional, continuous, or stochastic action spaces, but suffers from high gradient variance.
Value-based RL: Learns the expected return of states/actions, deriving the optimal policy implicitly. Exceptionally sample-efficient and stable.
Actor-Critic Fusion: The Actor proposes actions via a policy, and the Critic evaluates those actions using value-estimates, significantly reducing policy gradient variance.

The Feedback Loop

Critic calculates the **TD Error** (advantage signal) which guides policy updates in the Actor.

Mathematical Proof: Baseline Unbiasedness

Standard policy gradient algorithms suffer from high variance. Subtracted baselines reduce variance without introducing bias. Here is the complete proof that the expected value of the baseline gradient term equals zero:

Baseline Expectation Proof

Expected Baseline Gradient:

Expanded Integral / Summation:

Applying Log-Derivative Identity:

Simplifying to Derivative of Total Probability:

Since the sum of probability over all actions is always equal to 1, its gradient with respect to policy parameters

is always 0.

This mathematically guarantees that adding any state-dependent baseline

will not introduce bias to our policy gradient estimate.

In Actor-Critic frameworks, we use the Critic value function

as the baseline, establishing the Advantage:

Interactive Actor-Critic Explorer

Watch a 1D Cliff walk agent update its Actor preferences and Critic values in real-time.

The Environment Grid (S = Start, G = Goal, X = Cliff)

V: 0.00

Actor Action Preferences

State 1:← 50% | 50% →

State 2:← 50% | 50% →

State 3:← 50% | 50% →

State 4:← 50% | 50% →

State 5:← 50% | 50% →

Training Status

Episodes Completed:0

Total Steps:0

Avg Episode Return:0.0

Over time, the average return increases as the agent learns to avoid the Cliff (State 5) and reach the Goal (State 6).

Live Math Signals

Last Action Taken:None

TD Error / Advantage (δ):0.00

, the action was better than expected. The Actor increases preference for that action, and the Critic increases state value.

Loading Lab...