TD(λ) and the Deadly Triad

TD learning updates a value estimate toward a guess built from the next estimate. Eligibility traces decide how far back one surprise reaches, and lambda is the dial. The last tab shows what goes wrong when you also approximate values with a function and train on data from a different policy. Everything here is computed live in your browser from small simulations, with fixed random seeds so you can reproduce what you see.

A walker starts in the middle of 19 states and steps left or right. Falling off the left end pays −1, off the right end pays +1, everything else pays 0 (undiscounted). The true value of each state runs from −0.9 to +0.9. All estimates start at 0, so the first surprise happens at the very end of an episode.

lambda 0.80

step size alpha 0.30

Trace

Walk

value estimate true value eligibility trace last update (up, down)

TD(λ) and the Deadly Triad

Error after 10 episodes, by step size

Best error each lambda can reach

Lowest error per lambda

The smallest example: w and 2w

Baird's counterexample

Weights over time (signed log scale)

The same experiment, seven ways