Overview of Supervised Learning

The chapter sets up least squares and nearest neighbours as the two ends of one axis, and almost everything later in the book is positioned somewhere between them.

The two extremes

• Least squares. Heavy structural assumptions, low variance, bias that does not go away with more data.
• Nearest neighbours. Almost no assumptions, low bias, and variance that only comes down if the neighbourhood stays small in a space that is getting emptier.

Stated that way, effective number of parameters becomes the useful currency, and k nearest neighbours having roughly N over k of them is the line that makes the comparison click.

The decomposition, written out

EPE(x) = sigma^2                     irreducible
       + [E f_hat(x) - f(x)]^2       bias squared
       + E[f_hat(x) - E f_hat(x)]^2  variance

Worth writing by hand once. The irreducible term is the part people quietly drop when they promise a model will keep improving.

Curse of dimensionality

The sampling argument is the one that stuck: to capture a fixed fraction of the volume of a unit cube in p dimensions, the edge length of the sub-cube you need grows toward 1 quickly. Local methods stop being local. Neighbours in high dimensions are not near.

Note to self

The chapter's numbering is Chapter 2 in the book. Chapter 1 is a two page introduction and there is nothing in it worth a separate note.